search

servo / rust-url

URL parser for Rust
https://docs.rs/url/
Apache License 2.0
1.27k stars 317 forks source link

Massive regression in compile times #939

Open CryZe opened 2 weeks ago

CryZe commented 2 weeks ago

After #923 the dependency tree massively increased. The total amount of crates during compilation went from 8 to 36. In particular the "critical path" to compilation now consists of 17 crates as opposed to 5 crates. This means that the url crate is now the sole reason why basically all projects depending on the url crate (indirectly) should now take a whole 20 seconds or so longer to compile than before.

You can compile an entire GUI framework in the time the URL crate compiles now.

Before: image

After: image

CryZe commented 1 week ago

I've seen multiple responses that syn is the main offender here... but while that looks like it may be the case, it's just not the full picture. The problem is that the url crate is fundamentally an "early" dependency, as in, almost all of the web backend ecosystem (which is the majority of Rust's usage) builds on top of the url crate, often by building their API bindings on top of reqwest which in turn depends on the url crate. This means that the majority of those crates require the url crate to have been built before any of those crates can start building. By having such a large dependency chain of 17 (and I believe even worse 19 with idna master), all those crates can't start compiling, effectively increasing compile times by a lot. And here's the thing... it has been argued that syn is usually already "sunk cost" by almost always being in the dependency tree already anyway, but that just doesn't matter if url and in turns almost all of the web ecosystem crates have to start compiling much later. That's why I said that real world projects that do already have syn are heavily affected by this as well. I almost feel like the icu family of crates got split up in many small crates to help with compilation times, but I'm almost certain that it had an overall negative effect on real world compile times, by causing an even longer chain, forcing the url crate to not be an early compilation crate anymore, which it definitely needs to be.

I think there's some widespread misunderstanding that the dependency count or something not already being in the dependency tree is what determines compile times, but the scheduling of when crates can start compiling is actually the most important and most overlooked problem.

hsivonen commented 1 week ago

Thanks for the extra context. Looking at the build of reqwest with default options and without any application code, it is indeed the case that syn gets absorbed into parallelism better than some other parts that show up sequentially in the image here.

I've filed https://github.com/unicode-org/icu4x/issues/5121 and https://github.com/unicode-org/icu4x/issues/5120 . (The latter thing should get LTOed out, so it's not an issue of ICU4X resulting in extra stuff in an optimized binary; you don't pay for what you don't use in binary size, but you do in compile time.)

hsivonen commented 1 week ago

Looking at the build of reqwest with default options and without any application code, it is indeed the case that syn gets absorbed into parallelism better than some other parts that show up sequentially in the image here.

This was a misreading on my part, and it seems that syn is still a part of the compile time problem.