Closed enolan closed 1 year ago
@bennofs
This PR does seem to improve the time nix-index
takes:
# before
real 9m13.026s
user 7m36.327s
sys 0m28.164s
# after
real 7m53.152s
user 9m7.771s
sys 0m25.164s
@enolan how are you getting 73 seconds? o_O I'm on a reasonably recent CPU (AMD Ryzen 5 4650U).
Headline: 198 seconds to 73 seconds with these changes. :tada:
I'll go through the changes in order.
Baseline:
I first change I made was to run the
nix-env
invocations in parallel. This gets us about 10s:Upgrading
zstd
gets us around 5s:Turning on parallel compression doesn't do anything, though:
But if you watch it in
htop
you see that it doesn't actually use more than one core. AFAICT, the "ultra" compression levels are incompatible with multithreading. Switching to compression level 19 from 22 shaves more than 100s:It increases the size of the database from 31 megs to 36, which I'd say is an acceptable tradeoff, and makes the indexing process no longer CPU bound on my machine. As a sanity check, we can try level 19 without multithreading:
Most of the speedup is from the parallelism.
This was all done against
v0.1.2
, since I can't buildmaster
due to #151. Feedback very welcome, I'm still somewhat new to Rust :)