I'm updating the TTFX benchmarks for the Julia 1.9 release and noting a big regression. I should note that I haven't dug deeply on this and have a packed schedule today and a bit into tomorrow, so apologies for the slapdash report. In particular it's worth noting that while my main suspect is some new source of invalidations, I could easily be wrong and this may not be LoopVectorization's fault.
(TTFX is 1.74s). I should note that this run occurred without @recompile_invalidations which matches the old SnoopPrecompile behavior. In other words, if you use the JuliaCon2022_Precompilation repo as-is you may not replicate my result, but you may want to test without the @recompile_invalidations. Note that with a bit of manual labor you can just copy that Startup package and run it in isolation, or you could just comment out all the other packages at the top of that runner.jl script, edit the execute.sh script to run just the Julia version(s) you want to test, and then ./execute.sh. For GLMakie you need a graphics window open so ssh -X may be necessary.
I'll dig into this when my schedule frees up, but given the impending release I would certainly appreciate any help that can be provided.
KristofferC
commented
1 year ago
I'm updating the TTFX benchmarks for the Julia 1.9 release and noting a big regression. I should note that I haven't dug deeply on this and have a packed schedule today and a bit into tomorrow, so apologies for the slapdash report. In particular it's worth noting that while my main suspect is some new source of invalidations, I could easily be wrong and this may not be LoopVectorization's fault.
The code to run the benchmarks, and specifically the "LV" package, is here: https://github.com/timholy/JuliaCon2022_Precompilation/blob/teh/jan_2023/analysis/cache_external/runner.jl#L160-L194. Note this is the
teh/jan_2023branch of the repo (it is a repo for a 2022 presentation and I want to keepmainthe branch relevant to that presentation). All the different packages are tested using a "Startup" package defined in https://github.com/timholy/JuliaCon2022_Precompilation/blob/teh/jan_2023/analysis/cache_external/runner.jl#L160-L194 where the LV setup and workload code are in https://github.com/timholy/JuliaCon2022_Precompilation/blob/f5630562455653b5933ed5da50df61b8875c0516/analysis/cache_external/runner.jl#L109-L119.When I last ran it several weeks ago, I got timing
(TTFX is ~20ms) and today I got
(TTFX is 1.74s). I should note that this run occurred without
@recompile_invalidationswhich matches the old SnoopPrecompile behavior. In other words, if you use the JuliaCon2022_Precompilation repo as-is you may not replicate my result, but you may want to test without the@recompile_invalidations. Note that with a bit of manual labor you can just copy that Startup package and run it in isolation, or you could just comment out all the other packages at the top of thatrunner.jlscript, edit theexecute.shscript to run just the Julia version(s) you want to test, and then./execute.sh. For GLMakie you need a graphics window open sossh -Xmay be necessary.I'll dig into this when my schedule frees up, but given the impending release I would certainly appreciate any help that can be provided.