sharkdp / numbat

A statically typed programming language for scientific computations with first class support for physical dimensions and units
https://numbat.dev
Apache License 2.0
1.26k stars 53 forks source link

Improve startup time (regression) #525

Open sharkdp opened 3 months ago

sharkdp commented 3 months ago

The new type checker from #443 introduced a severe regression in startup times:

hyperfine --warmup 3 --reference "./numbat-master -e '1+2'" "./numbat-1.12 -e '1+2'" --export-markdown -
Command Mean [ms] Min [ms] Max [ms] Relative
./numbat-master -e '1+2' 78.9 ± 1.2 77.1 83.8 2.28 ± 0.09
./numbat-1.12 -e '1+2' 34.5 ± 1.2 32.4 42.5 1.00

We should look into potential measures to improve this again.

If the constraint checker is the culprit, we might want to look into de-duplication of constraints before attempting to solve them. But it might be something else entirely (e.g. the mutation/traversal/cloning of ASTs during elaboration)

sharkdp commented 3 months ago

Thanks to the changes in #543 by @GuerricChupin, we've moved a significant step in the right direction:

Command Mean [ms] Min [ms] Max [ms] Relative
./numbat-master -e '1+2' 59.9 ± 1.2 57.9 64.3 1.78 ± 0.08
./numbat-1.12 -e '1+2' 33.7 ± 1.3 32.6 44.8 1.00
sharkdp commented 2 months ago

Thanks to the changes in #537 by @irevoire (which are not related to the type checker, but still), we've moved another significant step in the right direction:

Command Mean [ms] Min [ms] Max [ms] Relative
./numbat-master -e '1+2' 47.0 ± 0.9 45.5 50.3 1.37 ± 0.04
./numbat-1.12 -e '1+2' 34.2 ± 0.9 32.2 36.7 1.00
sharkdp commented 1 month ago

Thanks to various changes by @rben01 (#559, #569, #578, #579, #585, #597, #598 ), there is another 5% improvement in startup time. For more details, see https://github.com/sharkdp/numbat/pull/579#issuecomment-2389482040 (note: the absolute times are not comparable. I measured on different machines).

Also, note that some of the measurements above might be underestimating the true performance benefit, since Numbats prelude is growing. For a true comparison, we need to run both versions against the same version of the prelude (typically the older one, as the new one might not be supported on the old interpreter).