github-actions[bot]
commented
3 weeks ago 
Bencher Report
| Branch | 2081/merge |
| Testbed | ubuntu-latest |
⚠️ WARNING: The following Measure does not have a Threshold. Without a Threshold, no Alerts will ever be generated!
Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the--ci-only-thresholdsCLI flag.
Click to view all benchmark results
| Benchmark | Latency | nanoseconds (ns) |
|---|---|---|
| fibonacci 10 | 📈 view plot ⚠️ NO THRESHOLD | 483,550.00 |
| foldl arrays 50 | 📈 view plot ⚠️ NO THRESHOLD | 1,691,100.00 |
| foldl arrays 500 | 📈 view plot ⚠️ NO THRESHOLD | 6,555,000.00 |
| foldr strings 50 | 📈 view plot ⚠️ NO THRESHOLD | 6,980,000.00 |
| foldr strings 500 | 📈 view plot ⚠️ NO THRESHOLD | 61,483,000.00 |
| generate normal 250 | 📈 view plot ⚠️ NO THRESHOLD | 45,361,000.00 |
| generate normal 50 | 📈 view plot ⚠️ NO THRESHOLD | 2,049,700.00 |
| generate normal unchecked 1000 | 📈 view plot ⚠️ NO THRESHOLD | 3,314,100.00 |
| generate normal unchecked 200 | 📈 view plot ⚠️ NO THRESHOLD | 742,840.00 |
| pidigits 100 | 📈 view plot ⚠️ NO THRESHOLD | 3,161,600.00 |
| pipe normal 20 | 📈 view plot ⚠️ NO THRESHOLD | 1,491,700.00 |
| pipe normal 200 | 📈 view plot ⚠️ NO THRESHOLD | 9,941,900.00 |
| product 30 | 📈 view plot ⚠️ NO THRESHOLD | 809,980.00 |
| scalar 10 | 📈 view plot ⚠️ NO THRESHOLD | 1,492,300.00 |
| sum 30 | 📈 view plot ⚠️ NO THRESHOLD | 802,290.00 |
The goal is to have some coarse metrics to decide if switching the parser to the new AST and then converting back to the old one (to replace the current pipeline step by step) is going to have any noticeable performance hit. And also because it's quite useful in general to see which phases are the most costly.
As a first experiment, on the private benchmark of 24kLoC of Nickel (plus 10-15kLoc of JSON or YAML), it's already interesting to see that the evaluation cost trumps pretty much everything else measured (type checking, stdlib prep, transformation, etc.). Preparing (parse + import resolution + transformation + type checking) is < 2% of the eval time (not even the total time, which is probably also impacted by serialization, substitution and pretty printing). Preparing the stdlib is < 0.1%, typechecking the main file is < 0.5%, and parsing is in the same order of magnitude as those.
On a small example (GCC example), almost all operations are negligible anyway (a handful of ms for each stage). So there's a good chance that the conversion from the new ast to the old ast will be reasonable in both case (in small cases it should be fast, in large examples it should be trumped by evaluation). But we'll make a more thorough benchmarking anyway.