search

arithy / packcc

A parser generator for C
Other
347 stars 28 forks source link

Add benchmarking script #52

Closed dolik-rce closed 3 years ago

dolik-rce commented 3 years ago

Hello @arithy,

Here is the benchmarking script we have talked about. It allows to test various commits to see how the affect performance. See benchmark/benchmark.sh --help for more details on usage.

It should be simple to add more grammars over time, just add the peg file to grammars/ directory and corresponding input to inputs/ (basename must match, extension is ignored).

Example output:

./benchmark.sh $(git log --format=oneline | head -n8 | tac | cut -c1-8)
...

Generation times:
=================
            f325a479        69d05291        4cc96203        9a623bbf        5f1f1b2c        f583711f        0ce1963f        5b33636d        
calc        845 us (100%)   919 us (108%)   850 us (100%)   831 us (98%)    833 us (98%)    827 us (97%)    905 us (107%)   837 us (98%)    
json        904 us (100%)   764 us (84%)    562 us (62%)    794 us (87%)    806 us (89%)    796 us (88%)    825 us (91%)    816 us (90%)    
kotlin      4852 us (100%)  3317 us (68%)   2923 us (60%)   3294 us (67%)   2928 us (60%)   3060 us (63%)   3047 us (62%)   2935 us (60%)   

Build times:
============
            f325a479        69d05291        4cc96203        9a623bbf        5f1f1b2c        f583711f        0ce1963f        5b33636d        
calc        106 ms (100%)   117 ms (110%)   108 ms (102%)   109 ms (102%)   121 ms (113%)   108 ms (101%)   109 ms (102%)   109 ms (102%)   
json        118 ms (100%)   102 ms (86%)    101 ms (86%)    101 ms (85%)    102 ms (86%)    101 ms (85%)    102 ms (87%)    102 ms (86%)    
kotlin      1229 ms (100%)  1321 ms (107%)  1219 ms (99%)   1217 ms (99%)   1198 ms (97%)   1185 ms (96%)   1213 ms (98%)   1185 ms (96%)   

Run times:
==========
            f325a479        69d05291        4cc96203        9a623bbf        5f1f1b2c        f583711f        0ce1963f        5b33636d        
calc        491 ms (100%)   637 ms (129%)   564 ms (114%)   450 ms (91%)    449 ms (91%)    464 ms (94%)    164 ms (33%)    163 ms (33%)    
json        56 ms (100%)    51 ms (90%)     51 ms (90%)     44 ms (78%)     44 ms (77%)     44 ms (77%)     44 ms (77%)     43 ms (76%)     
kotlin      449 ms (100%)   442 ms (98%)    421 ms (93%)    383 ms (85%)    375 ms (83%)    373 ms (83%)    363 ms (80%)    365 ms (81%)

This was run for the last 8 commits in the repository. It nicely shows that 0ce1963f improved the run times greatly for calc, while other grammars were not much affected.

arithy commented 3 years ago

Oh, thank you very much for your cool work!