search

sebastienros / jint

Javascript Interpreter for .NET
BSD 2-Clause "Simplified" License
4.02k stars 556 forks source link

Performance improvements #451

Closed lahma closed 6 years ago

lahma commented 6 years ago

This issue is tracking various pull requests and performance issues to keep track of progress.

Baseline created of benchmarks using commit https://github.com/sebastienros/jint/commit/4a7fed1c5e2852ce94d12657b92126eafdc2d222


BenchmarkDotNet=v0.10.11, OS=Windows 10 Redstone 3 [1709, Fall Creators Update] (10.0.16299.125)
Processor=Intel Core i7-6820HQ CPU 2.70GHz (Skylake), ProcessorCount=8
Frequency=2648437 Hz, Resolution=377.5812 ns, Timer=TSC
.NET Core SDK=2.1.2
  [Host]     : .NET Core 2.0.3 (Framework 4.6.25815.02), 64bit RyuJIT

ArrayBenchmark

Method N Mean Error StdDev Gen 0 Allocated
Slice 100 7.344 ms 0.1141 ms 0.1068 ms 3562.5000 14.26 MB
Concat 100 6.174 ms 0.0582 ms 0.0516 ms 2515.6250 10.07 MB
Unshift 100 196.603 ms 3.9271 ms 3.6734 ms 47250.0000 189.24 MB
Push 100 37.944 ms 0.2475 ms 0.2315 ms 8125.0000 32.67 MB
Index 100 27.836 ms 0.2256 ms 0.2000 ms 5812.5000 23.32 MB
Map 100 13.526 ms 0.0419 ms 0.0392 ms 4218.7500 16.93 MB
Apply 100 7.946 ms 0.0679 ms 0.0635 ms 2968.7500 11.89 MB
JsonStringifyParse 100 15.247 ms 0.0311 ms 0.0260 ms 6437.5000 25.77 MB

ArrayStressBenchmark

Method N Mean Error StdDev Gen 0 Gen 1 Gen 2 Allocated
Jint 20 3.775 s 0.6523 s 0.0369 s 786750.0000 40250.0000 18500.0000 3.17 GB

EvaluationBenchmark

Method N Mean Error StdDev Gen 0 Allocated
Jint 20 1.846 ms 0.0183 ms 0.0171 ms 263.6719 1.06 MB

LinqJsBenchmark

Method N Mean Error StdDev Gen 0 Gen 1 Gen 2 Allocated
Jint 10 106.1 ms 3.802 ms 0.2148 ms 9312.5000 2687.5000 1062.5000 55.54 MB

MinimalScriptBenchmark

Method N Mean Error StdDev Gen 0 Allocated
Jint 10 27.71 us 0.0954 us 0.0893 us 9.4910 38.98 KB

SunSpiderBenchmark

Method FileName Mean Error StdDev Gen 0 Gen 1 Gen 2 Allocated
Run 3d-cube 1,184.2 ms 163.111 ms 9.2161 ms 210000.0000 250.0000 - 842.35 MB
Run 3d-morph 1,279.9 ms 6.912 ms 0.3905 ms 191500.0000 51500.0000 6750.0000 758.14 MB
Run 3d-raytrace 1,043.6 ms 17.447 ms 0.9858 ms 203500.0000 2250.0000 750.0000 818.78 MB
Run access-binary-trees 472.9 ms 70.924 ms 4.0073 ms 92750.0000 1000.0000 - 373.94 MB
Run access-fannkuch 3,730.5 ms 328.132 ms 18.5400 ms 794500.0000 250.0000 - 3178.96 MB
Run access-nbody 1,063.3 ms 76.921 ms 4.3462 ms 179000.0000 - - 716.74 MB
Run access-nsieve 1,807.5 ms 216.799 ms 12.2495 ms 257416.6667 73916.6667 10500.0000 1058.6 MB
Run bitops-3bit-bits-in-byte 884.8 ms 360.792 ms 20.3854 ms 167500.0000 - - 670.44 MB
Run bitops-bits-in-byte 1,387.0 ms 561.769 ms 31.7410 ms 240500.0000 - - 962.87 MB
Run bitops-bitwise-and 884.9 ms 574.365 ms 32.4527 ms 101750.0000 - - 407.43 MB
Run bitops-nsieve-bits 1,568.4 ms 310.924 ms 17.5678 ms 296000.0000 31750.0000 1750.0000 1188 MB
Run controlflow-recursive 766.9 ms 225.838 ms 12.7602 ms 147750.0000 2750.0000 - 599.25 MB
Run crypto-aes 1,145.9 ms 246.053 ms 13.9025 ms 259583.3333 250.0000 - 1042.38 MB
Run crypto-md5 677.5 ms 394.378 ms 22.2831 ms 123000.0000 2500.0000 500.0000 494.57 MB
Run crypto-sha1 689.1 ms 91.831 ms 5.1886 ms 126250.0000 1250.0000 250.0000 506.9 MB
Run date-format-tofte 793.1 ms 110.827 ms 6.2619 ms 153000.0000 250.0000 - 614.79 MB
Run date-format-xparb 527.0 ms 197.836 ms 11.1781 ms 51000.0000 250.0000 - 205.97 MB
Run math-cordic 1,774.1 ms 392.520 ms 22.1781 ms 297750.0000 - - 1191.14 MB
Run math-partial-sums 567.1 ms 22.894 ms 1.2936 ms 73000.0000 - - 292.71 MB
Run math-spectral-norm 884.5 ms 567.655 ms 32.0736 ms 144000.0000 - - 576.42 MB
Run regexp-dna 355.1 ms 281.066 ms 15.8807 ms 2833.3333 2333.3333 1833.3333 21.49 MB
Run string-base64 827.4 ms 393.957 ms 22.2593 ms 350000.0000 250.0000 - 1403.56 MB
Run string-fasta 1,109.0 ms 365.973 ms 20.6782 ms 204750.0000 - - 819.47 MB
Run string-tagcloud 905.4 ms 105.217 ms 5.9450 ms 197333.3333 121250.0000 114250.0000 1080.06 MB
Run string-unpack-code 355.2 ms 133.734 ms 7.5562 ms 61500.0000 4000.0000 1250.0000 261.57 MB
Run string-validate-input 2,830.8 ms 128.618 ms 7.2672 ms 1646166.6667 1562916.6667 1561666.6667 6460.55 MB

UncacheableExpressionsBenchmark

Method N Mean Error StdDev Gen 0 Gen 1 Allocated
Benchmark 500 1.181 s 0.0025 s 0.0036 s 281875.0000 10062.5000 1.12 GB
lahma commented 6 years ago

Status for the new release 3.0.0-beta-1101. Typos and miscalculations are my own.

BenchmarkDotNet=v0.10.11, OS=Windows 10 Redstone 3 [1709, Fall Creators Update] (10.0.16299.192)
Processor=Intel Core i7-6820HQ CPU 2.70GHz (Skylake), ProcessorCount=8
Frequency=2648435 Hz, Resolution=377.5815 ns, Timer=TSC
.NET Core SDK=2.1.4
  [Host]     : .NET Core 2.0.5 (Framework 4.6.26020.03), 64bit RyuJIT
  Job-ILFHZD : .NET Core 2.0.5 (Framework 4.6.26020.03), 64bit RyuJIT

Esprima.Benchmark.SunSpiderBenchmark

Diff Method FileName Mean Gen 0 Gen 1 Gen 2 Allocated
Old Run 3d-cube 1,167.7 ms 210000.0000 250.0000 - 842.35 MB
New 870.3 ms (-25%) 63750.0000 (-70%) 250.0000 (0%) - 257.9 MB (-69%)
Old Run 3d-morph 1,275.4 ms 191250.0000 52250.0000 6750.0000 758.14 MB
New 902.1 ms (-29%) 41500.0000 (-78%) 5500.0000 (-89%) 1250.0000 (-81%) 193.86 MB (-74%)
Old Run 3d-raytrace 1,054.0 ms 203500.0000 2250.0000 750.0000 818.78 MB
New 679.4 ms (-36%) 54250.0000 (-73%) 750.0000 (-67%) - 219.35 MB (-73%)
Old Run access-binary-trees 469.6 ms 92750.0000 1000.0000 - 373.94 MB
New 289.1 ms (-38%) 37750.0000 (-59%) 500.0000 (-50%) - 151.98 MB (-59%)
Old Run access-fannkuch 3,581.1 ms 794500.0000 250.0000 - 3178.96 MB
New 2,708.8 ms (-24%) 172250.0000 (-78%) 250.0000 (0%) - 689.77 MB (-78%)
Old Run access-nbody 1,032.7 ms 179000.0000 - - 716.74 MB
New 786.4 ms (-24%) 57500.0000 (-68%) - - 230.28 MB (-68%)
Old Run access-nsieve 1,801.7 ms 255250.0000 73000.0000 9250.0000 1058.6 MB
New 1,002.6 ms (-44%) 62750.0000 (-75%) 7750.0000 (-89%) 2750.0000 (-70%) 260.87 MB (-75%)
Old Run bitops-3bit-bits-in-byte 878.1 ms 167500.0000 - - 670.44 MB
New 597.8 ms (-32%) 56250.0000 (-66%) - - 225.67 MB (-66%)
Old Run bitops-bits-in-byte 1,325.2 ms 240500.0000 - - 962.87 MB
New 984.1 ms (-26%) 81500.0000 (-66%) - - 326.8 MB (-66%)
Old Run bitops-bitwise-and 837.5 ms 101750.0000 - - 407.43 MB
New 826.9 ms (-1%) 45500.0000 (-55%) - - 182.65 MB (-55%)
Old Run bitops-nsieve-bits 1,530.9 ms 296000.0000 31750.0000 1750.0000 1188 MB
New 1,102.1 ms (-28%) 66750.0000 (-77%) 2750.0000 (-91%) 250.0000 (-86%) 269.04 MB (-77%)
Old Run controlflow-recursive 771.4 ms 147750.0000 2750.0000 - 599.25 MB
New 384.2 ms (-50%) 58500.0000 (-60%) - - 234.15 MB (-61%)
Old Run crypto-aes 1,099.5 ms 259666.6667 333.3333 - 1042.39 MB
New 664.1 ms (-40%) 44500.0000 (-83%) 250.0000 (-25%) - 180.08 MB (-83%)
Old Run crypto-md5 648.6 ms 123000.0000 2500.0000 500.0000 494.57 MB
New 440.7 ms (-32%) 52500.0000 (-57%) 250.0000 (-90%) - 212.71 MB (-57%)
Old Run crypto-sha1 676.2 ms 126250.0000 1250.0000 250.0000 506.9 MB
New 436.8 ms (-35%) 49000.0000 (-61%) 250.0000 (-80%) - 197.82 MB (-61%)
Old Run date-format-tofte 794.8 ms 153000.0000 250.0000 - 614.79 MB
New 445.0 ms (-44%) 37250.0000 (-76%) - - 149.63 MB (-76%)
Old Run date-format-xparb 510.4 ms 51000.0000 250.0000 - 205.97 MB
New 386.7 ms (-24%) 18500.0000 (-64%) 250.0000 (0%) - 74.92 MB (-64%)
Old Run math-cordic 1,744.8 ms 297750.0000 - - 1191.14 MB
New 1,296.7 ms (-26%) 111250.0000 (-63%) - - 445.76 MB (-63%)
Old Run math-partial-sums 570.0 ms 73000.0000 - - 292.71 MB
New 458.8 ms (-20%) 26750.0000 (-63%) - - 107.29 MB (-63%)
Old Run math-spectral-norm 797.6 ms 144000.0000 - - 576.42 MB
New 489.1 ms (-39%) 49750.0000 (-65%) - - 199.05 MB (-65%)
Old Run regexp-dna 344.3 ms 3250.0000 2750.0000 2250.0000 21.48 MB
New 340.3 ms (-1%) 1500.0000 (-54%) 1250.0000 (-55%) 1000.0000 (-56%) 13.47 MB (-37%)
Old Run string-base64 790.0 ms 349916.6667 250.0000 - 1403.56 MB
New 389.2 ms (-51%) 24500.0000 (-93%) 250.0000 (0%) - 99.86 MB (-93%)
Old Run string-fasta 1,072.3 ms 204750.0000 - - 819.47 MB
New 646.6 ms (-40%) 59000.0000 (-71%) - - 236.35 MB (-71%)
Old Run string-tagcloud 965.8 ms 210083.3333 133833.3333 126916.6667 1080.11 MB
New 260.3 ms (-73%) 26250.0000 (-88%) 1750.0000 (-99%) 250.0000 (-100%) 115.39 MB (-89%)
Old Run string-unpack-code 322.3 ms 61500.0000 4000.0000 1250.0000 261.57 MB
New 205.5 ms (-36%) 28250.0000 (-54%) 8250.0000 (+106%) 1500.0000 (+20%) 123.15 MB (-53%)
Old Run string-validate-input 3,497.3 ms 1650833.3333 1567583.3333 1566333.3333 6460.11 MB
New 262.5 ms (-92%) 20750.0000 (-99%) 250.0000 (-100%) - 86.94 MB (-99%)

Jint.Benchmark.ArrayBenchmark

Diff Method N Mean Gen 0 Allocated
Old Slice 100 7.005 ms 3562.5000 14.26 MB
New 795.5 us (-89%) 150.3906 (-96%) 617.19 KB (-96%)
Old Concat 100 6.068 ms 2515.6250 10.07 MB
New 1,039.9 us (-83%) 166.9922 (-93%) 686.72 KB (-93%)
Old Unshift 100 188.747 ms 47250.0000 189.24 MB
New 33,962.7 us (-82%) 3687.5000 (-92%) 15214.06 KB (-92%)
Old Push 100 36.531 ms 8125.0000 32.67 MB
New 22,118.2 us (-39%) 1406.2500 (-83%) 5847.66 KB (-82%)
Old Index 100 27.245 ms 5812.5000 23.32 MB
New 20,948.8 us (-23%) 1250.0000 (-78%) 5172.66 KB (-78%)
Old Map 100 13.287 ms 4218.7500 16.93 MB
New 4,225.7 us (-68%) 1531.2500 (-64%) 6285.16 KB (-63%)
Old Apply 100 7.837 ms 2968.7500 11.89 MB
New 1,085.1 us (-86%) 183.5938 (-94%) 753.91 KB (-94%)
Old JsonStringifyParse 100 15.021 ms 6437.5000 25.77 MB
New 5,385.5 us (-64%) 1250.0000 (-81%) 5144.53 KB (-80%)

Jint.Benchmark.ArrayStressBenchmark

Diff Method N Mean Gen 0 Gen 1 Gen 2 Allocated
Old Jint 20 3.673 s 786750.0000 40250.0000 18500.0000 3.17 GB
New 1.935 s (-47%) 93250.0000 (-88%) 13000.0000 (-68%) - 388.02 MB (-88%)

Jint.Benchmark.EvaluationBenchmark

Diff Method N Mean Gen 0 Allocated
Old Jint 20 1.962 ms 263.6719 1.06 MB
New 1.615 ms (-18%) 138.6719 (-47%) 569.69 KB (-46%)

Jint.Benchmark.LinqJsBenchmark

Diff Method N Mean Gen 0 Gen 1 Gen 2 Allocated
Old Jint 10 104.6 ms 9312.5000 2687.5000 1062.5000 55.54 MB
New 71.69 ms (-31%) 5125.0000 (-45%) 2562.5000 (-5%) - 30.86 MB (-44%)

Jint.Benchmark.MinimalScriptBenchmark

Diff Method N Mean Gen 0 Allocated
Old Jint 10 28.47 us 9.4910 38.98 KB
New 22.68 us (-20%) 8.5144 (-10%) 34.92 KB (-10%)

Jint.Benchmark.StringIndexingBenchmark

Diff Method N Mean Gen 0 Gen 1 Allocated
Old Jint 1 2.992 s 108750.0000 250.0000 435.83 MB
New 2.606 s (-13%) 40500.0000 (-63%) 250.0000 (0%) 162.98 MB (-63%)

Jint.Benchmark.UncacheableExpressionsBenchmark

Diff Method N Mean Gen 0 Gen 1 Allocated
Old Benchmark 500 1.178 s 282466.6667 8466.6667 1.12 GB
New 511.4 ms (-57%) 100000.0000 (-65%) 1312.5000 (-84%) 400.31 MB (-64%)
sebastienros commented 6 years ago

I'd love to be able to create a custom project on appveyor that would only run the benchmarks before and after all the commits on a PR. That should not be that hard to do. Ideally generating a custom report, or adding a comment to the PR.

lahma commented 6 years ago

Yes, that would be nice. You can programmatically make BenchmarkDotNet to run specific cases with wanted output. The current output has way too much noise and it would be nice to have per-benchmark results to that values wouldn't be scaled to share same mins and maxes.

lahma commented 6 years ago

Status for the new release 3.0.0-beta-1101 against version 2.11.58. Typos and miscalculations are my own.

BenchmarkDotNet=v0.10.11, OS=Windows 10 Redstone 3 [1709, Fall Creators Update] (10.0.16299.192)
Processor=Intel Core i7-6820HQ CPU 2.70GHz (Skylake), ProcessorCount=8
Frequency=2648435 Hz, Resolution=377.5815 ns, Timer=TSC
.NET Core SDK=2.1.4
  [Host]     : .NET Core 2.0.5 (Framework 4.6.26020.03), 64bit RyuJIT
  Job-ILFHZD : .NET Core 2.0.5 (Framework 4.6.26020.03), 64bit RyuJIT

Esprima.Benchmark.DromaeoBenchmark

Diff Method FileName Mean Gen 0 Gen 1 Gen 2 Allocated
Old Run dromaeo-3d-cube 162.75 ms 32250.0000 250.0000 - 131.94 MB
New 109.84 ms (-33%) 7250.0000 (-78%) 250.0000 (0%) - 31.85 MB (-76%)
Old Run dromaeo-core-eval 33.69 ms 12250.0000 - - 49.68 MB
New 26.88 ms (-20%) 1500.0000 (-88%) - - 6.9 MB (-86%)
Old Run dromaeo-object-array 1,955.27 ms 435000.0000 7500.0000 2000.0000 1763.05 MB
New 286.11 ms (-85%) 45250.0000 (-90%) 2250.0000 (-70%) 1000.0000 (-50%) 186.78 MB (-89%)
Old Run dromaeo-object-regexp 3,462.88 ms 373000.0000 88750.0000 44166.6667 1939.8 MB
New 1,135.31 ms (-67%) 71666.6667 (-81%) 35083.3333 (-60%) 21416.6667 (-52%) 482.78 MB (-75%)
Old Run dromaeo-object-string 6,357.63 ms 4638000.0000 2800083.3333 2786416.6667 17418.93 MB
New 1,349.38 ms (-79%) 235750.0000 (-95%) 170166.6667 (-94%) 164833.3333 (-94%) 1460.63 MB (-92%)
Old Run dromaeo-string-base64 510.55 ms 247750.0000 1500.0000 500.0000 997.99 MB
New 261.90 ms (-49%) 16750.0000 (-93%) 250.0000 (-83%) - 67.54 MB (-93%)

Esprima.Benchmark.SunSpiderBenchmark

Diff Method FileName Mean Gen 0 Gen 1 Gen 2 Allocated
Old Run 3d-cube 1,130.3 ms 200000.0000 250.0000 - 803.08 MB
New 870.3 ms (-23%) 63750.0000 (-68%) 250.0000 (0%) - 257.9 MB (-68%)
Old Run 3d-morph 1,099.6 ms 173750.0000 47750.0000 8000.0000 685.08 MB
New 902.1 ms (-18%) 41500.0000 (-76%) 5500.0000 (-88%) 1250.0000 (-84%) 193.86 MB (-72%)
Old Run 3d-raytrace 987.0 ms 185000.0000 1250.0000 250.0000 743.91 MB
New 679.4 ms (-31%) 54250.0000 (-71%) 750.0000 (-40%) - 219.35 MB (-71%)
Old Run access-binary-trees 432.8 ms 83250.0000 750.0000 - 335.17 MB
New 289.1 ms (-33%) 37750.0000 (-55%) 500.0000 (-33%) - 151.98 MB (-55%)
Old Run access-fannkuch 3,322.7 ms 753750.0000 250.0000 - 3015.43 MB
New 2,708.8 ms (-18%) 172250.0000 (-77%) 250.0000 (0%) - 689.77 MB (-77%)
Old Run access-nbody 981.3 ms 159500.0000 - - 638.35 MB
New 786.4 ms (-20%) 57500.0000 (-64%) - - 230.28 MB (-64%)
Old Run access-nsieve 1,161.6 ms 241250.0000 21000.0000 5000.0000 1009.95 MB
New 1,002.6 ms (-14%) 62750.0000 (-74%) 7750.0000 (-63%) 2750.0000 (-45%) 260.87 MB (-74%)
Old Run bitops-3bit-bits-in-byte 703.7 ms 132000.0000 - - 528.68 MB
New 597.8 ms (-15%) 56250.0000 (-57%) - - 225.67 MB (-57%)
Old Run bitops-bits-in-byte 1,120.7 ms 195750.0000 - - 783.66 MB
New 984.1 ms (-12%) 81500.0000 (-58%) - - 326.8 MB (-58%)
Old Run bitops-bitwise-and 766.1 ms 86750.0000 - - 347.92 MB
New 826.9 ms (+8%) 45500.0000 (-48%) - - 182.65 MB (-48%)
Old Run bitops-nsieve-bits 1,299.6 ms 253000.0000 27250.0000 1750.0000 1015.1 MB
New 1,102.1 ms (-15%) 66750.0000 (-74%) 2750.0000 (-90%) 250.0000 (-86%) 269.04 MB (-73%)
Old Run controlflow-recursive 713.5 ms 135250.0000 2500.0000 - 544.98 MB
New 384.2 ms (-46%) 58500.0000 (-57%) - - 234.15 MB (-57%)
Old Run crypto-aes 1,022.6 ms 244250.0000 250.0000 - 981.45 MB
New 664.1 ms (-35%) 44500.0000 (-82%) 250.0000 (0%) - 180.08 MB (-82%)
Old Run crypto-md5 577.0 ms 108250.0000 2500.0000 500.0000 436.1 MB
New 440.7 ms (-24%) 52500.0000 (-52%) 250.0000 (-90%) - 212.71 MB (-51%)
Old Run crypto-sha1 601.2 ms 110500.0000 1000.0000 250.0000 444.44 MB
New 436.8 ms (-27%) 49000.0000 (-56%) 250.0000 (-75%) - 197.82 MB (-55%)
Old Run date-format-tofte 705.5 ms 145500.0000 250.0000 - 583.33 MB
New 445.0 ms (-37%) 37250.0000 (-74%) - - 149.63 MB (-74%)
Old Run date-format-xparb 488.0 ms 45500.0000 250.0000 - 184.11 MB
New 386.7 ms (-21%) 18500.0000 (-59%) 250.0000 (0%) - 74.92 MB (-59%)
Old Run math-cordic 1,646.3 ms 284750.0000 - - 1139.63 MB
New 1,296.7 ms (-21%) 111250.0000 (-61%) - - 445.76 MB (-61%)
Old Run math-partial-sums 516.1 ms 63000.0000 - - 252.49 MB
New 458.8 ms (-11%) 26750.0000 (-58%) - - 107.29 MB (-58%)
Old Run math-spectral-norm 733.9 ms 129000.0000 - - 516.31 MB
New 489.1 ms (-33%) 49750.0000 (-61%) - - 199.05 MB (-61%)
Old Run regexp-dna 344.3 ms 2750.0000 2250.0000 1750.0000 20.84 MB
New 340.3 ms (-1%) 1500.0000 (-45%) 1250.0000 (-44%) 1000.0000 (-43%) 13.47 MB (-35%)
Old Run string-base64 735.0 ms 335750.0000 250.0000 - 1346.89 MB
New 389.2 ms (-47%) 24500.0000 (-93%) 250.0000 (0%) - 99.86 MB (-93%)
Old Run string-fasta 1,054.2 ms 201000.0000 - - 804.94 MB
New 646.6 ms (-39%) 59000.0000 (-71%) - - 236.35 MB (-71%)
Old Run string-tagcloud 936.2 ms 204500.0000 136583.3333 128083.3333 1052.84 MB
New 260.3 ms (-72%) 26250.0000 (-87%) 1750.0000 (-99%) 250.0000 (-100%) 115.39 MB (-89%)
Old Run string-unpack-code 324.8 ms 60000.0000 4000.0000 1250.0000 255.12 MB
New 205.5 ms (-37%) 28250.0000 (-53%) 8250.0000 (+106%) 1500.0000 (+20%) 123.15 MB (-52%)
Old Run string-validate-input 3,326.3 ms 1641750.0000 1566083.3333 1565500.0000 6427.62 MB
New 262.5 ms (-92%) 20750.0000 (-99%) 250.0000 (-100%) - 86.94 MB (-99%)

Jint.Benchmark.ArrayBenchmark

Diff Method N Mean Gen 0 Allocated
Old Slice 100 6.998 ms 3562.5000 14.27 MB
New 795.5 us (-89%) 150.3906 (-96%) 617.19 KB (-96%)
Old Concat 100 6.146 ms 2515.6250 10.09 MB
New 1,039.9 us (-83%) 166.9922 (-93%) 686.72 KB (-93%)
Old Unshift 100 190.064 ms 47125.0000 188.62 MB
New 33,962.7 us (-82%) 3687.5000 (-92%) 15214.06 KB (-92%)
Old Push 100 36.007 ms 8000.0000 32.01 MB
New 22,118.2 us (-39%) 1406.2500 (-82%) 5847.66 KB (-82%)
Old Index 100 27.367 ms 5781.2500 23.15 MB
New 20,948.8 us (-23%) 1250.0000 (-78%) 5172.66 KB (-78%)
Old Map 100 13.261 ms 4187.5000 16.78 MB
New 4,225.7 us (-68%) 1531.2500 (-63%) 6285.16 KB (-63%)
Old Apply 100 8.070 ms 2968.7500 11.91 MB
New 1,085.1 us (-87%) 183.5938 (-94%) 753.91 KB (-94%)
Old JsonStringifyParse 100 14.856 ms 6437.5000 25.79 MB
New 5,385.5 us (-64%) 1250.0000 (-81%) 5144.53 KB (-80%)

Jint.Benchmark.ArrayStressBenchmark

Diff Method N Mean Gen 0 Gen 1 Gen 2 Allocated
Old Jint 20 3.633 s 771250.0000 40250.0000 16500.0000 3.11 GB
New 1.935 s (-47%) 93250.0000 (-88%) 13000.0000 (-68%) - 388.02 MB (-88%)

Jint.Benchmark.EvaluationBenchmark

Diff Method N Mean Gen 0 Allocated
Old Jint 20 1.051 ms 246.0938 1015.63 KB
New 1.615 ms (+54%) 138.6719 (-44%) 569.69 KB (-44%)

Jint.Benchmark.LinqJsBenchmark

Diff Method N Mean Gen 0 Gen 1 Gen 2 Allocated
Old Jint 10 113.8 ms 8562.5000 3000.0000 1437.5000 50.91 MB
New 71.69 ms (-37%) 5125.0000 (-40%) 2562.5000 (-15%) - 30.86 MB (-39%)

Jint.Benchmark.MinimalScriptBenchmark

Diff Method N Mean Gen 0 Allocated
Old Jint 10 28.36 us 9.1248 37.42 KB
New 22.68 us (-20%) 8.5144 (-7%) 34.92 KB (-7%)

Jint.Benchmark.StringIndexingBenchmark

Diff Method N Mean Gen 0 Gen 1 Allocated
Old Jint 1 2.920 s 102250.0000 250.0000 409.33 MB
New 2.606 s (-11%) 40500.0000 (-60%) 250.0000 (0%) 162.98 MB (-60%)

Jint.Benchmark.UncacheableExpressionsBenchmark

Diff Method N Mean Gen 0 Gen 1 Allocated
Old Benchmark 500 1.169 s 263458.3333 78020.8333 1.08 GB
New 511.4 ms (-56%) 100000.0000 (-62%) 1312.5000 (-98%) 400.31 MB (-63%)
lahma commented 6 years ago

I'm closing this issue now as the release has been made to NuGet and and next fixes probably deserve their own tracking. See numbers above for improvements against the starting point and current 2.x release.

ayende commented 6 years ago

Just to let you know that I'm drooling over these improvement numbers.

sebastienros commented 6 years ago

Should I file an issue to track why EvaluationBenchmark went up 50%

lahma commented 6 years ago

Seems like Esprima problem, was throwing a lot of exceptions for number parsing: https://github.com/sebastienros/esprima-dotnet/pull/20 .

sebastienros commented 6 years ago

Why is it throwing exceptions? I don't see anything that overflows in this script.

lahma commented 6 years ago

Long.parse threw for decimals. Could probably record seeing period and skip trying long.parse.

ayende commented 6 years ago

TryParse ? Or manually parse this, since this doesn't need to do anything sophisticated regarding cultures, etc.

lahma commented 6 years ago

Just updating status for latest NuGet release 3.0.0-beta-1138

2.11.58 stable vs 3.0.0-beta-1138


BenchmarkDotNet=v0.10.12, OS=Windows 10 Redstone 3 [1709, Fall Creators Update] (10.0.16299.192)
Intel Core i7-6820HQ CPU 2.70GHz (Skylake), 1 CPU, 8 logical cores and 4 physical cores
Frequency=2648436 Hz, Resolution=377.5813 ns, Timer=TSC
.NET Core SDK=2.1.100-preview-007326
  [Host]     : .NET Core 2.0.5 (Framework 4.6.26020.03), 64bit RyuJIT
  DefaultJob : .NET Core 2.0.5 (Framework 4.6.26020.03), 64bit RyuJIT

Esprima.Benchmark.DromaeoBenchmark

Diff Method FileName Mean Gen 0 Gen 1 Gen 2 Allocated
Old Run dromaeo-3d-cube 162.75 ms 32250.0000 250.0000 - 131.94 MB
New 107.09 ms (-34%) 7437.5000 (-77%) 250.0000 (0%) - 31.67 MB (-76%)
Old Run dromaeo-core-eval 33.69 ms 12250.0000 - - 49.68 MB
New 25.92 ms (-23%) 1718.7500 (-86%) - - 6.9 MB (-86%)
Old Run dromaeo-object-array 1,955.27 ms 435000.0000 7500.0000 2000.0000 1763.05 MB
New 270.44 ms (-86%) 45125.0000 (-90%) 2062.5000 (-73%) 1000.0000 (-50%) 186.35 MB (-89%)
Old Run dromaeo-object-regexp 3,462.88 ms 373000.0000 88750.0000 44166.6667 1939.8 MB
New 1,168.48 ms (-66%) 71812.5000 (-81%) 35312.5000 (-60%) 22250.0000 (-50%) 479.86 MB (-75%)
Old Run dromaeo-object-string 6,357.63 ms 4638000.0000 2800083.3333 2786416.6667 17418.93 MB
New 1,584.29 ms (-75%) 249500.0000 (-95%) 183437.5000 (-93%) 179187.5000 (-94%) 1457.82 MB (-92%)
Old Run dromaeo-string-base64 510.55 ms 247750.0000 1500.0000 500.0000 997.99 MB
New 236.82 ms (-54%) 16062.5000 (-94%) 375.0000 (-75%) - 65.37 MB (-93%)

Esprima.Benchmark.SunSpiderBenchmark

Diff Method FileName Mean Gen 0 Gen 1 Gen 2 Allocated
Old Run 3d-cube 1,130.3 ms 200000.0000 250.0000 - 803.08 MB
New 848.7 ms (-25%) 63687.5000 (-68%) 312.5000 (+25%) 62.5000 256.97 MB (-68%)
Old Run 3d-morph 1,099.6 ms 173750.0000 47750.0000 8000.0000 685.08 MB
New 796.1 ms (-28%) 41937.5000 (-76%) 5875.0000 (-88%) 1812.5000 (-77%) 193.86 MB (-72%)
Old Run 3d-raytrace 987.0 ms 185000.0000 1250.0000 250.0000 743.91 MB
New 648.6 ms (-34%) 53562.5000 (-71%) 1000.0000 (-20%) 125.0000 (-50%) 216.57 MB (-71%)
Old Run access-binary-trees 432.8 ms 83250.0000 750.0000 - 335.17 MB
New 260.5 ms (-40%) 36125.0000 (-57%) 437.5000 (-42%) - 145.24 MB (-57%)
Old Run access-fannkuch 3,322.7 ms 753750.0000 250.0000 - 3015.43 MB
New 2,707.8 ms (-19%) 172437.5000 (-77%) 250.0000 (0%) - 689.76 MB (-77%)
Old Run access-nbody 981.3 ms 159500.0000 - - 638.35 MB
New 764.4 ms (-22%) 57500.0000 (-64%) 62.5000 - 230.06 MB (-64%)
Old Run access-nsieve 1,161.6 ms 241250.0000 21000.0000 5000.0000 1009.95 MB
New 980.3 ms (-16%) 62937.5000 (-74%) 7937.5000 (-62%) 2937.5000 (-41%) 260.87 MB (-74%)
Old Run bitops-3bit-bits-in-byte 703.7 ms 132000.0000 - - 528.68 MB
New 560.9 ms (-20%) 54937.5000 (-58%) - - 219.81 MB (-58%)
Old Run bitops-bits-in-byte 1,120.7 ms 195750.0000 - - 783.66 MB
New 965.9 ms (-14%) 80625.0000 (-59%) 62.5000 - 322.7 MB (-59%)
Old Run bitops-bitwise-and 766.1 ms 86750.0000 - - 347.92 MB
New 653.2 ms (-15%) 45625.0000 (-47%) 62.5000 - 182.65 MB (-48%)
Old Run bitops-nsieve-bits 1,299.6 ms 253000.0000 27250.0000 1750.0000 1015.1 MB
New 1,047.0 ms (-19%) 66750.0000 (-74%) 4375.0000 (-84%) 375.0000 (-79%) 269.04 MB (-73%)
Old Run controlflow-recursive 713.5 ms 135250.0000 2500.0000 - 544.98 MB
New 357.1 ms (-50%) 55687.5000 (-59%) - - 222.91 MB (-59%)
Old Run crypto-aes 1,022.6 ms 244250.0000 250.0000 - 981.45 MB
New 643.3 ms (-37%) 44250.0000 (-82%) 312.5000 (+25%) 62.5000 179.2 MB (-82%)
Old Run crypto-md5 577.0 ms 108250.0000 2500.0000 500.0000 436.1 MB
New 421.5 ms (-27%) 51250.0000 (-53%) 312.5000 (-88%) 62.5000 (-88%) 207.1 MB (-53%)
Old Run crypto-sha1 601.2 ms 110500.0000 1000.0000 250.0000 444.44 MB
New 417.5 ms (-31%) 47687.5000 (-57%) 312.5000 (-69%) 62.5000 (-75%) 192.39 MB (-57%)
Old Run date-format-tofte 705.5 ms 145500.0000 250.0000 - 583.33 MB
New 433.6 ms (-39%) 36062.5000 (-75%) - - 144.36 MB (-75%)
Old Run date-format-xparb 488.0 ms 45500.0000 250.0000 - 184.11 MB
New 373.4 ms (-23%) 18062.5000 (-60%) 250.0000 (0%) - 72.86 MB (-60%)
Old Run math-cordic 1,646.3 ms 284750.0000 - - 1139.63 MB
New 1,253.7 ms (-24%) 110000.0000 (-61%) 125.0000 - 440.03 MB (-61%)
Old Run math-partial-sums 516.1 ms 63000.0000 - - 252.49 MB
New 401.5 ms (-22%) 26812.5000 (-57%) - - 107.29 MB (-58%)
Old Run math-spectral-norm 733.9 ms 129000.0000 - - 516.31 MB
New 475.1 ms (-35%) 48312.5000 (-63%) - - 193.43 MB (-63%)
Old Run regexp-dna 344.3 ms 2750.0000 2250.0000 1750.0000 20.84 MB
New 343.3 ms (0%) 2000.0000 (-27%) 1687.5000 (-25%) 1375.0000 (-21%) 13.46 MB (-35%)
Old Run string-base64 735.0 ms 335750.0000 250.0000 - 1346.89 MB
New 350.1 ms (-52%) 23812.5000 (-93%) 250.0000 (0%) - 96.61 MB (-93%)
Old Run string-fasta 1,054.2 ms 201000.0000 - - 804.94 MB
New 601.0 ms (-43%) 57875.0000 (-71%) - - 231.69 MB (-71%)
Old Run string-tagcloud 936.2 ms 204500.0000 136583.3333 128083.3333 1052.84 MB
New 257.0 ms (-73%) 26187.5000 (-87%) 2125.0000 (-98%) 750.0000 (-99%) 113.18 MB (-89%)
Old Run string-unpack-code 324.8 ms 60000.0000 4000.0000 1250.0000 255.12 MB
New 202.0 ms (-38%) 27187.5000 (-55%) 6250.0000 (+56%) 1375.0000 (+10%) 118.12 MB (-54%)
Old Run string-validate-input 3,326.3 ms 1641750.0000 1566083.3333 1565500.0000 6427.62 MB
New 241.7 ms (-93%) 20375.0000 (-99%) 375.0000 (-100%) 62.5000 (-100%) 84.8 MB (-99%)

Jint.Benchmark.ArrayBenchmark

Diff Method N Mean Gen 0 Allocated
Old Slice 100 6.998 ms 3562.5000 14.27 MB
New 672.9 us (-90%) 149.4141 (-96%) 615.63 KB (-96%)
Old Concat 100 6.146 ms 2515.6250 10.09 MB
New 827.7 us (-87%) 166.0156 (-93%) 683.59 KB (-93%)
Old Unshift 100 190.064 ms 47125.0000 188.62 MB
New 30,986.7 us (-84%) 3687.5000 (-92%) 15212.5 KB (-92%)
Old Push 100 36.007 ms 8000.0000 32.01 MB
New 17,942.8 us (-50%) 1406.2500 (-82%) 5846.09 KB (-82%)
Old Index 100 27.367 ms 5781.2500 23.15 MB
New 16,693.3 us (-39%) 1250.0000 (-78%) 5171.09 KB (-78%)
Old Map 100 13.261 ms 4187.5000 16.78 MB
New 4,334.2 us (-67%) 1437.5000 (-66%) 5903.13 KB (-65%)
Old Apply 100 8.070 ms 2968.7500 11.91 MB
New 942.3 us (-88%) 183.5938 (-94%) 752.34 KB (-94%)
Old JsonStringifyParse 100 14.856 ms 6437.5000 25.79 MB
New 5,271.2 us (-65%) 1250.0000 (-81%) 5140.63 KB (-80%)

Jint.Benchmark.ArrayStressBenchmark

Diff Method N Mean Gen 0 Gen 1 Gen 2 Allocated
Old Jint 20 3.633 s 771250.0000 40250.0000 16500.0000 3.11 GB
New 1.706 s (-53%) 92437.5000 (-88%) 12562.5000 (-69%) - 385.91 MB (-88%)

Jint.Benchmark.EvaluationBenchmark

Diff Method N Mean Gen 0 Allocated
Old Jint 20 1.051 ms 246.0938 1015.63 KB
New 765.5 us (-27%) 134.7656 (-45%) 554.69 KB (-45%)

Jint.Benchmark.LinqJsBenchmark

Diff Method N Mean Gen 0 Gen 1 Gen 2 Allocated
Old Jint 10 113.8 ms 8562.5000 3000.0000 1437.5000 50.91 MB
New 76.41 ms (-33%) 5187.5000 (-39%) 2500.0000 (-17%) 187.5000 (-87%) 30.71 MB (-40%)

Jint.Benchmark.MinimalScriptBenchmark

Diff Method N Mean Gen 0 Allocated
Old Jint 10 28.36 us 9.1248 37.42 KB
New 23.65 us (-17%) 8.5144 (-7%) 34.92 KB (-7%)

Jint.Benchmark.UncacheableExpressionsBenchmark

Diff Method N Mean Gen 0 Gen 1 Allocated
Old Benchmark 500 1.169 s 263458.3333 78020.8333 1.08 GB
New 481.7 ms (-59%) 94187.5000 (-64%) 250.0000 (-100%) 377.11 MB (-65%)
lahma commented 6 years ago

Updating the latest dev branch 5d408955dc904300e543d27bb12ad10c19327b84 performance against 2.11.58 stable in case anyone is interested.


BenchmarkDotNet=v0.10.14, OS=Windows 10.0.17134
Intel Core i7-6820HQ CPU 2.70GHz (Skylake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.1.200-preview-007576
  [Host]     : .NET Core 2.0.7 (CoreCLR 4.6.26328.01, CoreFX 4.6.26403.03), 64bit RyuJIT
  DefaultJob : .NET Core 2.0.7 (CoreCLR 4.6.26328.01, CoreFX 4.6.26403.03), 64bit RyuJIT

Esprima.Benchmark.DromaeoBenchmark

Diff Method FileName Mean Gen 0 Gen 1 Gen 2 Allocated
Old Run dromaeo-3d-cube 162.75 ms 32250.0000 250.0000 - 131.94 MB
New 81.43 ms (-50%) 1437.5000 (-96%) 500.0000 (+100%) - 7749.97 KB (-94%)
Old Run dromaeo-core-eval 33.69 ms 12250.0000 - - 49.68 MB
New 23.91 ms (-29%) 62.5000 (-99%) - - 299.02 KB (-99%)
Old Run dromaeo-object-array 1,955.27 ms 435000.0000 7500.0000 2000.0000 1763.05 MB
New 197.28 ms (-90%) 43312.5000 (-90%) 2000.0000 (-73%) 1000.0000 (-50%) 181953.89 KB (-90%)
Old Run dromaeo-object-regexp 3,462.88 ms 373000.0000 88750.0000 44166.6667 1939.8 MB
New 1,093.45 ms (-68%) 61750.0000 (-83%) 33000.0000 (-63%) 22500.0000 (-49%) 466736.45 KB (-76%)
Old Run dromaeo-object-string 6,357.63 ms 4638000.0000 2800083.3333 2786416.6667 17418.93 MB
New 1,245.35 ms (-80%) 220000.0000 (-95%) 176500.0000 (-94%) 173000.0000 (-94%) 1403845.43 KB (-92%)
Old Run dromaeo-string-base64 510.55 ms 247750.0000 1500.0000 500.0000 997.99 MB
New 184.81 ms (-64%) 6062.5000 (-98%) 437.5000 (-71%) - 26624.45 KB (-97%)

Esprima.Benchmark.SunSpiderBenchmark

Diff Method FileName Mean Gen 0 Gen 1 Gen 2 Allocated
Old Run 3d-cube 1,130.3 ms 200000.0000 250.0000 - 803.08 MB
New 651.2 ms (-42%) 12000.0000 (-94%) 312.5000 (+25%) - 50.06 MB (-94%)
Old Run 3d-morph 1,099.6 ms 173750.0000 47750.0000 8000.0000 685.08 MB
New 605.4 ms (-45%) 11812.5000 (-93%) 2500.0000 (-95%) 1062.5000 (-87%) 68.03 MB (-90%)
Old Run 3d-raytrace 987.0 ms 185000.0000 1250.0000 250.0000 743.91 MB
New 567.9 ms (-42%) 22562.5000 (-88%) 750.0000 (-40%) 125.0000 (-50%) 92.34 MB (-88%)
Old Run access-binary-trees 432.8 ms 83250.0000 750.0000 - 335.17 MB
New 213.8 ms (-51%) 17625.0000 (-79%) 312.5000 (-58%) 62.5000 73 MB (-78%)
Old Run access-fannkuch 3,322.7 ms 753750.0000 250.0000 - 3015.43 MB
New 1,973.5 ms (-41%) 9437.5000 (-99%) 187.5000 (-25%) - 37.87 MB (-99%)
Old Run access-nbody 981.3 ms 159500.0000 - - 638.35 MB
New 589.8 ms (-40%) 13125.0000 (-92%) - - 52.6 MB (-92%)
Old Run access-nsieve 1,161.6 ms 241250.0000 21000.0000 5000.0000 1009.95 MB
New 745.1 ms (-36%) 15312.5000 (-94%) 3937.5000 (-81%) 1875.0000 (-63%) 73.09 MB (-93%)
Old Run bitops-3bit-bits-in-byte 703.7 ms 132000.0000 - - 528.68 MB
New 452.4 ms (-36%) 19750.0000 (-85%) - - 79.06 MB (-85%)
Old Run bitops-bits-in-byte 1,120.7 ms 195750.0000 - - 783.66 MB
New 769.0 ms (-31%) 12437.5000 (-94%) 62.5000 - 49.86 MB (-94%)
Old Run bitops-bitwise-and 766.1 ms 86750.0000 - - 347.92 MB
New 454.4 ms (-41%) 13562.5000 (-84%) - - 54.32 MB (-84%)
Old Run bitops-nsieve-bits 1,299.6 ms 253000.0000 27250.0000 1750.0000 1015.1 MB
New 761.4 ms (-41%) 21625.0000 (-91%) 250.0000 (-99%) 62.5000 (-96%) 88.65 MB (-91%)
Old Run controlflow-recursive 713.5 ms 135250.0000 2500.0000 - 544.98 MB
New 302.3 ms (-58%) 25937.5000 (-81%) 62.5000 (-98%) - 103.78 MB (-81%)
Old Run crypto-aes 1,022.6 ms 244250.0000 250.0000 - 981.45 MB
New 489.4 ms (-52%) 5625.0000 (-98%) 250.0000 (0%) - 24.49 MB (-98%)
Old Run crypto-md5 577.0 ms 108250.0000 2500.0000 500.0000 436.1 MB
New 349.4 ms (-39%) 30562.5000 (-72%) 187.5000 (-93%) - 124.99 MB (-71%)
Old Run crypto-sha1 601.2 ms 110500.0000 1000.0000 250.0000 444.44 MB
New 344.9 ms (-43%) 24875.0000 (-77%) 312.5000 (-69%) - 101.48 MB (-77%)
Old Run date-format-tofte 705.5 ms 145500.0000 250.0000 - 583.33 MB
New 340.7 ms (-52%) 12625.0000 (-91%) - - 50.73 MB (-91%)
Old Run date-format-xparb 488.0 ms 45500.0000 250.0000 - 184.11 MB
New 335.6 ms (-31%) 10375.0000 (-77%) 250.0000 (0%) - 42.27 MB (-77%)
Old Run math-cordic 1,646.3 ms 284750.0000 - - 1139.63 MB
New 990.1 ms (-40%) 22562.5000 (-92%) 62.5000 - 90.25 MB (-92%)
Old Run math-partial-sums 516.1 ms 63000.0000 - - 252.49 MB
New 312.5 ms (-39%) 7250.0000 (-88%) - - 29.04 MB (-88%)
Old Run math-spectral-norm 733.9 ms 129000.0000 - - 516.31 MB
New 369.2 ms (-50%) 18062.5000 (-86%) - - 72.31 MB (-86%)
Old Run regexp-dna 344.3 ms 2750.0000 2250.0000 1750.0000 20.84 MB
New 341.6 ms (-1%) 2250.0000 (-18%) 1812.5000 (-19%) 1500.0000 (-14%) 13.55 MB (-35%)
Old Run string-base64 735.0 ms 335750.0000 250.0000 - 1346.89 MB
New 278.0 ms (-62%) 8875.0000 (-97%) 500.0000 (+100%) - 37.16 MB (-97%)
Old Run string-fasta 1,054.2 ms 201000.0000 - - 804.94 MB
New 478.5 ms (-55%) 27125.0000 (-87%) 62.5000 - 108.53 MB (-87%)
Old Run string-tagcloud 936.2 ms 204500.0000 136583.3333 128083.3333 1052.84 MB
New 219.3 ms (-77%) 14687.5000 (-93%) 1750.0000 (-99%) 687.5000 (-99%) 67.2 MB (-94%)
Old Run string-unpack-code 324.8 ms 60000.0000 4000.0000 1250.0000 255.12 MB
New 170.7 ms (-47%) 14062.5000 (-77%) 3937.5000 (-2%) 937.5000 (-25%) 68.64 MB (-73%)
Old Run string-validate-input 3,326.3 ms 1641750.0000 1566083.3333 1565500.0000 6427.62 MB
New 189.6 ms (-94%) 7312.5000 (-100%) 312.5000 (-100%) 62.5000 (-100%) 34.26 MB (-99%)

Jint.Benchmark.ArrayBenchmark

Diff Method N Mean Gen 0 Allocated
Old Slice 100 6.998 ms 3562.5000 14.27 MB
New 512.2 us (-93%) 161.1328 (-95%) 660.16 KB (-95%)
Old Concat 100 6.146 ms 2515.6250 10.09 MB
New 607.1 us (-90%) 175.7813 (-93%) 720.31 KB (-93%)
Old Unshift 100 190.064 ms 47125.0000 188.62 MB
New 20,136.0 us (-89%) 3562.5000 (-92%) 14672.66 KB (-92%)
Old Push 100 36.007 ms 8000.0000 32.01 MB
New 13,198.6 us (-63%) 515.6250 (-94%) 2134.38 KB (-93%)
Old Index 100 27.367 ms 5781.2500 23.15 MB
New 12,839.2 us (-53%) 390.6250 (-93%) 1637.5 KB (-93%)
Old Map 100 13.261 ms 4187.5000 16.78 MB
New 3,683.2 us (-72%) 765.6250 (-82%) 3151.56 KB (-81%)
Old Apply 100 8.070 ms 2968.7500 11.91 MB
New 707.6 us (-91%) 190.4297 (-94%) 782.81 KB (-93%)
Old JsonStringifyParse 100 14.856 ms 6437.5000 25.79 MB
New 4,687.4 us (-68%) 1273.4375 (-80%) 5233.59 KB (-80%)

Jint.Benchmark.ArrayStressBenchmark

Diff Method N Mean Gen 0 Gen 1 Gen 2 Allocated
Old Jint 20 3.633 s 771250.0000 40250.0000 16500.0000 3.11 GB
New 696.1 ms (-81%) 60000.0000 (-92%) 8312.5000 (-79%) - 256.64 MB (-92%)

Jint.Benchmark.EvaluationBenchmark

Diff Method N Mean Gen 0 Allocated
Old Jint 20 1.051 ms 246.0938 1015.63 KB
New 702.0 us (-33%) 120.1172 (-51%) 492.5 KB (-52%)

Jint.Benchmark.LinqJsBenchmark

Diff Method N Mean Gen 0 Gen 1 Gen 2 Allocated
Old Jint 10 113.8 ms 8562.5000 3000.0000 1437.5000 50.91 MB
New 73.84 ms (-35%) 5062.5000 (-41%) 2500.0000 (-17%) - 30.56 MB (-40%)

Jint.Benchmark.MinimalScriptBenchmark

Diff Method N Mean Gen 0 Allocated
Old Jint 10 28.36 us 9.1248 37.42 KB
New 18.97 us (-33%) 8.1482 (-11%) 33.44 KB (-11%)

Jint.Benchmark.UncacheableExpressionsBenchmark

Diff Method N Mean Gen 0 Gen 1 Allocated
Old Benchmark 500 1.169 s 263458.3333 78020.8333 1.08 GB
New 389.6 ms (-67%) 50187.5000 (-81%) 10000.0000 (-87%) 211.26 MB (-80%)
lahma commented 6 years ago

There's a new release on NuGet so time to share latest numbers. We are pretty much faster across the board.

2.11.58 vs 3.0.0-beta-1210


BenchmarkDotNet=v0.10.14, OS=Windows 10.0.17134
Intel Core i7-6820HQ CPU 2.70GHz (Skylake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.1.300
  [Host]     : .NET Core 2.0.7 (CoreCLR 4.6.26328.01, CoreFX 4.6.26403.03), 64bit RyuJIT
  Job-YBOZLP : .NET Core 2.0.7 (CoreCLR 4.6.26328.01, CoreFX 4.6.26403.03), 64bit RyuJIT

LaunchCount=1  TargetCount=15  WarmupCount=10

Esprima.Benchmark.DromaeoBenchmark

Diff Method FileName Mean Gen 0 Gen 1 Gen 2 Allocated
Old Run dromaeo-3d-cube 162.75 ms 32250.0000 250.0000 - 131.94 MB
New 74.51 ms (-54%) 1375.0000 (-96%) 250.0000 (0%) - 7662.21 KB (-94%)
Old Run dromaeo-core-eval 33.69 ms 12250.0000 - - 49.68 MB
New 15.75 ms (-53%) 62.5000 (-99%) - - 298.7 KB (-99%)
Old Run dromaeo-object-array 1,955.27 ms 435000.0000 7500.0000 2000.0000 1763.05 MB
New 157.05 ms (-92%) 42437.5000 (-90%) 1937.5000 (-74%) 937.5000 (-53%) 178313.12 KB (-90%)
Old Run dromaeo-object-regexp 3,462.88 ms 373000.0000 88750.0000 44166.6667 1939.8 MB
New 891.11 ms (-74%) 53625.0000 (-86%) 32312.5000 (-64%) 21687.5000 (-51%) 414359.71 KB (-79%)
Old Run dromaeo-object-string 6,357.63 ms 4638000.0000 2800083.3333 2786416.6667 17418.93 MB
New 1,198.01 ms (-81%) 218062.5000 (-95%) 174812.5000 (-94%) 172375.0000 (-94%) 1392917.21 KB (-92%)
Old Run dromaeo-string-base64 510.55 ms 247750.0000 1500.0000 500.0000 997.99 MB
New 169.39 ms (-67%) 6375.0000 (-97%) 312.5000 (-79%) - 27381.53 KB (-97%)

Esprima.Benchmark.SunSpiderBenchmark

Diff Method FileName Mean Gen 0 Gen 1 Gen 2 Allocated
Old Run 3d-cube 1,130.3 ms 200000.0000 250.0000 - 803.08 MB
New 602.3 ms (-47%) 11875.0000 (-94%) 312.5000 (+25%) - 49.56 MB (-94%)
Old Run 3d-morph 1,099.6 ms 173750.0000 47750.0000 8000.0000 685.08 MB
New 565.8 ms (-49%) 11812.5000 (-93%) 2500.0000 (-95%) 1062.5000 (-87%) 68.03 MB (-90%)
Old Run 3d-raytrace 987.0 ms 185000.0000 1250.0000 250.0000 743.91 MB
New 486.8 ms (-51%) 21875.0000 (-88%) 750.0000 (-40%) 125.0000 (-50%) 89.75 MB (-88%)
Old Run access-binary-trees 432.8 ms 83250.0000 750.0000 - 335.17 MB
New 203.4 ms (-53%) 16500.0000 (-80%) 437.5000 (-42%) 62.5000 68.51 MB (-80%)
Old Run access-fannkuch 3,322.7 ms 753750.0000 250.0000 - 3015.43 MB
New 1,872.9 ms (-44%) 9437.5000 (-99%) 125.0000 (-50%) - 37.87 MB (-99%)
Old Run access-nbody 981.3 ms 159500.0000 - - 638.35 MB
New 563.8 ms (-43%) 13062.5000 (-92%) - - 52.42 MB (-92%)
Old Run access-nsieve 1,161.6 ms 241250.0000 21000.0000 5000.0000 1009.95 MB
New 716.9 ms (-38%) 15375.0000 (-94%) 3937.5000 (-81%) 1875.0000 (-63%) 73.09 MB (-93%)
Old Run bitops-3bit-bits-in-byte 703.7 ms 132000.0000 - - 528.68 MB
New 404.7 ms (-42%) 17562.5000 (-87%) - - 70.27 MB (-87%)
Old Run bitops-bits-in-byte 1,120.7 ms 195750.0000 - - 783.66 MB
New 663.6 ms (-41%) 11562.5000 (-94%) 62.5000 - 46.44 MB (-94%)
Old Run bitops-bitwise-and 766.1 ms 86750.0000 - - 347.92 MB
New 427.2 ms (-44%) 9000.0000 (-90%) - - 36.01 MB (-90%)
Old Run bitops-nsieve-bits 1,299.6 ms 253000.0000 27250.0000 1750.0000 1015.1 MB
New 697.3 ms (-46%) 21625.0000 (-91%) 125.0000 (-100%) - 88.5 MB (-91%)
Old Run controlflow-recursive 713.5 ms 135250.0000 2500.0000 - 544.98 MB
New 292.0 ms (-59%) 23625.0000 (-83%) - - 94.61 MB (-83%)
Old Run crypto-aes 1,022.6 ms 244250.0000 250.0000 - 981.45 MB
New 435.4 ms (-57%) 5500.0000 (-98%) 250.0000 (0%) - 24.14 MB (-98%)
Old Run crypto-md5 577.0 ms 108250.0000 2500.0000 500.0000 436.1 MB
New 322.3 ms (-44%) 28000.0000 (-74%) 250.0000 (-90%) 62.5000 (-88%) 114.57 MB (-74%)
Old Run crypto-sha1 601.2 ms 110500.0000 1000.0000 250.0000 444.44 MB
New 306.3 ms (-49%) 22687.5000 (-79%) 312.5000 (-69%) 62.5000 (-75%) 92.28 MB (-79%)
Old Run date-format-tofte 705.5 ms 145500.0000 250.0000 - 583.33 MB
New 325.1 ms (-54%) 11875.0000 (-92%) - - 47.71 MB (-92%)
Old Run date-format-xparb 488.0 ms 45500.0000 250.0000 - 184.11 MB
New 328.9 ms (-33%) 10000.0000 (-78%) 250.0000 (0%) - 40.85 MB (-78%)
Old Run math-cordic 1,646.3 ms 284750.0000 - - 1139.63 MB
New 914.1 ms (-44%) 20562.5000 (-93%) 62.5000 - 82.43 MB (-93%)
Old Run math-partial-sums 516.1 ms 63000.0000 - - 252.49 MB
New 293.1 ms (-43%) 7000.0000 (-89%) - - 28.07 MB (-89%)
Old Run math-spectral-norm 733.9 ms 129000.0000 - - 516.31 MB
New 348.2 ms (-53%) 16875.0000 (-87%) - - 67.63 MB (-87%)
Old Run regexp-dna 344.3 ms 2750.0000 2250.0000 1750.0000 20.84 MB
New 342.7 ms (0%) 2187.5000 (-20%) 1875.0000 (-17%) 1500.0000 (-14%) 13.55 MB (-35%)
Old Run string-base64 735.0 ms 335750.0000 250.0000 - 1346.89 MB
New 256.9 ms (-65%) 9312.5000 (-97%) 375.0000 (+50%) - 38.79 MB (-97%)
Old Run string-fasta 1,054.2 ms 201000.0000 - - 804.94 MB
New 432.6 ms (-59%) 26750.0000 (-87%) - - 107.01 MB (-87%)
Old Run string-tagcloud 936.2 ms 204500.0000 136583.3333 128083.3333 1052.84 MB
New 205.8 ms (-78%) 14000.0000 (-93%) 1687.5000 (-99%) 625.0000 (-100%) 64.92 MB (-94%)
Old Run string-unpack-code 324.8 ms 60000.0000 4000.0000 1250.0000 255.12 MB
New 157.0 ms (-52%) 13750.0000 (-77%) 4750.0000 (+19%) 875.0000 (-30%) 64.88 MB (-75%)
Old Run string-validate-input 3,326.3 ms 1641750.0000 1566083.3333 1565500.0000 6427.62 MB
New 176.1 ms (-95%) 7312.5000 (-100%) 375.0000 (-100%) 62.5000 (-100%) 33.53 MB (-99%)

Jint.Benchmark.ArrayBenchmark

Diff Method N Mean Gen 0 Allocated
Old Slice 100 6.998 ms 3562.5000 14.27 MB
New 444.1 us (-94%) 161.1328 (-95%) 660.16 KB (-95%)
Old Concat 100 6.146 ms 2515.6250 10.09 MB
New 478.3 us (-92%) 175.7813 (-93%) 720.31 KB (-93%)
Old Unshift 100 190.064 ms 47125.0000 188.62 MB
New 17,875.9 us (-91%) 3562.5000 (-92%) 14672.66 KB (-92%)
Old Push 100 36.007 ms 8000.0000 32.01 MB
New 10,700.9 us (-70%) 343.7500 (-96%) 1438.28 KB (-96%)
Old Index 100 27.367 ms 5781.2500 23.15 MB
New 11,988.4 us (-56%) 390.6250 (-93%) 1637.5 KB (-93%)
Old Map 100 13.261 ms 4187.5000 16.78 MB
New 3,191.3 us (-76%) 691.4063 (-83%) 2833.59 KB (-83%)
Old Apply 100 8.070 ms 2968.7500 11.91 MB
New 577.0 us (-93%) 188.4766 (-94%) 774.22 KB (-93%)
Old JsonStringifyParse 100 14.856 ms 6437.5000 25.79 MB
New 4,517.9 us (-70%) 1273.4375 (-80%) 5225.78 KB (-80%)

Jint.Benchmark.ArrayStressBenchmark

Diff Method N Mean Gen 0 Gen 1 Gen 2 Allocated
Old Jint 20 3.633 s 771250.0000 40250.0000 16500.0000 3.11 GB
New 592.4 ms (-84%) 60000.0000 (-92%) 2937.5000 (-93%) - 245.9 MB (-92%)

Jint.Benchmark.EvaluationBenchmark

Diff Method N Mean Gen 0 Allocated
Old Jint 20 1.051 ms 246.0938 1015.63 KB
New 670.4 us (-36%) 119.1406 (-52%) 488.91 KB (-52%)

Jint.Benchmark.LinqJsBenchmark

Diff Method N Mean Gen 0 Gen 1 Gen 2 Allocated
Old Jint 10 113.8 ms 8562.5000 3000.0000 1437.5000 50.91 MB
New 71.95 ms (-37%) 5125.0000 (-40%) 2500.0000 (-17%) 125.0000 (-91%) 30.57 MB (-40%)

Jint.Benchmark.MinimalScriptBenchmark

Diff Method N Mean Gen 0 Allocated
Old Jint 10 28.36 us 9.1248 37.42 KB
New 18.42 us (-35%) 8.1482 (-11%) 33.44 KB (-11%)

Jint.Benchmark.UncacheableExpressionsBenchmark

Diff Method N Mean Gen 0 Gen 1 Allocated
Old Benchmark 500 1.169 s 263458.3333 78020.8333 1.08 GB
New 336.2 ms (-71%) 48625.0000 (-82%) 11250.0000 (-86%) 196.66 MB (-82%)
lahma commented 6 years ago

There's a new release on NuGet so time to share latest numbers. This time memory usage was further reduced and some micro optimizations were made.

2.11.58 vs 3.0.0-beta-1249


BenchmarkDotNet=v0.10.14, OS=Windows 10.0.17713
Intel Core i7-6820HQ CPU 2.70GHz (Skylake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=2.1.302
  [Host]     : .NET Core 2.0.9 (CoreCLR 4.6.26614.01, CoreFX 4.6.26614.01), 64bit RyuJIT
  DefaultJob : .NET Core 2.0.9 (CoreCLR 4.6.26614.01, CoreFX 4.6.26614.01), 64bit RyuJIT

Esprima.Benchmark.DromaeoBenchmark

Diff Method FileName Mean Gen 0 Gen 1 Gen 2 Allocated
Old Run dromaeo-3d-cube 162.75 ms 32250.0000 250.0000 - 131.94 MB
New 71.57 ms (-56%) 1312.5000 (-96%) 250.0000 (0%) - 7541784 B (-95%)
Old Run dromaeo-core-eval 33.69 ms 12250.0000 - - 49.68 MB
New 14.85 ms (-56%) 62.5000 (-99%) - - 304976 B (-99%)
Old Run dromaeo-object-array 1,955.27 ms 435000.0000 7500.0000 2000.0000 1763.05 MB
New 165.39 ms (-92%) 42437.5000 (-90%) 1937.5000 (-74%) 937.5000 (-53%) 182590916 B (-90%)
Old Run dromaeo-object-regexp 3,462.88 ms 373000.0000 88750.0000 44166.6667 1939.8 MB
New 887.14 ms (-74%) 53375.0000 (-86%) 31625.0000 (-64%) 21250.0000 (-52%) 424122202 B (-79%)
Old Run dromaeo-object-string 6,357.63 ms 4638000.0000 2800083.3333 2786416.6667 17418.93 MB
New 1,196.51 ms (-81%) 219562.5000 (-95%) 176875.0000 (-94%) 173937.5000 (-94%) 1426761422 B (-92%)
Old Run dromaeo-string-base64 510.55 ms 247750.0000 1500.0000 500.0000 997.99 MB
New NA N/A N/A N/A N/A

Esprima.Benchmark.SunSpiderBenchmark

Diff Method FileName Mean Gen 0 Gen 1 Gen 2 Allocated
Old Run 3d-cube 1,130.3 ms 200000.0000 250.0000 - 803.08 MB
New 555.9 ms (-51%) 11375.0000 (-94%) 312.5000 (+25%) - 49874152 B (-94%)
Old Run 3d-morph 1,099.6 ms 173750.0000 47750.0000 8000.0000 685.08 MB
New 546.9 ms (-50%) 11812.5000 (-93%) 2500.0000 (-95%) 1062.5000 (-87%) 71333952 B (-90%)
Old Run 3d-raytrace 987.0 ms 185000.0000 1250.0000 250.0000 743.91 MB
New 460.3 ms (-53%) 19000.0000 (-90%) 875.0000 (-30%) 125.0000 (-50%) 82207560 B (-89%)
Old Run access-binary-trees 432.8 ms 83250.0000 750.0000 - 335.17 MB
New 184.1 ms (-57%) 13000.0000 (-84%) 375.0000 (-50%) - 56692984 B (-84%)
Old Run access-fannkuch 3,322.7 ms 753750.0000 250.0000 - 3015.43 MB
New 1,765.4 ms (-47%) 9437.5000 (-99%) 125.0000 (-50%) - 39708288 B (-99%)
Old Run access-nbody 981.3 ms 159500.0000 - - 638.35 MB
New 566.0 ms (-42%) 12812.5000 (-92%) - - 53803424 B (-92%)
Old Run access-nsieve 1,161.6 ms 241250.0000 21000.0000 5000.0000 1009.95 MB
New 663.9 ms (-43%) 15375.0000 (-94%) 3937.5000 (-81%) 1875.0000 (-63%) 76642850 B (-93%)
Old Run bitops-3bit-bits-in-byte 703.7 ms 132000.0000 - - 528.68 MB
New 348.1 ms (-51%) 12625.0000 (-90%) - - 53208176 B (-90%)
Old Run bitops-bits-in-byte 1,120.7 ms 195750.0000 - - 783.66 MB
New 612.1 ms (-45%) 8187.5000 (-96%) - - 34358224 B (-96%)
Old Run bitops-bitwise-and 766.1 ms 86750.0000 - - 347.92 MB
New 477.2 ms (-38%) 9000.0000 (-90%) - - 37758744 B (-90%)
Old Run bitops-nsieve-bits 1,299.6 ms 253000.0000 27250.0000 1750.0000 1015.1 MB
New 674.9 ms (-48%) 21625.0000 (-91%) 125.0000 (-100%) - 92795080 B (-91%)
Old Run controlflow-recursive 713.5 ms 135250.0000 2500.0000 - 544.98 MB
New 243.2 ms (-66%) 16187.5000 (-88%) - - 68108952 B (-88%)
Old Run crypto-aes 1,022.6 ms 244250.0000 250.0000 - 981.45 MB
New 424.2 ms (-59%) 5125.0000 (-98%) 250.0000 (0%) - 23551011 B (-98%)
Old Run crypto-md5 577.0 ms 108250.0000 2500.0000 500.0000 436.1 MB
New 282.8 ms (-51%) 23875.0000 (-78%) 250.0000 (-90%) 62.5000 (-88%) 102475720 B (-78%)
Old Run crypto-sha1 601.2 ms 110500.0000 1000.0000 250.0000 444.44 MB
New 271.9 ms (-55%) 19125.0000 (-83%) 250.0000 (-75%) - 82209984 B (-82%)
Old Run date-format-tofte 705.5 ms 145500.0000 250.0000 - 583.33 MB
New 301.7 ms (-57%) 11187.5000 (-92%) - - 47181728 B (-92%)
Old Run date-format-xparb 488.0 ms 45500.0000 250.0000 - 184.11 MB
New 285.1 ms (-42%) 9062.5000 (-80%) 250.0000 (0%) - 38892896 B (-80%)
Old Run math-cordic 1,646.3 ms 284750.0000 - - 1139.63 MB
New 864.0 ms (-48%) 16687.5000 (-94%) 62.5000 - 70234680 B (-94%)
Old Run math-partial-sums 516.1 ms 63000.0000 - - 252.49 MB
New 282.0 ms (-45%) 7000.0000 (-89%) - - 29428272 B (-89%)
Old Run math-spectral-norm 733.9 ms 129000.0000 - - 516.31 MB
New 304.5 ms (-59%) 13125.0000 (-90%) - - 55206048 B (-90%)
Old Run regexp-dna 344.3 ms 2750.0000 2250.0000 1750.0000 20.84 MB
New 346.4 ms (+1%) 2187.5000 (-20%) 1875.0000 (-17%) 1500.0000 (-14%) 14197886 B (-35%)
Old Run string-base64 735.0 ms 335750.0000 250.0000 - 1346.89 MB
New NA N/A N/A N/A N/A
Old Run string-fasta 1,054.2 ms 201000.0000 - - 804.94 MB
New 394.3 ms (-63%) 24562.5000 (-88%) 62.5000 - 103246608 B (-88%)
Old Run string-tagcloud 936.2 ms 204500.0000 136583.3333 128083.3333 1052.84 MB
New 182.5 ms (-81%) 12375.0000 (-94%) 1562.5000 (-99%) 562.5000 (-100%) 61680540 B (-94%)
Old Run string-unpack-code 324.8 ms 60000.0000 4000.0000 1250.0000 255.12 MB
New 147.1 ms (-55%) 11000.0000 (-82%) 3375.0000 (-16%) 875.0000 (-30%) 57449168 B (-79%)
Old Run string-validate-input 3,326.3 ms 1641750.0000 1566083.3333 1565500.0000 6427.62 MB
New 168.1 ms (-95%) 6750.0000 (-100%) 375.0000 (-100%) 62.5000 (-100%) 32537144 B (-100%)

Jint.Benchmark.ArrayBenchmark

Diff Method N Mean Gen 0 Allocated
Old Slice 100 6.998 ms 3562.5000 14.27 MB
New 444.6 us (-94%) 161.1328 (-95%) 660.16 KB (-95%)
Old Concat 100 6.146 ms 2515.6250 10.09 MB
New 495.4 us (-92%) 175.7813 (-93%) 720.31 KB (-93%)
Old Unshift 100 190.064 ms 47125.0000 188.62 MB
New 18,753.2 us (-90%) 3562.5000 (-92%) 14672.66 KB (-92%)
Old Push 100 36.007 ms 8000.0000 32.01 MB
New 10,397.4 us (-72%) 343.7500 (-96%) 1438.28 KB (-96%)
Old Index 100 27.367 ms 5781.2500 23.15 MB
New 11,708.2 us (-58%) 390.6250 (-93%) 1637.5 KB (-93%)
Old Map 100 13.261 ms 4187.5000 16.78 MB
New 2,599.5 us (-81%) 488.2813 (-88%) 2008.59 KB (-88%)
Old Apply 100 8.070 ms 2968.7500 11.91 MB
New 609.9 us (-93%) 188.4766 (-94%) 774.22 KB (-94%)
Old JsonStringifyParse 100 14.856 ms 6437.5000 25.79 MB
New 4,623.7 us (-70%) 1273.4375 (-80%) 5242.19 KB (-80%)

Jint.Benchmark.ArrayStressBenchmark

Diff Method N Mean Gen 0 Gen 1 Gen 2 Allocated
Old Jint 20 3.633 s 771250.0000 40250.0000 16500.0000 3.11 GB
New 568.9 ms (-85%) 60000.0000 (-92%) 2937.5000 (-93%) - 245.9 MB (-92%)

Jint.Benchmark.EvaluationBenchmark

Diff Method N Mean Gen 0 Allocated
Old Jint 20 1.051 ms 246.0938 1015.63 KB
New 666.9 us (-38%) 116.2109 (-53%) 477.34 KB (-53%)

Jint.Benchmark.LinqJsBenchmark

Diff Method N Mean Gen 0 Gen 1 Gen 2 Allocated
Old Jint 10 113.8 ms 8562.5000 3000.0000 1437.5000 50.91 MB
New 72.58 ms (-36%) 5062.5000 (-41%) 2500.0000 (-17%) - 30.5 MB (-40%)

Jint.Benchmark.MinimalScriptBenchmark

Diff Method N Mean Gen 0 Allocated
Old Jint 10 28.36 us 9.1248 37.42 KB
New 20.25 us (-29%) 8.1482 (-11%) 33.44 KB (-11%)

Jint.Benchmark.UncacheableExpressionsBenchmark

Diff Method N Mean Gen 0 Gen 1 Allocated
Old Benchmark 500 1.169 s 263458.3333 78020.8333 1.08 GB
New 285.3 ms (-76%) 33750.0000 (-87%) - 135.01 MB (-88%)