Investigate performance of Octane in Servo vs. Firefox

[metajack]

Since we use the same JS engine as Firefox, we should expect roughly equivalent performance on JS benchmarks. I decided to investigate to verify this was true, and was a little surprised by the results.

My data is collected here: https://docs.google.com/spreadsheets/d/1NjhzcLVSv1vyf_n7EOdZOkOXrdQ4z0I8T1Q8thXbhmU/edit

I used the last published Firefox 39 nightly since current Servo master uses SM39. Servo in that data is master just before https://docs.google.com/spreadsheets/d/1NjhzcLVSv1vyf_n7EOdZOkOXrdQ4z0I8T1Q8thXbhmU/edit#gid=0 and ServoPlus is servo master from a few days ago.

The interesting comparison is in the Stats tab, which shows relative performance between pairs. For ServoPlus vs. Gecko, here are the numbers:

Mandreel 22.29% Deltablue 50.96% MandreelLatency 53.99% Richards 56.70% EarleyBoyer 75.04% Raytrace 77.76% Total 85.35% Crypto 88.91% Regexp 98.22% NavierStokes 100.66% zlib 102.48% Box2D 106.50% SplayLatency 108.46% Typescript 116.62% Splay 116.71% CodeLoad 116.97% pdf.js 132.40% Gameboy 135.85%

In some tests, we are very slow compared to Firefox, and in some quite a bit faster. This bug is for investigation into why this discrepancy exists. I suggest starting Mandreel and Gameboy as those are the most extreme examples.

One possible contributing factor is that current Servo does not enable incremental GC, and Firefox does.

[jdm]

Is there any noticeable difference if we disable incremental GC in Firefox?

[metajack]

The spreadsheet is updated. GeckoNoInc is clean profile of Nightly39 with javascript.options.mem.gc_incremental = false. Here's the comparison with ServoPlus and GeckoNoInc:

Mandreel    21.91%
MandreelLatency 46.36%
Richards    60.93%
Deltablue   67.25%
EarleyBoyer 73.93%
Total   88.11%
Crypto  90.02%
Raytrace    90.49%
Box2D   93.18%
pdf.js  98.86%
NavierStokes    101.99%
Gameboy 103.04%
zlib    103.52%
CodeLoad    106.94%
Regexp  115.42%
Typescript  121.53%
Splay   128.99%
SplayLatency    183.95%

[metajack]

It would be useful if someone else also tried to reproduce these numbers to rule out measurement error. I'd just run Servo master vs. FF39 with incremental GC disabled and skip the other versions.

[metajack]

I have rerun these tests with js-shell from mozilla-central and mozjs-shell. Here are the results:

key                  js-shell        mozjs-shell    
SplayLatency         21188.5         5594.0          26.40%
EarleyBoyer          27732.5         20296.5         73.19%
Splay                17728.5         13295.0         74.99%
Score (version 9)    28255.5         25500.5         90.25%
DeltaBlue            57731.0         54372.0         94.18%
PdfJS                12560.5         12126.0         96.54%
Box2D                47371.0         45995.0         97.10%
RegExp               3590.5          3517.5          97.97%
Mandreel             27063.5         27117.5         100.20%
Richards             29528.5         29680.0         100.51%
NavierStokes         35284.0         35468.0         100.52%
Crypto               27976.0         28165.0         100.68%
zlib                 78127.0         79120.0         101.27%
Gameboy              47769.5         48806.5         102.17%
RayTrace             102636.0        106558.0        103.82%
CodeLoad             16800.0         18059.0         107.49%
Typescript           26319.5         29045.0         110.36%
MandreelLatency      30737.0         34147.0         111.09%

The numbers reported are the median scores over 10 runs. Note that incremental GC is enabled in both.

[metajack]

@tschneidereit I'm unsure how worried or not worried to be about the numbers, but it certainly seems like something is wrong with SplayLatency and perhaps EarleyBoyer and Splay. At least the remaining ones are within 10% or so.

[tschneidereit]

@metajack, how do the numbers with iGC compare to those without? Both in m-c shell and mozjs shell. What I'm getting at is: do we know for certain that iGC works as expected?

I do think that we should be somewhat concerned about this, because the tests we do badly on all hammer on GC performance, and that's an area that's especially important for Servo because eagerly creating reflectors means that we cause more GC traffic than FF would.

[atouchet]

I tried retesting this using the latest versions of Servo and Firefox.

Benchmark	Servo Nightly 2020-05-02	Firefox 75.0	Comparison
Richards	19907	20475	97.23%
Deltablue	36074	37707	95.67%
Crypto	19901	20003	99.49%
Raytrace	57127	51947	109.97%
EarleyBoyer	25877	25826	100.20%
Regexp	4550	4541	100.20%
Splay	11767	13202	89.13%
SplayLatency	4525	12903	35.07%
NavierStokes	30540	31016	98.47%
pdf.js	14934	16344	91.37%
Mandreel	19403	19802	97.99%
MandreelLatency	21675	25051	86.52%
GB Emulator	57319	57547	99.60%
CodeLoad	15595	15929	97.90%
Box2DWeb	36613	38381	95.39%
zlib	49793	49761	100.06%
Typescript	20001	24167	82.76%
Octane Score	21078	23258	90.63%

Most results are similar to each other aside from a few. Notably SplayLatency is significantly slower in Servo than it is in Firefox.