krausest
commented
2 years ago I investigated puppeteer's performance hit a bit more in depth for the "append 1,000 rows" benchmark. I created a small repo that allows to play with this puppeteer benchmark on https://github.com/krausest/puppeteerPerformanceTest
The open questions were:
- What is the cost of page.tracing?
- Do the cost vary for the frameworks?
To answer the questions I had to introduce additional metrics, since the only metric we currently have is computing the duration from the timeline and this uses some kind of page.tracing...
I found two metrics that can help:
- Use the delta value of page.metrics().TaskDuration to approximate the duration for appending 1,000 rows.
- Use the client startMeasure/stopMeasure hack that we removed in 89b546fe1b772e0f9f0b94da81b83513b6759e4b. It tries to enqueue a callback after repainting has finsished. Some clever people have good reasons to consider this hack completely broken, however I think it works surprisingly well in many cases.
This gives us five metrics per framework:
- taskTracing: Increase of page.metrics().TaskDuration for a benchmark run after page.tracing is started
- taskNoTracing: Increase of page.metrics().TaskDuration for a benchmark run without page.tracing.
- clientTracing: The duration as reported from the client side startMeasure/stopMeasure hack after page.tracing is started
- clientNoTracing: The duration as reported from the client side startMeasure/stopMeasure hack without page.tracing.
- timeline: The original idea from the js-framework-benchmark: Extract the elapsed time between the start of the click event and the end of the paint event, i.e. this can only be measured with page.tracing. The metrics are reported as milliseconds and the std deviation put in brackets.
From those metrics we can compute taskFactor = taskTracing / taskNoTracing, i.e. the overhead of tracing for the task duration and clientFactor = clientTracing / clientNoTracing, the tracing overhead for the client hack duration mesurement
Here are results for my MacBook Air: Default categories:
┌─────────┬───────────────┬───────────────────┬──────────────────┬──────────────────┬──────────────────┬──────────────────┬────────────┬──────────────┐
│ (index) │ framework │ taskTracing │ taskNoTracing │ clientTracing │ clientNoTracing │ timeline │ taskFactor │ clientFactor │
├─────────┼───────────────┼───────────────────┼──────────────────┼──────────────────┼──────────────────┼──────────────────┼────────────┼──────────────┤
│ 0 │ 'vanillajs' │ '53.625 (1.628)' │ '50.250 (4.679)' │ '45.150 (0.400)' │ '47.525 (1.456)' │ '44.781 (0.457)' │ '1.067' │ '0.950' │
│ 1 │ 'svelte' │ '98.859 (8.283)' │ '58.609 (4.733)' │ '80.000 (2.840)' │ '54.087 (0.759)' │ '79.531 (2.853)' │ '1.687' │ '1.479' │
│ 2 │ 'react-hooks' │ '82.359 (9.738)' │ '70.250 (5.638)' │ '66.013 (2.640)' │ '65.912 (2.453)' │ '65.125 (2.011)' │ '1.172' │ '1.002' │
│ 3 │ 'domvm' │ '79.406 (10.420)' │ '61.406 (7.198)' │ '57.688 (2.897)' │ '58.513 (3.892)' │ '56.750 (1.313)' │ '1.293' │ '0.986' │
└─────────┴───────────────┴───────────────────┴──────────────────┴──────────────────┴──────────────────┴──────────────────┴────────────┴──────────────┘Only category 'devtools.timeline':
┌─────────┬───────────────┬───────────────────┬──────────────────┬──────────────────┬──────────────────┬──────────────────┬────────────┬──────────────┐
│ (index) │ framework │ taskTracing │ taskNoTracing │ clientTracing │ clientNoTracing │ timeline │ taskFactor │ clientFactor │
├─────────┼───────────────┼───────────────────┼──────────────────┼──────────────────┼──────────────────┼──────────────────┼────────────┼──────────────┤
│ 0 │ 'vanillajs' │ '48.891 (2.984)' │ '49.438 (4.369)' │ '46.913 (2.959)' │ '46.612 (1.360)' │ '46.656 (3.106)' │ '0.989' │ '1.006' │
│ 1 │ 'svelte' │ '66.297 (4.835)' │ '56.969 (5.022)' │ '60.988 (1.556)' │ '53.863 (2.447)' │ '60.516 (1.585)' │ '1.164' │ '1.132' │
│ 2 │ 'react-hooks' │ '70.437 (10.051)' │ '67.656 (5.186)' │ '64.950 (5.838)' │ '63.763 (2.378)' │ '64.031 (4.452)' │ '1.041' │ '1.019' │
│ 3 │ 'domvm' │ '59.781 (4.260)' │ '60.094 (8.472)' │ '56.275 (1.443)' │ '55.888 (1.376)' │ '55.094 (0.734)' │ '0.995' │ '1.007' │
└─────────┴───────────────┴───────────────────┴──────────────────┴──────────────────┴──────────────────┴──────────────────┴────────────┴──────────────┘- Svelte has a very high performance penalty when all categories are enabled for both metrics and is slowest according to all metrics. When only category 'devtools.timeline' is used svelte is faster than react and the penalty is much lower, but still not on the level of other frameworks.
- clientTracing and timeline metrics are pretty close, i.e. the startMeasure/stopMeasure hack appears to work quite well here.
Here's the same for my Razer Blade 15 on Linux: Default categories:
┌─────────┬───────────────┬────────────────────┬────────────────────┬───────────────────┬────────────────────┬───────────────────┬────────────┬──────────────┐
│ (index) │ framework │ taskTracing │ taskNoTracing │ clientTracing │ clientNoTracing │ timeline │ taskFactor │ clientFactor │
├─────────┼───────────────┼────────────────────┼────────────────────┼───────────────────┼────────────────────┼───────────────────┼────────────┼──────────────┤
│ 0 │ 'vanillajs' │ '90.568 (3.505)' │ '86.557 (10.886)' │ '76.375 (2.040)' │ '82.275 (7.600)' │ '75.991 (2.047)' │ '1.046' │ '0.928' │
│ 1 │ 'svelte' │ '149.885 (17.859)' │ '105.280 (20.249)' │ '121.875 (6.575)' │ '96.837 (13.206)' │ '120.251 (6.501)' │ '1.424' │ '1.259' │
│ 2 │ 'react-hooks' │ '144.481 (16.758)' │ '128.734 (17.499)' │ '117.212 (5.654)' │ '122.900 (14.508)' │ '114.861 (4.227)' │ '1.122' │ '0.954' │
│ 3 │ 'domvm' │ '127.985 (15.459)' │ '108.558 (14.569)' │ '99.213 (6.213)' │ '102.200 (9.799)' │ '96.630 (2.584)' │ '1.179' │ '0.971' │
└─────────┴───────────────┴────────────────────┴────────────────────┴───────────────────┴────────────────────┴───────────────────┴────────────┴──────────────┘Only category 'devtools.timeline':
┌─────────┬───────────────┬────────────────────┬────────────────────┬────────────────────┬────────────────────┬────────────────────┬────────────┬──────────────┐
│ (index) │ framework │ taskTracing │ taskNoTracing │ clientTracing │ clientNoTracing │ timeline │ taskFactor │ clientFactor │
├─────────┼───────────────┼────────────────────┼────────────────────┼────────────────────┼────────────────────┼────────────────────┼────────────┼──────────────┤
│ 0 │ 'vanillajs' │ '94.773 (19.760)' │ '85.492 (10.156)' │ '87.950 (12.915)' │ '81.162 (6.288)' │ '87.807 (13.154)' │ '1.109' │ '1.084' │
│ 1 │ 'svelte' │ '119.704 (13.236)' │ '102.597 (9.550)' │ '110.137 (9.083)' │ '94.225 (4.419)' │ '109.541 (9.215)' │ '1.167' │ '1.169' │
│ 2 │ 'react-hooks' │ '130.983 (20.206)' │ '127.957 (21.973)' │ '120.675 (11.985)' │ '121.175 (20.117)' │ '118.993 (11.998)' │ '1.024' │ '0.996' │
│ 3 │ 'domvm' │ '116.202 (21.280)' │ '112.047 (15.712)' │ '107.063 (11.305)' │ '104.738 (8.388)' │ '104.627 (10.534)' │ '1.037' │ '1.022' │
└─────────┴───────────────┴────────────────────┴────────────────────┴────────────────────┴────────────────────┴────────────────────┴────────────┴──────────────┘- It appears that svelte is hit similarly on linux.
- The std deviation was pretty high on linux, but it didn't get much better for other runs.
yisar
:heavy_check_mark: Looks like #916 is hard to fix with selenium/chromedriver since there's no API to dispose a element reference. Puppeteer has such an dispose method.
:heavy_check_mark: Puppeteer also has a nice api to get a timeline trace that can be loaded in the chrome performance tab which would be very welcome to allow a better understanding of the reported values.
:heavy_check_mark: Puppeteer-core supports nicely the M1 mac, for which I haven't found a smooth path for chromedriver yet
:heavy_check_mark: For chromedriver I have to disable site isolation on M1, which is not needed for puppeteer
So I was pretty enthusiastic migrating to puppeteer which went mostly nice (handlich shadow dom is always ugly and e.g. calling click in the browser context didn't work for all frameworks, I had to call click on the element handle).
I've pushed a branch for puppeteer that should work on OSX and Linux.
However I have a few issues that currently prohibit replacing chromedriver with puppeteer.
[ ] Overall values from puppeteer are a bit slower than from chromedriver. Seems like the tracing functionality comes with additional costs. (I could live with if all frameworks were similarly slower)
[ ] Some frameworks are hit really badly by running the trace in puppeteer, see the screenshot below (left chromedriver, right puppeteer on OSX):
And the same for Linux:
We had similar behaviour with #549 with chromedriver which also impacted svelte. For chromedriver it was sufficient to reduce the trace categories, but it didn't help with puppeteer.
(I used only slow down factors that can be selected in the chrome performance tab thus some benchmarks are almost too fast to measure.)
Results for:
So currently switching to puppeteer is not the solution we're looking for!