Dedice on interesting Lighthouse metrics

krausest / js-framework-benchmark

A comparison of the performance of a few popular javascript frameworks

https://krausest.github.io/js-framework-benchmark/

Apache License 2.0

6.76k stars 836 forks source link

Dedice on interesting Lighthouse metrics #330

Closed krausest closed 6 years ago

krausest commented 6 years ago

@paulirish submitted a PR adding lighthouse creating lots of interesting information for the benchmark startup - thanks Paul! As an experiment I added them all to the results table.

We should discuss here which rows we want to keep. My preferred metrics are:

script bootup time
total byte weight (though I'm not sure we should mix duration and size in the table and average over both...)

See also #326 for Paul's recommendations.

krausest commented 6 years ago

@leeoniya @adamhaile @Bobris @localvoid Looking forward to hearing your opinion.

leeoniya commented 6 years ago

some quick observations:

estimated input latency looks useless
first meaningful paint and first contentful paint are the same
first interactive and consistently interactive are the same
onload looks suspect where total byte weight does not match (vidom, elm?)
marionette's first meaningful paint looks wrong - might be the result of rendering a skeleton and placing the script at the bottom. i think this should actually be changed for all impls (put script at bottom - may help bring some things in line)

krausest commented 6 years ago

@leeoniya

first interactive and consistently interactive are the same

Feared this too, but it's not. hyperapp-v0.16.0-keyed has a small difference!

leeoniya commented 6 years ago

hyperapp-v0.16.0-keyed has a small difference!

i think you mean first contentful paint and first meaningful paint. a 0.1ms difference in one lib is neither significant nor interesting enough to burden the entire bench runtime. it's very likely just noise.

paulirish commented 6 years ago

agree with the above observations.

I'd suggest Consistently Interactive, script bootup, main thread work cost, and total byte weight (to keep the network costs in the picture). FMP could also be considered, but I think it's going to deliver the same point as Consistently Interactive too often for it to be called out individually.

paulirish commented 6 years ago

it's worth pointing out the dependencies between the above metrics, too.

CI will be no earlier than FMP.
Main thread work cost includes all script bootup time.

So the most-top level choices for runtime would be Consistently Interactive and main thread work cost. There will be a correlation between these two but it's possible to have a fast CI even with lots of main thread work; so I think it's good to represent both.

And yah I do like including total byte weight. (And agree bytes and milliseconds shouldn't be averaged together. :)

krausest commented 6 years ago

There are now three result tables: Duration, startup and memory.

leeoniya commented 6 years ago

looking at the data, paul's advice is spot on. i've opened #334 to reduce the noise.

changing STARTUP_SLEEP_DURATION from 1000 to 0 should likely be re-evaulated too, i think.

kaste commented 6 years ago

Just want to mention that 'main thread work cost' is a bit random on my computer.

On the left you see vanillajs, in the middle my lib, on the right lit-html.