search

WebKit / Speedometer

An open source repository for the Speedometer benchmark
Other
592 stars 70 forks source link

Add an option to prime the network cache by loading each test suite in an iframe first. #330

Closed rniwa closed 10 months ago

bgrins commented 10 months ago

IIRC the server doesn't set cache headers (and if it did, then the reported effect would be at least mitigated by being contained within the first iteration without requiring a first pass like this)

camillobruni commented 10 months ago

I think it would still make sense to have this step for slower network speeds. We've occasionally seen different scores on v2.1 depending on how slow/fast the network is. Hopefully this would limit the effects a bit?

We could give it a shot with ts_proxy or a similar setup?

bgrins commented 10 months ago

I think it would still make sense to have this step for slower network speeds.

But would this have any effect today, since the server isn't sending cache headers? If/when we started caching then any potential benefit to this change shrinks since 9/10 iterations will already be able skip the network.

And on the margin I worry this change will incentive engines to do too-aggressive per-process/tab caching the first time a page is loaded (because it will improve the score by pulling work out of measured iterations). That's not really a good outcome - lots of pages are just loaded once and it'd be better have that be faster than to speed up potential subsequent load+interactions that may not ever happen.

camillobruni commented 10 months ago

On our perf waterfall we measure with a clean profile, which adds additional noise for the first load (and even worse for local runs). I'd say it's not something we want to cover directly in Speedometer v3, so I'd vouch for limiting these effects.

bgrins commented 10 months ago

The most straightforward way to mitigate any network speed side effects would be to update browserbench.org to set cache headers for static content. Without doing that, AFAICT this PR won't have any practical effect other than incentivizing engines to warm up JIT caches etc during page load (which seems like a bad incentive).

bgrins commented 10 months ago

Here are a couple of preview URLs with headers for all resources:

Cache-Control: public, max-age=15552000: https://with-cache-headers--speedometer-preview.netlify.app/ Cache-Control: no-store: https://with-no-store--speedometer-preview.netlify.app

camillobruni commented 10 months ago

At least the effects on v2.1 are not mitigate with cache-headers (maybe this is a different issue here?)

The main issue is that on slower-networks you keep the CPU busy for longer (and thus warm), while sometimes on faster networks we drop to the lower clockspeeds in the setTimeout delays.

Let's have this as an option in the developerMode so we can experiment easily with it. Whether or not we enable this by default is another discussion.

rniwa commented 10 months ago

The consensus here seems to abandon this PR, and instead pursue adding caching header on browserbench.org instead.