Webperf-se / webperf_core

webperf-core is an open-source testing suite tailored to help you improve your digital presence in areas like web performance, security and accessibility to email best practice using many small improvements.
https://webperf.se/articles/webperf-core/
MIT License
19 stars 30 forks source link

Add Unittest for Sitespeed based test using cached/modified browsertime.har files #600

Open 7h3Rabbit opened 1 month ago

7h3Rabbit commented 1 month ago

Feature/improvement

We need to improve stability for our sitespeed based tests.

We need to:

  1. Verify that browsers (Chrome and Firefox) generate the same browsertime.har file between versions (at least fields we use).
  2. Verify that we always identify the same features for the same browsertime.har file when our logic changes.

We can solve 1 by:

We can solve 2 by:

Creating/manufacturing/crafting a browsertime.har file for every feature we identify (and don't identify) and store them and later let our tests use them as "cached" versions and compare results. Verifying feature X is recognized when it should and verifying feature Y is not pressnt when it should not. Easiest way is probably by using following url schema: https://example.com/?webperf-core-test-y-feature-x-is-enabled=true. Where y is the name of test we want to test. Where x is the name of feature we want to test. Where true is true or false depending if the feature should be enabled or not.

Is this not overkill?

Probably, but if we want to rewrite this all as a sitespeed plugin, we need to have a good verification implemented before that rewrite.