Bad VMs leading to Inaccurate results

treosh / lighthouse-ci-action

Audit URLs using Lighthouse and test performance with Lighthouse CI.

MIT License

1.16k stars 82 forks source link

Bad VMs leading to Inaccurate results #14

Open exterkamp opened 4 years ago

exterkamp commented 4 years ago

Running this a few times with only 1 run on my personal site gh repo I got inconsistent results. This seemed to be due to the underlying VMs.

100 score & 1026 benchmark idx 98 score & 347 benchmark idx 88 score & 106 benchmark idx

Ideas:

Less than a 500 benchmark index, the run should be killed. These results won't be accurate and can't be asserted against.
We should run a pre-flight check with the benchmarker Lighthouse uses (it's some pretty simple js to run, seen here: https://benchmark.exterkamp.codes) and check if the VM we got for our action was DOA. If it was, then exit code 1 and print out something about re-running the action because of a bad VM.

alekseykulikov commented 4 years ago

Thank you @exterkamp for a great research! I hope it will go away as the Github Actions platform matures. Otherwise, we have to request a new VM to provide consistent performance testing results. Any ideas on how to force a new VM and not fail the workflow?

paulirish commented 4 years ago

Reading https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops it doesn't provide many expectations around CPU load.. or any options for it.

But it does say "(Linux only) Run steps in a cgroup that offers 6 GB of physical memory and 13 GB of total memory" so that's something.

cc @connorjclark

paulirish commented 4 years ago

Oh and much like azure pipelines... github actions apparently also allows self-hosted runners: https://help.github.com/en/actions/automating-your-workflow-with-github-actions/about-self-hosted-runners

alekseykulikov commented 4 years ago

As a solution, we may collect Lighthouse reports using PageSpeed Insights. It's more consistent (https://treo.sh/demo/6) and will allow running audits in parallel to speed up a build.

- name: Audit URLs using Lighthouse
  uses: treosh/lighthouse-ci-action@v2
  with:
   urls: |
      https://example.com/
      https://example.com/demo
      https://example.com/dashboard
   psiToken: page-speed-insights-token
   runs: 3 # use median for performance runs