Open exterkamp opened 4 years ago
Thank you @exterkamp for a great research! I hope it will go away as the Github Actions platform matures. Otherwise, we have to request a new VM to provide consistent performance testing results. Any ideas on how to force a new VM and not fail the workflow?
Reading https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops it doesn't provide many expectations around CPU load.. or any options for it.
But it does say "(Linux only) Run steps in a cgroup that offers 6 GB of physical memory and 13 GB of total memory" so that's something.
cc @connorjclark
Oh and much like azure pipelines... github actions apparently also allows self-hosted runners: https://help.github.com/en/actions/automating-your-workflow-with-github-actions/about-self-hosted-runners
As a solution, we may collect Lighthouse reports using PageSpeed Insights. It's more consistent (https://treo.sh/demo/6) and will allow running audits in parallel to speed up a build.
- name: Audit URLs using Lighthouse
uses: treosh/lighthouse-ci-action@v2
with:
urls: |
https://example.com/
https://example.com/demo
https://example.com/dashboard
psiToken: page-speed-insights-token
runs: 3 # use median for performance runs
Running this a few times with only 1 run on my personal site gh repo I got inconsistent results. This seemed to be due to the underlying VMs.
100 score & 1026 benchmark idx 98 score & 347 benchmark idx 88 score & 106 benchmark idx
Ideas:
Less than a 500 benchmark index, the run should be killed. These results won't be accurate and can't be asserted against.
We should run a pre-flight check with the benchmarker Lighthouse uses (it's some pretty simple js to run, seen here: https://benchmark.exterkamp.codes) and check if the VM we got for our action was DOA. If it was, then exit code 1 and print out something about re-running the action because of a bad VM.