Setup sandbox to check flackiness and test selfhosted github action

[ChristopherPHolder]

Motivation

Document the variance in performance scores between Lighthouse User Flow runs on GitHub runners.

To improve a users ability to draw conclusions about a performance-impacting change:

• They need to understand objectivity how much variability impacts scores
• They need to reduce the sources of variability between audits

Lighthouse has documented that sometimes the performance score is different between audits. They have also identified and documented the main sources of variance and how to mitigate them.

Source	Impact	Typical End User	PageSpeed Insights	Controlled Lab
Page nondeterminism	High	LIKELY	LIKELY	LIKELY
Local network variability	High	LIKELY	UNLIKELY	UNLIKELY
Tier-1 network variability	Medium	POSSIBLE	POSSIBLE	POSSIBLE
Web server variability	Low	LIKELY	LIKELY	LIKELY
Client hardware variability	High	LIKELY	UNLIKELY	UNLIKELY
Client resource contention	High	LIKELY	POSSIBLE	UNLIKELY
Browser nondeterminism	Medium	CERTAIN	CERTAIN	CERTAIN

Implementation

TODO - Improve this description

Using a sandbox environment we will run a user-flow audit chronologically in GitHub Actions. The results will be compiled and used to generate generate charts. These will help users understand how much results can vary using a similar setup.

Sandbox Environment Setup

Questions & Answers

1. What source of variance do we want to document?

• We want to document the impact of Client hardware variability and Client resource contention when running user-flow on a GitHub Actions Runner.

3. How are we mitigating the impact of other sources of variance?

• Page Nondeterminism: Ensuring that the exact same version of the page is being tested between different runs.
• Local Network Variability: Serve application locally.
• Tier-1 Network Variability: Serve application locally.
• Web Server Variability: Serve application locally.
• Browser Nondeterminism: Served application will not contain complex logic, nor compression.

4. How will this sandbox environment be generated?

• Target application generated is Coffee cart served locally.
• User-flow is generated with npx @push-based/user-flow init and adding the url and serve command.

Note: Regarding the chose of target application

• Coffee cart it is the example used by google and ourselves.
• It is open-source repo and we can easily build and serve it locally.

5. Where will the sandbox be located?

• Fully separate repo to not pollute the user-flow repository and allow user to reproduce the test easily

Github Workflow Configuration

Implementation details:

• It will run every 15min
• It will run user flow using an npx command
• It will run scripts to compile and cleanup the results
• It will commit the updated/compiled results

Data Collection & Compilation

The data will be collected running npx user-flow in a GitHub action with the following configuration in the .user-flowrc.json:

{
  "collect": {
    "url": "http://localhost:4173/",
    "ufPath": "./user-flows",
    "serveCommand" : "npm run serve",
    "awaitServeStdout": "4173",
    "config": {
      "settings": {
        "onlyCategories": ["performance"]
      }
    }
  },
  "persist": { "outPath": "./measures", "format": ["json"] },
  "assert": {}
}

After the completion of the user-flow audit a script will be run to compile and clean up the data. The script will perform the following actions:

• Open the audit report located at ~/measures/*
• Extract the fetchTime and performance score
• Open the compiled results located in ~/data/sample-1.json
• Append the extracted result data to the compiled results
• Save the updated compiled results
• Delete the audit report located at ~/measures/*

The script will be located at ~/scripts/compile-and-cleanup.js

The compiled report will be stored in a file at ~/data/sample-1.json. The file will contain an array of the fetchTime and performance score of all the audit runs.

Example compiled results object

[
  {
    "fetchTime": "2022-04-21T21:00:00.000Z",
    "performance": 0.93
  }
  {
    "fetchTime": "2022-04-21T21:15:00.000Z",
    "performance": 0.94
  }
  {
    "fetchTime": "2022-04-21T21:30:00.000Z",
    "performance": 0.92
  }
]

The results will be saved in the repo by committing the updated compiled report file as a step CI job.

Results Documentation

TODO

• Explain what will be the final outcome
• What charts will be displayed
  • How the chart will be generated

TODOS

TODO - Add more steps based on information above

• Sandbox Environment Setup
  • [x] Setup a repo for the project
  • [x] Setup the target application
  • [x] Setup user-flow
• Github Workflow Configuration
  • [x] Setup and build application
  • [x] Run user-flow audit
  • [x] Run compilation and cleanup script
  • [x] Commit update compiled results
  • [x] Run as a cron job
• Data Collection & Compilation
  • [x] Run only performance audit and save in json format
  • [x] Script that compiles and cleans up the data
• [x] Results Documentation

[push-based-bot]

We also have to reference the outcome inc. cool img in the action repo