Ci flake finder for unit tests

mikehardy commented 2 weeks ago

Purpose / Description

We had a hard time with windows unit test flake reproduction. I threw in a crude hack on the workflow as a temporary commit that iterated over windows a lot of times, and that allowed for reproduction confidence

This is an attempt to make that a general feature - so you can select either all 3 operating systems we run or focus on one of them, and you can run an arbitrary number of iterations of unit tests on it (okay, 255 limit, but that's a lot)

Fixes

Related #16197

Approach

first make our matrix generation dynamic
second make operating system default to all 3 like normal but allow selection if desired
third add idea of iterations to unit tests, and allow specification of more than 1 if desired

How Has This Been Tested?

On my separate fork: https://github.com/mikehardy/Anki-Android/actions/workflows/tests_unit.yml

I made sure that the "push" case was normal, it ran all 3 systems with one iteration
I made sure you could select an operating system and it would run just that one
I made sure you could specify an iteration count and it would add iterations if not just 1

Learning (optional, can help others)

dynamic matrices are fun: https://www.peterbe.com/plog/programmatically-control-the-matrix-in-a-github-action-workflow
getting inputs in javascript in actions is fun https://stackoverflow.com/questions/71155641/github-actions-how-to-view-inputs-for-workflow-dispatch

Checklist

Please, go through these checks before submitting the PR.

[x] You have a descriptive commit message with a short title (first line, max 50 chars).
[x] You have commented your code, particularly in hard-to-understand areas
[x] You have performed a self-review of your own code
[ ] UI changes: include screenshots of all affected screens (in particular showing any new or changed strings)
[ ] UI Changes: You have tested your change using the Google Accessibility Scanner

mikehardy commented 2 weeks ago

Each commit works by itself, I had to build this very very carefully since the merger of matrices + yaml + javascript + json is really touchy. So I went slowly and each commit was tested by itself for correct functionality - rebase merge will be fine

mikehardy commented 2 weeks ago

Note: will require a settings change such that branch protections are now looking for the required status from the 3 runs but with "Unit Tests / " prepended to the check names

Also has run count in there, so perhaps the branch protection can be a regex 🤔 - investigating

Okay: investigated

We will need to temporarily disable the specific status checks for unit tests while this goes in Then after one set of runs we can re-add required status checks as:

Unit Tests / JUnit Tests (macos run 1)
Unit Tests / JUnit Tests (ubuntu run 1)
Unit Tests / JUnit Tests (windows run 1)

That should work because we want to see at least 1 iteration of each, and that is the default set produced from a pull request event trigger on this workflow

mikehardy commented 2 weeks ago

@krmanik just tagging you because you might be interested in this - you've done so much github workflow + github-script and it's a potential useful trick to know how to get inputs into the javascript in a usable way, and use javascript to pass information to future steps/jobs

mikehardy commented 2 weeks ago

I liked the suggestions - had just enough time for the docs polishes and re-pushed, re-tested on my branch, the default expansion was still as expected (all 3, 1 iteration) and a manual run of windows + 3 worked, both visible here https://github.com/mikehardy/Anki-Android/actions/workflows/tests_unit.yml

mikehardy commented 2 weeks ago

I hate to merge this then ghost so I'm going to wait until I have time or someone else can commit to the branch protection temp disable / re-enable with new names I mention above with link to settings area https://github.com/ankidroid/Anki-Android/pull/16279#issuecomment-2080223140

krmanik commented 2 weeks ago

@krmanik just tagging you because you might be interested in this - you've done so much github workflow + github-script and it's a potential useful trick to know how to get inputs into the javascript in a usable way, and use javascript to pass information to future steps/jobs

The input can be provided similar to https://github.com/ankidroid/Anki-Android/blob/9036f1611380e2952630527dd26483e9d4772805/.github/workflows/opencollective_notices.yml#L30

We can use @actions/core to set output.

core.setOutput('outputKey', 'outputVal');

To access the value in same job,

${{ steps.[STEP ID].outputs.[KEY] }}

To access the value in different job,

${{ needs.[JOB ID].outputs.[KEY] }}

Defining outputs for jobs

mikehardy commented 2 weeks ago

The toJSON I used but still need to parse the os array because it was still a string inside inputs

The output was just using default return from the script so didn't need anything fancy theree

mikehardy commented 2 weeks ago

Okay, I'm going to merge this now then do the required baby-sitting on branch protection rules

mikehardy commented 2 weeks ago

New branch protection rules are in place. Existing PRs will need a close/reopen cycle to kick off the new CI tests expected by the new rules

ankidroid / Anki-Android