Open reimic opened 5 months ago
How slow is the manual workflow? Couldn't it just run automatically? Typically everyone forgets about these manual workflows and they get stale and start failing soon.
Typically everyone forgets about these manual workflows and they get stale and start failing soon.
Couldn't it just run automatically?
I see why you would inquire about that. To clarify: it's not possible for them to get stale or desync in any way - they run the same test suites that are run automatically. This is not a different set of tests - it is a different way of running tests, with modularity in mind.
When considering their usefulness - think "dev tool" not quality gate. When pushing or PR-ing all test suites do run automatically.
How slow is the manual workflow?
On my machine e2e tests already take a while (with just 3/4 implemented). Mind that this is for a single php - os run. The containers on GitHub are a bit faster, since the download times are much better. (Probably due to mirrors and data transfer speed.) Also they do try to run in parallel, yet they start to queue up quite quickly. To speak beyond numbers - I have noticed the issue almost instantly. And this worried me to the extent that I've tackled it on two fronts - see "observations" here: #108. If not addressed it will hinder our goal to offer superb reliability. Since that requires tests for various complex blueprints with larger downloads.
To reiterate:
Can you make it fail fast and also produce the full error output?
Eg mark the step as failed to make the job red, and then, on failure, start a full test run that won't stop after the first failure
Eg mark the step as failed to make the job red, and then, on failure, start a full test run that won't stop after the first failure
Sure. GitHub steps produce a conclusion
and an outcome
that can be used in logical expressions within a job. (Side note: a workflow has jobs that may include steps) There is a faint difference between one and another and they might be helpful here. Workflows share this feature and can trigger workflows depending on their status:
on:
workflow_run:
workflows: [Build]
types: [completed]
jobs:
on-success:
runs-on: ubuntu-latest
if: ${{ github.event.workflow_run.conclusion == 'success' }}
steps:
- run: echo 'The triggering workflow passed'
on-failure:
runs-on: ubuntu-latest
if: ${{ github.event.workflow_run.conclusion == 'failure' }}
steps:
- run: echo 'The triggering workflow failed'
Help me see your point. What you are saying is that the initial automatic fast-failing workflow would trigger another more robust workflow. Which is fine and clearly doable, but the new jobs will compete over the same resource pool.
I could explore 2 strategies here:
Of course all this can notify interested parties and all that jazz. But that is beyond the point here.
I am on the fence here. With the current state of this PR, if an automatic workflow fails. You can decide if you care about further development and if you do - for the cheap price of 2 clicks - run the manual workflow on your own fork to see how your code performs for the php-os combination of your choosing.
I see how this could be improved upon, but my recommendation is as follows: Marge this as it is. I'll add several e2e tests next and once I am done and execution times are more apparent we'll circle back to this issue and evaluate how useful it has proven to be and how it could be enhanced.
I mean the tests we have took 37s to finish, this fast/slow mode seems a but premature. Let's just ship the full version and scale back as needed.
Summary
Rework CI with 2 dispatch workflows: automatic as a quick and thick quality gate for all pushes and pull requests and manual for a nice and easy way of geeting details on failing tests.
The first step toward achieving robust end-to-end (#108) coverage.
Major Changes
The manual-dispatch allows for control over the branch it will run on, php version, os and test suite. Every default is set to
all
and the parameterFAIL_FAST
is always set tofalse
. This dispatches the run with a configuration that is broad and offers maximum info on the condition of the tested branch.The automatic-disptach is more rigid and always tests all php versions, os and with all test suites. It also has the
FAIL_FAST
parameter set totrue
. This also introduces--stop-on-defect
--fail-on-warning
--fail-on-risky
PHPUnit flags for enchanced quality control. Yet, this is a job-level feature. This means that the whole test matrix will always run (3 os * 7 php versions => 21 jobs). The benefit of theFAIL_FAST
parameter is to run as few tests as possible to ascertain if a os-php version combination is stable.I'd hope this will enable a workflow going more or less like that:
vendor/bin/phpunit
, but unless you are really into PHP versions this will run on the single version you have configured as your default project interpreterC:\php\php-8.3.exe
. Obviously, the local test run also only tests the code performance on your OS. For examples sake, let's say you are developting on a Mac and that all tests have passed.windows-latest
as the os to run on and decide to checkall
php versions. With an initial idea on the scope of your changes and data from the automatic runs you might have also conclued that only e2e tests actually failed. To save a few precious seconds you decide to only run thee2e
suite.Example Usage
Actions
tab, pickManual Tests Dispatch
.Run workflow
button:inputs
:all
is default and will use all versions)all
is default and will run on all os)all
is default and will run both unit and e2e tests)