f1ames
commented
3 years ago Workflow:
Switch to your working branch at
ckeditor4-workflows-common,Run
./test.sh
- Reset a master branch to a given commit. It could be any commit that leaves the master branch in the state needed for testing.
- Delete all running (not queued!) actions
- Commit all files inside
workflowsdirectory and optionallyworkflows-settings.jsonto target testing repository.- Gather and dispatch every action. Please notice, that actions have to be
dispatchable.
That's a good start and you have already proven that it makes testing easier in #12 :+1:, but I think it will be good to polish testing procedure slightly. There are few problematic assumptions here:
- Resetting branch to specific (hardcoded) commit.
- Dispatching all workflows at once (what if conflicting things happens?).
- Not checking workflow results at all.
We should go with the simplest procedure possible but at the same time make sure it's stable, workflows will not be conflicting one another and new tests could be added easily. So from my perspective:
- Instead of resetting test branch, maybe we can reset its contents by simply pushing new commit there? That way the commit can have proper description (like "Test workflow something, something") and then it will be clearly visible at what branch state each workflow was launched.
- Workflows should be launched in a sequence so we are sure there are now two workflows executing at the same time and interfering each other.
- Each workflow has specific outcome. But the first thing we can check automatically is if workflow was successful (so exit code 0). I hope GH API allows that (looks like it could be done). If as a result workflow creates a branch or PR we can check if it was created. Or if new commit was pushed to specific branch. I think as a first step we could focus on checking if workflow was successful only and then see how it works and what can be improved.
As for more details on the above:
Resetting branch contents
It will be good to do this without force pushing so we have entire history preserved for easier workflows "debugging". I think it could be done like:
git clone test-repo
cd test-repo
rm -rf *
// move files required for specific test
git add .
git commit -m "Test..."
git push origin branch-nameThis way for each test we will have files required but at the same time the history is preserved and can be checked later. TBH this looks like a beforeEach step so probably should be run before each test.
Launching workflows in a sequence and checking its result
I would imagine each workflow run as a separate test in a test suite - so beforeEach is run then a single test (dispatching workflow and waiting for it to end) and then go to next test/workflow. This prevents workflows from interfering each other and makes it possible to run single workflow more than once with different configuration.
And naturally this requires checking if and how each workflow ended. As mentioned above - I think as a first step we could focus on checking if workflow was successful only and then see how it works and what can be improved.
Further issues
For now I see at least two issue with the current and proposed approach:
- Since we use single repo for tests it might be problematic if two people would run tests at the same time (or when we add CI here). But I guess we will think about it later since it's not an issue for now.
- Clean up phase - some workflows create branches or PRs, which means second run will behave different due to those leftovers. For the beginning we could cleanup manually (as we will be checking runs results manually too), but later it might be good to think about clean-up step.
That's my suggestions only, so feel free to propose different approach or any improvements or simplifications. And well, I didn't plan to create such a "wall of text", but hopefully it's descriptive enough. If not, we could talk F2F to clarify any details.
sculpt0r
I create a script that uses GH CLI & GH API
Requirements: pre-prepared repository with at least one commit.
Workflow:
ckeditor4-workflows-common,./test.shworkflowsdirectory and optionallyworkflows-settings.jsonto target testing repository.dispatchable.This script could be used in the current master branch state. If we provide more
.yml's that require e.g. push an event to be triggered - we will have to upgrade this script.The script needs to be modified inside, at least with the testing repository and default init commit hash. So if we need to test workflow in a certain repo state - we can provide such a commit and the master branch will be always reset to this point.
The last point is to unfortunately manually check logs for each workflow. However, it's still faster to set up the initial environment & upload workflows files.
Closes #14