The issue arose from a post-mortem conducted by the Goat team, because unit tests had not been running on pull requests for several days.
In case a not allowed version of a GitHub Action is used in a GitHub Actions Workflow, this workflow will fail silently on startup. In this specific case the blocked Action was part of the Workflow running on pull requests and as part of this Workflow also unit tests are executed. The failed Workflow was not considered as a Check required by pull requests and therefore was not blocking the merge of pull requests. This led to a state where pull requests could be merged without the requirement that unit tests were executed.
This can be easily reproduced and there is no prevention that this can happen again.
The following steps led to faulty state:
PR with an update of golangci/golangci-lint-action to version 9d1e0624a798bb64f6c3cea93db47765312263dc was created by Dependabot
The new version 9d1e0624a798bb64f6c3cea93db47765312263dc of golangci/golangci-lint-action was not available as allowed action in the api-gateway repository. It's important to consider, that the availability of this allowed action can only be checked by repository admins, so not everyone on the team might be able to do this.
The pull request Workflow in api-gateway started to run into a Startup failure and PRs did no report the failed pull request workflow, but were mergeable.(Failed pull-request run).
Multiple teams already mentioned problems with allowed versions propagation and silently failing GitHub Workflows in the past. So in the post mortem we made the assumption that the propagation of allowed versions is done manually.
We consider it a critical issue that Workflows failing with status Startup failure are not considered relevant for pull request checks. This means a Startup failure, e.g. by using a not allowed action can prevent running Workflows silently and therefore failing our testing strategy.
Additionally, we get regularly Dependabot PRs to update GitHub Actions versions. Since we want to keep those dependencies up to date, we need to add the new versions in the allowed_actions.json. Since the propagation of allowed versions seems to be done manually, there is a risk that one or more of our workflows will break with each new version of a GitHub Actions.
Expected result
Propagation of Allowed GitHub Actions is reliable.
Actual result
Propagation of allowed versions and execution of GitHub Workflows is not reliable.
Steps to reproduce
Can be found in the description.
Description
The issue arose from a post-mortem conducted by the Goat team, because unit tests had not been running on pull requests for several days. In case a not allowed version of a GitHub Action is used in a GitHub Actions Workflow, this workflow will fail silently on startup. In this specific case the blocked Action was part of the Workflow running on pull requests and as part of this Workflow also unit tests are executed. The failed Workflow was not considered as a Check required by pull requests and therefore was not blocking the merge of pull requests. This led to a state where pull requests could be merged without the requirement that unit tests were executed.
This can be easily reproduced and there is no prevention that this can happen again. The following steps led to faulty state:
golangci/golangci-lint-actionto version9d1e0624a798bb64f6c3cea93db47765312263dcwas created by Dependabot9d1e0624a798bb64f6c3cea93db47765312263dcwas added to the allowed_actions.json in this PR.9d1e0624a798bb64f6c3cea93db47765312263dcofgolangci/golangci-lint-actionwas not available as allowed action in the api-gateway repository. It's important to consider, that the availability of this allowed action can only be checked by repository admins, so not everyone on the team might be able to do this.Startup failureand PRs did no report the failed pull request workflow, but were mergeable.(Failed pull-request run).Multiple teams already mentioned problems with allowed versions propagation and silently failing GitHub Workflows in the past. So in the post mortem we made the assumption that the propagation of allowed versions is done manually.
We consider it a critical issue that Workflows failing with status
Startup failureare not considered relevant for pull request checks. This means a Startup failure, e.g. by using a not allowed action can prevent running Workflows silently and therefore failing our testing strategy.Additionally, we get regularly Dependabot PRs to update GitHub Actions versions. Since we want to keep those dependencies up to date, we need to add the new versions in the allowed_actions.json. Since the propagation of allowed versions seems to be done manually, there is a risk that one or more of our workflows will break with each new version of a GitHub Actions.
Expected result Propagation of Allowed GitHub Actions is reliable.
Actual result Propagation of allowed versions and execution of GitHub Workflows is not reliable.
Steps to reproduce Can be found in the description.
Troubleshooting