Race condition between TACOs apply and PR merge. Cannot recover.

fpacifici commented 5 months ago

Environment

https://github.com/getsentry/tacos-gha/tree/stable on the ops repo This is the PR where the issue happened: https://github.com/getsentry/ops/pull/11131

Steps to Reproduce

On any slice:

make a change that triggers a successful plan
trigger tacos apply
while apply is running merge the PR

Expected Result

One of the following:

merge would be rejected.
the apply operation would still be executed
I could run apply after the PR was merged if merge did not happen
Nothing of the above should be relevant as apply should automatically happen on the merge queue

Actual Result

The apply action was canceled as the PR was merged.
The change was not applied
I could not re-run apply because it requires the PR to be open.
drift. https://github.com/getsentry/ops/actions/runs/9653132198/job/26624804891

mwarkentin commented 5 months ago

Is there a way for us to configure GHA to not cancel on merge? Feels like an optimization for unit test runtime or something..

mwarkentin commented 5 months ago

Re: apply on merge, I think we need to consider it very carefully as it can introduce some catastrophic failures.

An example:

A GCP disk is scaled up out of band. There is no way to shrink the disk, so the way it would apply will be to destroy the disk and recreate it. Hope your backups are working and tested. 😅

getsentry / tacos-gha