Github Actions Runner Fails To Launch Scaleset

[ben851]

Describe the bug

SEV-2 The GitHub ARC controller fails to run if the actions runner docker image is deprecated. We did not get advance notice of this, and there was no obvious error in the existing pods.

Eventually the actions runner scaleset fills up with failed pods and no more runners will be launched.

Temporary fix:

1. Cancel all pending actions
2. Helmfile -e staging -l app=arc,tier=scaleset destroy
3. Update the arc runner image in notification-terraform/aws/ecr/github_runner to latest
4. Docker build/push
5. Helmfile -e staging -l app=arc,tier=scaleset apply

To Reproduce

Build and push docker image with older version of GHA runner, launch an affected workflow file

Expected behavior

The workflow should complete successfully

Impact

Impact on Notify team:

We will be unable to release the application due to workflows not functioning

Additional context

There was no immediate error to be found in either the scaleset listener or the controller. Instead, I had to apply the above steps manually (except for the updating of docker image). When first run, the arc controller spins up pods. I tailed the logs on one of those pods and that is where the error stood out.

Action Items

• [x] Build a Github workflow that will build and push a new version of the ARC runner when the Dockerfile is updated
• [x] Build a generic workflow in the manifests repo that will do a helmfile apply based off of a selector provided as an input
• [x] Build a script in terraform that calls the manifests individual rolllout, and integrate it with the docker build and push workflow

[ben851]

Created a workflow that builds and pushes the docker image to the account repository.

Working on the workflow that will deploy an individual application via helmfile.

[sastels]

Will circle back with Ben on this when he's back.

[ben851]

Workflows created.