ben851 commented 3 months ago

Description

As a developer of notify, I would like to be able to simplify our deployment process so that it is easier to troubleshoot and be more reliable. Currently we are running internal Github ARC runners on our kubernetes cluster, which are prone to breaking due to aggressive deprecation policies by github. It also adds additional complexity when building and managing our github workflows.

WHY are we building?

We have had a few incidents regarding the github arc runners not working, and this causes our releases to be blocked.

WHAT are we building?

Implement the new certificate based client vpn in addition to the existing SAML based client vpn.

We will have two VPN's in this case, but we can actually reduce costs by removing subnet associations on the existing VPN and only adding one association with the new VPN.

VALUE created by our solution

Simplified release process Less outages and management Removes the chicken and egg scenario for github actions deployments to kubernetes

Tasks

Staging

[x] New VPN deployed
[x] Admin workflow updated to use VPN
[x] API workflow updated to use VPN
[x] Document Download workflow updated to use VPN
[x] Documentation workflow updated to use VPN
[x] IPv4 Geolocate workflow updated to use VPN
[x] Manifests workflows updated to use VPN
[x] Remove callManifests.sh from all application repositories
[x] Remove the individual rollout workflows from manifests
[x] Remove Github ARC from Staging

Production

[x] New VPN deployed
[x] Manifests workflows updated to use VPN
[x] Remove Github ARC from Prod

QA Steps

[x] Verify that we can still deploy
[x] Verify that the github-arc-runner namespace is no longer in staging and produciton eks

Acceptance Criteria

[x] Scenario: VPN deployment
- Given the new certificate based client VPN is deployed
- When the existing SAML based client VPN has subnet associations removed and only one association with the new VPN added
- Then the new VPN should be functional and used for Github Actions deployments to kubernetes
- [x] Scenario: Admin workflow updated
- Given the admin workflow is updated to use the new VPN
- When the workflow runs
- Then it should successfully complete using the new VPN
- [x] Scenario: API workflow updated
- Given the API workflow is updated to use the new VPN
- When the workflow runs
- Then it should successfully complete using the new VPN
- [x] Scenario: Document Download workflow updated
- Given the Document Download workflow is updated to use the new VPN
- When the workflow runs
- Then it should successfully complete using the new VPN
- [x] Scenario: Documentation workflow updated
- Given the Documentation workflow is updated to use the new VPN
- When the workflow runs
- Then it should successfully complete using the new VPN
- [x] Scenario: IPv4 Geolocate workflow updated
- Given the IPv4 Geolocate workflow is updated to use the new VPN
- When the workflow runs
- Then it should successfully complete using the new VPN
- [x] Scenario: Manifests workflows updated
- Given the manifests workflows are updated to use the new VPN
- When the workflows run
- Then they should successfully complete using the new VPN
- [x] Scenario: Remove callManifests.sh from all application repositories
- Given callManifests.sh is removed from all application repositories
- When the change is made
- Then it should be verified that callManifests.sh is no longer present in any repository
- [x] Scenario: Remove individual rollout workflows from manifests
- Given the individual rollout workflows are removed from manifests
- When the change is made
- Then it should be verified that the workflows are no longer present in manifests
- [x] Nobody cries

ben851 commented 3 months ago

All staging deploys in manifests are now tied to VPN