Move EKS administration to private endpoint

[ben851]

Description

As a developer of GC Notify, I would like our administrative endpoints to be as secure as possible so that we can stay out of the news.

It is best practice that the EKS admin API not be exposed publicly. With the introduction of our VPN, we can now move this behind the VPN.

WHY are we building?

• Complies with ITSG-22 & 33
• Increased security

WHAT are we building?

• We will need to implement github action runners inside the EKS cluster

VALUE created by our solution

Increased security and reliability Policy conformance

Acceptance Criteria

Given some context, when (X) action occurs, then (Y) outcome is achieved.

• [x] K8s is behind a private endpoint
• VPN works and we can still connect

QA Steps

• [x] verify that we can connect to k8s through the vpn and execute any kubectl commands in production

[ben851]

I've got the github ARC runner working inside the cluster, but it needs some additional config in order to work with our existing workflows. Working on that today.

[ben851]

Implemented private EKS with ARC runners working on custom runner image, verified workflow works. Wrote the beginning of the private EKS ADR.

[ben851]

Completed ADR. Need to follow up with next steps.

[ben851]

Jimmy will review the ADR once more, and I will proceed with the Terraform and Manifest PRs

[ben851]

Running into issues with the docker build step in notification-api which uses kubectl. Working on solutions.

[ben851]

Working on centralizing the kubernetes rollouts to manifest, and have other workflows call that workflow remotely. I have this working with CURL, and am trying to integrate it into the api workflow.

[ben851]

Integrated into the API workflow. I've created 5 PRs to move the remaining components. Note that the manifests one must be merged first, or the other PRs will break functionality.

https://github.com/cds-snc/notification-manifests/pull/2443

https://github.com/cds-snc/notification-api/pull/2138 https://github.com/cds-snc/notification-admin/pull/1775 https://github.com/cds-snc/notification-document-download-api/pull/161 https://github.com/cds-snc/notification-documentation/pull/145

[ben851]

PR for TF on ARC https://github.com/cds-snc/notification-terraform/pull/1187

[jimleroyer]

Many PRs to review and approve for Ben to move on.

[ben851]

A ton of work done over the last two days. Github ARC is in staging, and is working on several workflows.

The merge to main staging (and production) manifest kustomize code is too old for the latest version of kubectl. I will have to update it before proceeding.

[jimleroyer]

Many trials and errors yesterday. Got a few issues with kubectl and need to update the kustomize code today to re-organize and get past troubled waters and trying times.

[ben851]

Kustomize refactored for dev and staging and now running on internal GHA properly!

Next step is to do the app workflow updates, which will occur tomorrow.

[jimleroyer]

Ben: "It's all working!" Staging environment is good with the manifests changes. A few more repositories to migrate for today and more PRs coming in.

[ben851]

PRs for Monday: https://github.com/cds-snc/notification-terraform/pull/1205 https://github.com/cds-snc/notification-manifests/pull/2467

[sastels]

Some access token issues that were fixed last week. Work almost ready to switch!

[ben851]

Converted ARC to use GitHub App (Notify PR Bot) instead of PAT. Updated the ADR to reflect this. Private EKS is ready to go out today after notify dev review

[sastels]

in staging, ready for testing. Will let it sit in staging for a week before pushing changes to prod

[sastels]

Had an issue deploying k8s to staging. fixed now but needs improvement.

[ben851]

https://github.com/cds-snc/notification-api/pull/2145

[ben851]

https://github.com/cds-snc/notification-admin/pull/1782 https://github.com/cds-snc/notification-documentation/pull/148 https://github.com/cds-snc/notification-document-download-api/pull/164

[P0NDER0SA]

This is deployed to staging so we're letting it sit for a week to stew. (we're getting a stew going)

The 3 PRs above need to be merged 👆

[P0NDER0SA]

The PRs were merged and now we're looking to review the work and make sure it's all good

[P0NDER0SA]

waiting til tuesday to get the prod prs going

[P0NDER0SA]

we have to change some production code

[jimleroyer]

This is done in staging and dev. We need to change a release pipeline in production. Waiting for today's release. Won't be enabled in production, we just want to verify first if the github controllers work in production.

[ben851]

Started work on prod: New version of github arc runner image to include jq https://github.com/cds-snc/notification-terraform/pull/1237

Updated production kustomize to build against latest kubectl https://github.com/cds-snc/notification-manifests/pull/2515

[ben851]

Still waiting for a review on 2515

[ben851]

2515 Merged, to be released tomorrow

[sastels]

released, just a bit more work debugging the prod release workflow.

[ben851]

Made changes to production workflows yesterday. Will be tested during release process today. If they work, I can switch prod to private eks.

[ben851]

Workflow worked on github arc, created PR to move to private eks.

[sastels]

ready for review! https://github.com/cds-snc/notification-terraform/pull/1259

[sastels]

merged, ready for release then QA in prod

[ben851]

Released in prod, ready for QA

[jimleroyer]

Pond is QA'ing.

[P0NDER0SA]

Verified. connectivity to Production K8s only works when the vpn is turned on