[Vets-API / EKS] Verify configs between BRD and EKS

[mleclerc00]

Description

Double check that all of the needed configurations and files are put in place for the dev vets-api deployment

in order to do so I will be comparing dev

BRD (mainly in deployment) - https://github.com/department-of-veterans-affairs/devops/tree/master/ansible EKS - https://github.com/department-of-veterans-affairs/vsp-infra-application-manifests/tree/main/apps/vsp-tools-backend/vets-api

to make sure that everything is synced properly so we aren't diving down rabbit holes if anything doesn't work.

Background/context

https://github.com/department-of-veterans-affairs/va.gov-team/issues/35039

---

Tasks

• [x] create a checklist to make sure all files and configurations match between BRD and EKS dev
• [x] verify that all of the appropriate secrets are also matching between credstash and parameter store

Acceptance Criteria

• [x] all of the files and configurations that exist in BRD are up to date and exist in EKS dev

---

Reminders

• [ ] Please attach your team label and any other appropriate label(s) (operations, devops, and needs-grooming will automatically be applied as part of the template)
• [ ] Please connect to an epic (this will typically be done by the Platform Operations PM or TL)

[patrickvinograd]

Re the issue with fwdproxy connection getting an SSL error: vets-api BRD instances get the forward proxy's self.crt added to their trust store in ansible/build/roles/vets-api/tasks/main.yml

- name: Copy self signed forward proxy ELB cert (wildcard) to anchors
  copy:
    src: ../deployment/config/fwdproxy/self.crt
    dest: /etc/pki/ca-trust/source/anchors/forward-proxy.crt
    owner: root
    group: root
    mode: 0600

and then subsequently update-ca-trust and update-ca-trust-enable are executed.

I don't know what the equivalent would be for EKS.

[patrickvinograd]

I think the right approach would be to add the self.crt to config/ca-certs at Docker image build time. I see the code_checks.yml GHA job is doing this with the VA PKI cert: cp /etc/ssl/certs/VA-Internal-S2-RCA1-v1.cer.pem config/ca-trust/ And then it gets consumed from there in the main Dockerfile

But there's nothing equivalent in the Build & Push to ECR action. Wondering if both self.cert and the above VA cert should be injected there?

[patrickvinograd]

Digging further - looks like VA-Internal-S2-RCA1-v1.cer gets added to the eks base image (and the gha image) via the packer builds in the devops repo, which is why that is not needed in the Build & Push github action. So figuring out the right place to do likewise for self.crt should address this?

[patrickvinograd]

If there's a more k8s-style way of managing this cert that might be cool, but since the forward proxy is itself not in k8s, then it might be hard to split the difference.

[mleclerc00]

Thank you for the comments Patrick, definitely something to keep in mind as we continue to evolve this migration

[oseasmoran73]

Had a chat with Matt the following has to be done (for my notes and tracking purposes):

• Do they exists between BRD & EKS? -- Potentially create a script that will compare against
• Are the values correct?

The more difficult one that would need to be addressed is Jinja EKS against BRD

In an ideal world, we would probably want a script so that we can reuse once we are ready for higher environments

Additional notes:

One things that we could potentially address is moving away those currently in BRD from Credstash over onto SSM.

The other thing is currently there are some files that live in devops in SSM. We both were in agreement that some of them should not fall into our path, but rather vets-api path.

[mleclerc00]

as it pertains to the deployment_files in BRD land here -> https://github.com/department-of-veterans-affairs/devops/blob/cb4a947401bea743545a64cbc15d1a6d3c895ac5/ansible/deployment/config/vets-api-server-vagov-dev.yml#L113

There are definitely some differences going on which I will work on cleaning up today.

Going to aim at making sure everything is in parameter store at the correct paths and remove all references to credstash for good measure. It also looks like we missed a directory in the first couple of passes (/app/secret/pre)

I used a dumb little "one liner" to capture the files being deployed and output ls commands that I can just bulk copy and paste into the terminal to check for them.

cat vets-api-server-vagov-dev.yml| yq e '.deployment_files[].dest' | sed 's/\/srv\/vets-api/\/app/g' | xargs -I {} echo "ls "{}

[oseasmoran73]

Checked pki tls private directory and the values have the correct format

[oseasmoran73]

Checked pki tls certificate directory and the values have the correct format

[madebydna]

@mleclerc00 / @oseasmoran73 Not sure if you guys saw this vets-api PR but it essentially represents our attempt to ensure that the EKS version of vets-api (on the k8s branch) also trusts the forward proxy certificate. We haven't gotten much feedback on it, so I was wondering if you could take a quick look (cc @patrickvinograd)?

[oseasmoran73]

@madebydna LGTM. Us throwing an approval does not mean much since we are not code owners of vets-api repo

[oseasmoran73]

EKS & BRD are now aligned. See new screenshot

[oseasmoran73]

Given that there is not much verification aside that is it still working as intended in dev, I feel the requirements have been fulfilled. Closing ticket