EKS Upgrade Testing Plan: Step 3, PART II: Establish automation connectivity between vets-api & Postgres/RDS

[rmtolmach]

Issue Description

Part 1 of this ticket: https://github.com/department-of-veterans-affairs/va.gov-team/issues/74637, created a rake file that tests pg connectivity when it's run. This issue is to automate the running and reporting of that rake task.

Two options:

1. Update admin_controller.rb and app_info.rb to extend https://api.va.gov/v0/status to return a true/false value if pg is up.
2. One idea was to do this in a Liveness or Readiness probe (docs) possibly in the parent-helm-chart server-deployment. However, a downside of this solution is that changing the version becomes a three-step process: changing the version in the parent help chart, bumping the version in vets-api and also in the manifest repo. Lindsey and I thought this could be cumbersome and have decided on solution 1 above. But just to document it, Rachal came up with something like this:

   {{- if gt .Capabilities.KubeVersion.Minor "26" }}
   exec:
   command: ["bundle"]
    args:  ["exec", "rails", "db:check"]
   initialDelaySeconds: 5
   periodSeconds: 5
   {{- else }}
   ... the rest of the code that was already in the probe ...
   {{- end }}

Tasks

• [ ] https://api.va.gov/v0/status returns a true or false value for the pg_up key.
• [ ] Success or failure is reported through a Datadog synthetic test.
• [ ] Document the steps that will need to be followed by the person doing the upgrade testing: https://vfs.atlassian.net/wiki/spaces/PPT/pages/3127246924/Steps+for+EKS+Upgrade+Test

Success Metrics

• [ ] Postgress connectivity test is friction-less for the EKS upgrade tester.

Acceptance Criteria

• [ ] The pg connection tests runs automatically.
• [ ] The pg connection tests reports its success and failure automatically.

• [ ] The documentation is updated with the necessary steps and instructions for this step.

Validation

Assignee to add steps to this section. List the actions that need to be taken to confirm this issue is complete. Include any necessary links or context. State the expected outcome.

[rmtolmach]

Consider adding more information to the json returned at v0/status if it would be helpful (more than just true or false)