department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:

https://depo-platform-documentation.scrollhelp.site/index.html

284 stars 206 forks source link

Spike: Access `vets-api` endpoints from DD #83746

Open pjhill opened 6 months ago

pjhill commented 6 months ago

Description

In order to be able to connect DD synthetic tests and monitors to vets-api endpoints hosted in an arbitrary EKS cluster, we need to prove that we can do so with an example EKS cluster other than the typical named VA.gov instances such as dev, staging, prod, etc.

Tasks

[x] Create or discover a candidate EKS cluster hosting a VA.gov with vets-api
[ ] Try various solutions for connecting vagov.ddog-go.com instance to the vets-api endpoints
[ ] Create a document describing the process and technology that works to connect DD and vets-api in a VA gov platform EKS cluster

Acceptance Criteria

[ ] A spike connecting DD and vets-api is operating
[ ] A document describing the spike solution is in Confluence

Additional Information

We self-hosted Datadog synthetic nodes that we could use to access the private network for testing. We should explore that.

pjhill commented 6 months ago

I created a synthetic test that hits the status endpoint of the dev api here -- https://vagov.ddog-gov.com/synthetics/details/6e4-6k5-4qc?from_ts=1716567250688&to_ts=1717172050688&live=true

pjhill commented 6 months ago

Here's the EKS cluster that the Product team has been using to build and test -- https://argo-vetstest-next-eks.vfs.va.gov/applications/default/vets-api-dev?view=tree&conditions=false&operation=false&resource=

pjhill commented 6 months ago

I created this draft PR, but it probably doesn't contain everything that we need to happen within the EKS cluster to establish communication between Datadog and the vets-api endpoints within the eks-next cluster we are targeting for this spike.

pjhill commented 5 months ago

For the moment, I am stuck. I am unable to determine why only some applications are deployed to the test EKS cluster by the ArgoCD instance that is handling deploys -- https://argo-vetstest-next-eks.vfs.va.gov/

I also am not sure how to create routes to the Private Location / Datadog apps that will be created by the Helm chart additions. I do see various Route 53 and Unbound rule configurations in the dev versions of these Datadog applications. Those applications are deployed into the primary EKS cluster used for dev, staging, and prod.

pjhill commented 5 months ago

I started this discussion with folks in the Devops COP -- https://dsva.slack.com/archives/C04CYC4LMU6/p1717598591086829

The outcome of that discussion is --

I should not proceed with attempting to configure the Private Location
IST will be creating the private location that enables Datadog synthetic testing of applications within the cluster in the next sprint hopefully
Once the private location is available, we expect to be able to create synthetics pointed at vets-api endpoints within the next cluster without additional effort

pjhill commented 5 months ago

Given that this ticket depends on this ticket being worked by IST, we will need to carry this ticket over into the next sprint. If IST resolves the ticket to create the private location in the test cluster next sprint then we will be able to proceed with this ticket.