Discovery: Create Ability to Access `vets-api` Resources in EKS Cluster Externally

[pjhill]

Description

We need to discover how to make resources within test EKS clusters routable from our vagov.ddog-gov.com Datadog instance. In order for Datadog synthetics to function properly when they target a resource within an arbitrary EKS cluster operating vets-api, the resources inside of EKS clusters need to be routable from the Datadog infrastructure operating the synthetics.

Refinement Discussion Notes

• Per Ben -- recommend attempting a spike
• Per Ben -- consider DD APM test agent
• Per Patrick -- need to be more specific about what resources are required
  • For example, does not need access to the "Kyle" resource -- does need access to the vets-api API endpoints

Acceptance Criteria

• [ ] Discover how to route Datadog synthetics through to any EKS cluster
• [ ] Prototype a configuration that allows Datadog synthetics to reach a vets-api resource in an upgraded EKS cluster

[IGallupSoCo]

First points of reference on bridging this prospective gap: https://docs.datadoghq.com/containers/cluster_agent/ https://www.datadoghq.com/blog/eks-monitoring-datadog/#enable-datadogs-aws-integrations

These docs suggest that a single instance of Datadog Agent can be spun up to monitor all of the nodes in an active cluster.

My understanding of Synthetics is that tests can call the Datadog API and pull log data that might be usable for identifying all active clusters that need to be tested.