Datadog - Set up Private Location and Dashboard

Value Statement

As a DevOps Engineer I want to set up monitoring for our public and private endpoints So that traffic can be monitored to eliminate mystery to how users are using our networking

[ ] I have talked with QA

Datadog offers value statements in it's documentation on synthetic monitoring

Create custom locations in mission-critical areas of your business.
Verify the application performance in your internal testing environment before you release new features to production with Synthetic tests in your CI/CD pipelines.
Compare the application performance from inside and outside your internal network.
Also, ["private locations are integral to testing internal-facing applications"]

Acceptance Criteria

The goal is a Datadog Dashboard that shows a comparative metric of latency and traffic between private and public endpoints This is an enhancement to our codebase and implements non-breaking changes. New task definitions may need to be created.

Checklist

[ ] Set up a private location in ecs (using the Getting Started docs), perhaps deployed with our Datadog agent.
[ ] Track Private location performance with SLOs
[ ] Create two widgets on the dashboard, one that displays the potential latency differences, and one that monitors compares requests/second.

QA Considerations

As the regression validates every private endpoint, QA should be consulted wrt to setting up synthetic monitoring of the private endpoints

Additional Info/Resources

Getting Started with Private Locations
Monitor your Synthetic private locations with Datadog
Establishing effective SLOs
TIC Gateway Status and Latency - an example developed by Senior DevOps Engineer at Oddbal
Run Syntehtic Tests from Private Locations
"Since private locations are a critical part of test infrastructure, your SRE and platform teams need to ensure that they are not only performant but always available for testing. Datadog enables teams to track the performance of all of your deployed private locations and set objectives for improving their reliability over time with highly customizable SLOs." - Datadog Blog

department-of-veterans-affairs / notification-api