Move service catalog into source control and provide a mechanism for updating it in Datadog

[zachgoldfine44]

We've been creating service catalog entries in the Datadog UI, which is a fine way to get started, but we should export it as YAML, add it to vets-api source control, and then create a mechanism for it to be pushed to Datadog when updated (e.g. via Github Action on merges to main).

Having it in machine-readable format will also allow us to build additional validations like making sure that all controllers/jobs are associated with an extant service, by checking against what's present in the YAML file.

Definition of done

• [x] Export service catalog entries as YAML
• [x] Add YAML to vets-api source control
• [x] Create a programmatic mechanism for entries to be pushed to Datadog when updated (e.g. via Github Action on merges to main)

[zachgoldfine44]

Does this need to be a part of code yellow or can it be done after?

[zachgoldfine44]

Patrick S: No way to export service catalog out of Datadog as yaml, only csv. Need someone w/ permissions to make Patrick S an API key with scope that enables this.

(Steve, Bill, Aparna have admin)

Aparna can make and share API key.

[zachgoldfine44]

@lexicalninja Any updates to share?

[mattpointzxer0]

Still in progress, update on Monday.

[lexicalninja]

I have been able to do a few things in DataDog and have this information:

Options

GH Action from Marketplace

There is a github action option that can be dropped in, but the action is for a single service and would require an action file for each service as far as i can see.

Direct Github Connection

There are built in DataDog connections to GitHub that can be added. This would use a single service.datadog.yaml file at the root of the repository in order to update the services. This requires GH admins to make a GH app for the repo and then to set up webhooks in DD and GH to talk back and forth.

API + GHA

The Service definition API allows for the uploading of metadata to create and update services. The docs appear to indicate that you would update a single service at a time, but I tested it with a multi-yaml document and it created and updated multiple services at once.

There is a risk that this really is unsupported behavior, but it currently works.

This option needs to figure out where the yaml lives. Is it one file in the root of the api? is it a service.datadog.yaml in each services subfolder and the CICD checks the change set on merge for these? Some other option?

Recommendation

API + GHA is probably the easiest without getting GH admins involved and the easiest to modify and change. V1 can be a simple curl request on merge of the master file and then from there OCTO can figure out better design and governance of the file.

One thing that I am not sure of yet is whether all the services live in one place or if we will need some other way to catch strays that aren't in vets api code(if they exist)

[zachgoldfine44]

Update: verifying. Aiming for draft PR later this week.

[lexicalninja]

Draft PR is up

[mattpointzxer0]

Patrick S: This should be finalized Monday, with the validations in their own PR.

Bill: Reach out to Platform in slack and cc: @BillChapmanUSDS for admin help.