edxapp hosts are not tagged correctly

[timmc-edx]

Currently there are no hosts tagged service:edx-edxapp-lms in Datadog. (We do have APM with the correct tags.)

Filtering by env:prod (or stg) and grouping by service,services,cluster shows these groups:

• service:edx-edxapp-lms-workers, services:lms, cluster:edxapp
• service:edx-edxapp-cms-workers, services:workers, cluster:worker
• service:edx-edxapp-cms, services:cms, cluster:studio

Whereas filtering by env:edge shows these groups:

• service:[none], services:lms, cluster:edxapp
• service:[none], services:cms, cluster:studio
• service:[none], services:workers, cluster:worker

What we expect to see:

• Groups with service tags of edx-edxapp-lms (currently missing), edx-edxapp-cms, edx-edxapp-lms-workers, edx-edxapp-cms-workers.
  • This is probably the only one Arch-BOM cares about
  • LMS might be mistagged as LMS-workers
• Consistent services tags (TBD -- not clear what we want for these, but they should match up)
• Consistent cluster tags, with no use of the generic edxapp (should be lms or cms) or the outdated studio (should be cms)

[robrap]

• Ticket https://2u-internal.atlassian.net/browse/GSRE-1771 was closed, but didn't exactly address this issue.
• As part of the remap ticket, there is a linked DD support ticket: https://help.datadoghq.com/hc/en-us/requests/1762193
  • This ticket is still open, but they are recommending dropping the host level tagging, which is sort of noted in these DD tagging docs. The support ticket is still open and it is unclear whether this will mess with infrastructure tagging. This should all be a part of any new GSRE ticket for this issue.
• Do we know if parts of this are blocked on the containerization work to get the right tags?

[robrap]

@timmc-edx: SRE fixed the Infrastructure tab, but we still have the host-level double tagging that is inaccurate. I think we should punt on trying to fix this in ansible, and make this a part of the pipeline work, so I am updating the epic.

[timmc-edx]

I don't see any cms-workers in the env view -- is that still expected?