[Discovery] Reporting telemetry to Datadog

[timmc-edx]

Acceptance criteria

• [x] POC for delivering Datadog and New Relic telemetry simultaneously from the same process
• [x] POC for delivering metrics from a local service to Datadog and New Relic (for developing)
  • https://2u-internal.atlassian.net/wiki/spaces/~840928901/pages/813793298/OpenTelemetry+New+Relic+and+Datadog covers simultaneous DD and NR from devstack

Background

In order to deliver edx.org metrics to Datadog, we want to implement Datadog integration in our IDAs (possibly via OpenTelemetry). This should allow us to add a new telemetry backend without adding another hardcoded service and while still supporting New Relic during the transition period. In particular, we want to deliver telemetry to New Relic and Datadog simultaneously so that we can rebuild our dashboards and alerting in Datadog while we are still relying on New Relic. Broadly, this will affect:

• What happens when edx-django-utils monitoring APIs are called (e.g. set_custom_attribute)
• How we configure and start server processes

Questions

• Can OpenTelemetry report to multiple collectors from the same process?
• What are the various ways to use OpenTelemetry with Datadog?
  • What are the pros/cons of instrumenting the application with the OTEL library vs the DD-trace library?
  • OTEL seems reasonable for open source, but do we lose anything?
  • What are the pros/cons of our using the OTEL Collector/Datadog Exporter vs the Datadog agent?
  • Do we need SRE to help with this? (I think they did some experimenting.)
  • How will this look for AWS instances (using configuration repo) vs k8s?
• OTEL configs questions:
  • How will we get OTEL config for Kafka deployed? How well does it work?
  • See MIT ODL implementation below.
• Where will we install new dependencies? As part of edx-django-utils, or as private dependencies?

Notes/Resources

• https://docs.datadoghq.com/opentelemetry/ (high-level picture with various deployment options)
• https://www.datadoghq.com/blog/instrument-python-apps-with-datadog-and-opentelemetry/
• https://lumigo.io/opentelemetry/using-opentelemetry-with-datadog-a-practical-guide/
• https://ddtrace.readthedocs.io/en/stable/api.html#opentelemetry-api
• https://www.datadoghq.com/blog/correlate-traces-datadog-rum-otel/
• See https://discuss.openedx.org/t/opentelemetry-integration-coming-soon/11404/7 for PR of OTEL config in Open edX.

[robrap]

@timmc-edx: Do both New Relic and Datadog need to be a part of the actual process startup command, and is it possible to do both simultaneously? I guess I'm realizing that a POC on getting APM data to both New Relic and Datadog simultaneously is important, to prove out if that will be an issue. I really hope we don't need a big-bang switch from one to the other. UPDATE: Do you want a POC of getting APM data to both NR and DD in parallel to be part of this ticket, or a new ticket?

[timmc-edx]

@robrap I've filled out the epic with new tickets as best I could figure -- not sure if there's anything else to add proactively. See what you think.

[timmc-edx]

I've scoped this down to just the POC of "how do we get stuff into DD", which is done.