[APM] Datadog custom APM

[timmc-edx]

Get custom attributes and such into Datadog (edx-django-utils monitoring).

AC:

• [x] LMS and CMS are sending APM to Datadog
  • [x] Spans and traces
  • [x] Span attributes
  • [x] Exceptions (including both NR and DD receiving them)
  • May have to test record_exception by provoking an error in Kafka in stage
  • Confirmed locally
• [x] Document how teams can enable this for their IDAs
  • Done: https://2u-internal.atlassian.net/wiki/spaces/ENG/pages/1008500757/How+to+migrate+from+New+Relic+to+Datadog

Implementation:

• [x] Allow putting custom span tags into OTel or DD: https://github.com/openedx/edx-django-utils/pull/397
• [x] Upgrade edxapp to new edx-django-utils: https://github.com/openedx/edx-platform/pull/34666
• [x] Turn on new edx-django-utils feature in edxapp to try it out:
  • [x] LMS stage: https://github.com/edx/edx-internal/pull/10558
  • [x] Studio stage: https://github.com/edx/edx-internal/pull/10749
  • [x] Prod: https://github.com/edx/edx-internal/pull/10771
  • [x] Edge: https://github.com/edx/edge-internal/pull/695

Notes:

• opentelemetry-instrument will not work with edxapp, since it is incompatible with gunicorn's forking model. Instead, we'll need to use one of the following:
  • https://github.com/mitodl/open-edx-plugins/tree/main/src/ol_openedx_otel_monitoring/ or similar to explicitly initialize otel
  • ddtrace's built-in DD_TRACE_OTEL_ENABLED: https://ddtrace.readthedocs.io/en/stable/api.html#opentelemetry-api
  • Or just use ddtrace directly to send metrics to the agent.

[robrap]

@timmc-edx: Reminder to clean-up or close this ticket.

[timmc-edx]

This ticket looks like it is still accurate; it's a bit more of a coordination ticket covering both code and config.

[robrap]

Remaining work: Test record_exception locally or see if our actual usage has triggered.

[timmc-edx]

Confirmed using a locally provoked exception.