Disabling NR APM causes trace concatenation in Datadog

[timmc-edx]

Ultimately, this ticket is for disabling New Relic APM across edxapp. We ran into trace related issues in DD when first attempting to disable NR APM. We later caused the same issue in Edge when simply disabling NR Tracing.

This bug has been observed in edxapp (LMS and CMS), enterprise-catalog, and registrar. It can be identified by searching for spans matching operation_name:django.request -@_top_level:*.

Acceptance criteria

• [x] Understand and fix the trace concatenation issue
  • Fixed with ddtrace 2.14.2
• [x] Undo the below temporary changes, or make them more permanent or robust
• [x] Replay NR disablement PR: https://github.com/edx/configuration/pull/33

Things we have already tried

These should be checked off once they have already been either reverted or made permanent:

• [x] [keep] Set DD_DJANGO_INSTRUMENT_MIDDLEWARE to reduce the noise when debugging huge traces.
  • No compelling reason to change this back any time soon. Added to Datadog docs as an option.
• [x] [reverted] Set DD_TRACE_HEADER_TAGS to debug tracing headers
• [x] [keep] We've changed a number of monitors and dashboards from using Service Entry Spans to operation_name:django.request on All Spans since service entry spans were unreliable.
  • No reason to change this back.
• [x] [reverted] We removed the NR license key for stage and edge LMS
• [x] [reverted] We set up a free-tier key as a test in stage and edge LMS
  • We've reverted all uses of the key (EDXAPP_NEWRELIC_LICENSE_TEST_FREE) and removed from AWS secrets manager
• [x] [reverted] Experimental patched newrelic agent and the enablement flag
• [x] [reverted] Set DD_TRACE_PROPAGATION_STYLE_EXTRACT=none
• [x] DatadogDiagnostics plugin app; addition of that package's middleware via remote-config; the settings and waffle flags that activate or control them. Removal strictly in this order unless otherwise specified:
  • [x] Watch for activity with service:edx-edxapp-* dirname:"/edx/var/log/supervisor" "[edx_arch_experiments.datadog_diagnostics.middleware]"
  • [x] Remove middleware references from remote-config (EXTRA_MIDDLEWARE_CLASSES Django setting) [stage and prod, edge]
  • [x] Remove entire app and middleware from edx-arch-experiments [PR, release, stg/prod bump, edge bump]
  • [x] Remove any Django settings that those used (all prefixed with DATADOG_DIAGNOSTICS_) -- if it's just DATADOG_DIAGNOSTICS_ENABLE it can be merged in any order, as it's just controlling noisy logs we don't have any more. [prod LMS was only instance]
  • [x] Delete Waffle flags (all prefixed with datadog.diagnostics.) -- merge in any order, as these turn features on
• [x] [reverted] Disable Celery instrumentation via DD_TRACE_CELERY_ENABLED=false, because some of the request spans in anomalous traces have missing parent spans that were celery-related.
• [x] [reverted] Debug logging of celery signals via DATADOG_DIAGNOSTICS_CELERY_LOG_SIGNALS (using edx-arch-experiments 4.3.0)
• [x] [reverted] Log span ancestry on anomalous traces to find out what operations are involved.
• [x] Use a release-candidate of ddtrace (via EDXAPP_DDTRACE_PIP_SPEC) that closes celery spans using a fallback
  • [x] Revert overrides of EDXAPP_DDTRACE_PIP_SPEC
  • [x] Revert inclusion of a ddtrace build in vendor directory

    Details

When we disabled NR APM in edxapp on June 6 we observed two anomalies with traces:

• At about 17:34, the rate of service entry spans (and traces) in service:edx-edxapp-lms env:prod dropped precipitously by 2-3x.
• At the same time, the rate of all spans increased by about 5x. (This anomaly appears to have disappeared, retroactively!)

However, we believe the actual traffic was unchanged. This is corroborated by the Django hit metrics remaining steady, as seen in the Service Catalog. We cannot find any relevant code or config changes that would have been deployed around that time.

Our current understanding is that the majority of Django web requests that are traced are not recorded as service entry spans, but are instead parented to a different trace. This causes several problems:

• Monitors and dashboards that are based on traces or service entry spans are unreliable.
• It's difficult to impossible to view most of our traces because they become concatenated into 50,000-span-long monsters that can't be loaded in the browser.

We can also reproduce this issue by setting "Tracing type: None" in the application settings in NR (usually set to Distributed Tracing).

Links

• Private support ticket: https://help.datadoghq.com/hc/en-us/requests/1722518
• Diagnostic dashboard: https://app.datadoghq.com/dashboard/fzr-3jk-nsf

[robrap]

Other notes:

• Here is a google doc of potential plan for debugging and resolving.
• The current plan is:
  • [x] enable the new headers (2024-06-18: 20:49 UTC)
  • Config change went out in deploy 6f17017-5957
  • [x] disable tracing in NR (2024-06-18: 21:30 UTC) and confirm that DD tracing is broken again.
  • Set "Transaction tracing: Off" and "Tracing type: None" in edge LMS
  • [ ] Leave broken for a short window before re-enabling tracing in NR (record time).
  • [ ] Disable NR frontends in Stage and Edge and ensure it doesn't break DD tracing.
  • [x] https://github.com/edx/configuration/pull/40 (disable legacy)
  • [x] https://github.com/edx/edx-internal/pull/11100 (disable MFEs stage)
  • [ ] https://github.com/edx/edge-internal/pull/719 (disable MFEs edge)
    • [X] Authn deployed to Edge
    • [X] Learning deployed to Edge
    • [ ] Discussions deployed to Edge (waiting on owning team)
  • [ ] Confirm that DD tracing can still be broken with disabling tracking in NR (record times).

[robrap]

Additional thoughts, questions, ideas:

• We should double-check distributed tracing settings for tracing type (want updated Distributed Tracing).
  • No other settings were seen in doc.
• If we decide to try to use the NR test account, we should ensure it is safe. We can remove other users (if they are still there).
• What does the nginx request header set? Could this be interfering?
• APM Profiling caused a different problem with additional spans, but is it possible it could heal this issue without NR?
• If it comes to this, can we find a way to replicate locally?
• Could there be important information buried in the logs that we are no longer seeing? See https://github.com/edx/edx-arch-experiments/issues/610.
• Could the invalid traceparent log messages be related? See https://github.com/edx/edx-arch-experiments/issues/665.

[robrap]

Currently, we're investigating if using a NR Free Tier account for edxapp is enough to get DD traces working.

Other possibilities may include trying to get tracing (or APM) disabled everywhere in Edge. This includes where Spans were found in the last day:

prod-edge-edxapp-cms 9.28 M
prod-edge-analytics-api 2.35 M
prod-edge-notes 2.3 M
prod-edge-edxapp-workers-lms 755 k
prod-edge-forum 675 k
prod-edge-edxapp-workers-cms 188 k

[robrap]

[idea] We might want 3 modes for our hacked NR agent:

1. Send no data to NR, but add tracing info (fixes DD traces).
2. Send no data to NR, and fake a bad account id so we don't even take a performance hit (breaks DD traces).
3. Send data to NR (costs money, but if we want to verify anything temporarily, we can.

[timmc-edx]

We'll leave the datadog_diagnostics cleanup task for about a week, for cleanup 2024-10-14 or later. Moving to blocked for until then.