Reduce Datadog spend for July 2024

[robrap]

The Datadog contract has special constraints for July. For example, it is possible that our reduction of Browser Synthetics won't kick in as a benefit until August, when it no longer has a minimum spend.

It probably makes sense at this point to wait until our mid-July check-in with Datadog. Hopefully that won't be too late.

Notes:

• One place that we can target for July is Log Ingest. We noted that there may be really large messages that aren't helping our cause. See https://github.com/openedx/edx-platform/issues/35151.
• A really long Slack thread about June billing, which relates to July as well.
  • Note: I [Robert] am still quite confused on the June/July credits and thresholds, etc.
• From Shannon: "Recommendations from reviewing usage - I would focus primarily on adjusting RUM, API testing, and potentially adjusting ingested spans"
  • Some API reduction was completed, and that may be all that we can do.
  • RUM reduction happened end-of-June, and that may already be enough.
  • Ingested spans are the main item we can probably still affect, but requires some thought.
  • Important, ingested spans are not indexed spans, although they are all that could be indexed (or seen under the Live view).
  • Here's the top Live Spans for edx-edxapp-lms, the service creating the most spans.
    • Maybe we reduce the number of /event spans ingested? Other?
    • Could that start killing traces?
  • See https://github.com/edx/edx-arch-experiments/issues/653 for additional guidance
  • See APM Usage Dashboard for more details.

[timmc-edx]

Will have more info after the July 15 checkin.

[robrap]

• I created https://help.datadoghq.com/hc/en-us/requests/1779698 which is a feature request to be able to search for long log messages, in case they are affecting ingest.
• We also have https://github.com/openedx/edx-platform/issues/35151 that may fix some large LMS logs.
• I update this Slack thread to get more of the RUM config changes deployed. It was already deployed for some of the larger usage MFEs.
• I don't see specifically what I would target to reduce edx-course-discovery ingest (in terms of larger logs), so I think we should wait to see August before creating any requests around this, unless we just generally want to have them attempt to reduce logging.
• SRE is working on https://2u-internal.atlassian.net/browse/GSRE-1920.

@jristau1984: Do you think we should close this at this point, and July will be what it is, and we'll try again come August?

[robrap]

I'm closing this at this point. Beyond the changes that are in-progress (including SRE), I don't think changes will be worth the effort at this point for July. We'll come back to this in August, if there is anything left to do.