Support metrics collection in DD

[timmc-edx]

Acceptance criteria:

• [ ] edx-django-utils supports a new add_metric (or similar) monitoring function, complementing span tag/custom attribute collection
• [ ] The new function is implemented at least for Datadog, but is general enough that it could be implemented for New Relic and OpenTelemetry as well.
• [ ] 2U: Communicate to Sonic team when this is released (https://2u-internal.atlassian.net/browse/SONIC-409)

Implementation notes:

• For Datadog, we'd want to pull in the datadog Python package and install the DogStatsD agent.
  • Types of metrics: https://docs.datadoghq.com/metrics/custom_metrics/dogstatsd_metrics_submission/ (gauge, count, distribution, timer, etc. and optional tags)
  • NB: Ignore ddtrace.Span.set_metric in ddtrace! That's not actually metrics.
• Compatibility and abstraction:
  • OpenTelemetry requires that you first construct a named Meter (of a particular type: gauge, counter, etc.) and then make calls to it. edx-django-utils could transparently create and cache these.
  • New Relic has yet another approach, with push-based metrics via record_custom_metric and pull-based via register_data_source -- and each value can be a number or a dictionary of statistical aggregations (count/total/min/max) that don't match the other systems' metric breakdowns. It also has a naming scheme that may conflict with other implementations. It might be that NR users should just use the OTel integration. See comparison of support.

[timmc-edx]

Requesting clarification from Datadog about whether set_metric is even what we would want to call: https://help.datadoghq.com/hc/en-us/requests/1773961 (2U private link)

[timmc-edx]

Got an answer:

• set_metric in ddtrace is for span tags (with numeric values), not actual metrics.
• We would need to add dogstatsd or similar. For that, I think we'd want to switch to extras requirements.

[timmc-edx]

We may hold off on this and have Purchase try out dogstatsd in a private plugin if it turns out they need it, and then consider building out an abstraction layer in edx-django-utils based on what we learn from that.

[robrap]

Thanks. I would guess that SRE has experience deploying dogstatsd, which could be confirmed. I think Purchase should be pointed to SRE for support around the setup.

[robrap]

https://2u-internal.atlassian.net/browse/SONIC-409 is being used to PoC this functionality, rather than jumping to adding anything to the edx-django-utils API.

[robrap]

I'm going to close this ticket as unplanned, and we'll revisit when and if it is needed in the future.