DEPR and move code owner monitoring code with updates

[robrap]

• DEPR of code owner code should happen because this is likely used by 2U-only, so it is just the right thing to do.
• Also, having 2U-only-code would allow us to refactor more easily:
  • We could drop theme. (e.g.code_owner and code_owner_squad would have the same value, and code_owner_theme would be dropped. In theory, we could also drop code_owner_squad, but that might force too much unnecessary work on other teams.
  • We could refactor how celery tasks get code_owner without having to support anything other than Datadog:
  • See https://github.com/edx/edx-arch-experiments/issues/729

Implementation thoughts:

• We should experiment with a replacement solution before creating a DEPR ticket, so we know if we have any blockers before we sign ourselves up.
• CodeOwnerMonitoringMiddleware could no longer be added directly into edx-platform here.
  • We could try using edx-platform's mechanism for adding additional middleware, assuming it doesn't run too late. Note: we could experimentally set a temporary span tag and see how many spans get code_owner, but don't get our new test tag.
  • If this didn't work out, we could consider introducing an openedx-event from DeploymentMonitoringMiddleware that allows operators to add their own custom monitoring.
• Could be a separate ticket, but we used to have a New Relic search script for alerts and dashboards. It would be nice to have the same for DD for this type of work, and other monitoring refactors that have come up, like service name changes.
  • Also, see this separate and related ticket: https://github.com/edx/edx-arch-experiments/issues/786

ACs:

• [ ] Code has been moved from edx-django-utils to edx-arch-experiments plugin, so it is no longer public.
• [ ] Theme functionality is removed from the codebase, as it is no longer valuable.

[robrap]

Giving this a P3 because it is important, but it is not urgent.

[robrap]

Presumably this was marked blocked while I was out. Marking in-progress again.

[robrap]

Phase 1:

• [ ] Roll out new _2 monitoring from https://github.com/edx/edx-arch-experiments/pull/838
• [ ] Check DD to see if _2 tags seem to match the existing span tags Phase 2:
• [ ] Add ability to disable code-owner code from edx-django-utils by fixing bug related to reading settings.
• [ ] Determine what we want going forward. Options.
  • Keep existing tags.
  • Remove theme, and re-use but change data in code_owner tag,
  • Remove theme and come up with a new span tag name.
• [ ] Load new tag off separate settings, so we can enable new code and disable old code separately.
• [ ] Update jobs to also write new settings.
• [ ] Possibly create DD search script if we want to change code_owner, and see if it is in use.
  • [ ] If it is not in use, and we want to change it, we may want to add a separate tag to disable it from the old code while leaving the rest, for simpler expand/contract.