[DD] Migrate Python/Django versions Dashboard to Datadog

[iamsobanjaved]

Migrate the following mentioned New Relic Dashboard to Datadog. The Datadog dashboard version should have all widgets currently in New Relic's version.

New Relic URL (private URL): https://one.newrelic.com/dashboards/detail/ODgxNzh8VklafERBU0hCT0FSRHxkYTo0ODk2MDI?account=88178&duration=7776000000&filters=%28name%20LIKE%20%27python%27%20OR%20id%20%3D%20%27python%27%20OR%20domainId%20%3D%20%27python%27%29&state=8bcd531e-9d9e-477b-3c14-91f3eb9162cb

### A/C
- [x] Create a new Dashboard on Datadog with the same name
- [x] Copy over all widgets from the New Relic Dashboard

[iamsobanjaved]

Datadog Dashboard URL: https://app.datadoghq.com/dashboard/sjj-6xu-pcj

[iamsobanjaved]

I have migrated all widgets to the Datadog version but not getting data related to Python version from all services. I noticed that some services are missing spans, so we have very limited data as of now.

[robrap]

@iamsobanjaved: Can we get more specific about what is missing? Not sure iif we want to track here, or if we should use the DD migration tracking spreadsheet? I would have hoped that we wouldn't be missing much.

@jristau1984 @spencertiberi: What kind of follow-up do we want to do with teams if we find incomplete migrations?

[robrap]

@iamsobanjaved: First, thank you so much for setting up the dashboard.

Some feedback on the dashboard:

• The table "Python/Django versions in stage" is actually showing all environments.
• [proposal] What would you think about all tables showing all environments (including the env tag), and the dashboard simply having an env filter at the top?

Additionally, for my hackathon, I wanted to depend on this dashboard, but couldn't yet for the reasons you stated. So I switched to seeing how to get this dashboard working.

• Regarding missing data, I discovered there were 2 main problems for each service:
  • Missing required middleware.
  • Missing the DatadogBackend required to get custom span tags.
• I created a tracking sheet for what is needed, and I will probably post about this in Slack to try to get maintainers to fix (at least to start).

[jristau1984]

[proposal] What would you think about all tables showing all environments (including the env tag), and the dashboard simply having an env filter at the top?

I went ahead and made this update, so the dashboard widgets now all reflect the environment selected in the filter with Prod being the default.

[robrap]

Thanks @jristau1984. Since you added the filter, I made the other fixes (e.g. removed the redundant table, added env column to some tables, and fixed widget titles).

[robrap]

[inform] I just added the following ticket to your backlog, because I think we can improve observability for future upgrades:

• https://github.com/edx/edx-arch-experiments/issues/785

[robrap]

@jristau1984: I think this ticket is complete. However, all the data we need to the next upgrade is not yet on the dashboard because of config that needs to be updated for many services (see earlier messages). For now, I added a TODO to the dashboard widget, but you could also ticket this work if you wish. I'm not sure if you want arbi-bom to complete this so we have the info we need come the next upgrade, or if you want Spencer to poke owners until they do it, or if you want to simply kick this problem down the road. :)

[robrap]

@iamsobanjaved: I think the necessary follow-up was ticketed in https://github.com/edx/edx-arch-experiments/issues/815. I think this ticket can probably be closed in favor of that new ticket to actually get the data on the board. If you agree, please close this ticket. Thank you!