[APM][Perf] Determine performance impact of DD APM agent

[robrap]

We've seen impact in NR when adding the DD agent. This ticket is to confirm that we see a nearly equal performance win in DD when we drop NR APM.

Notes:

• This won't affect the contract numbers directly, but may affect the contract negotiation in general. Especially if we note a problem. Maybe we get a better deal? Maybe we get fixes on their roadmap?
• The idea is to disable NR in LMS (or edxapp) Edge for enough time to see a change in DD. This may require actual GoCD deployments to enbale/disable.
• The performance hit/improvement can be seen in NR from when Tim enabled/disabled DD APM on edxapp prod. See apdex changes, or chart duration changes for different percentiles.
• In Datadog, we are looking for a similar improvement when NR APM is dropped. If we don't, we know that NR APM seems to be better performant. Does this affect our decision (hopefully not)? Can we tune the agent? Can we request DD make improvements? The sooner the better to get POV attention on anything we find.
• There were questions about auto-scaling, but Edge is already at the minimum number of hosts, even with NR=DD, so there is no room to scale down anyway. This can be ignored, and is here for historical purposes only.

Acceptance Criteria:

• [x] Ensure that DD APM alone has similar improvement over DD+NR as NR APM alone.
  • [x] If not, follow-up with DD appropriately (see notes).

[robrap]

Other things we might want to look for:

• We seem to be missing lots of indexed spans. Is NR interfering?
• There was something else we discussed Wednesday, Apr 24, but I can't remember. Maybe the trace version errors?

[timmc-edx]

Disabling NR on Edge showed roughly 15% improvement in median latency. Graphs below for LMS, CMS, and "Django" show the drop in latency around 16:00:

(Bear in mind that these are log graphs to help make the change more obvious among various latency spikes. The improvement numbers are taken from averages extracted from various time spans.)

This appears to be in the same ballpark as the latency increase when adding DD to NR.

[timmc-edx]

Removed A/C "Confirm that code-owner information is still attached to DD traces (and that we weren't piggybacking on NR for this)" -- we actually aren't shipping this to DD yet. It does appear that we would be missing about 1% of code owner info with just DD, though: from Transaction select count(*) facet code_owner_transaction_name where code_owner_transaction_name is not null and appName = 'prod-edx-edxapp-lms' since 1 hour ago

[robrap]

Yay! Thanks @timmc-edx.