feat(infra): add datadog + custom metrics

[seaerchin]

Problem

Isomer does not have any custom metrics at the moment. This presents difficulty for the team when, for example, deciding when to deprecate endpoints or to quantify impact/active users. This PR solves this by adding custom metrics to our codebase using datadog. This is done through adding the hot-shots library together with creating a StatsService/StatsMiddleware that tracks each specified metrics.

Solution

1. use hot-shots library to submit custom metrics to datadog. This was chosen over the official API sdk because of the difference in transport. hot-shots uses UDP, which means that we don't require any waits.
2. Rather than allowing the caller to pass in the specified metric, StatsService computes the metric itself. This is because our metrics are a snapshot of system state. This means that the additional logic (to compute said state) should be hidden in order to avoid false positives where the caller passes in an incorrect metric to be submitted

Notes We log the individual path for each v1 api call with the exception of users. This is because the router at that prefix is a v2 router so if we tag the individual path, it would lead to the v2 calls being logged as well (leading to v1/users never being deprecated)

New dependencies:

• hot-shots : statsD client for datadog. this requires the dogStatsD daemon to be running, which is true for us (see the datadog config for .ebextentions) because we use 7.x.
  • follow the instructions here to have datadog agent on your local machine

[seaerchin]

use hot-shots library to submit custom metrics to datadog. This was chosen over the official API sdk because of the difference in transport. hot-shots uses UDP, which means that we don't require any waits.

Hmm but this is not true right? The documentation states "Because it uses UDP, your application can send metrics to DogStatsD and resume its work without waiting for a response. If DogStatsD ever becomes unavailable, your application doesn’t experience an interruption."

think you've a misconception! DogStatsD is the one that hot-shots is currently using and the official sdk uses the rest API. see here and the below 2 screenshots.

After downloading the datadog in my local machine, what is the fastest way that I can test that this code works ya? just run npm run dev on frontend and backend, then go to the all sites page. This should cause the agent to submit metrics. Thereafter, navigate to the isomer product dashboard here. Take note that you need to wait for the agent to flush so it's not immediate