Integrate statsd metric collection with lambda API

[andrewleith]

Description

Get statsd back up and working for the Lambda API to bring back lost metrics to track how our API is performing.

Since moving the API to lambda, we are no longer getting all of our business analytics metrics collected via statsd.

1. We can deploy the cloudwatch agent by deploying our lambda function using a container image which includes the cloudwatch agent binary package.

Details can be found here: https://aws.amazon.com/premiumsupport/knowledge-center/lambda-linux-binary-package/

2. Another option is to forgo using statsd and use Lambda Insights instead. Details can be found here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Lambda-Insights-metrics.html

Acceptance Criteria** (Definition of done)

To be refined through discussion with the team

Given some context, when (X) action occurs, then (Y) outcome is achieved

• [ ] HTTP stats are once again being collected (like service callbacks complaints)
• [ ] Over daily limit alarm is working again
• [ ] make sure all alarms that depend on statsd are working again

QA Steps

• [ ] Check in staging that the statsd metrics previously dropped are now collected
• [ ] Check in production that the statsd metrics previously dropped are now collected

Additional info

AWS Support case: https://support.console.aws.amazon.com/support/home?region=ca-central-1#/case/?displayId=11155697721&language=en

[jimleroyer]

Hey team! Please add your planning poker estimate with Zenhub @andrewleith @jzbahrai @sastels @smcmurtry @whabanks

[yaelberger-commits]

Any overlap with other sources? NewRelic for example? More fine grain metrics available through statsd

[yaelberger-commits]

Try option 1 with container image in a timeboxed timeframe and then try option 2 only if needed.

[sastels]

AWS doesn't have a Python 3.10 base image yet :/ https://docs.aws.amazon.com/lambda/latest/dg/python-image.html

[yaelberger-commits]

Can also create a separate task for adding new statsd metrics for new code

[sastels]

We can manually push a metric value as we do here for sending notifications to redis. Would work for the "over daily limit" ones.

[jimleroyer]

@sastels This could be another way and considering that most statsd metric are going through an annotation, it might not be that difficult a task.