#2041 - Implement datadog APM for celery

[MackHalliday]

Description

issue #2041

• Script Updates: Refactored health check and run scripts to adapt to the new CELERY_CONCURRENCY environment variable and added support for ddtrace-run, enabling APM in DataDog.
• Introduction of CELERY_CONCURRENCY: Set to "16" in various environments, both in .env and task definitions.
• Increased CPU Allocation: Increased the CPU allocation from 1024 to 2048 for the Celery task definitions in all environments.
• Updated Local Docker: Update local Docker compose file and local envs to function correctly with updates to scripts.

Note
The ECS Celery health check (scripts/run_celery.py) was updated because appending ddtrace run made Celery a child of ddtrace. Therefore, the ECS Celery health check could not properly find the Celery PIDs and failed.

How Has This Been Tested?

Confirm Locally Docker Build OK

• Ran usual commands to build and compose local Dockerfile without issue

Confirmed Deployed OK

• Ran Github Action to deploy feature branch

Confirmed Celery Service Available in APM

• After deploying, I confirmed Celery tasks show in APM under the Celery service.
• If I hit the POST /v2/notifications/email, I can see its related Celery tasks in APM.

Confirmed CPU and Memory OK in ECS Celery Service

• CPU and Memory does not max out.
• CPU only spikes during deployment as expected.

Confirmed Shutdown Logs Appears as Expected

• Per AC, the shutdown logs should still appear when the ECS container is re-deployed
• I deployed a second time from the same feature branch.
• After the second deployment, I check the DataDog logs for the shutdown logs after the new ECS Celery service was deployed.

Confirmed Health Check Works as Expected

• I hard-coded the value of Celery works to be 2 in the check_celery.sh script. This would cause the health check to fail as the CELERY_CONCURRENCY is set to 16 in the task definitions.

  

• I deployed the update and confirm the ECS Celery service deploy failed.

Checklist

• [x] I have assigned myself to this PR
• [x] PR has an appropriate title: #9999 - What the thing does
• [x] PR has a detailed description, including links to specific documentation
• [x] I have added the appropriate labels to the PR.
• [x] I did not remove any parts of the template, such as checkboxes even if they are not used
• [x] My code follows the style guidelines of this project
• [x] I have performed a self-review of my own code
• [x] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to any documentation
• [x] My changes generate no new warnings
• [ ] I have added tests that prove my fix is effective or that my feature works. Testing guidelines
• [x] I have ensured the latest main is merged into my branch and all checks are green prior to review
• [x] New and existing unit tests pass locally with my changes
• [x] Any dependent changes have been merged and published in downstream modules
• [x] The ticket was moved into the DEV test column when I began testing this change