P0NDER0SA commented 1 month ago

Description

As an ops lead, I want to link all requests going through the Notify stack that was triggered by the same user / same origin IP, So that I can quickly analyze their behavior and actions on our stack.

WHY are we building?

Better tracing support through our stack.

WHAT are we building?

Implementing correlation / trace IDs through the various Notify components. Potentially using aws x-ray.

VALUE created by our solution

Quicker support and incident response. Better routing / behavior analysis of the requests made against Notify.

Acceptance Criteria** (Definition of done)

[ ] Filter correlated logs in various Notify components coming from the same request.

QA Steps

[x] login to AWS and verify that tracing is being done for Celery pods
[x] login to AWS and verify that the celery pods are showing up in the trace map
[x] login to AWS and verify that the xray map is showing traffic from Admin to the API nodes
[x] https://ca-central-1.console.aws.amazon.com/cloudwatch/home?region=ca-central-1#xray:service-map/map

Additional info

AWS X-Ray
Notify used zipkin in the past and it has traces of it in the code. Look into the utils component among others. That might facilitate the implementation as the feature is present for another product. Ideally we can just tweak and enable it.

Celery integration with X-Ray:

The one area I am uncertain with xray is how we will plug this with Celery. This and that seems interesting.

P0NDER0SA commented 1 month ago

The very basic middleware code has been added. The Daemon is awaiting information as well. More work is required.

jimleroyer commented 1 month ago

GDS has this previous utility class to trace their request IDs through the application starting from the admin. We can leverage it and modify it for xray (whereas this was meant initially for zipkin from Twitter): https://github.com/cds-snc/notification-utils/blob/623d109f9145996d7f32eacdada190384d8a5ffa/notifications_utils/request_helper.py

P0NDER0SA commented 1 month ago

Jimmy Royer :spiral_calendar_pad: 1 hour ago https://flask.palletsprojects.com/en/2.3.x/patterns/celery/ You can use Celery without any integration with Flask, but it’s convenient to configure it through Flask’s config, and to let tasks access the Flask application. Celery uses similar ideas to Flask, with a Celery app object that has configuration and registers tasks. While creating a Flask app, use the following code to create and configure a Celery app as well.

NEW

1 reply

Jimmy Royer :spiral_calendar_pad: 1 hour ago For a distributed application, X-Ray combines nodes from all services that process requests with the same trace ID into a single service graph. The first service that the request hits adds a tracing header that is propagated between the front end and services that it calls. For example, Scorekeep runs a web API that calls a microservice (an AWS Lambda function) to generate a random name by using a Node.js library. The X-Ray SDK for Java generates the trace ID and includes it in calls to Lambda. Lambda sends tracing data and passes the trace ID to the function. The X-Ray SDK for Node.js also uses the trace ID to send data. As a result, nodes for the API, the Lambda service, and the Lambda function all appear as separate, but connected, nodes on the trace map. Service graph data is retained for 30 days.

P0NDER0SA commented 1 month ago

https://github.com/cds-snc/notification-api/pull/2248

jimleroyer commented 3 weeks ago

Next steps:

Implement a switch to disable x-ray on demand.
Merge the PR that adds more logging for the Celery custom SDK.

jimleroyer commented 3 weeks ago

Potential fix for the Celery errors, based on an analysis of the enabled log statements: https://github.com/cds-snc/notification-api/pull/2259

The rationale of the fix is contained in the PR description.