search

scoutapp / scout_apm_python

ScoutAPM Python Agent. Supports Django, Flask, and many other frameworks.
https://scoutapm.com
MIT License
69 stars 21 forks source link

Track a chain of Celery tasks as a single request #373

Open jacotri77 opened 4 years ago

jacotri77 commented 4 years ago

The use case:

Trying to manually instrument the start and stop of a background celery task that is a chain of Celery tasks and we want to instrument the full chain.

Task-One starts, does some work and completes, handing over the rest of the work to Task-Two. Task-Two does some work, completes and passes off the next bit of work to Task-Three and so on.

The details are handed over to the next task as arguments. Initializing the scout_apm.api.BackgroundTask object in Task-One and sending the details as an argument doesn't work as the variable passed to Task-Two is None.

adamchainz commented 4 years ago

So as I understand, the customer is trying to record a single trace for the chain of tasks by:

I don't think we can easily support this case.

Firstly, we don't support serializing and restoring the state. It would be very hard to do reliably. The current state is naturally related to the current function call stack, with layers added per level of function call. Restoring that state in essentially a different part of the program whilst continuing to close all the open spans would be very hard. Also hard would be that the code may have changed arbitrarily in between. Essentially this is a continuation for Scout state.

Secondly, I'm not sure the data would much sense. Celery makes no guarantee about when the tasks in a chain run, apart from "after each other." It can't make stronger guarantees because its broker may have a large queue, or it may be shut down between tasks, etc. Therefore the slow traces Scout would record for such a chain would mostly be those where there was randomly more delay between the tasks, and not hot spots which are open for optimization.

The built-in Celery integration traces tasks and adds the task ID and parent task ID to the trace context. This should be a useful starting point for checking on the tasks in the chain. I'm sure there is some extra context we could be adding for chains or other workflows, so I'll keep the issue open to look into that.

Thanks,

Adam

jacotri77 commented 4 years ago

Thanks, @adamchainz for your response. I will follow up.