PolicyStat / jobtastic

Make your user-responsive long-running Celery jobs totally awesomer.
http://policystat.github.com/jobtastic/
MIT License
644 stars 61 forks source link

Add easy memory logging #14

Closed winhamwr closed 11 years ago

winhamwr commented 11 years ago

One of the common debugging issues with Celery tasks are that they appear to "leak memory." Usually, this has nothing to do with Celery and everything to do with the code written in the task itself, but it's very difficult to pinpoint what tasks under what situations actually cause the problem.

For tasks that increase memory usage by more than a threshold amount during the executation of the task, spit out a log message right before returning the result.

This can all be done with the psutil module. Here is an example for getting RAM usage of current process in python.

Before calculate_result is called, grab the RAM usage. Then, before we return the result, get RAM usage again. If it's increased by more than the threshold amount, spit out a WARNING message with:

jlward commented 11 years ago

Looks good once my two comments are addressed.