Measuring in Celery task

[michel-cf]

For a project of mine, I want to measure inside Celery tasks. But it seems that measured data from the Celery workers does not get reported. (It actually does get printed if I trigger that from within the worker)

A minimal example: celerypyformance.py

from pyformance import MetricsRegistry
from pyformance.reporters import ConsoleReporter

from celery import Celery

app = Celery('celerypyformance', backend='redis://localhost', broker='redis://localhost')

registry = MetricsRegistry()
reporter = ConsoleReporter(registry, reporting_interval=2)
reporter.start()

registry.counter('started').inc()

@app.task
def task(argument):
    registry.counter('task.%s' % argument).inc()
    print('task.%s' % argument)

test.py

from celerypyformance import task

if __name__ == '__main__':
    task.delay("delayed")

It keeps printing only the start action, never the counter from within the task:

== 2016-11-29 10:49:20 ===================================
started:
               count = 1

[2016-11-29 11:49:21,986: WARNING/Worker-3] task.delayed
== 2016-11-29 10:49:22 ===================================
started:
               count = 1

To run: Install Celery using pip, install redis. Start the redis server. Run celery -A celerypyformance worker and leave this running while executing the next command. Execute python test.py to execute the celery task.

Is there something I've forgotten or is there a small fix that can resolve this issue? Thanks in advance

[omergertel]

If I understand correctly, you are asking about getting measurements from a different process (the celery worker). I don't think I have anything like this implemented. I'm pretty sure that's an overkill for something as simple as Pyformance, and also might cause measurements to be expensive and cause delays in the software running.