OvalMoney / celery-exporter

Prometheus metrics exporter for Celery
MIT License
142 stars 37 forks source link

Fix throwing of an exception in an exception handler #30

Closed itsx closed 4 years ago

itsx commented 4 years ago

Function setup_metrics was throwing an exception in the exception handler of _monitor method of TaskThread:

https://github.com/OvalMoney/celery-exporter/blob/497c8d8bb39d4a808805d1f3340fd84eaff6d0b1/celery_exporter/monitor.py#L64

Exception:

File "/usr/local/lib/python3.6/site-packages/celery_exporter/monitor.py", line 122, in setup_metrics
    for name, labels, cnt in metric.samples:
ValueError: too many values to unpack (expected 3)

Problem was that metrics.samples iterable contains 5 items instead of three ('timestamp' and 'exemplar' were not anticipated): https://github.com/OvalMoney/celery-exporter/blob/497c8d8bb39d4a808805d1f3340fd84eaff6d0b1/celery_exporter/monitor.py#L122

Major problem: This exception thrown in an exception handler caused hanging of the task thread after lost of the connection to a broker. Normally exception from the lost connection is handled in an exception handler and recovery loop: https://github.com/OvalMoney/celery-exporter/blob/497c8d8bb39d4a808805d1f3340fd84eaff6d0b1/celery_exporter/monitor.py#L53 starts new connection. Because of the second exception in the handler, this has never happened.

After lost of a connection to a broker, other metrics from other threads were collected without problems.

Fixes #29