Spike: Investigate methods of Redis/Celery-to-Prometheus

[infinitewarp]

Summary

As a cloudigrade sysop, I want monitoring and alerting for Redis availability and queue sizes so that I can react to new problems as they arise.

How do we get the useful data into Prometheus?

Options to consider and research on this issue:

• celery-exporter?
  • @infinitewarp found discussion in StackOverflow suggesting its use or maybe a fork of it?
  • need to verify which fork is active and maintained
  • need to verify if it actually works with Celery/Kombu/etc. latest versions.
• manually sending commands to Redis?
  • where would we do this? part of our health check?
  • would we expose the values we poll into the existing Prometheus metrics exporter endpoint?
  • via Python libraries? If so, which?
  • or via execing out shell commands to redis-cli? hopefully not, but a last resort.
• other ideas?

@abellotti is also asking around to see if/how Koku and friends are handling this.

Expected output: Answers to the above questions with some basic prototype experiments.

Time box: 1 week

[abellotti]

Koku seems to be gathering all celery queue info and metrics directly via the celery app instance, i.e. they are not using the redis library or redis-cli to talk to redis but just have celery do the fetching, i.e. using celery_app.connection, celery_app.control.inspect, and subsequent calls.

See:

• https://github.com/project-koku/koku/blob/main/koku/masu/api/status.py
• https://github.com/project-koku/koku/blob/main/koku/masu/api/running_celery_tasks.py

and the rest of the management api command exposed here:

• https://github.com/project-koku/koku/tree/main/koku/masu/api

Methods of interest:

• get_status: https://github.com/project-koku/koku/blob/3495df3108f3865705e3b5ed2a9982bf4d3172f9/koku/masu/api/status.py#L41
• celery_status: https://github.com/project-koku/koku/blob/3495df3108f3865705e3b5ed2a9982bf4d3172f9/koku/masu/api/status.py#L76
• celery_task_status: https://github.com/project-koku/koku/blob/3495df3108f3865705e3b5ed2a9982bf4d3172f9/koku/masu/api/status.py#L98
• collect_queue_metrics: https://github.com/project-koku/koku/blob/3495df3108f3865705e3b5ed2a9982bf4d3172f9/koku/masu/celery/tasks.py#L385

[abellotti]

It's a beautiful thing:

$ poetry run manage.py shell
...
Python 3.10.1 (main, Dec  6 2021, 22:25:40) [Clang 13.0.0 (clang-1300.0.29.3)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.31.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from config.celery import app

In [2]: conn = app.connection()

In [3]: inspector = app.control.inspect(connection=conn, timeout=10)

In [4]: print(inspector.active())
{'celery@aab-macbook': [{'id': '48d7e5d9-33b0-4c2a-ae5f-16723c30e110', 'name': 'api.clouds.aws.tasks.analyze_log', 'args': [], 'kwargs': {}, 'type': 'api.clouds.aws.tasks.analyze_log', 'hostname': 'celery@aab-macbook', 'time_start': 1644519575.8304832, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'analyze_log', 'priority': 0, 'redelivered': None}, 'worker_pid': 71090}, {'id': '91882c7b-db1d-442f-b36c-7474b7001812', 'name': 'api.tasks.persist_inspection_cluster_results_task', 'args': [], 'kwargs': {}, 'type': 'api.tasks.persist_inspection_cluster_results_task', 'hostname': 'celery@aab-macbook', 'time_start': 1644519575.8046281, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'persist_inspection_cluster_results_task', 'priority': 0, 'redelivered': None}, 'worker_pid': 71098}]}

In [5]: 

There's hope.

[abellotti]

Here's a prototype for one approach where we are incorporating the celery-exporter metrics logic as part of our Cloudigrade prometheus metrics: https://github.com/cloudigrade/cloudigrade/compare/1118_add_celery_metrics, you can see the combined metrics (current python, django) with the celery_ metrics exposed via our current /internal/metrics endpoint:

curl localhost:8000/internal/metrics

# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 2774.0
python_gc_objects_collected_total{generation="1"} 1195.0
python_gc_objects_collected_total{generation="2"} 97.0
# HELP python_gc_objects_uncollectable_total Uncollectable object found during GC
# TYPE python_gc_objects_uncollectable_total counter
python_gc_objects_uncollectable_total{generation="0"} 0.0
python_gc_objects_uncollectable_total{generation="1"} 0.0
python_gc_objects_uncollectable_total{generation="2"} 0.0
# HELP python_gc_collections_total Number of times this generation was collected
# TYPE python_gc_collections_total counter
python_gc_collections_total{generation="0"} 399.0
python_gc_collections_total{generation="1"} 36.0
python_gc_collections_total{generation="2"} 3.0
# HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="3",minor="10",patchlevel="1",version="3.10.1"} 1.0
# HELP django_model_inserts_total Number of insert operations by model.
# TYPE django_model_inserts_total counter
# HELP django_model_updates_total Number of update operations by model.
# TYPE django_model_updates_total counter
# HELP django_model_deletes_total Number of delete operations by model.
# TYPE django_model_deletes_total counter
# HELP django_migrations_unapplied_total Count of unapplied migrations by database connection
# TYPE django_migrations_unapplied_total gauge
django_migrations_unapplied_total{connection="default"} 0.0
# HELP django_migrations_applied_total Count of applied migrations by database connection
# TYPE django_migrations_applied_total gauge
django_migrations_applied_total{connection="default"} 95.0
# HELP django_db_new_connections_total Counter of created connections by database and by vendor.
# TYPE django_db_new_connections_total counter
django_db_new_connections_total{alias="default",vendor="postgresql"} 3.0
# HELP django_db_new_connections_created Counter of created connections by database and by vendor.
# TYPE django_db_new_connections_created gauge
django_db_new_connections_created{alias="default",vendor="postgresql"} 1.645129436949567e+09
# HELP django_db_new_connection_errors_total Counter of connection failures by database and by vendor.
# TYPE django_db_new_connection_errors_total counter
# HELP django_db_execute_total Counter of executed statements by database and by vendor, including bulk executions.
# TYPE django_db_execute_total counter
django_db_execute_total{alias="default",vendor="postgresql"} 7.0
# HELP django_db_execute_created Counter of executed statements by database and by vendor, including bulk executions.
# TYPE django_db_execute_created gauge
django_db_execute_created{alias="default",vendor="postgresql"} 1.6451294369539788e+09
# HELP django_db_execute_many_total Counter of executed statements in bulk operations by database and by vendor.
# TYPE django_db_execute_many_total counter
# HELP django_db_errors_total Counter of execution errors by database, vendor and exception type.
# TYPE django_db_errors_total counter
# HELP django_db_query_duration_seconds Histogram of query duration by database and vendor.
# TYPE django_db_query_duration_seconds histogram
django_db_query_duration_seconds_bucket{alias="default",le="0.005",vendor="postgresql"} 7.0
django_db_query_duration_seconds_bucket{alias="default",le="0.01",vendor="postgresql"} 7.0
django_db_query_duration_seconds_bucket{alias="default",le="0.025",vendor="postgresql"} 7.0
django_db_query_duration_seconds_bucket{alias="default",le="0.05",vendor="postgresql"} 7.0
django_db_query_duration_seconds_bucket{alias="default",le="0.075",vendor="postgresql"} 7.0
django_db_query_duration_seconds_bucket{alias="default",le="0.1",vendor="postgresql"} 7.0
django_db_query_duration_seconds_bucket{alias="default",le="0.25",vendor="postgresql"} 7.0
django_db_query_duration_seconds_bucket{alias="default",le="0.5",vendor="postgresql"} 7.0
django_db_query_duration_seconds_bucket{alias="default",le="0.75",vendor="postgresql"} 7.0
django_db_query_duration_seconds_bucket{alias="default",le="1.0",vendor="postgresql"} 7.0
django_db_query_duration_seconds_bucket{alias="default",le="2.5",vendor="postgresql"} 7.0
django_db_query_duration_seconds_bucket{alias="default",le="5.0",vendor="postgresql"} 7.0
django_db_query_duration_seconds_bucket{alias="default",le="7.5",vendor="postgresql"} 7.0
django_db_query_duration_seconds_bucket{alias="default",le="10.0",vendor="postgresql"} 7.0
django_db_query_duration_seconds_bucket{alias="default",le="+Inf",vendor="postgresql"} 7.0
django_db_query_duration_seconds_count{alias="default",vendor="postgresql"} 7.0
django_db_query_duration_seconds_sum{alias="default",vendor="postgresql"} 0.008750976994633675
# HELP django_db_query_duration_seconds_created Histogram of query duration by database and vendor.
# TYPE django_db_query_duration_seconds_created gauge
django_db_query_duration_seconds_created{alias="default",vendor="postgresql"} 1.645129436954027e+09
# HELP django_http_requests_before_middlewares_total Total count of requests before middlewares run.
# TYPE django_http_requests_before_middlewares_total counter
django_http_requests_before_middlewares_total 1.0
# HELP django_http_requests_before_middlewares_created Total count of requests before middlewares run.
# TYPE django_http_requests_before_middlewares_created gauge
django_http_requests_before_middlewares_created 1.6451294372967381e+09
# HELP django_http_responses_before_middlewares_total Total count of responses before middlewares run.
# TYPE django_http_responses_before_middlewares_total counter
django_http_responses_before_middlewares_total 0.0
# HELP django_http_responses_before_middlewares_created Total count of responses before middlewares run.
# TYPE django_http_responses_before_middlewares_created gauge
django_http_responses_before_middlewares_created 1.645129437296777e+09
# HELP django_http_requests_latency_including_middlewares_seconds Histogram of requests processing time (including middleware processing time).
# TYPE django_http_requests_latency_including_middlewares_seconds histogram
django_http_requests_latency_including_middlewares_seconds_bucket{le="0.005"} 0.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="0.01"} 0.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="0.025"} 0.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="0.05"} 0.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="0.075"} 0.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="0.1"} 0.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="0.25"} 0.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="0.5"} 0.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="0.75"} 0.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="1.0"} 0.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="2.5"} 0.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="5.0"} 0.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="7.5"} 0.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="10.0"} 0.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="+Inf"} 0.0
django_http_requests_latency_including_middlewares_seconds_count 0.0
django_http_requests_latency_including_middlewares_seconds_sum 0.0
# HELP django_http_requests_latency_including_middlewares_seconds_created Histogram of requests processing time (including middleware processing time).
# TYPE django_http_requests_latency_including_middlewares_seconds_created gauge
django_http_requests_latency_including_middlewares_seconds_created 1.645129437296798e+09
# HELP django_http_requests_unknown_latency_including_middlewares_total Count of requests for which the latency was unknown (when computing django_http_requests_latency_including_middlewares_seconds).
# TYPE django_http_requests_unknown_latency_including_middlewares_total counter
django_http_requests_unknown_latency_including_middlewares_total 0.0
# HELP django_http_requests_unknown_latency_including_middlewares_created Count of requests for which the latency was unknown (when computing django_http_requests_latency_including_middlewares_seconds).
# TYPE django_http_requests_unknown_latency_including_middlewares_created gauge
django_http_requests_unknown_latency_including_middlewares_created 1.645129437296845e+09
# HELP django_http_requests_latency_seconds_by_view_method Histogram of request processing time labelled by view.
# TYPE django_http_requests_latency_seconds_by_view_method histogram
# HELP django_http_requests_unknown_latency_total Count of requests for which the latency was unknown.
# TYPE django_http_requests_unknown_latency_total counter
django_http_requests_unknown_latency_total 0.0
# HELP django_http_requests_unknown_latency_created Count of requests for which the latency was unknown.
# TYPE django_http_requests_unknown_latency_created gauge
django_http_requests_unknown_latency_created 1.645129437296887e+09
# HELP django_http_ajax_requests_total Count of AJAX requests.
# TYPE django_http_ajax_requests_total counter
django_http_ajax_requests_total 0.0
# HELP django_http_ajax_requests_created Count of AJAX requests.
# TYPE django_http_ajax_requests_created gauge
django_http_ajax_requests_created 1.645129437296899e+09
# HELP django_http_requests_total_by_method_total Count of requests by method.
# TYPE django_http_requests_total_by_method_total counter
django_http_requests_total_by_method_total{method="GET"} 1.0
# HELP django_http_requests_total_by_method_created Count of requests by method.
# TYPE django_http_requests_total_by_method_created gauge
django_http_requests_total_by_method_created{method="GET"} 1.645135065531822e+09
# HELP django_http_requests_total_by_transport_total Count of requests by transport.
# TYPE django_http_requests_total_by_transport_total counter
django_http_requests_total_by_transport_total{transport="http"} 1.0
# HELP django_http_requests_total_by_transport_created Count of requests by transport.
# TYPE django_http_requests_total_by_transport_created gauge
django_http_requests_total_by_transport_created{transport="http"} 1.645135065531841e+09
# HELP django_http_requests_total_by_view_transport_method_total Count of requests by view, transport, method.
# TYPE django_http_requests_total_by_view_transport_method_total counter
django_http_requests_total_by_view_transport_method_total{method="GET",transport="http",view="prometheus-django-metrics"} 1.0
# HELP django_http_requests_total_by_view_transport_method_created Count of requests by view, transport, method.
# TYPE django_http_requests_total_by_view_transport_method_created gauge
django_http_requests_total_by_view_transport_method_created{method="GET",transport="http",view="prometheus-django-metrics"} 1.645135065532011e+09
# HELP django_http_requests_body_total_bytes Histogram of requests by body size.
# TYPE django_http_requests_body_total_bytes histogram
django_http_requests_body_total_bytes_bucket{le="0.0"} 1.0
django_http_requests_body_total_bytes_bucket{le="1.0"} 1.0
django_http_requests_body_total_bytes_bucket{le="2.0"} 1.0
django_http_requests_body_total_bytes_bucket{le="4.0"} 1.0
django_http_requests_body_total_bytes_bucket{le="8.0"} 1.0
django_http_requests_body_total_bytes_bucket{le="16.0"} 1.0
django_http_requests_body_total_bytes_bucket{le="32.0"} 1.0
django_http_requests_body_total_bytes_bucket{le="64.0"} 1.0
django_http_requests_body_total_bytes_bucket{le="128.0"} 1.0
django_http_requests_body_total_bytes_bucket{le="256.0"} 1.0
django_http_requests_body_total_bytes_bucket{le="512.0"} 1.0
django_http_requests_body_total_bytes_bucket{le="1024.0"} 1.0
django_http_requests_body_total_bytes_bucket{le="2048.0"} 1.0
django_http_requests_body_total_bytes_bucket{le="4096.0"} 1.0
django_http_requests_body_total_bytes_bucket{le="8192.0"} 1.0
django_http_requests_body_total_bytes_bucket{le="16384.0"} 1.0
django_http_requests_body_total_bytes_bucket{le="32768.0"} 1.0
django_http_requests_body_total_bytes_bucket{le="65536.0"} 1.0
django_http_requests_body_total_bytes_bucket{le="131072.0"} 1.0
django_http_requests_body_total_bytes_bucket{le="262144.0"} 1.0
django_http_requests_body_total_bytes_bucket{le="524288.0"} 1.0
django_http_requests_body_total_bytes_bucket{le="1.048576e+06"} 1.0
django_http_requests_body_total_bytes_bucket{le="2.097152e+06"} 1.0
django_http_requests_body_total_bytes_bucket{le="4.194304e+06"} 1.0
django_http_requests_body_total_bytes_bucket{le="8.388608e+06"} 1.0
django_http_requests_body_total_bytes_bucket{le="1.6777216e+07"} 1.0
django_http_requests_body_total_bytes_bucket{le="3.3554432e+07"} 1.0
django_http_requests_body_total_bytes_bucket{le="6.7108864e+07"} 1.0
django_http_requests_body_total_bytes_bucket{le="1.34217728e+08"} 1.0
django_http_requests_body_total_bytes_bucket{le="2.68435456e+08"} 1.0
django_http_requests_body_total_bytes_bucket{le="5.36870912e+08"} 1.0
django_http_requests_body_total_bytes_bucket{le="+Inf"} 1.0
django_http_requests_body_total_bytes_count 1.0
django_http_requests_body_total_bytes_sum 0.0
# HELP django_http_requests_body_total_bytes_created Histogram of requests by body size.
# TYPE django_http_requests_body_total_bytes_created gauge
django_http_requests_body_total_bytes_created 1.6451294372969658e+09
# HELP django_http_responses_total_by_templatename_total Count of responses by template name.
# TYPE django_http_responses_total_by_templatename_total counter
# HELP django_http_responses_total_by_status_total Count of responses by status.
# TYPE django_http_responses_total_by_status_total counter
# HELP django_http_responses_total_by_status_view_method_total Count of responses by status, view, method.
# TYPE django_http_responses_total_by_status_view_method_total counter
# HELP django_http_responses_body_total_bytes Histogram of responses by body size.
# TYPE django_http_responses_body_total_bytes histogram
django_http_responses_body_total_bytes_bucket{le="0.0"} 0.0
django_http_responses_body_total_bytes_bucket{le="1.0"} 0.0
django_http_responses_body_total_bytes_bucket{le="2.0"} 0.0
django_http_responses_body_total_bytes_bucket{le="4.0"} 0.0
django_http_responses_body_total_bytes_bucket{le="8.0"} 0.0
django_http_responses_body_total_bytes_bucket{le="16.0"} 0.0
django_http_responses_body_total_bytes_bucket{le="32.0"} 0.0
django_http_responses_body_total_bytes_bucket{le="64.0"} 0.0
django_http_responses_body_total_bytes_bucket{le="128.0"} 0.0
django_http_responses_body_total_bytes_bucket{le="256.0"} 0.0
django_http_responses_body_total_bytes_bucket{le="512.0"} 0.0
django_http_responses_body_total_bytes_bucket{le="1024.0"} 0.0
django_http_responses_body_total_bytes_bucket{le="2048.0"} 0.0
django_http_responses_body_total_bytes_bucket{le="4096.0"} 0.0
django_http_responses_body_total_bytes_bucket{le="8192.0"} 0.0
django_http_responses_body_total_bytes_bucket{le="16384.0"} 0.0
django_http_responses_body_total_bytes_bucket{le="32768.0"} 0.0
django_http_responses_body_total_bytes_bucket{le="65536.0"} 0.0
django_http_responses_body_total_bytes_bucket{le="131072.0"} 0.0
django_http_responses_body_total_bytes_bucket{le="262144.0"} 0.0
django_http_responses_body_total_bytes_bucket{le="524288.0"} 0.0
django_http_responses_body_total_bytes_bucket{le="1.048576e+06"} 0.0
django_http_responses_body_total_bytes_bucket{le="2.097152e+06"} 0.0
django_http_responses_body_total_bytes_bucket{le="4.194304e+06"} 0.0
django_http_responses_body_total_bytes_bucket{le="8.388608e+06"} 0.0
django_http_responses_body_total_bytes_bucket{le="1.6777216e+07"} 0.0
django_http_responses_body_total_bytes_bucket{le="3.3554432e+07"} 0.0
django_http_responses_body_total_bytes_bucket{le="6.7108864e+07"} 0.0
django_http_responses_body_total_bytes_bucket{le="1.34217728e+08"} 0.0
django_http_responses_body_total_bytes_bucket{le="2.68435456e+08"} 0.0
django_http_responses_body_total_bytes_bucket{le="5.36870912e+08"} 0.0
django_http_responses_body_total_bytes_bucket{le="+Inf"} 0.0
django_http_responses_body_total_bytes_count 0.0
django_http_responses_body_total_bytes_sum 0.0
# HELP django_http_responses_body_total_bytes_created Histogram of responses by body size.
# TYPE django_http_responses_body_total_bytes_created gauge
django_http_responses_body_total_bytes_created 1.645129437297098e+09
# HELP django_http_responses_total_by_charset_total Count of responses by charset.
# TYPE django_http_responses_total_by_charset_total counter
# HELP django_http_responses_streaming_total Count of streaming responses.
# TYPE django_http_responses_streaming_total counter
django_http_responses_streaming_total 0.0
# HELP django_http_responses_streaming_created Count of streaming responses.
# TYPE django_http_responses_streaming_created gauge
django_http_responses_streaming_created 1.645129437297185e+09
# HELP django_http_exceptions_total_by_type_total Count of exceptions by object type.
# TYPE django_http_exceptions_total_by_type_total counter
# HELP django_http_exceptions_total_by_view_total Count of exceptions by view.
# TYPE django_http_exceptions_total_by_view_total counter
# HELP celery_task_sent_total Sent when a task message is published.
# TYPE celery_task_sent_total counter
# HELP celery_task_received_total Sent when the worker receives a task.
# TYPE celery_task_received_total counter
celery_task_received_total{hostname="celery@aab-macbook",name="api.clouds.aws.tasks.analyze_log"} 188.0
celery_task_received_total{hostname="celery@aab-macbook",name="api.tasks.persist_inspection_cluster_results_task"} 93.0
celery_task_received_total{hostname="celery@aab-macbook",name="api.tasks.inspect_pending_images"} 6.0
celery_task_received_total{hostname="celery@aab-macbook",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 1.0
celery_task_received_total{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 1.0
celery_task_received_total{hostname="celery@aab-macbook",name="api.tasks.recalculate_runs_for_cloud_account_id"} 12.0
celery_task_received_total{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 1.0
celery_task_received_total{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 4.0
celery_task_received_total{hostname="celery@aab-macbook",name="api.tasks.delete_inactive_users"} 1.0
celery_task_received_total{hostname="celery@aab-macbook",name="api.tasks.delete_cloud_accounts_not_in_sources"} 1.0
celery_task_received_total{hostname="celery@aab-macbook",name="api.tasks.delete_orphaned_cloud_accounts"} 1.0
# HELP celery_task_received_created Sent when the worker receives a task.
# TYPE celery_task_received_created gauge
celery_task_received_created{hostname="celery@aab-macbook",name="api.clouds.aws.tasks.analyze_log"} 1.6451294453228219e+09
celery_task_received_created{hostname="celery@aab-macbook",name="api.tasks.persist_inspection_cluster_results_task"} 1.645129485567157e+09
celery_task_received_created{hostname="celery@aab-macbook",name="api.tasks.inspect_pending_images"} 1.6451300765126379e+09
celery_task_received_created{hostname="celery@aab-macbook",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 1.645131876095266e+09
celery_task_received_created{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 1.645131876112152e+09
celery_task_received_created{hostname="celery@aab-macbook",name="api.tasks.recalculate_runs_for_cloud_account_id"} 1.6451318768339682e+09
celery_task_received_created{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 1.645131876950148e+09
celery_task_received_created{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 1.645131877193078e+09
celery_task_received_created{hostname="celery@aab-macbook",name="api.tasks.delete_inactive_users"} 1.645132544047013e+09
celery_task_received_created{hostname="celery@aab-macbook",name="api.tasks.delete_cloud_accounts_not_in_sources"} 1.645132544090635e+09
celery_task_received_created{hostname="celery@aab-macbook",name="api.tasks.delete_orphaned_cloud_accounts"} 1.645132544102581e+09
# HELP celery_task_started_total Sent just before the worker executes the task.
# TYPE celery_task_started_total counter
celery_task_started_total{hostname="celery@aab-macbook",name="api.clouds.aws.tasks.analyze_log"} 188.0
celery_task_started_total{hostname="celery@aab-macbook",name="api.tasks.persist_inspection_cluster_results_task"} 93.0
celery_task_started_total{hostname="celery@aab-macbook",name="api.tasks.inspect_pending_images"} 6.0
celery_task_started_total{hostname="celery@aab-macbook",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 1.0
celery_task_started_total{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 1.0
celery_task_started_total{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 1.0
celery_task_started_total{hostname="celery@aab-macbook",name="api.tasks.recalculate_runs_for_cloud_account_id"} 12.0
celery_task_started_total{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 4.0
celery_task_started_total{hostname="celery@aab-macbook",name="api.tasks.delete_inactive_users"} 1.0
celery_task_started_total{hostname="celery@aab-macbook",name="api.tasks.delete_cloud_accounts_not_in_sources"} 1.0
celery_task_started_total{hostname="celery@aab-macbook",name="api.tasks.delete_orphaned_cloud_accounts"} 1.0
# HELP celery_task_started_created Sent just before the worker executes the task.
# TYPE celery_task_started_created gauge
celery_task_started_created{hostname="celery@aab-macbook",name="api.clouds.aws.tasks.analyze_log"} 1.6451294453238342e+09
celery_task_started_created{hostname="celery@aab-macbook",name="api.tasks.persist_inspection_cluster_results_task"} 1.645129485568856e+09
celery_task_started_created{hostname="celery@aab-macbook",name="api.tasks.inspect_pending_images"} 1.6451300765174332e+09
celery_task_started_created{hostname="celery@aab-macbook",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 1.645131876098634e+09
celery_task_started_created{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 1.6451318761452e+09
celery_task_started_created{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 1.6451318769617689e+09
celery_task_started_created{hostname="celery@aab-macbook",name="api.tasks.recalculate_runs_for_cloud_account_id"} 1.6451318769626298e+09
celery_task_started_created{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 1.645131877213779e+09
celery_task_started_created{hostname="celery@aab-macbook",name="api.tasks.delete_inactive_users"} 1.645132544048255e+09
celery_task_started_created{hostname="celery@aab-macbook",name="api.tasks.delete_cloud_accounts_not_in_sources"} 1.6451325440923479e+09
celery_task_started_created{hostname="celery@aab-macbook",name="api.tasks.delete_orphaned_cloud_accounts"} 1.645132544104063e+09
# HELP celery_task_succeeded_total Sent if the task executed successfully.
# TYPE celery_task_succeeded_total counter
celery_task_succeeded_total{hostname="celery@aab-macbook",name="api.clouds.aws.tasks.analyze_log"} 187.0
celery_task_succeeded_total{hostname="celery@aab-macbook",name="api.tasks.persist_inspection_cluster_results_task"} 93.0
celery_task_succeeded_total{hostname="celery@aab-macbook",name="api.tasks.inspect_pending_images"} 6.0
celery_task_succeeded_total{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 1.0
celery_task_succeeded_total{hostname="celery@aab-macbook",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 1.0
celery_task_succeeded_total{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 1.0
celery_task_succeeded_total{hostname="celery@aab-macbook",name="api.tasks.recalculate_runs_for_cloud_account_id"} 12.0
celery_task_succeeded_total{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 4.0
celery_task_succeeded_total{hostname="celery@aab-macbook",name="api.tasks.delete_inactive_users"} 1.0
celery_task_succeeded_total{hostname="celery@aab-macbook",name="api.tasks.delete_orphaned_cloud_accounts"} 1.0
celery_task_succeeded_total{hostname="celery@aab-macbook",name="api.tasks.delete_cloud_accounts_not_in_sources"} 1.0
# HELP celery_task_succeeded_created Sent if the task executed successfully.
# TYPE celery_task_succeeded_created gauge
celery_task_succeeded_created{hostname="celery@aab-macbook",name="api.clouds.aws.tasks.analyze_log"} 1.6451294558855271e+09
celery_task_succeeded_created{hostname="celery@aab-macbook",name="api.tasks.persist_inspection_cluster_results_task"} 1.645129496029406e+09
celery_task_succeeded_created{hostname="celery@aab-macbook",name="api.tasks.inspect_pending_images"} 1.6451300765777469e+09
celery_task_succeeded_created{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 1.645131876950497e+09
celery_task_succeeded_created{hostname="celery@aab-macbook",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 1.6451318769535491e+09
celery_task_succeeded_created{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 1.645131877193237e+09
celery_task_succeeded_created{hostname="celery@aab-macbook",name="api.tasks.recalculate_runs_for_cloud_account_id"} 1.645131877194161e+09
celery_task_succeeded_created{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 1.64513187742946e+09
celery_task_succeeded_created{hostname="celery@aab-macbook",name="api.tasks.delete_inactive_users"} 1.6451325440608e+09
celery_task_succeeded_created{hostname="celery@aab-macbook",name="api.tasks.delete_orphaned_cloud_accounts"} 1.645132544159999e+09
celery_task_succeeded_created{hostname="celery@aab-macbook",name="api.tasks.delete_cloud_accounts_not_in_sources"} 1.645132544162857e+09
# HELP celery_task_failed_total Sent if the execution of the task failed.
# TYPE celery_task_failed_total counter
# HELP celery_task_rejected_total The task was rejected by the worker, possibly to be re-queued or moved to a dead letter queue.
# TYPE celery_task_rejected_total counter
# HELP celery_task_revoked_total Sent if the task has been revoked.
# TYPE celery_task_revoked_total counter
# HELP celery_task_retried_total Sent if the task failed, but will be retried in the future.
# TYPE celery_task_retried_total counter
# HELP celery_worker_up Indicates if a worker has recently sent a heartbeat.
# TYPE celery_worker_up gauge
celery_worker_up{hostname="celery@aab-macbook"} 1.0
# HELP celery_worker_tasks_active The number of tasks the worker is currently processing
# TYPE celery_worker_tasks_active gauge
celery_worker_tasks_active{hostname="celery@aab-macbook"} 1.0
# HELP celery_task_runtime Histogram of task runtime measurements.
# TYPE celery_task_runtime histogram
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.005",name="api.clouds.aws.tasks.analyze_log"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.01",name="api.clouds.aws.tasks.analyze_log"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.025",name="api.clouds.aws.tasks.analyze_log"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.05",name="api.clouds.aws.tasks.analyze_log"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.075",name="api.clouds.aws.tasks.analyze_log"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.1",name="api.clouds.aws.tasks.analyze_log"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.25",name="api.clouds.aws.tasks.analyze_log"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.5",name="api.clouds.aws.tasks.analyze_log"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.75",name="api.clouds.aws.tasks.analyze_log"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="1.0",name="api.clouds.aws.tasks.analyze_log"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="2.5",name="api.clouds.aws.tasks.analyze_log"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="5.0",name="api.clouds.aws.tasks.analyze_log"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="7.5",name="api.clouds.aws.tasks.analyze_log"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="10.0",name="api.clouds.aws.tasks.analyze_log"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="+Inf",name="api.clouds.aws.tasks.analyze_log"} 187.0
celery_task_runtime_count{hostname="celery@aab-macbook",name="api.clouds.aws.tasks.analyze_log"} 187.0
celery_task_runtime_sum{hostname="celery@aab-macbook",name="api.clouds.aws.tasks.analyze_log"} 1919.006366727408
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.005",name="api.tasks.persist_inspection_cluster_results_task"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.01",name="api.tasks.persist_inspection_cluster_results_task"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.025",name="api.tasks.persist_inspection_cluster_results_task"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.05",name="api.tasks.persist_inspection_cluster_results_task"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.075",name="api.tasks.persist_inspection_cluster_results_task"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.1",name="api.tasks.persist_inspection_cluster_results_task"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.25",name="api.tasks.persist_inspection_cluster_results_task"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.5",name="api.tasks.persist_inspection_cluster_results_task"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.75",name="api.tasks.persist_inspection_cluster_results_task"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="1.0",name="api.tasks.persist_inspection_cluster_results_task"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="2.5",name="api.tasks.persist_inspection_cluster_results_task"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="5.0",name="api.tasks.persist_inspection_cluster_results_task"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="7.5",name="api.tasks.persist_inspection_cluster_results_task"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="10.0",name="api.tasks.persist_inspection_cluster_results_task"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="+Inf",name="api.tasks.persist_inspection_cluster_results_task"} 93.0
celery_task_runtime_count{hostname="celery@aab-macbook",name="api.tasks.persist_inspection_cluster_results_task"} 93.0
celery_task_runtime_sum{hostname="celery@aab-macbook",name="api.tasks.persist_inspection_cluster_results_task"} 966.4502739268355
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.005",name="api.tasks.inspect_pending_images"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.01",name="api.tasks.inspect_pending_images"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.025",name="api.tasks.inspect_pending_images"} 2.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.05",name="api.tasks.inspect_pending_images"} 3.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.075",name="api.tasks.inspect_pending_images"} 4.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.1",name="api.tasks.inspect_pending_images"} 5.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.25",name="api.tasks.inspect_pending_images"} 6.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.5",name="api.tasks.inspect_pending_images"} 6.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.75",name="api.tasks.inspect_pending_images"} 6.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="1.0",name="api.tasks.inspect_pending_images"} 6.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="2.5",name="api.tasks.inspect_pending_images"} 6.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="5.0",name="api.tasks.inspect_pending_images"} 6.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="7.5",name="api.tasks.inspect_pending_images"} 6.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="10.0",name="api.tasks.inspect_pending_images"} 6.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="+Inf",name="api.tasks.inspect_pending_images"} 6.0
celery_task_runtime_count{hostname="celery@aab-macbook",name="api.tasks.inspect_pending_images"} 6.0
celery_task_runtime_sum{hostname="celery@aab-macbook",name="api.tasks.inspect_pending_images"} 0.33198631275445223
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.005",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.01",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.025",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.05",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.075",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.1",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.25",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.5",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.75",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="1.0",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="2.5",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="5.0",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="7.5",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="10.0",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="+Inf",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 1.0
celery_task_runtime_count{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 1.0
celery_task_runtime_sum{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 0.24318843707442284
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.005",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.01",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.025",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.05",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.075",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.1",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.25",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.5",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.75",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="1.0",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="2.5",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="5.0",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="7.5",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="10.0",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="+Inf",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 1.0
celery_task_runtime_count{hostname="celery@aab-macbook",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 1.0
celery_task_runtime_sum{hostname="celery@aab-macbook",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 0.26870434917509556
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.005",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.01",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.025",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.05",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.075",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.1",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.25",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.5",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.75",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="1.0",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="2.5",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="5.0",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="7.5",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="10.0",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="+Inf",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 1.0
celery_task_runtime_count{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 1.0
celery_task_runtime_sum{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 0.09012714307755232
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.005",name="api.tasks.recalculate_runs_for_cloud_account_id"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.01",name="api.tasks.recalculate_runs_for_cloud_account_id"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.025",name="api.tasks.recalculate_runs_for_cloud_account_id"} 7.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.05",name="api.tasks.recalculate_runs_for_cloud_account_id"} 11.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.075",name="api.tasks.recalculate_runs_for_cloud_account_id"} 11.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.1",name="api.tasks.recalculate_runs_for_cloud_account_id"} 12.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.25",name="api.tasks.recalculate_runs_for_cloud_account_id"} 12.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.5",name="api.tasks.recalculate_runs_for_cloud_account_id"} 12.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.75",name="api.tasks.recalculate_runs_for_cloud_account_id"} 12.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="1.0",name="api.tasks.recalculate_runs_for_cloud_account_id"} 12.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="2.5",name="api.tasks.recalculate_runs_for_cloud_account_id"} 12.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="5.0",name="api.tasks.recalculate_runs_for_cloud_account_id"} 12.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="7.5",name="api.tasks.recalculate_runs_for_cloud_account_id"} 12.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="10.0",name="api.tasks.recalculate_runs_for_cloud_account_id"} 12.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="+Inf",name="api.tasks.recalculate_runs_for_cloud_account_id"} 12.0
celery_task_runtime_count{hostname="celery@aab-macbook",name="api.tasks.recalculate_runs_for_cloud_account_id"} 12.0
celery_task_runtime_sum{hostname="celery@aab-macbook",name="api.tasks.recalculate_runs_for_cloud_account_id"} 0.377012986689806
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.005",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.01",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.025",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.05",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 2.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.075",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 4.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.1",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 4.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.25",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 4.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.5",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 4.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.75",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 4.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="1.0",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 4.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="2.5",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 4.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="5.0",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 4.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="7.5",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 4.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="10.0",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 4.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="+Inf",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 4.0
celery_task_runtime_count{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 4.0
celery_task_runtime_sum{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 0.19502167543396354
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.005",name="api.tasks.delete_inactive_users"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.01",name="api.tasks.delete_inactive_users"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.025",name="api.tasks.delete_inactive_users"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.05",name="api.tasks.delete_inactive_users"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.075",name="api.tasks.delete_inactive_users"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.1",name="api.tasks.delete_inactive_users"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.25",name="api.tasks.delete_inactive_users"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.5",name="api.tasks.delete_inactive_users"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.75",name="api.tasks.delete_inactive_users"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="1.0",name="api.tasks.delete_inactive_users"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="2.5",name="api.tasks.delete_inactive_users"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="5.0",name="api.tasks.delete_inactive_users"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="7.5",name="api.tasks.delete_inactive_users"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="10.0",name="api.tasks.delete_inactive_users"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="+Inf",name="api.tasks.delete_inactive_users"} 1.0
celery_task_runtime_count{hostname="celery@aab-macbook",name="api.tasks.delete_inactive_users"} 1.0
celery_task_runtime_sum{hostname="celery@aab-macbook",name="api.tasks.delete_inactive_users"} 0.012029502540826797
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.005",name="api.tasks.delete_orphaned_cloud_accounts"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.01",name="api.tasks.delete_orphaned_cloud_accounts"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.025",name="api.tasks.delete_orphaned_cloud_accounts"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.05",name="api.tasks.delete_orphaned_cloud_accounts"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.075",name="api.tasks.delete_orphaned_cloud_accounts"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.1",name="api.tasks.delete_orphaned_cloud_accounts"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.25",name="api.tasks.delete_orphaned_cloud_accounts"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.5",name="api.tasks.delete_orphaned_cloud_accounts"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.75",name="api.tasks.delete_orphaned_cloud_accounts"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="1.0",name="api.tasks.delete_orphaned_cloud_accounts"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="2.5",name="api.tasks.delete_orphaned_cloud_accounts"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="5.0",name="api.tasks.delete_orphaned_cloud_accounts"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="7.5",name="api.tasks.delete_orphaned_cloud_accounts"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="10.0",name="api.tasks.delete_orphaned_cloud_accounts"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="+Inf",name="api.tasks.delete_orphaned_cloud_accounts"} 1.0
celery_task_runtime_count{hostname="celery@aab-macbook",name="api.tasks.delete_orphaned_cloud_accounts"} 1.0
celery_task_runtime_sum{hostname="celery@aab-macbook",name="api.tasks.delete_orphaned_cloud_accounts"} 0.0551412426866591
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.005",name="api.tasks.delete_cloud_accounts_not_in_sources"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.01",name="api.tasks.delete_cloud_accounts_not_in_sources"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.025",name="api.tasks.delete_cloud_accounts_not_in_sources"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.05",name="api.tasks.delete_cloud_accounts_not_in_sources"} 0.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.075",name="api.tasks.delete_cloud_accounts_not_in_sources"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.1",name="api.tasks.delete_cloud_accounts_not_in_sources"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.25",name="api.tasks.delete_cloud_accounts_not_in_sources"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.5",name="api.tasks.delete_cloud_accounts_not_in_sources"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="0.75",name="api.tasks.delete_cloud_accounts_not_in_sources"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="1.0",name="api.tasks.delete_cloud_accounts_not_in_sources"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="2.5",name="api.tasks.delete_cloud_accounts_not_in_sources"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="5.0",name="api.tasks.delete_cloud_accounts_not_in_sources"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="7.5",name="api.tasks.delete_cloud_accounts_not_in_sources"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="10.0",name="api.tasks.delete_cloud_accounts_not_in_sources"} 1.0
celery_task_runtime_bucket{hostname="celery@aab-macbook",le="+Inf",name="api.tasks.delete_cloud_accounts_not_in_sources"} 1.0
celery_task_runtime_count{hostname="celery@aab-macbook",name="api.tasks.delete_cloud_accounts_not_in_sources"} 1.0
celery_task_runtime_sum{hostname="celery@aab-macbook",name="api.tasks.delete_cloud_accounts_not_in_sources"} 0.07062886003404856
# HELP celery_task_runtime_created Histogram of task runtime measurements.
# TYPE celery_task_runtime_created gauge
celery_task_runtime_created{hostname="celery@aab-macbook",name="api.clouds.aws.tasks.analyze_log"} 1.645129455885616e+09
celery_task_runtime_created{hostname="celery@aab-macbook",name="api.tasks.persist_inspection_cluster_results_task"} 1.6451294960295029e+09
celery_task_runtime_created{hostname="celery@aab-macbook",name="api.tasks.inspect_pending_images"} 1.645130076577836e+09
celery_task_runtime_created{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_all_users"} 1.6451318769517682e+09
celery_task_runtime_created{hostname="celery@aab-macbook",name="api.tasks.recalculate_runs_for_all_cloud_accounts"} 1.645131876953662e+09
celery_task_runtime_created{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_user_id"} 1.645131877193297e+09
celery_task_runtime_created{hostname="celery@aab-macbook",name="api.tasks.recalculate_runs_for_cloud_account_id"} 1.645131877194226e+09
celery_task_runtime_created{hostname="celery@aab-macbook",name="api.tasks.recalculate_concurrent_usage_for_user_id_on_date"} 1.645131877429517e+09
celery_task_runtime_created{hostname="celery@aab-macbook",name="api.tasks.delete_inactive_users"} 1.6451325440609028e+09
celery_task_runtime_created{hostname="celery@aab-macbook",name="api.tasks.delete_orphaned_cloud_accounts"} 1.645132544160079e+09
celery_task_runtime_created{hostname="celery@aab-macbook",name="api.tasks.delete_cloud_accounts_not_in_sources"} 1.645132544162931e+09

[abellotti]

Metrics gathered by https://github.com/cloudigrade/cloudigrade/pull/1128:

Metrics

Name	Description	Type
celery_task_sent_total	Sent when a task message is published.	Counter
celery_task_received_total	Sent when the worker receives a task.	Counter
celery_task_started_total	Sent just before the worker executes the task.	Counter
celery_task_succeeded_total	Sent if the task executed successfully.	Counter
celery_task_failed_total	Sent if the execution of the task failed.	Counter
celery_task_rejected_total	The task was rejected by the worker, possibly to be re-queued or moved to a dead letter queue.	Counter
celery_task_revoked_total	Sent if the task has been revoked.	Counter
celery_task_retried_total	Sent if the task failed, but will be retried in the future.	Counter
celery_worker_up	Indicates if a worker has recently sent a heartbeat.	Gauge
celery_worker_tasks_active	The number of tasks the worker is currently processing	Gauge
celery_task_runtime	Histogram of runtime measurements for each task	Histogram

[abellotti]

Long journey but this is now complete, with the following Repo, PRs & MRs:

• https://github.com/danihodovic/celery-exporter/pull/89
• https://github.com/danihodovic/celery-exporter/pull/90
• https://github.com/cloudigrade/celery-exporter-container
• https://github.com/cloudigrade/cloudigrade/pull/1152
• https://gitlab.cee.redhat.com/service/app-interface/-/merge_requests/36709
• https://gitlab.cee.redhat.com/service/app-interface/-/merge_requests/36914
• https://gitlab.cee.redhat.com/service/app-interface/-/merge_requests/37156
• https://gitlab.cee.redhat.com/service/app-interface/-/merge_requests/37161

With all the ducks above in order, the django metrics from cloudigrade-api and celery metrics from cloudigrade-metrics are properly scraped by Prometheus and can be introspected and queried via PromQL:

Examples:

• https://prometheus.crcs02ue1.devshift.net/graph?g0.expr=django_http_responses_body_total_bytes_count%7Bnamespace%3D%22cloudigrade-stage%22%7D&g0.tab=1&g0.stacked=0&g0.show_exemplars=0&g0.range_input=1h
• https://prometheus.crcs02ue1.devshift.net/graph?g0.expr=celery_task_runtime_count%7Bnamespace%3D%22cloudigrade-stage%22%7D&g0.tab=1&g0.stacked=0&g0.show_exemplars=0&g0.range_input=1h