Fix references in queue_metrics ETS table

[gerhard]

The core metrics collector reads properties incorrectly from the queue_metrics ETS table, by prepending queue_ to properties that don't have this: e.g. queue_consumers instead of consumers.

Review all queue_metrics properties and fix.

[gerhard]

Discovered in https://github.com/rabbitmq/rabbitmq-prometheus/issues/9#issuecomment-558233464

[gerhard]

@hrobertson this is currently going through the pipeline, I expect the fix to be out in v3.8.2 most likely, but definitely in v3.8.3.

Meanwhile, if you want to test this, provided you have Docker running locally, you can try running make overview in the root of this repo (make sure your master branch is up-to-date), then navigate to http://localhost:15693/metrics. This is what I get:

curl -s http://localhost:15693/metrics | grep '^rabbitmq_queue_'

rabbitmq_queue_messages_published_total{channel="<0.1044.0>",queue_vhost="/",queue="basic-get",exchange_vhost="/",exchange="direct"} 20
rabbitmq_queue_messages_published_total{channel="<0.1033.0>",queue_vhost="/",queue="greedy-consumer",exchange_vhost="/",exchange="direct"} 22253
rabbitmq_queue_index_read_ops_total 0
rabbitmq_queue_index_write_ops_total 0
rabbitmq_queue_index_journal_write_ops_total 0
rabbitmq_queue_messages_ready{vhost="/",queue="greedy-consumer"} 0
rabbitmq_queue_messages_ready{vhost="/",queue="basic-get"} 0
rabbitmq_queue_messages_unacked{vhost="/",queue="greedy-consumer"} 21
rabbitmq_queue_messages_unacked{vhost="/",queue="basic-get"} 0
rabbitmq_queue_messages{vhost="/",queue="greedy-consumer"} 21
rabbitmq_queue_messages{vhost="/",queue="basic-get"} 0
rabbitmq_queue_process_reductions_total{vhost="/",queue="greedy-consumer"} 13582375
rabbitmq_queue_process_reductions_total{vhost="/",queue="basic-get"} 79604
rabbitmq_queue_consumers{vhost="/",queue="greedy-consumer"} 1
rabbitmq_queue_consumers{vhost="/",queue="basic-get"} 0
rabbitmq_queue_consumer_utilisation{vhost="/",queue="greedy-consumer"} 1.0
rabbitmq_queue_consumer_utilisation{vhost="/",queue="basic-get"} NaN
rabbitmq_queue_process_memory_bytes{vhost="/",queue="greedy-consumer"} 4721052
rabbitmq_queue_process_memory_bytes{vhost="/",queue="basic-get"} 55552
rabbitmq_queue_messages_ram{vhost="/",queue="greedy-consumer"} 21
rabbitmq_queue_messages_ram{vhost="/",queue="basic-get"} 0
rabbitmq_queue_messages_ram_bytes{vhost="/",queue="greedy-consumer"} 252
rabbitmq_queue_messages_ram_bytes{vhost="/",queue="basic-get"} 0
rabbitmq_queue_messages_ready_ram{vhost="/",queue="greedy-consumer"} 0
rabbitmq_queue_messages_ready_ram{vhost="/",queue="basic-get"} 0
rabbitmq_queue_messages_unacked_ram{vhost="/",queue="greedy-consumer"} 21
rabbitmq_queue_messages_unacked_ram{vhost="/",queue="basic-get"} 0
rabbitmq_queue_messages_persistent{vhost="/",queue="greedy-consumer"} 0
rabbitmq_queue_messages_persistent{vhost="/",queue="basic-get"} 0
rabbitmq_queue_messages_persistent_bytes{vhost="/",queue="greedy-consumer"} 0
rabbitmq_queue_messages_persistent_bytes{vhost="/",queue="basic-get"} 0
rabbitmq_queue_messages_bytes{vhost="/",queue="greedy-consumer"} 252
rabbitmq_queue_messages_bytes{vhost="/",queue="basic-get"} 0
rabbitmq_queue_messages_ready_bytes{vhost="/",queue="greedy-consumer"} 0
rabbitmq_queue_messages_ready_bytes{vhost="/",queue="basic-get"} 0
rabbitmq_queue_messages_unacked_bytes{vhost="/",queue="greedy-consumer"} 252
rabbitmq_queue_messages_unacked_bytes{vhost="/",queue="basic-get"} 0
rabbitmq_queue_messages_paged_out{vhost="/",queue="greedy-consumer"} 0
rabbitmq_queue_messages_paged_out{vhost="/",queue="basic-get"} 0
rabbitmq_queue_messages_paged_out_bytes{vhost="/",queue="greedy-consumer"} 0
rabbitmq_queue_messages_paged_out_bytes{vhost="/",queue="basic-get"} 0
rabbitmq_queue_disk_reads_total{vhost="/",queue="greedy-consumer"} 0
rabbitmq_queue_disk_reads_total{vhost="/",queue="basic-get"} 0
rabbitmq_queue_disk_writes_total{vhost="/",queue="greedy-consumer"} 0
rabbitmq_queue_disk_writes_total{vhost="/",queue="basic-get"} 0

[hrobertson]

Hi @gerhard . We've just deployed 3.8.3 and our metrics don't have labels as in your output above:

curl -s http://node1:15692/metrics | grep '^rabbitmq_queue_'
rabbitmq_queue_messages_published_total 374
rabbitmq_queue_index_read_ops_total 0
rabbitmq_queue_index_write_ops_total 31
rabbitmq_queue_index_journal_write_ops_total 5870
rabbitmq_queue_messages_ready 0
rabbitmq_queue_messages_unacked 4
rabbitmq_queue_messages 4
rabbitmq_queue_process_reductions_total 6797927
rabbitmq_queue_consumers 610
rabbitmq_queue_consumer_utilisation 2.0
rabbitmq_queue_process_memory_bytes 1713596
rabbitmq_queue_messages_ram 4
rabbitmq_queue_messages_ram_bytes 294
rabbitmq_queue_messages_ready_ram 0
rabbitmq_queue_messages_unacked_ram 4
rabbitmq_queue_messages_persistent 4
rabbitmq_queue_messages_bytes 294
rabbitmq_queue_messages_ready_bytes 0
rabbitmq_queue_messages_unacked_bytes 294
rabbitmq_queue_messages_paged_out 0
rabbitmq_queue_messages_paged_out_bytes 0
rabbitmq_queue_disk_reads_total 0
rabbitmq_queue_disk_writes_total 1949

I've scoured the docs but can't see any mention of this being a configuration option etc. Is there something we need to do to enable labels or did they get removed since November? Thanks

Edit: Nevermind. Found https://github.com/rabbitmq/rabbitmq-prometheus/pull/28 Where is this documented?

[michaelklishin]

This plugin has a documentation guide. It is open source and we would consider contributions. Documenting #28 slipped through the cracks when 3.8.3 shipped.

[michaelklishin]

Per-object metric collection now has a brief documentation section.