Replace global labels with build_info & identity_info metrics

[gerhard]

This started in the context of prometheus/docs#1414, specifically https://github.com/prometheus/docs/pull/1414#issuecomment-520505757

Rather than labelling all metrics with the same label, we are introducing 2 new metrics: rabbitmq_build_info & rabbitmq_identity_info.

I suspect that we may want to revert deadtrickster/prometheus.erl#91 when we agree that the proposed alternative is better.

We are yet to see through changes in Grafana dashboards. I am most interested in how the updated queries will look like and, more importantly, if we will have the same panels as we do now. More commits to follow shortly, wanted to get this out the door first.

In summary, this commit changes:

# TYPE erlang_mnesia_held_locks gauge
# HELP erlang_mnesia_held_locks Number of held locks.
erlang_mnesia_held_locks{node="rabbit@920f1e3272af",cluster="rabbit@920f1e3272af",rabbitmq_version="3.8.0-alpha.806",erlang_version="22.0.7"} 0
# TYPE erlang_mnesia_lock_queue gauge
# HELP erlang_mnesia_lock_queue Number of transactions waiting for a lock.
erlang_mnesia_lock_queue{node="rabbit@920f1e3272af",cluster="rabbit@920f1e3272af",rabbitmq_version="3.8.0-alpha.806",erlang_version="22.0.7"} 0
...

To this:

# TYPE erlang_mnesia_held_locks gauge
# HELP erlang_mnesia_held_locks Number of held locks.
erlang_mnesia_held_locks 0
# TYPE erlang_mnesia_lock_queue gauge
# HELP erlang_mnesia_lock_queue Number of transactions waiting for a lock.
erlang_mnesia_lock_queue 0
...
# TYPE rabbitmq_build_info untyped
# HELP rabbitmq_build_info RabbitMQ & Erlang/OTP version info
rabbitmq_build_info{rabbitmq_version="3.8.0-alpha.809",prometheus_plugin_version="3.8.0-alpha.809-2019.08.15",prometheus_client_version="4.4.0",erlang_version="22.0.7"} 1
# TYPE rabbitmq_identity_info untyped
# HELP rabbitmq_identity_info Node & cluster identity info
rabbitmq_identity_info{node="rabbit@bc7aeb0c2564",cluster="rabbit@bc7aeb0c2564"} 1
...

[#167846096]

[gerhard]

Is the above output the alternative that you had in mind to global labels @brian-brazil ?

[gerhard]

Are there any code improvements that you would recommend @dcorbacho ?

[gerhard]

I am struggling to convert the following query into something that works when the cluster label is missing from the metric:

rabbitmq_memory_used_limit_bytes{cluster="$cluster"} - rabbitmq_memory_used_bytes{cluster="$cluster"}

This is what the metrics look like after the latest changes:

rabbitmq_memory_used_limit_bytes 805306368
rabbitmq_memory_used_bytes 252805120

This is what they looked like before:

rabbitmq_memory_used_limit_bytes{node="rabbit@rmq0",cluster="rabbitmq-overview"} 805306368
rabbitmq_memory_used_bytes{node="rabbit@rmq0",cluster="rabbitmq-overview"} 252805120

This is the only metric that now has the cluster info now:

rabbitmq_identity_info{node="rabbit@rmq0",cluster="rabbitmq-overview"} 1

[brian-brazil]

rabbitmq_memory_used_limit_bytes * on (instance, job) group_left(cluster) rabbitmq_identity_info

In reality the cluster would be coming from a target label.

[brian-brazil]

That looks about right, thanks.

I suspect that we may want to revert deadtrickster/prometheus.erl#91 when we agree that the proposed alternative is better.

Yes, this is not something a client library should offer.

[gerhard]

In reality the cluster would be coming from a target label.

A RabbitMQ node's internal knowledge of its name and which cluster it is part of trumps the target label.

The target label may be wrong, or maybe the target label is right, but the node is not what the target label says it is.

To prevent a whole category of support escalations, the only node & cluster label that can be trusted is what the RabbitMQ node outputs.

[brian-brazil]

The target label may be wrong, or maybe the target label is right, but the node is not what the target label says it is.

In which case most likely RabbitMQ is misconfigured, as SD tends to come from the source of truth of how things are meant to be.

[gerhard]

rabbitmq_memory_used_limit_bytes * on (instance, job) group_left(cluster) rabbitmq_identity_info

Based on the above, I ended up with this:

(rabbitmq_memory_used_limit_bytes * on(instance) group_left(cluster, node) rabbitmq_identity_info{cluster="$cluster"}) -
(rabbitmq_memory_used_bytes * on(instance) group_left(cluster, node) rabbitmq_identity_info{cluster="$cluster"})

The graph looks right, but the PromQL looks complex. Any improvements that can be done?

This is what we had with global labels:

rabbitmq_memory_used_limit_bytes{cluster="$cluster"} -
rabbitmq_memory_used_bytes{cluster="$cluster"}

[brian-brazil]

Any improvements that can be done?

That's as simple as it gets.

[gerhard]

In which case most likely RabbitMQ is misconfigured, as SD tends to come from the source of truth of how things are meant to be.

The system managing RabbitMQ may believe that everything is correct, but given the complex behaviour of a distributed stateful system, correctness is a moving target. Given how dynamic a node's lifecycle is, the only sane approach is to make it the source of truth, and consider everything else a middleman, with a partial understanding of what is going on at any point in time.

[gerhard]

How does the new output look like @brian-brazil ?

# TYPE rabbitmq_channel_consumers gauge
# HELP rabbitmq_channel_consumers Consumers on a channel
rabbitmq_channel_consumers{channel="<0.929.0>"} 1
rabbitmq_channel_consumers{channel="<0.988.0>"} 0
rabbitmq_channel_consumers{channel="<0.1020.0>"} 0
rabbitmq_channel_consumers{channel="<0.940.0>"} 0
rabbitmq_channel_consumers{channel="<0.977.0>"} 0
# TYPE rabbitmq_channel_messages_unacked gauge
# HELP rabbitmq_channel_messages_unacked Delivered but not yet acknowledged messages
rabbitmq_channel_messages_unacked{channel="<0.929.0>"} 213
rabbitmq_channel_messages_unacked{channel="<0.988.0>"} 0
rabbitmq_channel_messages_unacked{channel="<0.1020.0>"} 0
rabbitmq_channel_messages_unacked{channel="<0.940.0>"} 0
rabbitmq_channel_messages_unacked{channel="<0.977.0>"} 0
# TYPE rabbitmq_channel_messages_unconfirmed gauge
# HELP rabbitmq_channel_messages_unconfirmed Published but not yet confirmed messages
rabbitmq_channel_messages_unconfirmed{channel="<0.929.0>"} 0
rabbitmq_channel_messages_unconfirmed{channel="<0.988.0>"} 0
rabbitmq_channel_messages_unconfirmed{channel="<0.1020.0>"} 0
rabbitmq_channel_messages_unconfirmed{channel="<0.940.0>"} 0
rabbitmq_channel_messages_unconfirmed{channel="<0.977.0>"} 0
# TYPE rabbitmq_channel_messages_uncommitted gauge
# HELP rabbitmq_channel_messages_uncommitted Messages received in a transaction but not yet committed
rabbitmq_channel_messages_uncommitted{channel="<0.929.0>"} 0
rabbitmq_channel_messages_uncommitted{channel="<0.988.0>"} 0
rabbitmq_channel_messages_uncommitted{channel="<0.1020.0>"} 0
rabbitmq_channel_messages_uncommitted{channel="<0.940.0>"} 0
rabbitmq_channel_messages_uncommitted{channel="<0.977.0>"} 0
# TYPE rabbitmq_channel_acks_uncommitted gauge
# HELP rabbitmq_channel_acks_uncommitted Message acknowledgements in a transaction not yet committed
rabbitmq_channel_acks_uncommitted{channel="<0.929.0>"} 0
rabbitmq_channel_acks_uncommitted{channel="<0.988.0>"} 0
rabbitmq_channel_acks_uncommitted{channel="<0.1020.0>"} 0
rabbitmq_channel_acks_uncommitted{channel="<0.940.0>"} 0
rabbitmq_channel_acks_uncommitted{channel="<0.977.0>"} 0
# TYPE rabbitmq_consumer_prefetch gauge
# HELP rabbitmq_consumer_prefetch Limit of unacknowledged messages for each consumer
rabbitmq_consumer_prefetch{channel="<0.929.0>"} 2000
rabbitmq_consumer_prefetch{channel="<0.988.0>"} 0
rabbitmq_consumer_prefetch{channel="<0.1020.0>"} 0
rabbitmq_consumer_prefetch{channel="<0.940.0>"} 0
rabbitmq_consumer_prefetch{channel="<0.977.0>"} 0
# TYPE rabbitmq_channel_prefetch gauge
# HELP rabbitmq_channel_prefetch Total limit of unacknowledged messages for all consumers on a channel
rabbitmq_channel_prefetch{channel="<0.929.0>"} 0
rabbitmq_channel_prefetch{channel="<0.988.0>"} 0
rabbitmq_channel_prefetch{channel="<0.1020.0>"} 0
rabbitmq_channel_prefetch{channel="<0.940.0>"} 0
rabbitmq_channel_prefetch{channel="<0.977.0>"} 0
# TYPE rabbitmq_channel_messages_published gauge
# HELP rabbitmq_channel_messages_published Messages published into an exchange on a channel
rabbitmq_channel_messages_published{channel="<0.988.0>",vhost="/",exchange="direct"} 256
rabbitmq_channel_messages_published{channel="<0.940.0>",vhost="/",exchange="direct"} 270131
# TYPE rabbitmq_channel_messages_confirmed gauge
# HELP rabbitmq_channel_messages_confirmed Messages published into an exchange and confirmed on the channel
rabbitmq_channel_messages_confirmed{channel="<0.988.0>",vhost="/",exchange="direct"} 0
rabbitmq_channel_messages_confirmed{channel="<0.940.0>",vhost="/",exchange="direct"} 0
# TYPE rabbitmq_channel_messages_unroutable_returned gauge
# HELP rabbitmq_channel_messages_unroutable_returned Messages published as mandatory into an exchange and returned to the publisher as unroutable
rabbitmq_channel_messages_unroutable_returned{channel="<0.988.0>",vhost="/",exchange="direct"} 0
rabbitmq_channel_messages_unroutable_returned{channel="<0.940.0>",vhost="/",exchange="direct"} 0
# TYPE rabbitmq_channel_messages_unroutable_dropped gauge
# HELP rabbitmq_channel_messages_unroutable_dropped Messages published as non-mandatory into an exchange and dropped as unroutable
rabbitmq_channel_messages_unroutable_dropped{channel="<0.988.0>",vhost="/",exchange="direct"} 0
rabbitmq_channel_messages_unroutable_dropped{channel="<0.940.0>",vhost="/",exchange="direct"} 0
# TYPE rabbitmq_channel_process_reductions_total counter
# HELP rabbitmq_channel_process_reductions_total Total number of channel process reductions
rabbitmq_channel_process_reductions_total{channel="<0.929.0>"} 57225960
rabbitmq_channel_process_reductions_total{channel="<0.988.0>"} 195071
rabbitmq_channel_process_reductions_total{channel="<0.1020.0>"} 249874
rabbitmq_channel_process_reductions_total{channel="<0.940.0>"} 75659778
rabbitmq_channel_process_reductions_total{channel="<0.977.0>"} 273808
# TYPE rabbitmq_channel_get_ack_total counter
# HELP rabbitmq_channel_get_ack_total Total number of messages fetched with basic.get in manual acknowledgement mode
rabbitmq_channel_get_ack_total{channel="<0.977.0>",vhost="/",queue="basic-get"} 175
rabbitmq_channel_get_ack_total{channel="<0.1020.0>",vhost="/",queue="basic-get"} 0
rabbitmq_channel_get_ack_total{channel="<0.929.0>",vhost="/",queue="greedy-consumer"} 0
# TYPE rabbitmq_channel_get_total counter
# HELP rabbitmq_channel_get_total Total number of messages fetched with basic.get in automatic acknowledgement mode
rabbitmq_channel_get_total{channel="<0.977.0>",vhost="/",queue="basic-get"} 0
rabbitmq_channel_get_total{channel="<0.1020.0>",vhost="/",queue="basic-get"} 81
rabbitmq_channel_get_total{channel="<0.929.0>",vhost="/",queue="greedy-consumer"} 0
# TYPE rabbitmq_channel_messages_delivered_ack_total counter
# HELP rabbitmq_channel_messages_delivered_ack_total Total number of messages delivered to consumers in manual acknowledgement mode
rabbitmq_channel_messages_delivered_ack_total{channel="<0.977.0>",vhost="/",queue="basic-get"} 0
rabbitmq_channel_messages_delivered_ack_total{channel="<0.1020.0>",vhost="/",queue="basic-get"} 0
rabbitmq_channel_messages_delivered_ack_total{channel="<0.929.0>",vhost="/",queue="greedy-consumer"} 270131
# TYPE rabbitmq_channel_messages_delivered_total counter
# HELP rabbitmq_channel_messages_delivered_total Total number of messages delivered to consumers in automatic acknowledgement mode
rabbitmq_channel_messages_delivered_total{channel="<0.977.0>",vhost="/",queue="basic-get"} 0
rabbitmq_channel_messages_delivered_total{channel="<0.1020.0>",vhost="/",queue="basic-get"} 0
rabbitmq_channel_messages_delivered_total{channel="<0.929.0>",vhost="/",queue="greedy-consumer"} 0
# TYPE rabbitmq_channel_messages_redelivered_total counter
# HELP rabbitmq_channel_messages_redelivered_total Total number of messages redelivered to consumers
rabbitmq_channel_messages_redelivered_total{channel="<0.977.0>",vhost="/",queue="basic-get"} 0
rabbitmq_channel_messages_redelivered_total{channel="<0.1020.0>",vhost="/",queue="basic-get"} 0
rabbitmq_channel_messages_redelivered_total{channel="<0.929.0>",vhost="/",queue="greedy-consumer"} 0
# TYPE rabbitmq_channel_messages_acked_total counter
# HELP rabbitmq_channel_messages_acked_total Total number of messages acknowledged by consumers
rabbitmq_channel_messages_acked_total{channel="<0.977.0>",vhost="/",queue="basic-get"} 175
rabbitmq_channel_messages_acked_total{channel="<0.1020.0>",vhost="/",queue="basic-get"} 0
rabbitmq_channel_messages_acked_total{channel="<0.929.0>",vhost="/",queue="greedy-consumer"} 270131
# TYPE rabbitmq_channel_get_empty_total counter
# HELP rabbitmq_channel_get_empty_total Total number of times basic.get operations fetched no message
rabbitmq_channel_get_empty_total{channel="<0.977.0>",vhost="/",queue="basic-get"} 897
rabbitmq_channel_get_empty_total{channel="<0.1020.0>",vhost="/",queue="basic-get"} 991
rabbitmq_channel_get_empty_total{channel="<0.929.0>",vhost="/",queue="greedy-consumer"} 0
# TYPE rabbitmq_connection_opened_total counter
# HELP rabbitmq_connection_opened_total Total number of connections opened
rabbitmq_connection_opened_total 11
# TYPE rabbitmq_connection_closed_total counter
# HELP rabbitmq_connection_closed_total Total number of connections closed or terminated
rabbitmq_connection_closed_total 3
# TYPE rabbitmq_channel_opened_total counter
# HELP rabbitmq_channel_opened_total Total number of channels opened
rabbitmq_channel_opened_total 8
# TYPE rabbitmq_channel_closed_total counter
# HELP rabbitmq_channel_closed_total Total number of channels closed
rabbitmq_channel_closed_total 3
# TYPE rabbitmq_queue_declared_total counter
# HELP rabbitmq_queue_declared_total Total number of queues declared
rabbitmq_queue_declared_total 3
# TYPE rabbitmq_queue_created_total counter
# HELP rabbitmq_queue_created_total Total number of queues created
rabbitmq_queue_created_total 2
# TYPE rabbitmq_queue_deleted_total counter
# HELP rabbitmq_queue_deleted_total Total number of queues deleted
rabbitmq_queue_deleted_total 0
# TYPE rabbitmq_connection_incoming_bytes_total counter
# HELP rabbitmq_connection_incoming_bytes_total Total number of bytes received on a connection
rabbitmq_connection_incoming_bytes_total{channel="<0.980.0>"} 19505
rabbitmq_connection_incoming_bytes_total{channel="<0.998.0>"} 1902
rabbitmq_connection_incoming_bytes_total{channel="<0.909.0>"} 5670472
rabbitmq_connection_incoming_bytes_total{channel="<0.958.0>"} 31105
rabbitmq_connection_incoming_bytes_total{channel="<0.951.0>"} 1902
rabbitmq_connection_incoming_bytes_total{channel="<0.932.0>"} 21610921
rabbitmq_connection_incoming_bytes_total{channel="<0.1005.0>"} 27430
rabbitmq_connection_incoming_bytes_total{channel="<0.902.0>"} 1902
# TYPE rabbitmq_connection_outgoing_bytes_total counter
# HELP rabbitmq_connection_outgoing_bytes_total Total number of bytes sent on a connection
rabbitmq_connection_outgoing_bytes_total{channel="<0.980.0>"} 1994
rabbitmq_connection_outgoing_bytes_total{channel="<0.998.0>"} 1966
rabbitmq_connection_outgoing_bytes_total{channel="<0.909.0>"} 31876050
rabbitmq_connection_outgoing_bytes_total{channel="<0.958.0>"} 27009
rabbitmq_connection_outgoing_bytes_total{channel="<0.951.0>"} 1966
rabbitmq_connection_outgoing_bytes_total{channel="<0.932.0>"} 1994
rabbitmq_connection_outgoing_bytes_total{channel="<0.1005.0>"} 20335
rabbitmq_connection_outgoing_bytes_total{channel="<0.902.0>"} 1966
# TYPE rabbitmq_connection_process_reductions_total counter
# HELP rabbitmq_connection_process_reductions_total Total number of connection process reductions
rabbitmq_connection_process_reductions_total{channel="<0.980.0>"} 538550
rabbitmq_connection_process_reductions_total{channel="<0.998.0>"} 505329
rabbitmq_connection_process_reductions_total{channel="<0.909.0>"} 32494906
rabbitmq_connection_process_reductions_total{channel="<0.958.0>"} 910468
rabbitmq_connection_process_reductions_total{channel="<0.951.0>"} 505475
rabbitmq_connection_process_reductions_total{channel="<0.932.0>"} 52898599
rabbitmq_connection_process_reductions_total{channel="<0.1005.0>"} 889203
rabbitmq_connection_process_reductions_total{channel="<0.902.0>"} 506143
# TYPE rabbitmq_connection_incoming_packets_total counter
# HELP rabbitmq_connection_incoming_packets_total Total number of packets received on a connection
rabbitmq_connection_incoming_packets_total{channel="<0.980.0>"} 257
rabbitmq_connection_incoming_packets_total{channel="<0.998.0>"} 181
rabbitmq_connection_incoming_packets_total{channel="<0.909.0>"} 265316
rabbitmq_connection_incoming_packets_total{channel="<0.958.0>"} 1256
rabbitmq_connection_incoming_packets_total{channel="<0.951.0>"} 181
rabbitmq_connection_incoming_packets_total{channel="<0.932.0>"} 267748
rabbitmq_connection_incoming_packets_total{channel="<0.1005.0>"} 1081
rabbitmq_connection_incoming_packets_total{channel="<0.902.0>"} 181
# TYPE rabbitmq_connection_outgoing_packets_total counter
# HELP rabbitmq_connection_outgoing_packets_total Total number of packets sent on a connection
rabbitmq_connection_outgoing_packets_total{channel="<0.980.0>"} 183
rabbitmq_connection_outgoing_packets_total{channel="<0.998.0>"} 181
rabbitmq_connection_outgoing_packets_total{channel="<0.909.0>"} 267402
rabbitmq_connection_outgoing_packets_total{channel="<0.958.0>"} 1082
rabbitmq_connection_outgoing_packets_total{channel="<0.951.0>"} 181
rabbitmq_connection_outgoing_packets_total{channel="<0.932.0>"} 183
rabbitmq_connection_outgoing_packets_total{channel="<0.1005.0>"} 1082
rabbitmq_connection_outgoing_packets_total{channel="<0.902.0>"} 181
# TYPE rabbitmq_connection_pending_packets gauge
# HELP rabbitmq_connection_pending_packets Number of packets waiting to be sent on a connection
rabbitmq_connection_pending_packets{channel="<0.980.0>"} 0
rabbitmq_connection_pending_packets{channel="<0.998.0>"} 0
rabbitmq_connection_pending_packets{channel="<0.909.0>"} 0
rabbitmq_connection_pending_packets{channel="<0.958.0>"} 0
rabbitmq_connection_pending_packets{channel="<0.951.0>"} 0
rabbitmq_connection_pending_packets{channel="<0.932.0>"} 0
rabbitmq_connection_pending_packets{channel="<0.1005.0>"} 0
rabbitmq_connection_pending_packets{channel="<0.902.0>"} 0
# TYPE rabbitmq_connection_channels gauge
# HELP rabbitmq_connection_channels Channels on a connection
rabbitmq_connection_channels{channel="<0.980.0>"} 1
rabbitmq_connection_channels{channel="<0.998.0>"} 0
rabbitmq_connection_channels{channel="<0.909.0>"} 1
rabbitmq_connection_channels{channel="<0.958.0>"} 1
rabbitmq_connection_channels{channel="<0.951.0>"} 0
rabbitmq_connection_channels{channel="<0.932.0>"} 1
rabbitmq_connection_channels{channel="<0.1005.0>"} 1
rabbitmq_connection_channels{channel="<0.902.0>"} 0
# TYPE rabbitmq_queue_messages_published_total counter
# HELP rabbitmq_queue_messages_published_total Total number of messages published to queues
rabbitmq_queue_messages_published_total{channel="<0.988.0>",queue_vhost="/",queue="basic-get",exchange_vhost="/",exchange="direct"} 256
rabbitmq_queue_messages_published_total{channel="<0.940.0>",queue_vhost="/",queue="greedy-consumer",exchange_vhost="/",exchange="direct"} 270131
# TYPE rabbitmq_file_descriptors_open gauge
# HELP rabbitmq_file_descriptors_open Open file descriptors
rabbitmq_file_descriptors_open 45
# TYPE rabbitmq_tcp_sockets_open gauge
# HELP rabbitmq_tcp_sockets_open Open TCP sockets
rabbitmq_tcp_sockets_open 8
# TYPE rabbitmq_memory_used_bytes gauge
# HELP rabbitmq_memory_used_bytes Memory used in bytes
rabbitmq_memory_used_bytes 238411776
# TYPE rabbitmq_disk_space_available_bytes gauge
# HELP rabbitmq_disk_space_available_bytes Disk space available in bytes
rabbitmq_disk_space_available_bytes 45361704960
# TYPE rabbitmq_erlang_processes_used gauge
# HELP rabbitmq_erlang_processes_used Erlang processes used
rabbitmq_erlang_processes_used 620
# TYPE rabbitmq_erlang_gc_runs_total counter
# HELP rabbitmq_erlang_gc_runs_total Total number of Erlang garbage collector runs
rabbitmq_erlang_gc_runs_total 1626600
# TYPE rabbitmq_erlang_gc_reclaimed_bytes_total counter
# HELP rabbitmq_erlang_gc_reclaimed_bytes_total Total number of bytes of memory reclaimed by Erlang garbage collector
rabbitmq_erlang_gc_reclaimed_bytes_total 31296629456
# TYPE rabbitmq_erlang_scheduler_context_switches_total counter
# HELP rabbitmq_erlang_scheduler_context_switches_total Total number of Erlang scheduler context switches
rabbitmq_erlang_scheduler_context_switches_total 8431639
# TYPE rabbitmq_file_descriptors_open_limit gauge
# HELP rabbitmq_file_descriptors_open_limit Open file descriptors limit
rabbitmq_file_descriptors_open_limit 2000
# TYPE rabbitmq_tcp_sockets_open_limit gauge
# HELP rabbitmq_tcp_sockets_open_limit Open TCP sockets limit
rabbitmq_tcp_sockets_open_limit 1708
# TYPE rabbitmq_memory_used_limit_bytes gauge
# HELP rabbitmq_memory_used_limit_bytes Memory high watermark in bytes
rabbitmq_memory_used_limit_bytes 805306368
# TYPE rabbitmq_disk_space_available_limit_bytes gauge
# HELP rabbitmq_disk_space_available_limit_bytes Free disk space low watermark in bytes
rabbitmq_disk_space_available_limit_bytes 50000000
# TYPE rabbitmq_erlang_processes_limit gauge
# HELP rabbitmq_erlang_processes_limit Erlang processes limit
rabbitmq_erlang_processes_limit 1048576
# TYPE rabbitmq_uptime_milliseconds gauge
# HELP rabbitmq_uptime_milliseconds Node uptime in milliseconds
rabbitmq_uptime_milliseconds 5359871
# TYPE rabbitmq_erlang_scheduler_run_queue gauge
# HELP rabbitmq_erlang_scheduler_run_queue Erlang scheduler run queue
rabbitmq_erlang_scheduler_run_queue 1
# TYPE rabbitmq_erlang_net_ticktime_seconds gauge
# HELP rabbitmq_erlang_net_ticktime_seconds Inter-node heartbeat interval in seconds
rabbitmq_erlang_net_ticktime_seconds 60
# TYPE rabbitmq_io_read_ops_total counter
# HELP rabbitmq_io_read_ops_total Total number of I/O read operations
rabbitmq_io_read_ops_total 4121
# TYPE rabbitmq_io_read_bytes_total counter
# HELP rabbitmq_io_read_bytes_total Total number of I/O bytes read
rabbitmq_io_read_bytes_total 13853938369
# TYPE rabbitmq_io_read_time_microseconds_total counter
# HELP rabbitmq_io_read_time_microseconds_total Total I/O read time
rabbitmq_io_read_time_microseconds_total 7416931
# TYPE rabbitmq_io_write_ops_total counter
# HELP rabbitmq_io_write_ops_total Total number of I/O write operations
rabbitmq_io_write_ops_total 269359
# TYPE rabbitmq_io_write_bytes_total counter
# HELP rabbitmq_io_write_bytes_total Total number of I/O bytes written
rabbitmq_io_write_bytes_total 13891234623
# TYPE rabbitmq_io_write_time_microseconds_total gauge
# HELP rabbitmq_io_write_time_microseconds_total Total I/O write time
rabbitmq_io_write_time_microseconds_total 96147036
# TYPE rabbitmq_io_sync_ops_total counter
# HELP rabbitmq_io_sync_ops_total Total number of I/O sync operations
rabbitmq_io_sync_ops_total 269357
# TYPE rabbitmq_io_sync_time_microseconds_total counter
# HELP rabbitmq_io_sync_time_microseconds_total Total I/O sync time
rabbitmq_io_sync_time_microseconds_total 340374916
# TYPE rabbitmq_io_seek_ops_total counter
# HELP rabbitmq_io_seek_ops_total Total number of I/O seek operations
rabbitmq_io_seek_ops_total 2687
# TYPE rabbitmq_io_seek_time_microseconds_total counter
# HELP rabbitmq_io_seek_time_microseconds_total Total I/O seek time
rabbitmq_io_seek_time_microseconds_total 473849
# TYPE rabbitmq_io_open_attempt_ops_total counter
# HELP rabbitmq_io_open_attempt_ops_total Total number of file open attempts
rabbitmq_io_open_attempt_ops_total 1627929
# TYPE rabbitmq_io_open_attempt_time_microseconds_total counter
# HELP rabbitmq_io_open_attempt_time_microseconds_total Total file open attempts time
rabbitmq_io_open_attempt_time_microseconds_total 20349405
# TYPE rabbitmq_io_reopen_ops_total counter
# HELP rabbitmq_io_reopen_ops_total Total number of times files have been reopened
rabbitmq_io_reopen_ops_total 0
# TYPE rabbitmq_schema_db_ram_tx_total counter
# HELP rabbitmq_schema_db_ram_tx_total Total number of Schema DB memory transactions
rabbitmq_schema_db_ram_tx_total 21
# TYPE rabbitmq_schema_db_disk_tx_total counter
# HELP rabbitmq_schema_db_disk_tx_total Total number of Schema DB disk transactions
rabbitmq_schema_db_disk_tx_total 20
# TYPE rabbitmq_msg_store_read_total counter
# HELP rabbitmq_msg_store_read_total Total number of Message Store read operations
rabbitmq_msg_store_read_total 0
# TYPE rabbitmq_msg_store_write_total counter
# HELP rabbitmq_msg_store_write_total Total number of Message Store write operations
rabbitmq_msg_store_write_total 270013
# TYPE rabbitmq_queue_index_read_ops_total counter
# HELP rabbitmq_queue_index_read_ops_total Total number of Queue Index read operations
rabbitmq_queue_index_read_ops_total 0
# TYPE rabbitmq_queue_index_write_ops_total counter
# HELP rabbitmq_queue_index_write_ops_total Total number of Queue Index write operations
rabbitmq_queue_index_write_ops_total 1599
# TYPE rabbitmq_queue_index_journal_write_ops_total counter
# HELP rabbitmq_queue_index_journal_write_ops_total Total number of Queue Index Journal write operations
rabbitmq_queue_index_journal_write_ops_total 809741
# TYPE rabbitmq_raft_term gauge
# HELP rabbitmq_raft_term Raft member term
# TYPE rabbitmq_raft_log_snapshot_index gauge
# HELP rabbitmq_raft_log_snapshot_index Raft log snapshot index
# TYPE rabbitmq_raft_log_last_applied_index gauge
# HELP rabbitmq_raft_log_last_applied_index Raft log last applied index
# TYPE rabbitmq_raft_log_commit_index gauge
# HELP rabbitmq_raft_log_commit_index Raft log commit index
# TYPE rabbitmq_raft_log_last_written_index gauge
# HELP rabbitmq_raft_log_last_written_index Raft log last written index
# TYPE rabbitmq_raft_entry_commit_latency gauge
# HELP rabbitmq_raft_entry_commit_latency Time taken for an entry to be committed
# TYPE rabbitmq_queue_messages_ready gauge
# HELP rabbitmq_queue_messages_ready Messages ready to be delivered to consumers
rabbitmq_queue_messages_ready{vhost="/",queue="basic-get"} 0
rabbitmq_queue_messages_ready{vhost="/",queue="greedy-consumer"} 0
# TYPE rabbitmq_queue_messages_unacked gauge
# HELP rabbitmq_queue_messages_unacked Messages delivered to consumers but not yet acknowledged
rabbitmq_queue_messages_unacked{vhost="/",queue="basic-get"} 0
rabbitmq_queue_messages_unacked{vhost="/",queue="greedy-consumer"} 433
# TYPE rabbitmq_queue_messages gauge
# HELP rabbitmq_queue_messages Sum of ready and unacknowledged messages - total queue depth
rabbitmq_queue_messages{vhost="/",queue="basic-get"} 0
rabbitmq_queue_messages{vhost="/",queue="greedy-consumer"} 433
# TYPE rabbitmq_queue_process_reductions_total counter
# HELP rabbitmq_queue_process_reductions_total Total number of queue process reductions
rabbitmq_queue_process_reductions_total{vhost="/",queue="basic-get"} 930406
rabbitmq_queue_process_reductions_total{vhost="/",queue="greedy-consumer"} 165324711
# TYPE rabbitmq_queue_consumers gauge
# HELP rabbitmq_queue_consumers Consumers on a queue
# TYPE rabbitmq_erlang_process_memory_bytes gauge
# HELP rabbitmq_erlang_process_memory_bytes Memory in bytes used by the Erlang queue process
# TYPE rabbitmq_queue_messages_bytes gauge
# HELP rabbitmq_queue_messages_bytes Size in bytes of ready and unacknowledged messages
# TYPE rabbitmq_queue_messages_ram gauge
# HELP rabbitmq_queue_messages_ram Ready and unacknowledged messages stored in memory
# TYPE rabbitmq_queue_messages_ready_ram gauge
# HELP rabbitmq_queue_messages_ready_ram Ready messages stored in memory
# TYPE rabbitmq_queue_messages_ready_bytes gauge
# HELP rabbitmq_queue_messages_ready_bytes Size in bytes of ready messages
# TYPE rabbitmq_queue_messages_unacked_ram gauge
# HELP rabbitmq_queue_messages_unacked_ram Unacknowledged messages stored in memory
# TYPE rabbitmq_queue_messages_unacked_bytes gauge
# HELP rabbitmq_queue_messages_unacked_bytes Size in bytes of all unacknowledged messages
# TYPE rabbitmq_queue_messages_persistent gauge
# HELP rabbitmq_queue_messages_persistent Persistent messages
# TYPE rabbitmq_queue_messages_persistent_bytes gauge
# HELP rabbitmq_queue_messages_persistent_bytes Size in bytes of persistent messages
# TYPE rabbitmq_queue_messages_paged_out gauge
# HELP rabbitmq_queue_messages_paged_out Messages paged out to disk
# TYPE rabbitmq_queue_messages_paged_out_bytes gauge
# HELP rabbitmq_queue_messages_paged_out_bytes Size in bytes of messages paged out to disk
# TYPE rabbitmq_queue_disk_reads_total counter
# HELP rabbitmq_queue_disk_reads_total Total number of times queue read messages from disk
rabbitmq_queue_disk_reads_total{vhost="/",queue="basic-get"} 0
rabbitmq_queue_disk_reads_total{vhost="/",queue="greedy-consumer"} 0
# TYPE rabbitmq_queue_disk_writes_total counter
# HELP rabbitmq_queue_disk_writes_total Total number of times queue wrote messages to disk
rabbitmq_queue_disk_writes_total{vhost="/",queue="basic-get"} 0
rabbitmq_queue_disk_writes_total{vhost="/",queue="greedy-consumer"} 0
# TYPE rabbitmq_connections gauge
# HELP rabbitmq_connections Connections currently running
rabbitmq_connections 8
# TYPE rabbitmq_channels gauge
# HELP rabbitmq_channels Channels currently running
rabbitmq_channels 5
# TYPE rabbitmq_consumers gauge
# HELP rabbitmq_consumers Consumers currently connected
rabbitmq_consumers 1
# TYPE rabbitmq_queues gauge
# HELP rabbitmq_queues Queues available
rabbitmq_queues 2
# TYPE rabbitmq_build_info untyped
# HELP rabbitmq_build_info RabbitMQ & Erlang/OTP version info
rabbitmq_build_info{rabbitmq_version="3.8.0-alpha.819",prometheus_plugin_version="3.8.0-alpha.819-2019.08.27",prometheus_client_version="4.4.0",erlang_version="22.0.7"} 1
# TYPE rabbitmq_identity_info untyped
# HELP rabbitmq_identity_info RabbitMQ node & cluster identity info
rabbitmq_identity_info{rabbitmq_node="rabbit@rmq0",rabbitmq_cluster="rabbitmq-overview"} 1

Noop if queries are as good as they can be:

$ ag expr docker/grafana/dashboards/RabbitMQ-Overview.json
95:          "expr": "sum(rabbitmq_queues * on(instance) group_left(rabbitmq_cluster) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"})",
181:          "expr": "sum(rabbitmq_channel_consumers * on(instance) group_left(rabbitmq_cluster) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"})",
267:          "expr": "sum(rabbitmq_connections * on(instance) group_left(rabbitmq_cluster) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"})",
353:          "expr": "sum(rabbitmq_channels * on(instance) group_left(rabbitmq_cluster) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"})",
439:          "expr": "sum(rate(rabbitmq_channel_messages_published[60s]) * on(instance) group_left(rabbitmq_cluster) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"})",
525:          "expr": "sum(rate(rabbitmq_channel_messages_redelivered_total[60s]) * on(instance) group_left(rabbitmq_cluster) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) +\nsum(rate(rabbitmq_channel_messages_delivered_total[60s]) * on(instance) group_left(rabbitmq_cluster) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) +\nsum(rate(rabbitmq_channel_messages_delivered_ack_total[60s]) * on(instance) group_left(rabbitmq_cluster) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) +\nsum(rate(rabbitmq_channel_get_total[60s]) * on(instance) group_left(rabbitmq_cluster) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) +\nsum(rate(rabbitmq_channel_get_ack_total[60s]) * on(instance) group_left(rabbitmq_cluster) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"})",
612:          "expr": "sum(rabbitmq_queue_messages_ready * on(instance) group_left(rabbitmq_cluster) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"})",
699:          "expr": "sum(rabbitmq_queue_messages_unacked * on(instance) group_left(rabbitmq_cluster) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"})",
820:          "expr": "(rabbitmq_memory_used_limit_bytes * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) -\n(rabbitmq_memory_used_bytes * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"})",
971:          "expr": "rabbitmq_disk_space_available_bytes * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}",
1123:          "expr": "(rabbitmq_file_descriptors_open_limit * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) -\n(rabbitmq_file_descriptors_open * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"})",
1274:          "expr": "(rabbitmq_tcp_sockets_open_limit * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) -\n(rabbitmq_tcp_sockets_open * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"})",
1439:          "expr": "sum(rabbitmq_queue_messages_ready * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",
1574:          "expr": "sum(rabbitmq_queue_messages_unacked * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",
1722:          "expr": "sum(rate(rabbitmq_channel_messages_published[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",
1857:          "expr": "sum(rate(rabbitmq_channel_messages_confirmed[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",
1956:          "expr": "sum(rate(rabbitmq_channel_messages_unroutable_returned[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",
2100:          "expr": "sum(rate(rabbitmq_queue_messages_published_total[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",
2235:          "expr": "sum(rate(rabbitmq_channel_messages_unconfirmed[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",
2334:          "expr": "sum(rate(rabbitmq_channel_messages_unroutable_dropped[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",
2491:          "expr": "sum(\n  (rate(rabbitmq_channel_messages_delivered_total[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) +\n  (rate(rabbitmq_channel_messages_delivered_ack_total[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"})\n) by(rabbitmq_node)",
2626:          "expr": "sum(rate(rabbitmq_channel_messages_delivered_total[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",
2725:          "expr": "sum(rate(rabbitmq_channel_get_total[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",
2869:          "expr": "sum(rate(rabbitmq_channel_messages_redelivered_total[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",
3021:          "expr": "sum(rate(rabbitmq_channel_messages_delivered_ack_total[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",
3120:          "expr": "sum(rate(rabbitmq_channel_get_ack_total[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",
3264:          "expr": "sum(rate(rabbitmq_channel_messages_acked_total[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",
3364:          "expr": "sum(rate(rabbitmq_channel_get_empty_total[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",
3521:          "expr": "rabbitmq_queues * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}",
3656:          "expr": "sum(rate(rabbitmq_queue_declared_total[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",
3808:          "expr": "sum(rate(rabbitmq_queue_created_total[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",
3960:          "expr": "sum(rate(rabbitmq_queue_deleted_total[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",
4125:          "expr": "rabbitmq_channels * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}",
4260:          "expr": "sum(rate(rabbitmq_channel_opened_total[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",
4412:          "expr": "sum(rate(rabbitmq_channel_closed_total[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",
4577:          "expr": "rabbitmq_connections * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}",
4712:          "expr": "sum(rate(rabbitmq_connection_opened_total[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",
4865:          "expr": "sum(rate(rabbitmq_connection_closed_total[60s]) * on(instance) group_left(rabbitmq_cluster, rabbitmq_node) rabbitmq_identity_info{rabbitmq_cluster=\"$rabbitmq_cluster\"}) by(rabbitmq_node)",

[brian-brazil]

That looks better. "Connections currently open" might be more understandable for rabbitmq_connections.

rabbitmq_file_descriptors_open

This should be process_open_fds

rabbitmq_erlang_gc_runs_total

This is a metric from erlang, not rabbitmq, so should not have a rabbitmq_ prefix. Similarly with the others.

[gerhard]

rabbitmq_file_descriptors_open

This should be process_open_fds

How do we distinguish between what RabbitMQ thinks that the open file descriptors are vs the kernel? Even though these values should be the same, because RabbitMQ works them out in one way, and caches this value, it may differ from the value reported by e.g. node_exporter.

rabbitmq_erlang_gc_runs_total

This is a metric from erlang, not rabbitmq, so should not have a rabbitmq_ prefix. Similarly with the others.

You are right, but this is what the RabbitMQ core metrics collector reports, which may be different from e.g. prometheus.erl erlang_vm collector. I am keen on exposing what RabbitMQ thinks the state of the world is, and preserve this perspective, vs what a different collector reports. Is there precedence for this in the Prometheus world?

[brian-brazil]

How do we distinguish between what RabbitMQ thinks that the open file descriptors are vs the kernel?

Why would they be different? It's ultimately coming from the same place in the kernel, and there shouldn't be any caching. The node exporter doesn't expose this, as it's not a machine-level metric.

Is there precedence for this in the Prometheus world?

Metrics like this should be provided out of the box by your client library, if they're missing something then look at improving that upstream.

[michaelklishin]

@brian-brazil RabbitMQ tries to limit the number of file descriptors it uses, and file handle open/close/reopen sequences. The difference with the metric reported by the kernel is usually nil or not something meaningful but RabbitMQ nodes have their own metric nonetheless. The metric is particularly useful on Windows. It's been around since 2007 or so.

[brian-brazil]

I'd suggest upstreaming that then. In any case if the upstream library provides this, it should be as process_open_fds.

[gerhard]

The upstream library add-on, prometheus_process_collector, is not portable across platforms, while our core RabbitMQ metrics are.

fd_used is a core RabbitMQ metric, which gets updated every 5s by default into an internal, in-memory data structure, same as all other metrics. We expose this metric via the Prometheus API as rabbitmq_file_descriptors_open. Happy to rename it to rabbitmq_open_fds, but we must preserve the RabbitMQ context, meaning the rabbitmq_ prefix.

Same is applicable to all other metrics that start with rabbitmq_erlang_ or rabbitmq_io_. While they may point to subsystems, the values are what RabbitMQ observes and uses through the entire lifecycle.

[gerhard]

I have just noticed the process_open_fds metric in prometheus_process_collector example scrape. Are you suggesting @brian-brazil that we should use the same name, even if the metric is coming from RabbitMQ Core Metrics?

[gerhard]

This is done and ready to get merged. Last chance to disagree 😄

[gerhard]

The last commit should do it. Do not want to delay this anymore, happy to address any outstanding issues separately.