search

rabbitmq / rabbitmq-prometheus

A minimalistic Prometheus exporter of core RabbitMQ metrics
Other
145 stars 109 forks source link

Use the correct metric types & capture perspective when naming #6

Closed gerhard closed 5 years ago

gerhard commented 5 years ago

Some metrics were of type gauge while they should have been of type counter. Thanks @brian-brazil for making the distinction clear. This is now captured as a comment above the metric definitions.

Because all metrics are from RabbitMQ's perspective, cached for up to 5 seconds by default (configurable), we prepend rabbitmq_ to all metrics emitted by this collector. While Some metrics are for Erlang (erlang_), Mnesia (schemadb) or the System (io_), they are all observed & cached by RabbitMQ, hence the prefix.

This is the last PR which started in the context of prometheus/docs#1414

[#167846096]

gerhard commented 5 years ago

Missing the Grafana dashboard updates, but wanted to get this out first.

Does this look right @brian-brazil?

This is the full output from rabbitmq_prometheus after the latest changes:

# TYPE rabbitmq_channel_consumers gauge
# HELP rabbitmq_channel_consumers Consumers on a channel
rabbitmq_channel_consumers{channel="<0.793.0>"} 0
rabbitmq_channel_consumers{channel="<0.782.0>"} 1
# TYPE rabbitmq_channel_messages_unacked gauge
# HELP rabbitmq_channel_messages_unacked Delivered but not yet acknowledged messages
rabbitmq_channel_messages_unacked{channel="<0.793.0>"} 0
rabbitmq_channel_messages_unacked{channel="<0.782.0>"} 0
# TYPE rabbitmq_channel_messages_unconfirmed gauge
# HELP rabbitmq_channel_messages_unconfirmed Published but not yet confirmed messages
rabbitmq_channel_messages_unconfirmed{channel="<0.793.0>"} 0
rabbitmq_channel_messages_unconfirmed{channel="<0.782.0>"} 0
# TYPE rabbitmq_channel_messages_uncommitted gauge
# HELP rabbitmq_channel_messages_uncommitted Messages received in a transaction but not yet committed
rabbitmq_channel_messages_uncommitted{channel="<0.793.0>"} 0
rabbitmq_channel_messages_uncommitted{channel="<0.782.0>"} 0
# TYPE rabbitmq_channel_acks_uncommitted gauge
# HELP rabbitmq_channel_acks_uncommitted Message acknowledgements in a transaction not yet committed
rabbitmq_channel_acks_uncommitted{channel="<0.793.0>"} 0
rabbitmq_channel_acks_uncommitted{channel="<0.782.0>"} 0
# TYPE rabbitmq_consumer_prefetch gauge
# HELP rabbitmq_consumer_prefetch Limit of unacknowledged messages for each consumer
rabbitmq_consumer_prefetch{channel="<0.793.0>"} 0
rabbitmq_consumer_prefetch{channel="<0.782.0>"} 0
# TYPE rabbitmq_channel_prefetch gauge
# HELP rabbitmq_channel_prefetch Total limit of unacknowledged messages for all consumers on a channel
rabbitmq_channel_prefetch{channel="<0.793.0>"} 0
rabbitmq_channel_prefetch{channel="<0.782.0>"} 0
# TYPE rabbitmq_channel_messages_published_total counter
# HELP rabbitmq_channel_messages_published_total Total number of messages published into an exchange on a channel
rabbitmq_channel_messages_published_total{channel="<0.793.0>",vhost="/",exchange="direct"} 10
# TYPE rabbitmq_channel_messages_confirmed_total counter
# HELP rabbitmq_channel_messages_confirmed_total Total number of messages published into an exchange and confirmed on the channel
rabbitmq_channel_messages_confirmed_total{channel="<0.793.0>",vhost="/",exchange="direct"} 0
# TYPE rabbitmq_channel_messages_unroutable_returned_total counter
# HELP rabbitmq_channel_messages_unroutable_returned_total Total number of messages published as mandatory into an exchange and returned to the publisher as unroutable
rabbitmq_channel_messages_unroutable_returned_total{channel="<0.793.0>",vhost="/",exchange="direct"} 0
# TYPE rabbitmq_channel_messages_unroutable_dropped_total counter
# HELP rabbitmq_channel_messages_unroutable_dropped_total Total number of messages published as non-mandatory into an exchange and dropped as unroutable
rabbitmq_channel_messages_unroutable_dropped_total{channel="<0.793.0>",vhost="/",exchange="direct"} 0
# TYPE rabbitmq_channel_process_reductions_total counter
# HELP rabbitmq_channel_process_reductions_total Total number of channel process reductions
rabbitmq_channel_process_reductions_total{channel="<0.793.0>"} 5137
rabbitmq_channel_process_reductions_total{channel="<0.782.0>"} 3966
# TYPE rabbitmq_channel_get_ack_total counter
# HELP rabbitmq_channel_get_ack_total Total number of messages fetched with basic.get in manual acknowledgement mode
rabbitmq_channel_get_ack_total{channel="<0.782.0>",vhost="/",queue="amq.gen-g3OV1t997aanV8t7n8DUlQ"} 0
# TYPE rabbitmq_channel_get_total counter
# HELP rabbitmq_channel_get_total Total number of messages fetched with basic.get in automatic acknowledgement mode
rabbitmq_channel_get_total{channel="<0.782.0>",vhost="/",queue="amq.gen-g3OV1t997aanV8t7n8DUlQ"} 0
# TYPE rabbitmq_channel_messages_delivered_ack_total counter
# HELP rabbitmq_channel_messages_delivered_ack_total Total number of messages delivered to consumers in manual acknowledgement mode
rabbitmq_channel_messages_delivered_ack_total{channel="<0.782.0>",vhost="/",queue="amq.gen-g3OV1t997aanV8t7n8DUlQ"} 10
# TYPE rabbitmq_channel_messages_delivered_total counter
# HELP rabbitmq_channel_messages_delivered_total Total number of messages delivered to consumers in automatic acknowledgement mode
rabbitmq_channel_messages_delivered_total{channel="<0.782.0>",vhost="/",queue="amq.gen-g3OV1t997aanV8t7n8DUlQ"} 0
# TYPE rabbitmq_channel_messages_redelivered_total counter
# HELP rabbitmq_channel_messages_redelivered_total Total number of messages redelivered to consumers
rabbitmq_channel_messages_redelivered_total{channel="<0.782.0>",vhost="/",queue="amq.gen-g3OV1t997aanV8t7n8DUlQ"} 0
# TYPE rabbitmq_channel_messages_acked_total counter
# HELP rabbitmq_channel_messages_acked_total Total number of messages acknowledged by consumers
rabbitmq_channel_messages_acked_total{channel="<0.782.0>",vhost="/",queue="amq.gen-g3OV1t997aanV8t7n8DUlQ"} 10
# TYPE rabbitmq_channel_get_empty_total counter
# HELP rabbitmq_channel_get_empty_total Total number of times basic.get operations fetched no message
rabbitmq_channel_get_empty_total{channel="<0.782.0>",vhost="/",queue="amq.gen-g3OV1t997aanV8t7n8DUlQ"} 0
# TYPE rabbitmq_connections_opened_total counter
# HELP rabbitmq_connections_opened_total Total number of connections opened
rabbitmq_connections_opened_total 3
# TYPE rabbitmq_connections_closed_total counter
# HELP rabbitmq_connections_closed_total Total number of connections closed or terminated
rabbitmq_connections_closed_total 0
# TYPE rabbitmq_channels_opened_total counter
# HELP rabbitmq_channels_opened_total Total number of channels opened
rabbitmq_channels_opened_total 3
# TYPE rabbitmq_channels_closed_total counter
# HELP rabbitmq_channels_closed_total Total number of channels closed
rabbitmq_channels_closed_total 1
# TYPE rabbitmq_queues_declared_total counter
# HELP rabbitmq_queues_declared_total Total number of queues declared
rabbitmq_queues_declared_total 1
# TYPE rabbitmq_queues_created_total counter
# HELP rabbitmq_queues_created_total Total number of queues created
rabbitmq_queues_created_total 1
# TYPE rabbitmq_queues_deleted_total counter
# HELP rabbitmq_queues_deleted_total Total number of queues deleted
rabbitmq_queues_deleted_total 0
# TYPE rabbitmq_connection_incoming_bytes_total counter
# HELP rabbitmq_connection_incoming_bytes_total Total number of bytes received on a connection
rabbitmq_connection_incoming_bytes_total{channel="<0.755.0>"} 478
rabbitmq_connection_incoming_bytes_total{channel="<0.785.0>"} 1026
rabbitmq_connection_incoming_bytes_total{channel="<0.762.0>"} 825
# TYPE rabbitmq_connection_outgoing_bytes_total counter
# HELP rabbitmq_connection_outgoing_bytes_total Total number of bytes sent on a connection
rabbitmq_connection_outgoing_bytes_total{channel="<0.755.0>"} 546
rabbitmq_connection_outgoing_bytes_total{channel="<0.785.0>"} 574
rabbitmq_connection_outgoing_bytes_total{channel="<0.762.0>"} 1404
# TYPE rabbitmq_connection_process_reductions_total counter
# HELP rabbitmq_connection_process_reductions_total Total number of connection process reductions
rabbitmq_connection_process_reductions_total{channel="<0.755.0>"} 9301
rabbitmq_connection_process_reductions_total{channel="<0.785.0>"} 11303
rabbitmq_connection_process_reductions_total{channel="<0.762.0>"} 11814
# TYPE rabbitmq_connection_incoming_packets_total counter
# HELP rabbitmq_connection_incoming_packets_total Total number of packets received on a connection
rabbitmq_connection_incoming_packets_total{channel="<0.755.0>"} 4
rabbitmq_connection_incoming_packets_total{channel="<0.785.0>"} 11
rabbitmq_connection_incoming_packets_total{channel="<0.762.0>"} 16
# TYPE rabbitmq_connection_outgoing_packets_total counter
# HELP rabbitmq_connection_outgoing_packets_total Total number of packets sent on a connection
rabbitmq_connection_outgoing_packets_total{channel="<0.755.0>"} 3
rabbitmq_connection_outgoing_packets_total{channel="<0.785.0>"} 5
rabbitmq_connection_outgoing_packets_total{channel="<0.762.0>"} 15
# TYPE rabbitmq_connection_pending_packets gauge
# HELP rabbitmq_connection_pending_packets Number of packets waiting to be sent on a connection
rabbitmq_connection_pending_packets{channel="<0.755.0>"} 0
rabbitmq_connection_pending_packets{channel="<0.785.0>"} 0
rabbitmq_connection_pending_packets{channel="<0.762.0>"} 0
# TYPE rabbitmq_connection_channels gauge
# HELP rabbitmq_connection_channels Channels on a connection
rabbitmq_connection_channels{channel="<0.755.0>"} 0
rabbitmq_connection_channels{channel="<0.785.0>"} 1
rabbitmq_connection_channels{channel="<0.762.0>"} 1
# TYPE rabbitmq_queue_messages_published_total counter
# HELP rabbitmq_queue_messages_published_total Total number of messages published to queues
rabbitmq_queue_messages_published_total{channel="<0.793.0>",queue_vhost="/",queue="amq.gen-g3OV1t997aanV8t7n8DUlQ",exchange_vhost="/",exchange="direct"} 10
# TYPE rabbitmq_process_open_fds gauge
# HELP rabbitmq_process_open_fds Open file descriptors
rabbitmq_process_open_fds 108
# TYPE rabbitmq_process_open_sockets gauge
# HELP rabbitmq_process_open_sockets Open TCP sockets
rabbitmq_process_open_sockets 3
# TYPE rabbitmq_process_resident_memory_bytes gauge
# HELP rabbitmq_process_resident_memory_bytes Memory used in bytes
rabbitmq_process_resident_memory_bytes 100691968
# TYPE rabbitmq_disk_space_available_bytes gauge
# HELP rabbitmq_disk_space_available_bytes Disk space available in bytes
rabbitmq_disk_space_available_bytes 595666440192
# TYPE rabbitmq_erlang_processes_used gauge
# HELP rabbitmq_erlang_processes_used Erlang processes used
rabbitmq_erlang_processes_used 579
# TYPE rabbitmq_erlang_gc_runs_total counter
# HELP rabbitmq_erlang_gc_runs_total Total number of Erlang garbage collector runs
rabbitmq_erlang_gc_runs_total 4656
# TYPE rabbitmq_erlang_gc_reclaimed_bytes_total counter
# HELP rabbitmq_erlang_gc_reclaimed_bytes_total Total number of bytes of memory reclaimed by Erlang garbage collector
rabbitmq_erlang_gc_reclaimed_bytes_total 342969896
# TYPE rabbitmq_erlang_scheduler_context_switches_total counter
# HELP rabbitmq_erlang_scheduler_context_switches_total Total number of Erlang scheduler context switches
rabbitmq_erlang_scheduler_context_switches_total 137525
# TYPE rabbitmq_process_max_fds gauge
# HELP rabbitmq_process_max_fds Open file descriptors limit
rabbitmq_process_max_fds 256
# TYPE rabbitmq_process_max_sockets gauge
# HELP rabbitmq_process_max_sockets Open TCP sockets limit
rabbitmq_process_max_sockets 138
# TYPE rabbitmq_resident_memory_limit_bytes gauge
# HELP rabbitmq_resident_memory_limit_bytes Memory high watermark in bytes
rabbitmq_resident_memory_limit_bytes 27487790694
# TYPE rabbitmq_disk_space_available_limit_bytes gauge
# HELP rabbitmq_disk_space_available_limit_bytes Free disk space low watermark in bytes
rabbitmq_disk_space_available_limit_bytes 50000000
# TYPE rabbitmq_erlang_processes_limit gauge
# HELP rabbitmq_erlang_processes_limit Erlang processes limit
rabbitmq_erlang_processes_limit 1048576
# TYPE rabbitmq_erlang_scheduler_run_queue gauge
# HELP rabbitmq_erlang_scheduler_run_queue Erlang scheduler run queue
rabbitmq_erlang_scheduler_run_queue 1
# TYPE rabbitmq_erlang_net_ticktime_seconds gauge
# HELP rabbitmq_erlang_net_ticktime_seconds Inter-node heartbeat interval in seconds
rabbitmq_erlang_net_ticktime_seconds 60
# TYPE rabbitmq_io_read_ops_total counter
# HELP rabbitmq_io_read_ops_total Total number of I/O read operations
rabbitmq_io_read_ops_total 1
# TYPE rabbitmq_io_read_bytes_total counter
# HELP rabbitmq_io_read_bytes_total Total number of I/O bytes read
rabbitmq_io_read_bytes_total 1
# TYPE rabbitmq_io_write_ops_total counter
# HELP rabbitmq_io_write_ops_total Total number of I/O write operations
rabbitmq_io_write_ops_total 2
# TYPE rabbitmq_io_write_bytes_total counter
# HELP rabbitmq_io_write_bytes_total Total number of I/O bytes written
rabbitmq_io_write_bytes_total 5
# TYPE rabbitmq_io_sync_ops_total counter
# HELP rabbitmq_io_sync_ops_total Total number of I/O sync operations
rabbitmq_io_sync_ops_total 0
# TYPE rabbitmq_io_seek_ops_total counter
# HELP rabbitmq_io_seek_ops_total Total number of I/O seek operations
rabbitmq_io_seek_ops_total 0
# TYPE rabbitmq_io_open_attempt_ops_total counter
# HELP rabbitmq_io_open_attempt_ops_total Total number of file open attempts
rabbitmq_io_open_attempt_ops_total 11
# TYPE rabbitmq_io_reopen_ops_total counter
# HELP rabbitmq_io_reopen_ops_total Total number of times files have been reopened
rabbitmq_io_reopen_ops_total 0
# TYPE rabbitmq_schema_db_ram_tx_total counter
# HELP rabbitmq_schema_db_ram_tx_total Total number of Schema DB memory transactions
rabbitmq_schema_db_ram_tx_total 14
# TYPE rabbitmq_schema_db_disk_tx_total counter
# HELP rabbitmq_schema_db_disk_tx_total Total number of Schema DB disk transactions
rabbitmq_schema_db_disk_tx_total 0
# TYPE rabbitmq_msg_store_read_total counter
# HELP rabbitmq_msg_store_read_total Total number of Message Store read operations
rabbitmq_msg_store_read_total 0
# TYPE rabbitmq_msg_store_write_total counter
# HELP rabbitmq_msg_store_write_total Total number of Message Store write operations
rabbitmq_msg_store_write_total 0
# TYPE rabbitmq_queue_index_read_ops_total counter
# HELP rabbitmq_queue_index_read_ops_total Total number of Queue Index read operations
rabbitmq_queue_index_read_ops_total 0
# TYPE rabbitmq_queue_index_write_ops_total counter
# HELP rabbitmq_queue_index_write_ops_total Total number of Queue Index write operations
rabbitmq_queue_index_write_ops_total 0
# TYPE rabbitmq_queue_index_journal_write_ops_total counter
# HELP rabbitmq_queue_index_journal_write_ops_total Total number of Queue Index Journal write operations
rabbitmq_queue_index_journal_write_ops_total 0
# TYPE rabbitmq_raft_term_total counter
# HELP rabbitmq_raft_term_total Current Raft term number
# TYPE rabbitmq_raft_log_snapshot_index gauge
# HELP rabbitmq_raft_log_snapshot_index Raft log snapshot index
# TYPE rabbitmq_raft_log_last_applied_index gauge
# HELP rabbitmq_raft_log_last_applied_index Raft log last applied index
# TYPE rabbitmq_raft_log_commit_index gauge
# HELP rabbitmq_raft_log_commit_index Raft log commit index
# TYPE rabbitmq_raft_log_last_written_index gauge
# HELP rabbitmq_raft_log_last_written_index Raft log last written index
# TYPE rabbitmq_raft_entry_commit_latency gauge
# HELP rabbitmq_raft_entry_commit_latency Time taken for an entry to be committed
# TYPE rabbitmq_queue_messages_ready gauge
# HELP rabbitmq_queue_messages_ready Messages ready to be delivered to consumers
rabbitmq_queue_messages_ready{vhost="/",queue="amq.gen-g3OV1t997aanV8t7n8DUlQ"} 0
# TYPE rabbitmq_queue_messages_unacked gauge
# HELP rabbitmq_queue_messages_unacked Messages delivered to consumers but not yet acknowledged
rabbitmq_queue_messages_unacked{vhost="/",queue="amq.gen-g3OV1t997aanV8t7n8DUlQ"} 0
# TYPE rabbitmq_queue_messages gauge
# HELP rabbitmq_queue_messages Sum of ready and unacknowledged messages - total queue depth
rabbitmq_queue_messages{vhost="/",queue="amq.gen-g3OV1t997aanV8t7n8DUlQ"} 0
# TYPE rabbitmq_queue_process_reductions_total counter
# HELP rabbitmq_queue_process_reductions_total Total number of queue process reductions
rabbitmq_queue_process_reductions_total{vhost="/",queue="amq.gen-g3OV1t997aanV8t7n8DUlQ"} 8900
# TYPE rabbitmq_queue_consumers gauge
# HELP rabbitmq_queue_consumers Consumers on a queue
# TYPE rabbitmq_queue_process_memory_bytes gauge
# HELP rabbitmq_queue_process_memory_bytes Memory in bytes used by the Erlang queue process
# TYPE rabbitmq_queue_messages_bytes gauge
# HELP rabbitmq_queue_messages_bytes Size in bytes of ready and unacknowledged messages
# TYPE rabbitmq_queue_messages_ram gauge
# HELP rabbitmq_queue_messages_ram Ready and unacknowledged messages stored in memory
# TYPE rabbitmq_queue_messages_ready_ram gauge
# HELP rabbitmq_queue_messages_ready_ram Ready messages stored in memory
# TYPE rabbitmq_queue_messages_ready_bytes gauge
# HELP rabbitmq_queue_messages_ready_bytes Size in bytes of ready messages
# TYPE rabbitmq_queue_messages_unacked_ram gauge
# HELP rabbitmq_queue_messages_unacked_ram Unacknowledged messages stored in memory
# TYPE rabbitmq_queue_messages_unacked_bytes gauge
# HELP rabbitmq_queue_messages_unacked_bytes Size in bytes of all unacknowledged messages
# TYPE rabbitmq_queue_messages_persistent gauge
# HELP rabbitmq_queue_messages_persistent Persistent messages
# TYPE rabbitmq_queue_messages_persistent_bytes gauge
# HELP rabbitmq_queue_messages_persistent_bytes Size in bytes of persistent messages
# TYPE rabbitmq_queue_messages_paged_out gauge
# HELP rabbitmq_queue_messages_paged_out Messages paged out to disk
# TYPE rabbitmq_queue_messages_paged_out_bytes gauge
# HELP rabbitmq_queue_messages_paged_out_bytes Size in bytes of messages paged out to disk
# TYPE rabbitmq_queue_disk_reads_total counter
# HELP rabbitmq_queue_disk_reads_total Total number of times queue read messages from disk
rabbitmq_queue_disk_reads_total{vhost="/",queue="amq.gen-g3OV1t997aanV8t7n8DUlQ"} 0
# TYPE rabbitmq_queue_disk_writes_total counter
# HELP rabbitmq_queue_disk_writes_total Total number of times queue wrote messages to disk
rabbitmq_queue_disk_writes_total{vhost="/",queue="amq.gen-g3OV1t997aanV8t7n8DUlQ"} 0
# TYPE rabbitmq_erlang_uptime_seconds gauge
# HELP rabbitmq_erlang_uptime_seconds Node uptime
rabbitmq_erlang_uptime_seconds 17.481
# TYPE rabbitmq_io_read_time_seconds_total counter
# HELP rabbitmq_io_read_time_seconds_total Total I/O read time
rabbitmq_io_read_time_seconds_total 1.3e-5
# TYPE rabbitmq_io_write_time_seconds_total counter
# HELP rabbitmq_io_write_time_seconds_total Total I/O write time
rabbitmq_io_write_time_seconds_total 4.6e-5
# TYPE rabbitmq_io_sync_time_seconds_total counter
# HELP rabbitmq_io_sync_time_seconds_total Total I/O sync time
rabbitmq_io_sync_time_seconds_total 0.0
# TYPE rabbitmq_io_seek_time_seconds_total counter
# HELP rabbitmq_io_seek_time_seconds_total Total I/O seek time
rabbitmq_io_seek_time_seconds_total 0.0
# TYPE rabbitmq_io_open_attempt_time_seconds_total counter
# HELP rabbitmq_io_open_attempt_time_seconds_total Total file open attempts time
rabbitmq_io_open_attempt_time_seconds_total 3.54e-4
# TYPE rabbitmq_connections gauge
# HELP rabbitmq_connections Connections currently open
rabbitmq_connections 3
# TYPE rabbitmq_channels gauge
# HELP rabbitmq_channels Channels currently open
rabbitmq_channels 2
# TYPE rabbitmq_consumers gauge
# HELP rabbitmq_consumers Consumers currently connected
rabbitmq_consumers 1
# TYPE rabbitmq_queues gauge
# HELP rabbitmq_queues Queues available
rabbitmq_queues 1
# TYPE rabbitmq_rabbitmq_build_info untyped
# HELP rabbitmq_rabbitmq_build_info RabbitMQ & Erlang/OTP version info
rabbitmq_rabbitmq_build_info{rabbitmq_version="3.8.0+beta.7.9.gcd114c4",prometheus_plugin_version="3.8.0-alpha.857-2019.09.04",prometheus_client_version="4.4.0",erlang_version="22.0.7"} 1
# TYPE rabbitmq_rabbitmq_identity_info untyped
# HELP rabbitmq_rabbitmq_identity_info RabbitMQ node & cluster identity info
rabbitmq_rabbitmq_identity_info{rabbitmq_node="rabbit@focker",rabbitmq_cluster="rabbit@focker"} 1
brian-brazil commented 5 years ago

That looks about right, I'd add tcp to rabbitmq_process_open_sockets for clarity as there's more than one type of socket.