search

deadtrickster / prometheus_rabbitmq_exporter

Prometheus.io exporter as a RabbitMQ Managment Plugin plugin
MIT License
291 stars 72 forks source link

Rabbitmq specific metrics (e.g. rabbitmq_queues ) are missing from metrics response #33

Closed choyuri closed 7 years ago

choyuri commented 7 years ago

On a 3-node clustered rabbitmq version 3.6.11, I enabled prometheus plugins on all three nodes I see all the beam metrics, but not any of the rabbitmq metrics

Initially tried prometheus_rabbitmq_exporter v3.6.12.1

then tried prometheus_rabbitmq_exporter v3.6.9.1

there are no errors in rabbit logs. prometheus metrics are working.. just missing the rabbitmq metrics

thank you

deadtrickster commented 7 years ago

Hi, that's strange. Could you please show output of rabbitmq-plugins list -v?

choyuri commented 7 years ago 
[E*] accept
     Version:       0.3.3
     Description:   Accept header(s) for Erlang/Elixir

[e*] amqp_client
     Version:       3.6.11
     Description:   RabbitMQ AMQP Client

[e*] cowboy
     Version:       1.0.4
     Dependencies:  [cowlib]
     Description:   Small, fast, modular HTTP server.

[e*] cowlib
     Version:       1.0.2
     Description:   Support library for manipulating Web protocols.

[E*] prometheus
     Version:       3.4.0
     Description:   Prometheus.io client in Erlang

[E*] prometheus_httpd
     Version:       2.1.4
     Dependencies:  [prometheus,accept]
     Description:   Prometheus.io inets httpd exporter

[  ] prometheus_process_collector
     Version:       1.1.0
     Description:   Prometheus.io process collector
    Collector exports the current state of process metrics including cpu, memory,
    file descriptor usage and native threads count as well as the process start and up times.

[E*] prometheus_rabbitmq_exporter
     Version:       v3.6.12.1
     Dependencies:  [rabbitmq_management,prometheus]
     Description:   RabbitMQ Prometheus.io metrics exporter

[  ] rabbitmq_amqp1_0
     Version:       3.6.11
     Dependencies:  [amqp_client]
     Description:   AMQP 1.0 support for RabbitMQ

[E*] rabbitmq_auth_backend_http
     Version:       3.6.7
     Dependencies:  [amqp_client]
     Description:   RabbitMQ HTTP Authentication Backend

[  ] rabbitmq_auth_backend_ldap
     Version:       3.6.11
     Description:   RabbitMQ LDAP Authentication Backend

[  ] rabbitmq_auth_mechanism_ssl
     Version:       3.6.11
     Description:   RabbitMQ SSL authentication (SASL EXTERNAL)

[  ] rabbitmq_consistent_hash_exchange
     Version:       3.6.11
     Description:   Consistent Hash Exchange Type

[E*] rabbitmq_event_exchange
     Version:       3.6.11
     Description:   Event Exchange Type

[  ] rabbitmq_federation
     Version:       3.6.11
     Dependencies:  [amqp_client]
     Description:   RabbitMQ Federation

[  ] rabbitmq_federation_management
     Version:       3.6.11
     Dependencies:  [rabbitmq_management,rabbitmq_federation]
     Description:   RabbitMQ Federation Management

[  ] rabbitmq_jms_topic_exchange
     Version:       3.6.11
     Description:   RabbitMQ JMS topic selector exchange plugin

[E*] rabbitmq_management
     Version:       3.6.11
     Dependencies:  [amqp_client,cowboy,cowlib,rabbitmq_web_dispatch,
                         rabbitmq_management_agent]
     Description:   RabbitMQ Management Console

[e*] rabbitmq_management_agent
     Version:       3.6.11
     Description:   RabbitMQ Management Agent

[  ] rabbitmq_management_visualiser
     Version:       3.6.11
     Dependencies:  [rabbitmq_management]
     Description:   RabbitMQ Visualiser

[  ] rabbitmq_mqtt
     Version:       3.6.11
     Dependencies:  [amqp_client]
     Description:   RabbitMQ MQTT Adapter

[  ] rabbitmq_recent_history_exchange
     Version:       3.6.11
     Description:   RabbitMQ Recent History Exchange

[  ] rabbitmq_sharding
     Version:       3.6.11
     Description:   RabbitMQ Sharding Plugin

[  ] rabbitmq_shovel
     Version:       3.6.11
     Dependencies:  [amqp_client]
     Description:   Data Shovel for RabbitMQ

[  ] rabbitmq_shovel_management
     Version:       3.6.11
     Dependencies:  [rabbitmq_management,rabbitmq_shovel]
     Description:   Management extension for the Shovel plugin

[e*] rabbitmq_stomp
     Version:       3.6.11
     Dependencies:  [amqp_client]
     Description:   RabbitMQ STOMP plugin

[  ] rabbitmq_top
     Version:       3.6.11
     Dependencies:  [amqp_client,rabbitmq_management]
     Description:   RabbitMQ Top

[  ] rabbitmq_tracing
     Version:       3.6.11
     Dependencies:  [rabbitmq_management]
     Description:   RabbitMQ message logging / tracing

[  ] rabbitmq_trust_store
     Version:       3.6.11
     Description:   Client X.509 certificates trust store

[e*] rabbitmq_web_dispatch
     Version:       3.6.11
     Dependencies:  [cowboy]
     Description:   RabbitMQ Web Dispatcher

[  ] rabbitmq_web_mqtt
     Version:       3.6.11
     Dependencies:  [cowboy,rabbitmq_mqtt]
     Description:   RabbitMQ MQTT-over-WebSockets adapter

[  ] rabbitmq_web_mqtt_examples
     Version:       3.6.11
     Dependencies:  [rabbitmq_web_dispatch,rabbitmq_web_mqtt]
     Description:   Rabbit WEB-MQTT - examples

[E*] rabbitmq_web_stomp
     Version:       3.6.11
     Dependencies:  [cowboy,sockjs,rabbitmq_stomp]
     Description:   Rabbit WEB-STOMP - WebSockets to Stomp adapter

[  ] rabbitmq_web_stomp_examples
     Version:       3.6.11
     Dependencies:  [rabbitmq_web_dispatch,rabbitmq_web_stomp]
     Description:   Rabbit WEB-STOMP - examples

[e*] sockjs
     Version:       0.3.4
     Description:   SockJS
deadtrickster commented 7 years ago

Thanks!

More questions:

choyuri commented 7 years ago

yes checked on all three. what happened was.. I did a test deploy on node1.. didnt see rabbit metrics tried node2 same issue tried node 3 same issue

thanks

node1: [prometheus_boolean,prometheus_counter,prometheus_gauge,prometheus_histogram, prometheus_mnesia_collector,prometheus_summary, prometheus_vm_memory_collector,prometheus_vm_statistics_collector, prometheus_vm_system_info_collector]

node2: [prometheus_boolean,prometheus_counter,prometheus_gauge,prometheus_histogram, prometheus_mnesia_collector,prometheus_summary, prometheus_vm_memory_collector,prometheus_vm_statistics_collector, prometheus_vm_system_info_collector]

node3:

[prometheus_boolean,prometheus_counter,prometheus_gauge,prometheus_histogram, prometheus_mnesia_collector,prometheus_summary, prometheus_vm_memory_collector,prometheus_vm_statistics_collector, prometheus_vm_system_info_collector]

deadtrickster commented 7 years ago

So rabbitmq collectors just not registered for some reason... Could you please now try this call prometheus_registry:register_collector(prometheus_rabbitmq_overview_collector). and look at /metrics again?

choyuri commented 7 years ago

wooh! that worked

deadtrickster commented 7 years ago

ok, so the problem is that rabbitmq_ collectors aren't registered automatically on node startup...

Did you strip debug_info? What if you restart a node?

Anyway will fix registering soon. Nice catch :-)

choyuri commented 7 years ago

thank you for debugging!

so i just followed the documentation and performed enable plugin steps only it's a pretty hefty cluster so takes non trivial amount of time for each node to bounce and rejoin :) so was trying to avoid restarting the workaround is great in the meantime.

thanks again!

deadtrickster commented 7 years ago

Should be fixed here: https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/tag/rabbitmq-3.6.14.1.