search

deadtrickster / prometheus_rabbitmq_exporter

Prometheus.io exporter as a RabbitMQ Managment Plugin plugin
MIT License
292 stars 72 forks source link

Cluster Deployment #4

Closed neildunlop closed 8 years ago

neildunlop commented 8 years ago

I'm deploying a Rabbit cluster and the plugin works fine on the first node that its deployed on but all subsequent nodes return:

{"error":"Internal Server Error","reason":"{error,\n {error,badarg,\n [{ets,select,\n [connectionstats,\n [{{{'','$1'},'$2','_'},[{'==',create,'$1'}],['$2']}]],\n []},\n {prometheus_rabbitmq_overview_collector,created_events,1,\n [{file,\n \"src/collectors/prometheus_rabbitmq_overview_collector.erl\"},\n {line,63}]},\n {prometheus_rabbitmq_overview_collector,collect_metrics,2,\n [{file,\n \"src/collectors/prometheus_rabbitmq_overview_collector.erl\"},\n {line,39}]},\n {prometheus_model_helpers,create_mf,5,\n [{file,\"src/model/prometheus_model_helpers.erl\"},{line,28}]},\n {prometheus_rabbitmq_overview_collector,collect_mf,2,\n [{file,\n \"src/collectors/prometheus_rabbitmq_overview_collector.erl\"},\n {line,32}]},\n {prometheus_registry,'-collect/2-lc$^0/1-0-',3,\n [{file,\"src/prometheus_registry.erl\"},{line,14}]},\n {prometheus_registry,'-collect/2-lc$^0/1-0-',3,\n [{file,\"src/prometheus_registry.erl\"},{line,14}]},\n {prometheus_text_format,format,1,\n [{file,\"src/formats/prometheus_text_format.erl\"},{line,28}]}]}}\n"}

Am I doing something wrong? (All the cluster nodes are configured identically through an ansible playbook).

deadtrickster commented 8 years ago

Hi,

  1. Are they all the same version?
  2. What plugin version you are using?

I'm pretty sure I saw something like this already when RabbitMQ/plugin versions didn't match. They change internals from time to time.

deadtrickster commented 8 years ago

I mean are you sure all servers in the cluster are the same version?

neildunlop commented 8 years ago

All the servers use the same Dockerfile which has:

rabbitmq:3.6.5-management as a base image and

prometheus-1.6.0.ez prometheus_process_collector-0.1.2.ez prometheus_rabbitmq_exporter-v3.6.5.1.ez

Could the difference between 3.6.5 and 3.6.5.1 be an issue?

deadtrickster commented 8 years ago

3.6.5.1 is a plugin version (like version 1 for rabbitmq 3.6.5). Maybe it's a bug... I'll take a look at the code.

neildunlop commented 8 years ago

awesome.. thanks!

neildunlop commented 8 years ago

The docker image is https://hub.docker.com/r/neildunlop/rabbitmonitored/

deadtrickster commented 8 years ago

Yep, can confirm this problem on my setup

deadtrickster commented 8 years ago

Fix: https://github.com/deadtrickster/prometheus_rabbitmq_exporter/commit/f043cdb8f7b020747b484bfa2f578cdddad6df75

Release: https://github.com/deadtrickster/prometheus_rabbitmq_exporter/releases/tag/rabbitmq-3.6.5.2

Let me know if error persists.

neildunlop commented 8 years ago

Awesome. I'll spin it up and let you know shortly.

neildunlop commented 8 years ago

Works perfectly! Thanks for the rapid fix.