rabbitmq / rabbitmq-server

Open source RabbitMQ: core server and tier 1 (built-in) plugins
https://www.rabbitmq.com/
Other
12.1k stars 3.9k forks source link

`rabbitmq_queue_exchange_messages_published_total` in `/metrics/per-object` has a duplicate `vhost` label #12347

Open Gama11 opened 7 hours ago

Gama11 commented 7 hours ago

Describe the bug

Prometheus seems to choke on duplicate labels:

image

Note how vhost appears twice here:

# TYPE rabbitmq_queue_exchange_messages_published_total counter
# HELP rabbitmq_queue_exchange_messages_published_total Total number of messages published into a queue through an exchange
rabbitmq_queue_exchange_messages_published_total{vhost="/",queue="<queue-name>",vhost="/",exchange="<exchange-name>"} 19

Versions used:

This seems to be a regression compared to 3.13.7 which we upgraded from.

Reproduction steps

  1. Enable the rabbitmq_prometheus plugin
  2. Send some messages to a queue so the rabbitmq_queue_exchange_messages_published_total metric is included
  3. Query the /metrics/per-object endpoint

Expected behavior

There should not be duplicate labels.

Additional context

No response

Gama11 commented 7 hours ago

Possibly relevant, our rabbitmq.config:

[
  {
    rabbit,
      [
        { loopback_users, [] }
      ]
  },
  {
    rabbitmq_management,
      [
        { load_definitions, "/etc/rabbitmq/definitions.json" }
      ]
  }
].

definitions.json:

{
  "users": [
    {
      "name": "test",
      "password_hash": "<redacted>",
      "hashing_algorithm": "rabbit_password_hashing_sha256",
      "tags": "administrator,monitoring"
    }
  ],
  "vhosts": [
    {
      "name": "/"
    }
  ],
  "permissions": [
    {
      "user": "test",
      "vhost": "/",
      "configure": ".*",
      "write": ".*",
      "read": ".*"
    }
  ],
  "topic_permissions": [],
  "parameters": [],
  "global_parameters": [],
  "policies": [],
  "queues": [],
  "exchanges": [],
  "bindings": []
}
michaelklishin commented 5 hours ago

Thank you. This looks like a blank node's config with a pre-configured user for remote access.

I was able to reproduce with a blank node from main:

# HELP rabbitmq_queue_exchange_messages_published_total Total number of messages published into a queue through an exchange
rabbitmq_queue_exchange_messages_published_total{vhost="/",queue="qq.1",vhost="/",exchange=""} 100000
gomoripeti commented 4 hours ago

I think this is related to https://github.com/rabbitmq/rabbitmq-server/pull/11559 the first vhost belongs to the queue, the second to the exchange but obviously they are the same so need to special handle this new case to eliminate the redundancy