queue_master_locator=min-masters not creating queues evenly when bindings are involved

[jharting]

I am observing what I believe is a bug:

I am running a 2-node RabbitMQ cluster in OpenShift with rabbitmq-peer-discovery-k8s plugin. A template I use to deploy the cluster can be found here. I am setting queue_master_locator=min-masters in rabbitmq.conf for the queues to be created evenly across the cluster nodes.

The problem

Queues are not distributed evenly. Here are the steps to reproduce: 1) Start a fresh 2-node rabbitmq cluster 2) Create topic exchange 3) Create queue0 4) Create 6 different bindings from the exchange to queue0 5) Create 6 more queues (with no bindings)

Expected

roughly half of the clues resides on node0, the rest on node1

Actual

queue0 is the only queue on node0, all the other queues end up on node1

<rabbit@rabbitmq-cluster-0.rabbitmq-cluster.bravo.svc.cluster.local.2.2017.0>   queue0
<rabbit@rabbitmq-cluster-1.rabbitmq-cluster.bravo.svc.cluster.local.2.1858.0>   queue4
<rabbit@rabbitmq-cluster-1.rabbitmq-cluster.bravo.svc.cluster.local.2.1850.0>   queue2
<rabbit@rabbitmq-cluster-1.rabbitmq-cluster.bravo.svc.cluster.local.2.1850.0>   queue1
<rabbit@rabbitmq-cluster-1.rabbitmq-cluster.bravo.svc.cluster.local.2.1862.0>   queue5
<rabbit@rabbitmq-cluster-1.rabbitmq-cluster.bravo.svc.cluster.local.2.1854.0>   queue3
<rabbit@rabbitmq-cluster-1.rabbitmq-cluster.bravo.svc.cluster.local.2.1866.0>   queue6

If I keep adding queues (with no bindings) the 7th or 8th will finally land on node0 making queue0 and queue7/8 the only queues on node0.

It looks as if the bindings counted into the number of queues on each node RabbitMQ is deciding which node a queue should land on.

In our app this causes one queue (which uses a lot of bindings) to take an entire cluster node for itself while all othe other queues end up being created on the other node.

If I don't use bindings in the reproducer then queues are created evenly as expected.

Reproducer:

'use strict';

const amqplib = require('amqplib');
const MQ_URL = process.env.MQ_URL;

void async function () {
    const connection = await amqplib.connect(MQ_URL);
    const channel = await connection.createChannel();

    await channel.assertExchange('exchange0', 'topic');
    await channel.assertQueue('queue0', {exclusive: true});

    for (const i of [1, 2, 3, 4, 5, 6]) {
        channel.bindQueue('queue0', 'exchange1', `test.${i}`);
    }

    for (const i of [1, 2, 3, 4, 5, 6, 7, 8]) {
        channel.assertQueue(`queue${i}`, {exclusive: true});
    }

    console.log('done');
}();

• RabbitMQ version: 3.7.3
• Erlang version: 20.2.2
• RabbitMQ plugin information via rabbitmq-plugins list

  [  ] rabbitmq_amqp1_0                  3.7.3
  [  ] rabbitmq_auth_backend_cache       3.7.3
  [  ] rabbitmq_auth_backend_http        3.7.3
  [  ] rabbitmq_auth_backend_ldap        3.7.3
  [  ] rabbitmq_auth_mechanism_ssl       3.7.3
  [  ] rabbitmq_consistent_hash_exchange 3.7.3
  [  ] rabbitmq_event_exchange           3.7.3
  [  ] rabbitmq_federation               3.7.3
  [  ] rabbitmq_federation_management    3.7.3
  [  ] rabbitmq_jms_topic_exchange       3.7.3
  [E*] rabbitmq_management               3.7.3
  [e*] rabbitmq_management_agent         3.7.3
  [  ] rabbitmq_mqtt                     3.7.3
  [  ] rabbitmq_peer_discovery_aws       3.7.3
  [e*] rabbitmq_peer_discovery_common    3.7.3
  [  ] rabbitmq_peer_discovery_consul    3.7.3
  [  ] rabbitmq_peer_discovery_etcd      3.7.3
  [E*] rabbitmq_peer_discovery_k8s       3.7.3
  [  ] rabbitmq_random_exchange          3.7.3
  [  ] rabbitmq_recent_history_exchange  3.7.3
  [  ] rabbitmq_sharding                 3.7.3
  [  ] rabbitmq_shovel                   3.7.3
  [  ] rabbitmq_shovel_management        3.7.3
  [  ] rabbitmq_stomp                    3.7.3
  [  ] rabbitmq_top                      3.7.3
  [  ] rabbitmq_tracing                  3.7.3
  [  ] rabbitmq_trust_store              3.7.3
  [e*] rabbitmq_web_dispatch             3.7.3
  [  ] rabbitmq_web_mqtt                 3.7.3
  [  ] rabbitmq_web_mqtt_examples        3.7.3
  [  ] rabbitmq_web_stomp                3.7.3
  [  ] rabbitmq_web_stomp_examples       3.7.3

• Client library version (for all libraries used): amqplib 0.5.2
• Operating system, version, and patch level: Fedora 27, Kernel 4.15.3-300.fc27.x86_64

[lukebakken]

Thank you for the report and the detailed (and correct) reproduction steps.

https://gist.github.com/lukebakken/7459fbb4fcbfa1062198c36ca0d8fd41

[michaelklishin]

A fix will be included into 3.7.4 and 3.6.16.

[jharting]

@michaelklishin that's awesome! Appreciate the quick turnaround.