rabbitmq / messaging-topology-operator

RabbitMQ messaging topology operator
Mozilla Public License 2.0
123 stars 64 forks source link

Federation not working for quorum #844

Open pay-11-m opened 1 month ago

pay-11-m commented 1 month ago

Hi Team,

I have deployed 3 node RMQ cluster and operator on Azure kubernetes cluster.

Versions: RMQ: 3.13.2-management RMQ cluster operator: rabbitmqoperator/cluster-operator:2.9.0 controller-gen.kubebuilder.io/version: v0.14.0 RMQ messaging topology operator: rabbitmqoperator/messaging-topology-operator:1.14.2 controller-gen.kubebuilder.io/version: v0.15.0 mtls is on in downstream, and certificates are given in upstream.

Since classic has been deprecated, we have upgraded to quorum. The federation used before was as below: apiVersion: rabbitmq.com/v1beta1 kind: Federation metadata: name: federation-example spec: name: "origin" vhost: "downstream" uriSecret: name: federation-uri ackMode: "on-confirm" rabbitmqClusterReference: name: example-rabbit

But it was showing error. The federation link created in Upstream queue refers to classic, whereas all the queues are in quorum. Also the vhost in both Upstream and downstream have defaultQueueType is quorum.

Could you please support?

But when I added it through dashboard, and applied queuetype as quorum, it is working fine. But, how can I add queuetype in yaml file. I added as below but even if it didn't show any error while applying, it is not taking that configuration. apiVersion: rabbitmq.com/v1beta1 kind: Federation metadata: name: federation-example spec: name: "origin" vhost: "downstream" queue: quorum

I would really appreciate if you could support at earliest.

Zerpet commented 3 weeks ago

It would be great if you could provide formatted YAML, that we can just apply and reproduce this issue. Read this on how to format code blocks, and this one on basic formatting. It would be helpful if you could provide the error you are observing.

pay-11-m commented 2 weeks ago

Hi @Zerpet .

Please find below details for recreating the issue:

Earlier versions we used in both RabbitMQ before upgrade were as below ->

RABBITMQ Image : 3.11.18-management CRDS:

  1. RABBITMQ-CLUSTER-OPERATOR controller-gen.kubebuilder.io/version: v.13.0 rabbitmqoperator/messaging-topology-operator:1.13.0
  2. MESSAGING-TOPOLOGY-OPERATOR controller-gen.kubebuilder.io/version: v.14.0 rabbitmqoperator/cluster-operator:2.7.0

Versions we used in both RabbitMQ after upgrade are as below ->

RABBITMQ Image : 3.13.2-management CRDS:

  1. RABBITMQ-CLUSTER-OPERATOR controller-gen.kubebuilder.io/version: v.14.0 rabbitmqoperator/messaging-topology-operator:1.14.2
  2. MESSAGING-TOPOLOGY-OPERATOR controller-gen.kubebuilder.io/version: v.15.0 rabbitmqoperator/cluster-operator:2.9.0

We have enabled mtls in one of the rabbitmq.

Scenario:

  1. Before upgrade Federation and Shovel was working fine with both amqp and amqps in URI. URI federation: amqps://username>:<password>@<hostname/ URI shovel: destination -> amqps://username>:<password>@<hostname/ source -> amqp://username>@/<vhost Both were working
  2. After upgrade, both shovel and federation were stuck in error/starting. So, we changed amqps to amqp in URI and at least Shovel started working. URI shovel: destination -> amqp://username>:<password>@<hostname/ source -> amqp://username>@/<vhost

    But, even after changing amqps to amqp in federation URI, it still stucked in starting state. Format for federation in yaml is same as given in above question.

  3. Since we couldn't create it again using yaml, we tried to create it from dashboard and still faced the error. But, when we applied queue type = quorum from available option, we were able to get the federation in working. Screenshot 2024-07-10 163724

Issue is, I am unabke to get the federated queue type as quorum in upstream RMQ. As shown in attached screenshot, federated queue type is quorum for the one created by dashboard with parameter queue type = quorum.

I will really appreciate if you could you let me know how to add parameter queue type in kind: federation? https://www.rabbitmq.com/docs/federation-reference

Thank you so much for the support.

Zerpet commented 2 weeks ago

I'm afraid it is not possible to define the queue type in Federation YAML spec. We need a new feature for that. AMQPs and mTLS may not work correctly in RabbitMQ image 3.13.x because it has Erlang 26, which is more strict about TLS configuration. I can try to reproduce this problem locally.