Implement the ability in Ansible Kolla to deploy RabbitMQ with inter-node comunication encrypted via TLS

[fdobrovolny]

Epic #462

As an SCS Operator, I want to be able to deploy RabbitMQ with TLS encryption enabled between the RabbitMQ cluster nodes so that I can have OpenStack services communication E2E encrypted.

When this is implemented #469 is immidietly required as cli would stop working:

Once a node has inter-node connection configured with TLS, CLI tools such as rabbitmqctl and rabbitmq-diagnostics also must use TLS to talk to the node. Plain TCP connections will be fail.

Therefore:

As an SCS Operator, I want to be able to use rabbitmqctl with the RabbitMQ cluster with TLS encryption enabled between the RabbitMQ cluster nodes, so that I can use CLI tools for debugging.

https://www.rabbitmq.com/clustering-ssl.html

As pointed out below there is some capability regarding TLS of Kolla in RabbitMQ as of now mainly rabbitmq_enable_tls and om_rabbitmq_enable_tls. This should be extended to fulfill all objectives.

Definition of Ready:

• [ ] User Story is clear and understood by the whole team
• [ ] Acceptance criteria are defined
• [ ] Acceptance criteria are clear and understood by the whole team

Definition of Done:

• [x] Ansible Kolla is able to deploy RabbitMQ with TLS encryption enabled between nodes
• [x] CLI tools are configured to use TLS to talk to the RabbitMQ cluster.
• [ ] The patch has been submitted to upstream and merged.
• [ ] Change has been included in the testbed.
• [ ] CI tests have run successfully
• [ ] Documentation has been updated
• [ ] Release Notes have been updated

[berendt]

I think this is already implemented:

https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/rabbitmq/templates/rabbitmq.conf.j2#L23-L30

[fdobrovolny]

@berendt I think you are correct, I was going of off the docs page:

https://docs.openstack.org/kolla-ansible/latest/reference/message-queues/rabbitmq.html

But I think you commented under the wrong issue I think it is #469.

[berendt]

Should be the right issue. I will enable the RabbitMQ inter-node encryption. This will also enable the client-server traffic encryption.

When om_rabbitmq_enable_tls is true (it defaults to the value of rabbitmq_enable_tls), applicable OpenStack services will be configured to use oslo.messaging with TLS enabled.

[fdobrovolny]

I think: rabbitmq_enable_tls enables SSL listener rather then https://www.rabbitmq.com/clustering-ssl.html om_rabbitmq_enable_tls enables Oslo messaging to use the TLS endpoint.

As the docs state:

Encryption of the following channels is not currently supported:

• RabbitMQ cluster traffic between RabbitMQ server nodes
• RabbitMQ CLI communication with RabbitMQ server nodes
• RabbitMQ Management API and UI (backend connection from HAProxy to RabbitMQ)

But I think I wrote the #469 wrong as it was supposed to address the "RabbitMQ CLI communication with RabbitMQ server nodes" and now looking into it closer seems like that was meant like configure the CLI client to use the TLS endpoint.

As for the inter-node communication it requires the inet_tls flag is not present https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/rabbitmq/templates/rabbitmq-env.conf.j2#L5

[berendt]

om_rabbitmq_enable_tls defaults to the value of rabbitmq_enable_tls. With https://github.com/osism/testbed/issues/1826 I will set rabbitmq_enable_tls to true (this is related to this issue) and this will results in om_rabbitmq_enable_tls = true as well.

[berendt]

As for the inter-node communication it requires the inet_tls flag is not present https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/rabbitmq/templates/rabbitmq-env.conf.j2#L5

I think we should add this with a check on rabbitmq_enable_tls.

[MatusJenca2]

This issue progress (December):

• rabbitmq internode tls configured properly outside Kolla in Docker
• TLS configuration file and task added to Ansible playbook Problems:
• can't start RabbitMQ, possibly some config error
• I have no capacity to test in HA mode (the internode communication itself needs it), I need testbed or something

[horazont]

As mentioned in https://github.com/SovereignCloudStack/issues/issues/462#issuecomment-1884472190 we have seen spurious TLS errors with RabbitMQ cluster encryption ("certificate has expired" even though it hadn't). This was not reproducible. In case you can reproduce this, we'd be interested to hear about that.

For general configuration of RabbitMQ TLS encryption, you can look at the steps the YAOOK project did to enable cluster encryption for RabbitMQ here:

• https://gitlab.com/yaook/operator/-/blob/devel/yaook/op/infra/templates/amqp-statefulset.yaml?ref_type=heads#L60-79
• https://gitlab.com/yaook/operator/-/blob/devel/yaook/op/infra/templates/amqp-inter_node_tls.config?ref_type=heads

[fdobrovolny]

In yaook there is basicaly all stuff about the RabbitMQ already done we can use it as inspiration for ansiblle kolla.

[berendt]

In yaook there is basicaly all stuff about the RabbitMQ already done we can use it as inspiration for ansiblle kolla.

Is there any progress on the inter RabbbitMQ communication?

[MatusJenca2]

I just started working on this again, while I am waiting for the other upstream work to be reviewed/merged Progress

• [x] Create test environment (3 RabbitMQ nodes)
• [x] Configure intertnode TLS
• [x] Verify if internode TLS is utilized (tcpdump?)
• [x] Create upstream PR/patch
• [ ] Upstream patch merged

[MatusJenca2]

Upstream PR: https://review.opendev.org/c/openstack/kolla-ansible/+/921381

Upstream PR (certificates):https://review.opendev.org/c/openstack/kolla-ansible/+/921380