search

mher / flower

Real-time monitor and web admin for Celery distributed task queue
https://flower.readthedocs.io
Other
6.5k stars 1.09k forks source link

Unable to connect to rabbitmq broker that runs with self-signed certificates #1349

Open henkesde92 opened 10 months ago

henkesde92 commented 10 months ago

Describe the bug I'm unable to run flower when rabbitmq is running with TLS with self-signed certificates. However, when I disable TLS on both components, everything is running correctly.

When using rabbitmq with TLS, I run it on port 5671/15671, without it's running on 15672/5672.

Here are both commands how I want to run flower respectively:

TLS: /usr/local/bin/celery --app celery_flower --broker=amqp://<username>:<password>@<servername>:5671//?ssl=1 flower --broker_api=https://<username>:<password>@<servername>:15671/api/ --persistent --db="/root/flower/data/flower.db"

non-TLS: /usr/local/bin/celery --app celery_flower --broker="pyamqp://<username>:<password>@<servername>:5672//" flower --broker_api="http://<username>:<password>@<servername>:15672/api/" --persistent --db="/root/flower/data/flower.db"

The "non-TLS" example also works when I use "amqp" instead of "pyamqp". I already tried a lot of different approaches to get it working, but somehow I think that the issue is correlated to the self-signed certificates on the rabbitmq side.

I can also find the following logs on the rabbitmq-server:

2024-01-16 10:56:18.043985+01:00 [notice] <0.1962.0> TLS server: In state wait_finished at tls_record_1_3.erl:213 generated SERVER ALERT: Fatal - Bad Record MAC 2024-01-16 10:56:18.043985+01:00 [notice] <0.1962.0> - {record_type_mismatch,21} 2024-01-16 10:56:18.239173+01:00 [notice] <0.1968.0> TLS server: In state wait_finished at tls_record_1_3.erl:213 generated SERVER ALERT: Fatal - Bad Record MAC 2024-01-16 10:56:18.239173+01:00 [notice] <0.1968.0> - {record_type_mismatch,21} 2024-01-16 10:56:18.387667+01:00 [notice] <0.1974.0> TLS server: In state wait_finished at tls_record_1_3.erl:213 generated SERVER ALERT: Fatal - Bad Record MAC 2024-01-16 10:56:18.387667+01:00 [notice] <0.1974.0> - {record_type_mismatch,21} 2024-01-16 10:56:18.414899+01:00 [notice] <0.1986.0> TLS server: In state wait_finished at tls_record_1_3.erl:213 generated SERVER ALERT: Fatal - Bad Record MAC 2024-01-16 10:56:18.414899+01:00 [notice] <0.1986.0> - {record_type_mismatch,21} 2024-01-16 10:56:18.415532+01:00 [notice] <0.1980.0> TLS server: In state wait_finished at tls_record_1_3.erl:213 generated SERVER ALERT: Fatal - Bad Record MAC 2024-01-16 10:56:18.415532+01:00 [notice] <0.1980.0> - {record_type_mismatch,21} 2024-01-16 10:56:18.591205+01:00 [notice] <0.1993.0> TLS server: In state wait_finished at tls_record_1_3.erl:213 generated SERVER ALERT: Fatal - Bad Record MAC 2024-01-16 10:56:18.591205+01:00 [notice] <0.1993.0> - {record_type_mismatch,21} 2024-01-16 10:56:18.591379+01:00 [notice] <0.2002.0> TLS server: In state wait_finished at tls_record_1_3.erl:213 generated SERVER ALERT: Fatal - Bad Record MAC 2024-01-16 10:56:18.591379+01:00 [notice] <0.2002.0> - {record_type_mismatch,21} 2024-01-16 10:56:18.591668+01:00 [notice] <0.2001.0> TLS server: In state wait_finished at tls_record_1_3.erl:213 generated SERVER ALERT: Fatal - Bad Record MAC 2024-01-16 10:56:18.591668+01:00 [notice] <0.2001.0> - {record_type_mismatch,21} 2024-01-16 10:56:22.913618+01:00 [notice] <0.2015.0> TLS server: In state wait_finished at tls_record_1_3.erl:213 generated SERVER ALERT: Fatal - Bad Record MAC 2024-01-16 10:56:22.913618+01:00 [notice] <0.2015.0> - {record_type_mismatch,21} 2024-01-16 10:56:27.918545+01:00 [notice] <0.2021.0> TLS server: In state wait_finished at tls_record_1_3.erl:213 generated SERVER ALERT: Fatal - Bad Record MAC 2024-01-16 10:56:27.918545+01:00 [notice] <0.2021.0> - {record_type_mismatch,21}

These logs don't show up anymore as soon as I stop the flower server. Consumer and producers are working as expected with rabbitmq running with TLS.

Can somebody help me to figure out my issue?

gabrielmocanu commented 3 months ago

Hey @henkesde92,

I have the same problem. Have you found any way to resolve this?

michaelklishin commented 1 month ago

record_type_mismatch from a TLS 1.3 module in Erlang/OTP usually suggests that the Erlang version is not compatible with the OpenSSL/LibreSSL version used.

This question completely leaves out all version information, so the only thing that can be added is that there is a dedicated doc guide on troubleshooting TLS, which has been around for a few years.

For example, if openssl s_client triggers the same exception in node logs, then something is clearly off with the TLS implementation in your Erlang/OTP installation. RabbitMQ does not implement TLS.

OpenSSL 3.x is compatible with Erlang 26+, which is the only series modern RabbitMQ release series support.