flower broker page shows error 500

[tobiasdam]

Accessing the /broker page results in a HTTP 500 status code:

The flower flogs show the following errors:

[I 240716 13:07:19 mixins:228] Connected to redis://:**@redis-service:6379/0
[E 240717 16:46:46 broker:31] Unable to get queues: 'Error while reading from redis-service:6379 : (104, 'Connection reset by peer')'
[E 240717 16:46:46 web:1871] Uncaught exception GET /broker (10.1.75.6)
    HTTPServerRequest(protocol='http', host='<redacted>', method='GET', uri='/broker', version='HTTP/1.1', remote_ip='<redacted>')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.11/site-packages/tornado/web.py", line 1786, in _execute
        result = await result
                 ^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/flower/views/broker.py", line 35, in get
        queues=queues)
               ^^^^^^
    UnboundLocalError: cannot access local variable 'queues' where it is not associated with a value
client_loop: send disconnect: Broken pipe

The second one is just a follow-up error caused by the first one. As the logs say Connected to redis just a line before, the Connection reset by peer error is curious.

Debugging showed that the code resulting in Connected to redis actually opens a ssl connection to redis.

After some debugging, the logs of redis give further info:

# Error accepting a client connection: error:0A00010B:SSL routines::wrong version number

Rechecking the docs made clear that I used a wrong connector url. I used the following connector url for both celery workers and flower:

redis://:$RAUTH@redis-service:6379/0?ssl_cert_reqs=required

According to the docs you need to use rediss instead of redis for ssl/tls connections.

There are 2 confusing things about the behaviour shown:

• using redis: in conjunction with ?ssl_cert_reqs=required works and establishes a tls connection for celery workers. It does not work with flower as is uses its own code to establish a redis connection and as far as I know does not take ?ssl_cert_reqs=required into account. The behaviour of flower makes more sense, the error message could be better though.
• The info log message stating Connected to redis followed by a connection error to redis is misleading. It results from celery and flower using different code to establish a redis connection. Maybe flower can somehow use the broker connection from celery directly?

Solution

Use the correct connection url with rediss: instead of redis: