Unable to configure flower with sentinel redis server ( MasterNotFoundError )

[arunporwal]

Describe the bug Created one sentinel Redis server and given the below configuration as:

CELERY_BROKER_URL = 'sentinel://default:<pass>@10.4.158.111:26379/0;sentinel://default:<pass>@10.4.158.128:26379/0;sentinel://default:<pass>@10.4.158.168:26379/0'
CELERY_BROKER_TRANSPORT_OPTIONS = {
    "master_name": "airflowmaster",
    "sentinel_kwargs": {"password": "<pass>"},
}

To Reproduce Steps to reproduce the behavior: Installed airflow and flower Given configuration but getting errors.

Expected behavior A clear and concise description of what you expected to happen. It should connect to Redis Sentinel

Errors :

Mar  2 03:49:55 ubuntu-earth-base airflow[19529]: [#033[34m2022-03-02 03:49:55,628#033[0m] {#033[34mbase_events.py:#033[0m1707} ERROR#033[0m - Future exception was never retrieved
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]: future: <Future finished exception=OperationalError("No master found for 'airflowmaster'")>#033[0m
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]: Traceback (most recent call last):
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:   File "/usr/local/lib/python3.8/dist-packages/kombu/transport/virtual/base.py", line 923, in create_channel
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:     return self._avail_channels.pop()
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]: IndexError: pop from empty list
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]: During handling of the above exception, another exception occurred:
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]: Traceback (most recent call last):
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:   File "/usr/local/lib/python3.8/dist-packages/kombu/connection.py", line 447, in _reraise_as_library_errors
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:     yield
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:   File "/usr/local/lib/python3.8/dist-packages/kombu/connection.py", line 434, in _ensure_connection
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:     return retry_over_time(
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:   File "/usr/local/lib/python3.8/dist-packages/kombu/utils/functional.py", line 312, in retry_over_time
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:     return fun(*args, **kwargs)
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:   File "/usr/local/lib/python3.8/dist-packages/kombu/connection.py", line 878, in _connection_factory
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:     self._connection = self._establish_connection()
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:   File "/usr/local/lib/python3.8/dist-packages/kombu/connection.py", line 813, in _establish_connection
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:     conn = self.transport.establish_connection()
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:   File "/usr/local/lib/python3.8/dist-packages/kombu/transport/virtual/base.py", line 947, in establish_connection
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:     self._avail_channels.append(self.create_channel(self))
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:   File "/usr/local/lib/python3.8/dist-packages/kombu/transport/virtual/base.py", line 925, in create_channel
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:     channel = self.Channel(connection)
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:   File "/usr/local/lib/python3.8/dist-packages/kombu/transport/redis.py", line 679, in __init__
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:     self.client.ping()
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:   File "/usr/local/lib/python3.8/dist-packages/redis/commands/core.py", line 954, in ping
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:     return self.execute_command("PING", **kwargs)
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:   File "/usr/local/lib/python3.8/dist-packages/redis/client.py", line 1173, in execute_command
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:     conn = self.connection or pool.get_connection(command_name, **options)
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:   File "/usr/local/lib/python3.8/dist-packages/redis/connection.py", line 1370, in get_connection
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:     connection.connect()
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:   File "/usr/local/lib/python3.8/dist-packages/redis/sentinel.py", line 54, in connect
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:     return self.retry.call_with_retry(
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:   File "/usr/local/lib/python3.8/dist-packages/redis/retry.py", line 50, in call_with_retry
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:     raise error
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:   File "/usr/local/lib/python3.8/dist-packages/redis/retry.py", line 45, in call_with_retry
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:     return do()
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:   File "/usr/local/lib/python3.8/dist-packages/redis/sentinel.py", line 44, in _connect_retry
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:     self.connect_to(self.connection_pool.get_master_address())
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:   File "/usr/local/lib/python3.8/dist-packages/redis/sentinel.py", line 117, in get_master_address
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:     master_address = self.sentinel_manager.discover_master(self.service_name)
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:   File "/usr/local/lib/python3.8/dist-packages/redis/sentinel.py", line 251, in discover_master
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]:     raise MasterNotFoundError(f"No master found for {service_name!r}")
Mar  2 03:49:55 ubuntu-earth-base airflow[19529]: redis.sentinel.MasterNotFoundError: No master found for 'airflowmaster'

Screenshots If applicable, add screenshots to help explain your problem.

System information Output of python -c 'from flower.utils import bugreport; print(bugreport())' command

flower   -> flower:1.0.0 tornado:6.1 humanize:3.13.1
software -> celery:5.2.3 (dawn-chorus) kombu:5.2.3 py:3.8.10
            billiard:3.6.4.0 redis:4.1.4
platform -> system:Linux arch:64bit, ELF
            kernel version:5.4.0-100-generic imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:sentinel results:disabled

deprecated_settings: None

Also tried with code and it is working in python code.

root@airflow-mn-1:~/airflow# python3
Python 3.8.10 (default, Nov 26 2021, 20:14:08)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from redis.sentinel import Sentinel
>>> sentinel = Sentinel([('10.4.158.128', 26379),
...                      ('10.4.158.168',26379),
...                      ('10.4.158.111',26379)], socket_timeout = None, sentinel_kwargs={'password': '<pass>'})
>>> r = sentinel.discover_master('airflowmaster')
>>> print(r)
('10.4.158.128', 6379)
>>>

[arunporwal]

Can anyone help here?

[arunporwal]

The issue with the Redis cluster version. Airflow is not supporting the latest Redis Sentinel cluster.

[ahmedaljawahiry]

Also experiencing this issue.

Looking at the broker for RedisSentinel, here, the sentinel_kwargs are never passed through when instantiating Sentinel, which leads to this error (related: https://github.com/redis/redis-py/issues/1388#issuecomment-681960193).

[ross919]

@ahmedaljawahiry how to pass master_name? I run flower with docker compose. From this example, https://github.com/mher/flower/blob/master/docker-compose.yml how to pass master_name?

[ross919]

@mher could you help me with this issue?

[lookandhate]

@ross919 Have you figured it out?