Airflow 1.10.14 - celery worker doesn't want to connect to redis.

Apache Airflow version: 1.10.14

Kubernetes version (if you are using kubernetes) (use kubectl version):

Environment: Red Hat Enterprise Linux 8.2 (VMware)

Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release): Red Hat Enterprise Linux 8.2
Kernel (e.g. uname -a): 4.18.0-193.19.1.el8_2.x86_64 #1 SMP Wed Aug 26 15:29:02 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
Install tools: Airflow 1.10.14, redis.x86_64 (version 5.0.3.2), redis client installed with airflow (redis 3.5.3)
Others:

What happened:

We have upgraded version of airflow from 1.10.2 to 1.10.14 and there is a problem to connect to redis by celery worker. This is really wierd problem because when we run a airflow celery worker (without -D parameter) it's connecting without problem. In Flower tool we can see redis connection is online, but when we try do it with -D, airflow celery worker -D there are errors like below.

[2021-02-04 17:33:00,442: ERROR/MainProcess] consumer: Cannot connect to redis://localhost:6379/0: Error 107 connecting to redis:6379. Operation on closed file.. Trying again in 12.00 seconds... (6/100) Before upgrade on version 1.10.2, we didin't have a problems like that. Our configuration has been working with settings broker_url = redis://localhost:6379/0

Of course our redis is working on default port on this same machine netstat-ntlp

tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN -

Now we have a settings IN airflow cfg we have setting like below:

broker_url = redis://localhost/0 We have tried also settings like below:

broker_url = redis://localhost:6379/0 broker_url = redis://127.0.0.1/0 broker_url = redis://127.0.0.1:6379/0 broker_url = redis://redis:6379/0 broker_url = redis://redis/0

What you expected to happen:

We would like to connect to redis on airflow celery deamon functionality.

We do not really know, redis server version is this same, when airflow 1.10.2 was connecting sucessfully. Later Version of airflow and redis client has been upgraded.

How to reproduce it:

Anything else we need to know:

apache / airflow

Airflow 1.10.14 - celery worker doesn't want to connect to redis. #14100