Web client hanging forever

[yadid]

Im trying to run flower on my local celery deployment. When I try to connect via web browser to localhost:5555 or via curl , it just hangs forever (waiting for localhost). I can telnet to port 5555, but other than that no response. Below is some debug info:

celery -A myproject flower --broker=redis://localhost:6379/0 --broker_api=redis://localhost:6379/0 --debug --inspect_timeout=10 [I 161120 20:08:01 command:136] Visit me at http://localhost:5555 [I 161120 20:08:01 command:141] Broker: redis://localhost:6379/0 [I 161120 20:08:01 command:144] Registered tasks: [u'app.tasks.generate_scheduled', u'celery.accumulate', u'celery.backend_cleanup', u'celery.chain', u'celery.chord', u'celery.chord_unlock', u'celery.chunks', u'celery.group', u'celery.map', u'celery.starmap', u'app.tasks.generate_result', 'normalize_time_task', u'myplatform.celery.debug_task', u'tasks.bake_block', 'tasks.baker', 'tasks.mailman'] [D 161120 20:08:01 command:146] Settings: {'cookie_secret': 'pnKuJkmuRPW43p1N4KYmvRAZjwFJtE+dvDWUNGrHTl8=', 'debug': True, 'login_url': '/login', 'static_path': '/usr/local/lib/python2.7/site-packages/flower-0.9.1-py2.7.egg/flower/static', 'static_url_prefix': '/static/', 'template_path': '/usr/local/lib/python2.7/site-packages/flower-0.9.1-py2.7.egg/flower/templates'} [D 161120 20:08:01 control:29] Updating all worker's cache... [I 161120 20:08:01 mixins:224] Connected to redis://localhost:6379/0

[toorsukhmeet]

Having the same issue with RabbitMQ broker. Workers work as expected but flower (0.9.1) hangs.

[thenewguy]

I have recently started observing this as well and have not been able to track down what started the issue

[thenewguy]

Master did not fix the issue

[thenewguy]

My problem was ssl connection problems. Unrelated to flower... was a certificates permissions error on the filesystem

[enriquefernandez]

I'm having the same issue with Redis in OSX El Capitan.

Going to: http://localhost:5555/ hangs. It doesn't display anything. In the terminal, flower doesn't complain about anything. It seems to start normally and it even displays the correct names of the registered tasks.

[philejmath]

I've having the same issue. Flower hanging indefinitely. Seems to start normally and displays all registered task.

[Jahangir-Sh]

Same issue, hangs forever. There is nothing descriptive logs on debug mode --debug. Broker is Redis, celery worker runs locally, tasks are registered successfully. When installed worked as expected then appeared hanging, reinstalled, on changes.

[kellanburket]

I ran into this issue while using Redis Sentinel to connect to Flower. Celery couldn't resolve a DNS name I passed in but Flower didn't display an error message. It would hang when I tried to connect to it. It's extremely frustrating that there's no error message when Flower fails to connect to its broker and it's unclear to me why this failure results in an inability to set up the web client.

Here's my start up log:

python3.5 -m celery flower --port=5555 --address=0.0.0.0 --app=app --loglevel=INFO INFO:flower.command:Visit me at http://0.0.0.0:5555 INFO:flower.command:Broker: redis-sentinel://redis-sentinel:16379/0 INFO:flower.command:Registered tasks: ['app.tasks.do_archive_elasticsearch_index_task', 'app.tasks.do_close_elasticsearch_index_task', 'celery.accumulate', 'celery.backend_cleanup', 'celery.chain', 'celery.chord', 'celery.chord_unlock', 'celery.chunks', 'celery.group', 'celery.map', 'celery.starmap']

[MegacoderKim]

Every time i have experienced this is usually a celery config issue with the log suppressed by flower. try invoking the worker alone and see if the logs give any leads celery -A app worker --loglevel=inf0

[ocervell]

Same issue here ... Happened after I switched to eventlet or gevent to run my tasks.

[senyuuri]

I'm having the same issue here.

If you are using WSL/Bash on Windows, this might be a result of a known compatibility issue between amqp 2.1.1 - 2.2.2 and WSL. (It's fixed in amqp 2.2.3 which has not been released at this moment of writing.)

In my case, I was using celery==4.1.0 and amqp==2.1.1. Everything was working fine. However, if I install flower through pip install flower, run celery -A myprojectname flower and open http:\\localhost:5555 in the browser, the page just hangs.

I quickly ran a pip freeze and noticed my amqp version was replace with the current latest 2.2.2. Reinstalling an older version of amqp simply solves the issue. http:\\localhost:5555 now correctly shows the flower dashboard.

Hope this can help.

[djlambert]

Just ran into this issue. I added the following lines to the beginning of my Flask app to fix an issue with websockets.

import eventlet
eventlet.monkey_patch()

Commenting this out resolved the issue with Flower, so as @ocervell mentioned eventlet is involved somehome. Will need to find a workaround.

Update: Downgrading amqp to 2.1.1 did not resolve the issue. Update 2: Starting Flower without specifying the app worked.

[gwsampso]

Having similar problems with apache-airflow==1.9.0 amqp==1.4.9 celery==3.1.25 flower==0.9.2

[shashankkr9]

I encountered similar problem as well with: amqp==2.2.2 celery==4.1.0 flower==0.9.2 kombu==4.1.0

[mberacochea]

Same issue:

flower==0.9.2 celery==4.1.1 amqp==2.3.1 kombu==4.2.0

[erindy]

Having the same problem: kombu (4.1.0) Django (2.0.4) celery (4.1.0) amqp (2.2.2)

[djlambert]

I switched from eventlet to gevent (dependency of another package) and updated the bin/celery script to import gevent and monkey patch before all the other imports. I can now run flower with celery flower --app myapp.app --pool gevent without issue. Also my flower==1.0.0 and celery==4.2.0 now.

[thestevenbell]

Flower was hanging for me, added gevent via pip3 and started with celery flower -A src.celery --address=127.0.0.1 --port=5555 --logLevel=debug --pool gevent requirements.txt

# requirements.txt 
celery[redis]==4.2.1
python-dotenv==0.9.1
cfenv==0.5.3
tornado==5.1
flower==0.9.2
gevent==1.3.6

[tomli1981]

Any solution for this issue ?

[zetaab]

with --pool gevent I cannot see any workers online

[sivang]

I'm getting this when trying to click one the workers and inspect it's configs and details , any idea how to solve this or how to make flower be able to resolve DNS for the worker if that is indeed the problem?

[maciejzukowski]

I performed a 'FLUSHALL' on redis via the redis-cli and then restarted flower and it started to work. EDIT: I think this had to do with BROKER_POOL_LIMIT. When i set it to 0 Flower seems to work.

[maribedran]

Having the same problem with docker. Running directly on the machine flower works normally, but inside the container it hangs forever.

amqp==2.3.2
celery==4.1.1
flower==0.9.2
kombu==4.2.1
tornado==5.1.1

[pdoshi-ns]

I believe you have to start the redis server first. If there is no active redis server, flower does not give you an error but it will just keep looking for one and trying to connect to it resulting in it hanging. So to start a simple redis server just type redis-server in your terminal. After that run flower.

[chenliang0571]

I have the same issue, in my case it can be resolved by adding --config argument.

flower -b 'sentinel://192.168.1.1:26379;sentinel://192.168.1.2:26379' --address=0.0.0.0 --port=5555 --config=airflow.config_templates.default_celery

$ flower --version
0.9.2
4.1.1 (latentcall)
flower   -> flower:0.9.2 tornado:5.1.1 babel:2.6.0
software -> celery:4.1.1 (latentcall) kombu:4.3.0 py:3.6.5
            billiard:3.5.0.5 py-amqp:2.4.1
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:disabled

checkout this jira for detail: https://issues.apache.org/jira/browse/AIRFLOW-5272

[gainskills]

For my case, I examed the connection between flower and rabbitmq / redis, got the issue resolved after the connectivity works well.

[sizzlorox]

I'm not sure if this is the same issue or not but I've been having problems where my celery was getting heartbeat timeout issues with rabbitmq as a broker. I followed a few workarounds where people disabled the hearbeat for rabbitmq and celery. This fixed the timeout problem but ended up causing this bug with flower where about 5 seconds after booting up flower, the webclient would hang forever. After adding the heartbeat back, my flowers webclient has gone back to normal.

[bartolootrit]

I have flower-0.9.3 and tornado-5.1.1. After downgrading Tornado to v4.5.2 Flower works correctly (see https://github.com/mher/flower/issues/788#issuecomment-373622016)

[mher]

Can someone try https://github.com/mher/flower/compare/issue639 fix?

[sillycube]

Any fix in 2020? Mine is working well in dev server, but it won't work in prod server. docker-compose.override.yml

flower:
    image: mher/flower
    environment:
      - CELERY_BROKER_URL=redis://redis:6379
      - FLOWER_PORT=5555
      - FLOWER_BASIC_AUTH=${FLOWER_USER}:${FLOWER_PASSWORD}
    ports:
      - "5555:5555"

[ckcollab]

@sillycube any output from flower? Everything appears to be running fine on prod for us, but we're also experiencing no response/hanging..

docker-compose.yml:

  rabbit:
    image: rabbitmq:3.6-management
    env_file: .env
    ports:
      - ${RABBITMQ_MANAGEMENT_PORT:-15672}:15672
      - ${RABBITMQ_PORT:-5672}:5672

  flower:
    image: mher/flower
    env_file: .env
    environment:
      - CELERY_BROKER_URL=pyamqp://${RABBITMQ_DEFAULT_USER}:${RABBITMQ_DEFAULT_PASS}@rabbit:${RABBITMQ_PORT}//
      - FLOWER_BROKER_API=http://${RABBITMQ_DEFAULT_USER}:${RABBITMQ_DEFAULT_PASS}@rabbit:${RABBITMQ_MANAGEMENT_PORT}/api/
    ports:
      - ${FLOWER_PUBLIC_PORT:-5555}:5555

With RABBITMQ_PORT=9002 this breaks. With RABBITMQ_PORT=5672 it works?!

_EDIT: I added FLOWER_BROKER_API because I'm using a special port for rabbitmq management as well.. still hanging!_

[ckcollab]

I believe the problem has been narrowed down to RABBITMQ_PORT running on a non-default port causing problems? As far as I know CELERY_BROKER_URL is the only variable that needs to account for the different port, am I missing another change that's needed?

[sillycube]

Hi @ckcollab,

I tried to change the ports from 5555:5555 to 5556:5555 and I can access flower with the new link. But I don't know what I did wrong

[ckcollab]

The problem for us ended up being the hostname wasn't resolving over docker properly, on the different port. On port 5672 rabbit docker container hostname worked fine, on 9002 it didn't work.. we had to use docker.for.mac.localhost or the IP of the actual server with port 9002... strangely..

[timothywangdev]

This is due to flower not being able to connect to the broker. If you are docker-compose file, make sure that your broker ip address is the broker container name.

version: '3'
services:
    rabbitmq:
        restart: always
        image: rabbitmq:management
        volumes:
            - "./data/rabbitmq:/var/lib/rabbitmq"
        ports:
            - "5672:5672"
            - "15672:15672"
    flower:  
        image: mher/flower
        command: ["flower", "--broker=amqp://rabbitmq:5672"]  
        ports:  
            - 5555:5555  
        depends_on:
            - rabbitmq

Here, "--broker=amqp://rabbitmq:5672" is used to connect to the "rabbitmq" container Also, turn on the debug option in flower might be helpful.

[cmoiccool]

Hi,

In my case it was able to connect with the broker, so this was not the issue. I tried the version from branch issue639 but it did not solve the problem. The issue i was having was some incompatibility with the Elastic APM client that was installed in my Django App. Removing 'elasticapm.contrib.django', from INSTALLED_APPS solved it.

So for the ones that still have not found how to solve this, this might be one of the possible cause to investigate.

[NBardelot]

I have the same issue with Airflow configured with CeleryExecutor and Flower.

We run everything using the stable Airflow Helm chart, and in the pod that runs Flower I can confirm that I can access my Redis broker using OpenSSL s_client (as our Redis is a TLS-encrypted endpoint):

s_client -connect <redis host>:6379

I provide the remote certificate as a PEM in /etc/ssl/certs. And the encrypted connexion is successfuly created, and I can use commands like AUTH and INFO to talk to my Redis server.

That said, the Flower server itself does not seems to work. It hangs after some initial logs. It is not reachable through port 5555 (even if the socket is listening).

I've published a couple of PRs for puckel's Docker image of Airflow in order to be able to use manual configuration for Airflow using Docker:

https://github.com/puckel/docker-airflow/pull/495 and https://github.com/puckel/docker-airflow/pull/503

And I use the following configuration for every Airflow component including Flower:

AIRFLOW__CELERY__BROKER_URL=redis://:<basic auth>@<redis host>:6379/<queue number>

While it works for Airflow itself (i.e. the scheduler), it does not work for Flower. In my opinion the issue is that Flower does not decrypt the TLS connection the same way as the other Airflow components.

As stated in the Celery documentation:

https://docs.celeryproject.org/en/stable/userguide/configuration.html#id22

This backend requires the result_backend setting to be set to a Redis or Redis over TLS URL: ... Use the rediss:// protocol to connect to redis over TLS:

result_backend = 'rediss://:[email protected]:port/db?ssl_cert_reqs=required'

Note that the ssl_cert_reqs string should be one of required, optional, or none (though, for backwards compatibility, the string may also be one of CERT_REQUIRED, CERT_OPTIONAL, CERT_NONE).

But the issue here is that Airflow does not seem to handle that kind of configuration very well. So at the moment, the way the Airflow Helm chart is done, it's either Airflow working, or Flower working, but not both...

TL;DR

• Flower should log connection issues to the broker
• it should also 'look' at TLS connections and at least warn us if it does not handle it properly
• ... or even better, know how to automatically switch to TLS-enabled (with the system's certificates) if the remote endpoint asks for it

Edit:

See airflow/config_templates/default_celery.py in the Airflow project, where it is clear that rediss:// is not handled for TLS connections, instead of the AIRFLOW__CELERY__SSL_ACTIVE option.

    if celery_ssl_active:
        if 'amqp://' in broker_url:
            broker_use_ssl = {'keyfile': conf.get('celery', 'SSL_KEY'),
                              'certfile': conf.get('celery', 'SSL_CERT'),
                              'ca_certs': conf.get('celery', 'SSL_CACERT'),
                              'cert_reqs': ssl.CERT_REQUIRED}
        elif 'redis://' in broker_url:
            broker_use_ssl = {'ssl_keyfile': conf.get('celery', 'SSL_KEY'),
                              'ssl_certfile': conf.get('celery', 'SSL_CERT'),
                              'ssl_ca_certs': conf.get('celery', 'SSL_CACERT'),
                              'ssl_cert_reqs': ssl.CERT_REQUIRED}
        else:
            raise AirflowException('The broker you configured does not support SSL_ACTIVE to be True. '
                                   'Please use RabbitMQ or Redis if you would like to use SSL for broker.')

Thus, Airflow and Flower cannot share the same configuration. And there is no easy way to provide an alternate configuration to Flower that I can see.

Here is a ticket I've just opened in the Airflow JIRA: https://issues.apache.org/jira/browse/AIRFLOW-6985

[mher]

Please try the latest master version. I've pushed a fix for this issue.

[northeastsquare]

@mher I clone down latest master, but no use