search

mher / flower

Real-time monitor and web admin for Celery distributed task queue
https://flower.readthedocs.io
Other
6.48k stars 1.09k forks source link

flower not showing anything in task and monitor tab #895

Open Ian-Re opened 5 years ago

Ian-Re commented 5 years ago

Hey everyone,

I've been searching the web and github for a while plus asking on forums but to no avail can I get this working. I feel like I am missing something.

Capture Capture1 Capture3 Capture4

Here you can see that my dashboard shows everything concerning tasks, even when retried, failed,etc.

sadly the monitor and tasks tab always remain empty and the broker tab shows info but it never changes. what is seen on that screenshot of the broker is all I ever see.

All my workers return pongs when i ping them and there status is always up and running.

Here is the config of my project:

 celery -A myproject report

software -> celery:4.3.0 (rhubarb) kombu:4.5.0 py:3.6.7
            billiard:3.6.0.0 py-amqp:2.4.2
platform -> system:Linux arch:64bit, ELF
            kernel version:4.15.0-47-generic imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:pyamqp results:rpc:///

broker_url: 'amqp://ian:********@localhost:5672/iansvhost'
result_backend: 'rpc:///'
task_serializer: 'pickle'
result_serializer: 'pickle'
accept_content: ['json', 'pickle']
enable_utc: True
timezone: 'Africa/Johannesburg'
worker_send_task_events: True
task_send_sent_event: True
task_track_started: True

Furthermore here's how I start flower:

celery flower -A myproject --broker=pyamqp://ian:***@localhost:5672/iansvhost --broker_api=http://ian:***@localhost:15672/api/

Rabbitmq is running on port 5672 with nginx acting as reverse proxy.

I do see an occasional error when running flower from the terminal:


[I 190422 05:56:53 command:136] Visit me at http://localhost:5555
[I 190422 05:56:53 command:141] Broker: amqp://ian:**@localhost:5672/iansvhost
[I 190422 05:56:53 command:144] Registered tasks:
    ['celery.accumulate',
     'celery.backend_cleanup',
     'celery.chain',
     'celery.chord',
     'celery.chord_unlock',
     'celery.chunks',
     'celery.group',
     'celery.map',
     'celery.starmap',
     'myproject.defer_me']
[I 190422 05:56:53 mixins:229] Connected to amqp://ian:**@127.0.0.1:5672/iansvhost
[E 190422 05:59:53 events:125] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
[I 190422 05:59:54 mixins:229] Connected to amqp://ian:**@127.0.0.1:5672/iansvhost

Please let me know if any other information is needed. I'm still learning and this is my first exposure to celery and flower so any help will really be appreciated.

Thanks in advance

Ian-Re commented 5 years ago

Still haven't been able to figure this out, but I've noticed I'm getting an enormous amount of these errors:

May 20 07:42:24 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [E 190520 07:42:24 events:125] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
May 20 07:42:25 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [I 190520 07:42:25 mixins:229] Connected to amqp://ian:**@127.0.0.1:5672/iansvhost
May 20 07:45:25 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [E 190520 07:45:25 events:125] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
May 20 07:45:26 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [I 190520 07:45:26 mixins:229] Connected to amqp://ian:**@127.0.0.1:5672/iansvhost
May 20 07:48:26 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [E 190520 07:48:26 events:125] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
May 20 07:48:28 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [I 190520 07:48:28 mixins:229] Connected to amqp://ian:**@127.0.0.1:5672/iansvhost
May 20 07:51:28 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [E 190520 07:51:28 events:125] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
May 20 07:51:29 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [I 190520 07:51:29 mixins:229] Connected to amqp://ian:**@127.0.0.1:5672/iansvhost
May 20 07:54:29 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [E 190520 07:54:29 events:125] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
May 20 07:54:30 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [I 190520 07:54:30 mixins:229] Connected to amqp://ian:**@127.0.0.1:5672/iansvhost
May 20 07:57:30 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [E 190520 07:57:30 events:125] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
May 20 07:57:31 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [I 190520 07:57:31 mixins:229] Connected to amqp://ian:**@127.0.0.1:5672/iansvhost
May 20 08:00:31 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [E 190520 08:00:31 events:125] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
May 20 08:00:32 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [I 190520 08:00:32 mixins:229] Connected to amqp://ian:**@127.0.0.1:5672/iansvhost
May 20 08:03:32 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [E 190520 08:03:32 events:125] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
May 20 08:03:33 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [I 190520 08:03:33 mixins:229] Connected to amqp://ian:**@127.0.0.1:5672/iansvhost
May 20 08:06:33 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [E 190520 08:06:33 events:125] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
May 20 08:06:34 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [I 190520 08:06:34 mixins:229] Connected to amqp://ian:**@127.0.0.1:5672/iansvhost
May 20 08:09:34 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [E 190520 08:09:34 events:125] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
May 20 08:09:35 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [I 190520 08:09:35 mixins:229] Connected to amqp://ian:**@127.0.0.1:5672/iansvhost
May 20 08:12:35 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [E 190520 08:12:35 events:125] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
May 20 08:12:36 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [I 190520 08:12:36 mixins:229] Connected to amqp://ian:**@127.0.0.1:5672/iansvhost
May 20 08:15:36 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [E 190520 08:15:36 events:125] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
May 20 08:15:37 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [I 190520 08:15:37 mixins:229] Connected to amqp://ian:**@127.0.0.1:5672/iansvhost
May 20 08:18:37 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [E 190520 08:18:37 events:125] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
May 20 08:18:38 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [I 190520 08:18:38 mixins:229] Connected to amqp://ian:**@127.0.0.1:5672/iansvhost
May 20 08:21:38 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [E 190520 08:21:38 events:125] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
May 20 08:21:39 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [I 190520 08:21:39 mixins:229] Connected to amqp://ian:**@127.0.0.1:5672/iansvhost
May 20 08:24:39 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [E 190520 08:24:39 events:125] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
May 20 08:24:40 ubuntu-s-1vcpu-1gb-fra1-01 flower[888]: [I 190520 08:24:40 mixins:229] Connected to amqp://ian:**@127.0.0.1:5672/iansvhost
Kub-AT commented 5 years ago

Since 2 weeks I also have problem with monitoring tab ( "Queued tasks" graph is not available) and going to worker details gives me "Unknown worker ".

Celery inspect method fails.

[D 190610 08:19:16 selector_events:53] Using selector: EpollSelector
[I 190610 08:19:16 command:136] Visit me at http://0.0.0.0:6666
[I 190610 08:19:16 command:141] Broker: redis://AMAZON-REDIS
[I 190610 08:19:16 command:144] Registered tasks: 
    ['celery.accumulate',
     'celery.backend_cleanup',
     'celery.chain',
     'celery.chord',
     'celery.chord_unlock',
     'celery.chunks',
     'celery.group',
     'celery.map',
     'celery.starmap']
[D 190610 08:19:16 command:146] Settings: {'cookie_secret': b'COOKIE',
     'debug': True,
     'login_url': '/testy/login',
     'static_path': '/usr/local/lib/python3.7/site-packages/flower/static',
     'static_url_prefix': '/testy/static/',
     'template_path': '/usr/local/lib/python3.7/site-packages/flower/templates'}
[D 190610 08:19:16 control:29] Updating all worker's cache...

[I 190610 08:19:16 mixins:229] Connected to redis://....
[W 190610 08:19:19 control:44] 'stats' inspect method failed
[W 190610 08:19:19 control:44] 'active_queues' inspect method failed
[W 190610 08:19:19 control:44] 'registered' inspect method failed
[W 190610 08:19:19 control:44] 'scheduled' inspect method failed
[W 190610 08:19:19 control:44] 'active' inspect method failed
[W 190610 08:19:19 control:44] 'reserved' inspect method failed
[W 190610 08:19:19 control:44] 'revoked' inspect method failed
[W 190610 08:19:19 control:44] 'conf' inspect method failed
[D 190610 08:19:19 app:62] Successfully updated worker cache
Kub-AT commented 5 years ago

@Ian-Re try downgrade or upgrade kombu https://github.com/mher/flower/issues/909

Ian-Re commented 5 years ago

Hey @Kub-AT , thanks for the help!

Sadly I've upgraded and downgraded the packages, especially kombu as shown in that post but the issue persists ¯_(ツ)_/¯

stormlifter commented 5 years ago

Experiencing same issue given same setup. Tried upgrade and downgrade of kombu

DAIXK commented 5 years ago

@Ian-Re I've same issue, if you use flower with nginx, you can try it. my nginx config location ~ ^/flower/? { rewrite ^/flower/?(.*)$ /$1 break; sub_filter '="/' '="/flower/'; sub_filter_last_modified on; sub_filter_once off; proxy_pass http://127.0.0.1:5555; proxy_redirect off; proxy_set_header Host $host; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_http_version 1.1; } my flower: Name: flower Version: 0.9.3 Summary: Celery Flower Home-page: https://github.com/mher/flower Author: Mher Movsisyan Author-email: mher.movsisyan@gmail.com License: BSD Location: /usr/local/lib/python3.5/dist-packages Requires: pytz, celery, futures, tornado, babel Required-by:

============================

static/js/flower.js

function url_prefix() { var url_prefix = $('#url_prefix').val(); if (url_prefix) { if (url_prefix.startsWith('/')) { return url_prefix; } else { return '/' + url_prefix; } } return ''; }

change to >>>>>>>>>>>>>>>>>>>>>>

function url_prefix() { var url_prefix = $('#url_prefix').val(); if (url_prefix) { if (url_prefix.startsWith('/')) { return '/flower'+url_prefix; } else { return '/flower'+'/' + url_prefix; } } return '/flower'; }

alanrhannah commented 5 years ago

I concur that the change highlighted by @DAIXK above works for me, when I insert a breakpoint and update the return statement in the function. The issue is that I re-install flower on every deployment, so I'd have to make this change every time.

function url_prefix() {
    var url_prefix = $('#url_prefix').val();
    if (url_prefix) {
        if (url_prefix.startsWith('/')) {
            return '/flower' + url_prefix;
        } else {
            return '/flower' + '/' + url_prefix;
        }
    }
    return '/flower';
}
Ian-Re commented 5 years ago

I missed the "static/js/flower.js" in the code linked by @DAIXK , It seems to be going a lot better now. Here's some screenshots of what it looks like now:

5 1 2 4


Here's the status output of flower:
celery flower -A myproject --broker=pyamqp://user:password@localhost:5672/iansvhost --broker_api=http://user:password,@localhost:15672/api/

Aug 02 11:25:56 ubuntu-s-1vcpu-1gb-fra1-01 flower[4500]:      'celery.chord',
Aug 02 11:25:56 ubuntu-s-1vcpu-1gb-fra1-01 flower[4500]:      'celery.chord_unlock',
Aug 02 11:25:56 ubuntu-s-1vcpu-1gb-fra1-01 flower[4500]:      'celery.chunks',
Aug 02 11:25:56 ubuntu-s-1vcpu-1gb-fra1-01 flower[4500]:      'celery.group',
Aug 02 11:25:56 ubuntu-s-1vcpu-1gb-fra1-01 flower[4500]:      'celery.map',
Aug 02 11:25:56 ubuntu-s-1vcpu-1gb-fra1-01 flower[4500]:      'celery.starmap',
Aug 02 11:25:56 ubuntu-s-1vcpu-1gb-fra1-01 flower[4500]:      'myproject.defer_me',
Aug 02 11:25:56 ubuntu-s-1vcpu-1gb-fra1-01 flower[4500]:      'myproject.edata',
Aug 02 11:25:56 ubuntu-s-1vcpu-1gb-fra1-01 flower[4500]:      'myproject.jdata']
Aug 02 11:25:56 ubuntu-s-1vcpu-1gb-fra1-01 flower[4500]: [I 190802 11:25:56 mixins:229] Aug 02 11:25:56 flower[4500]: [I 190802 11:25:56 mixins:229] Connected to amqp://user:**@127.0.0.1:5672/iansvhost
Aug 02 11:28:56 ubuntu-s-1vcpu-1gb-fra1-01 flower[4500]: [E 190802 11:28:56 events:125] Failed to capture events: '[Errno 104] Connection reset by peer', trying again in 1 seconds.
Aug 02 11:28:57 ubuntu-s-1vcpu-1gb-fra1-01 flower[4500]: [I 190802 11:28:57 mixins:229] Connected to amqp://user:**@127.0.0.1:5672/iansvhost

It looks like everything has started to work except that I can't see anything in the graphs on the monitor tab.

Anyone know what else could be wrong! Thanks for all the help thus far!

sshishov commented 5 years ago

I will try to investigate and fix in free time

nabobalis commented 5 years ago

Yeah, this happened when I was reverse proxying flower behind NGINX like DAIXK has.

sobolevn commented 5 years ago

This also happens to me. I have inconsistency between local (docker-compose) and production (nginx) versions. And while I am able to fix the production version with ugly hacks (like rewriting URLs), I am not able to fix the development version. Since there's literally no way to rewrite a url / response content.

So, I ended up with different FLOWER_URL_PREFIX env options for development and production. I set FLOWER_URL_PREFIX='' in dev and FLOWER_URL_PREFIX='/flower in production. It is ugly, but it works.

luvwinnie commented 4 years ago

@Ian-Re Hi i'm facing the issues of flower not showing anything in realtime with nginx reverse proxy, can you provide some advices for settings the nginx setting for flower? the below is my nginx configuration.

flower --port=5555 --broker=redis://redis:6379 --loglevel=info --basic_auth=$FLOWER_ID:$FLOWER_PASS --broker=redis://redis:6379'
location ~ ^/flower/? {
    auth_basic "Restricted";
    auth_basic_user_file /etc/nginx/.htpasswd;
    rewrite ^/flower/?(.*)$ /$1 break;

    sub_filter '="/' '="/flower/';
    sub_filter_last_modified on;
    sub_filter_once off;

    proxy_pass http://flower_ip;
    proxy_redirect off;
    proxy_set_header Host $host;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Protocol $scheme;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_http_version 1.1;
  }

This is my flower page, without nginx it supppose the celery@e086bacebeb5 is clickable. Although i can get my flower page, i can't realtime monitoring my tasks. Does this need to configure something in order to make the flower's WebSocket works?

Screen Shot 2020-01-04 at 19 01 57

rafaelcascalho commented 4 years ago

@luvwinnie I'm not sure what is your issue, but here is my docker-compose file config for Flower and my Nginx, as well with my custom config file. I was having the same issue of not seeing anything in the task and other tabs, but this config solved for me.

Nginx config .conf file:

...

upstream flower {
        server flower:5555;
}

server{
        listen 80;

        ...

        location ~ ^/flower/? {
                sub_filter_last_modified on;
                sub_filter_once off;

                rewrite ^/flower/(.*)$ /$1 break;
                proxy_pass http://flower;
                proxy_redirect off;
                proxy_http_version 1.1;
                proxy_set_header Host $host;
                proxy_set_header Connection "upgrade";
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header X-Real-IP $remote_addr;
        }
}

docker-compose file:

version: '3.6'

services:
...
  nginx:
    restart: always
    ports:
      - 80:80

  flower:
    image: mher/flower
    restart: always
    expose:
      - 5555
    environment:
      - CELERY_BROKER_URL=...
      - FLOWER_PORT=5555
      - FLOWER_URL_PREFIX=/flower
      - FLOWER_BASIC_AUTH=user:password
...

Here is what I see now: screencapture-localhost-90-flower-dashboard-2020-03-31-17_46_20

screencapture-localhost-90-flower-tasks-2020-03-31-17_46_36

One thing I'm dealing with is: when I click in my worker, which appears as a link in my dashboard, I'm redirected to a page which says 'Unknow worker', but even though I'm pretty ok and it's working fine.

diegopedroso1 commented 4 years ago

@Ian-Re seems that is a Flower bug. When you use --broker=amqp the worker's connections are lost due protocol, when use HTTP you lost broker connection. That's why your Broker page is empty. Try to monitor broker using RabbitMQ web Interface and control worker using Flower, but without broker's option above.

ShahBinoy commented 3 years ago

I cannot see monitor tab on flower being run inside a docker container too for version 0.9.7 I have to fallback on 0.9.4 to see the monitor tab @mher

ziptnf commented 3 years ago

@ShahBinoy the monitor tab was removed on purpose, to force the prometheus integration with flower.

ShahBinoy commented 3 years ago

@ShahBinoy the monitor tab was removed on purpose, to force the prometheus integration with flower.

It would be good to see a Prometheus dashboard for such. do you have a Reference Prometheus dashboard @ziptnf for flower.

lioshi commented 2 years ago

I concur that the change highlighted by @DAIXK above works for me, when I insert a breakpoint and update the return statement in the function. The issue is that I re-install flower on every deployment, so I'd have to make this change every time.

function url_prefix() {
    var url_prefix = $('#url_prefix').val();
    if (url_prefix) {
        if (url_prefix.startsWith('/')) {
            return '/flower' + url_prefix;
        } else {
            return '/flower' + '/' + url_prefix;
        }
    }
    return '/flower';
}

Good idea but I resolved issue with launch flower like this, and then no need of JS patch : flower --url-prefix=/flower

Note the "/" before "flower", if it could help.

espoirMur commented 2 years ago

Same issue here...

I can see my tasks when I call them via the APIs but the dashboard is empty

sshishov commented 2 years ago

The problem is represented here: https://github.com/mher/flower/issues/787 Dashboard is not storing any data, as soon as you refresh the page, all data lost. There was a PR to bring the persistency for the dashboard but it was ignored by maintainer. I guess their solution is to use integration with Prometheus. But I agree that not everyone wants to install heavy Prometheus with greedy memory usage just to get some metrics or info...

eljeffeg commented 2 years ago

I'm having this issue as well using docker & nginx. Would love some assistance. The worker shows up fine, but clicking on Processed, Failed, Succeeded just goes to an empty task page. I've tried to follow @rafaelcascalho configuration, but that doesn't appear to work (though I'm similarly using a subdirectory of /flower). Not sure how his is working as nginx doesn't specify a port, show he should be getting a 502 error. But even specifying a port, the - FLOWER_URL_PREFIX=/flower breaks it for me even though that's what I'm doing. Removing that brings up the site, but only the dashboard works showing the worker and the stats. Using docker image: mher/flower

Screen Shot 2022-07-31 at 7 57 15 PM Screen Shot 2022-07-31 at 7 57 22 PM Screen Shot 2022-07-31 at 7 57 34 PM
eljeffeg commented 2 years ago

I got it working with this config.. but the tasks don't show results or the responding status after starting.

flower:
    restart: always
    image: mher/flower
    environment:
      - CELERY_BROKER_URL=amqp://${RABBITMQ_DEFAULT_USER}:${RABBITMQ_DEFAULT_PASS}@rabbitmq:5672/
      - CELERY_RESULT_BACKEND=rpc://${RABBITMQ_DEFAULT_USER}:${RABBITMQ_DEFAULT_PASS}@rabbitmq:5672/
      - FLOWER_URL_PREFIX=flower
location ~ ^/flower/? {
        proxy_pass http://flower:5555;
        proxy_set_header Host $host;
        proxy_redirect off;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
}