search

mher / flower

Real-time monitor and web admin for Celery distributed task queue
https://flower.readthedocs.io
Other
6.5k stars 1.09k forks source link

Browser hanging up until ctrl-c #1390

Closed shizidushu closed 2 months ago

shizidushu commented 3 months ago

Describe the bug I try to monitor tasks of dify. After start flower in the command line, open the browser but it keep hangs up (nothing shown).

After press one Ctrl-C, the browser opened the page, though it shows the worker is offline. And the flower webpage seems working fine, i.e., I can check the queues and config of the worker.

To Reproduce Steps to reproduce the behavior:

  1. Follow the steps in https://github.com/langgenius/dify/blob/main/api/README.md
  2. Install flower: poetry add flower
  3. Open Firefox browser, open the flower webpage, i.e., xx.xx.xx.xx:5555
  4. Nothing appears in the browser.
  5. CTRL-C, the terminal did not end the program and then the flower webpage loads.

Here is the logs:

(venv) hy@s107:~/dify/api$ poetry run python -m celery -A app.celery --broker=redis://:difyai123456@localhost:6379/1 flower --port=5555 --address='0.0.0.0' --debug --enable_events
Warning: Found deprecated key 'default' or 'secondary' in pyproject.toml configuration for source aliyun. Please provide the key 'priority' instead. Accepted values are: 'default', 'primary', 'secondary', 'supplemental', 'explicit'.
Warning: Found deprecated priority 'default' for source 'aliyun' in pyproject.toml. You can achieve the same effect by changing the priority to 'primary' and putting the source first.
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
/home/hy/dify/api/venv/lib/python3.10/site-packages/pydub/utils.py:170: RuntimeWarning: Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work
  warn("Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work", RuntimeWarning)
2024-08-05 08:38:18,118.118 WARNING [MainThread] [ext_mail.py:50] - MAIL_TYPE is not set
2024-08-05 08:38:18,974.974 DEBUG [MainThread] [selector_events.py:54] - Using selector: GeventSelector
2024-08-05 08:38:18,980.980 INFO [MainThread] [command.py:168] - Visit me at http://0.0.0.0:5555
2024-08-05 08:38:18,985.985 INFO [MainThread] [command.py:176] - Broker: redis://:**@localhost:6379/1
2024-08-05 08:38:18,986.986 DEBUG [MainThread] [functional.py:335] - 
def clean_embedding_cache_task():
    return 1

2024-08-05 08:38:18,986.986 DEBUG [MainThread] [functional.py:335] - 
def add_document_to_index_task(dataset_document_id):
    return 1

2024-08-05 08:38:18,986.986 DEBUG [MainThread] [functional.py:335] - 
def chunks(task, it, n):
    return 1

2024-08-05 08:38:18,987.987 DEBUG [MainThread] [functional.py:335] - 
def group(self, tasks, result, group_id, partial_args, add_to_parent=0):
    return 1

2024-08-05 08:38:18,987.987 DEBUG [MainThread] [functional.py:335] - 
def recover_document_indexing_task(dataset_id, document_id):
    return 1

2024-08-05 08:38:18,987.987 DEBUG [MainThread] [functional.py:335] - 
def delete_annotation_index_task(annotation_id, app_id, tenant_id, collection_binding_id):
    return 1

2024-08-05 08:38:18,988.988 DEBUG [MainThread] [functional.py:335] - 
def enable_segment_to_index_task(segment_id):
    return 1

2024-08-05 08:38:18,988.988 DEBUG [MainThread] [functional.py:335] - 
def update_annotation_to_index_task(annotation_id, question, tenant_id, app_id, collection_binding_id):
    return 1

2024-08-05 08:38:18,988.988 DEBUG [MainThread] [functional.py:335] - 
def chain(*args, **kwargs):
    return 1

2024-08-05 08:38:18,989.989 DEBUG [MainThread] [functional.py:335] - 
def delete_segment_from_index_task(segment_id, index_node_id, dataset_id, document_id):
    return 1

2024-08-05 08:38:18,989.989 DEBUG [MainThread] [functional.py:335] - 
def chord(self, header, body, partial_args=0, interval=1, countdown=2, max_retries=3, eager=4, **kwargs):
    return 1

2024-08-05 08:38:18,989.989 DEBUG [MainThread] [functional.py:335] - 
def clean_notion_document_task(document_ids, dataset_id):
    return 1

2024-08-05 08:38:18,990.990 DEBUG [MainThread] [functional.py:335] - 
def clean_unused_datasets_task():
    return 1

2024-08-05 08:38:18,990.990 DEBUG [MainThread] [functional.py:335] - 
def document_indexing_update_task(dataset_id, document_id):
    return 1

2024-08-05 08:38:18,990.990 DEBUG [MainThread] [functional.py:335] - 
def remove_app_and_related_data_task(self, tenant_id, app_id):
    return 1

2024-08-05 08:38:18,991.991 DEBUG [MainThread] [functional.py:335] - 
def process_trace_tasks(tasks_data):
    return 1

2024-08-05 08:38:18,991.991 DEBUG [MainThread] [functional.py:335] - 
def clean_dataset_task(dataset_id, tenant_id, indexing_technique, index_struct, collection_binding_id, doc_form):
    return 1

2024-08-05 08:38:18,991.991 DEBUG [MainThread] [functional.py:335] - 
def add_annotation_to_index_task(annotation_id, question, tenant_id, app_id, collection_binding_id):
    return 1

2024-08-05 08:38:18,992.992 DEBUG [MainThread] [functional.py:335] - 
def remove_document_from_index_task(document_id):
    return 1

2024-08-05 08:38:18,992.992 DEBUG [MainThread] [functional.py:335] - 
def disable_annotation_reply_task(job_id, app_id, tenant_id):
    return 1

2024-08-05 08:38:18,992.992 DEBUG [MainThread] [functional.py:335] - 
def retry_document_indexing_task(dataset_id, document_ids):
    return 1

2024-08-05 08:38:18,993.993 DEBUG [MainThread] [functional.py:335] - 
def send_invite_member_mail_task(language, to, token, inviter_name, workspace_name):
    return 1

2024-08-05 08:38:18,993.993 DEBUG [MainThread] [functional.py:335] - 
def document_indexing_sync_task(dataset_id, document_id):
    return 1

2024-08-05 08:38:18,993.993 DEBUG [MainThread] [functional.py:335] - 
def disable_segment_from_index_task(segment_id):
    return 1

2024-08-05 08:38:18,994.994 DEBUG [MainThread] [functional.py:335] - 
def backend_cleanup():
    return 1

2024-08-05 08:38:18,994.994 DEBUG [MainThread] [functional.py:335] - 
def duplicate_document_indexing_task(dataset_id, document_ids):
    return 1

2024-08-05 08:38:18,994.994 DEBUG [MainThread] [functional.py:335] - 
def batch_create_segment_to_index_task(job_id, content, dataset_id, document_id, tenant_id, user_id):
    return 1

2024-08-05 08:38:18,994.994 DEBUG [MainThread] [functional.py:335] - 
def accumulate(self, *args, **kwargs):
    return 1

2024-08-05 08:38:18,995.995 DEBUG [MainThread] [functional.py:335] - 
def batch_import_annotations_task(job_id, content_list, app_id, tenant_id, user_id):
    return 1

2024-08-05 08:38:18,995.995 DEBUG [MainThread] [functional.py:335] - 
def clean_document_task(document_id, dataset_id, doc_form, file_id):
    return 1

2024-08-05 08:38:18,995.995 DEBUG [MainThread] [functional.py:335] - 
def deal_dataset_vector_index_task(dataset_id, action):
    return 1

2024-08-05 08:38:18,996.996 DEBUG [MainThread] [functional.py:335] - 
def unlock_chord(self, group_id, callback, interval=0, max_retries=1, result=2, Result=3, GroupResult=4, result_from_tuple=5, **kwargs):
    return 1

2024-08-05 08:38:18,996.996 DEBUG [MainThread] [functional.py:335] - 
def sync_website_document_indexing_task(dataset_id, document_id):
    return 1

2024-08-05 08:38:18,997.997 DEBUG [MainThread] [functional.py:335] - 
def enable_annotation_reply_task(job_id, app_id, user_id, tenant_id, score_threshold, embedding_provider_name, embedding_model_name):
    return 1

2024-08-05 08:38:18,997.997 DEBUG [MainThread] [functional.py:335] - 
def send_reset_password_mail_task(language, to, token):
    return 1

2024-08-05 08:38:18,997.997 DEBUG [MainThread] [functional.py:335] - 
def xmap(task, it):
    return 1

2024-08-05 08:38:18,998.998 DEBUG [MainThread] [functional.py:335] - 
def document_indexing_task(dataset_id, document_ids):
    return 1

2024-08-05 08:38:18,998.998 DEBUG [MainThread] [functional.py:335] - 
def xstarmap(task, it):
    return 1

2024-08-05 08:38:18,999.999 INFO [MainThread] [command.py:177] - Registered tasks: 
['celery.accumulate',
 'celery.backend_cleanup',
 'celery.chain',
 'celery.chord',
 'celery.chord_unlock',
 'celery.chunks',
 'celery.group',
 'celery.map',
 'celery.starmap',
 'schedule.clean_embedding_cache_task.clean_embedding_cache_task',
 'schedule.clean_unused_datasets_task.clean_unused_datasets_task',
 'tasks.add_document_to_index_task.add_document_to_index_task',
 'tasks.annotation.add_annotation_to_index_task.add_annotation_to_index_task',
 'tasks.annotation.batch_import_annotations_task.batch_import_annotations_task',
 'tasks.annotation.delete_annotation_index_task.delete_annotation_index_task',
 'tasks.annotation.disable_annotation_reply_task.disable_annotation_reply_task',
 'tasks.annotation.enable_annotation_reply_task.enable_annotation_reply_task',
 'tasks.annotation.update_annotation_to_index_task.update_annotation_to_index_task',
 'tasks.batch_create_segment_to_index_task.batch_create_segment_to_index_task',
 'tasks.clean_dataset_task.clean_dataset_task',
 'tasks.clean_document_task.clean_document_task',
 'tasks.clean_notion_document_task.clean_notion_document_task',
 'tasks.deal_dataset_vector_index_task.deal_dataset_vector_index_task',
 'tasks.delete_segment_from_index_task.delete_segment_from_index_task',
 'tasks.disable_segment_from_index_task.disable_segment_from_index_task',
 'tasks.document_indexing_sync_task.document_indexing_sync_task',
 'tasks.document_indexing_task.document_indexing_task',
 'tasks.document_indexing_update_task.document_indexing_update_task',
 'tasks.duplicate_document_indexing_task.duplicate_document_indexing_task',
 'tasks.enable_segment_to_index_task.enable_segment_to_index_task',
 'tasks.mail_invite_member_task.send_invite_member_mail_task',
 'tasks.mail_reset_password_task.send_reset_password_mail_task',
 'tasks.ops_trace_task.process_trace_tasks',
 'tasks.recover_document_indexing_task.recover_document_indexing_task',
 'tasks.remove_app_and_related_data_task.remove_app_and_related_data_task',
 'tasks.remove_document_from_index_task.remove_document_from_index_task',
 'tasks.retry_document_indexing_task.retry_document_indexing_task',
 'tasks.sync_website_document_indexing_task.sync_website_document_indexing_task']
2024-08-05 08:38:18,999.999 DEBUG [MainThread] [command.py:181] - Settings: {'cookie_secret': '2mlxcoRvsugyqbduN9V-TG99R_0Po7PFx8C_cE3_OaZ23io8cwqZUKHrn5eopREKYCO8XJ1WNUWmQK_YvEtlRg',
 'debug': True,
 'login_url': '/login',
 'static_path': '/home/hy/dify/api/venv/lib/python3.10/site-packages/flower/static',
 'static_url_prefix': '/static/',
 'template_path': '/home/hy/dify/api/venv/lib/python3.10/site-packages/flower/templates'}
2024-08-05 08:38:19,001.001 DEBUG [MainThread] [events.py:152] - Starting enable events timer...
2024-08-05 08:38:19,037.037 DEBUG [ThreadPoolExecutor-4_0] [inspector.py:34] - Sending stats inspect command
2024-08-05 08:38:19,038.038 DEBUG [ThreadPoolExecutor-4_1] [inspector.py:34] - Sending active_queues inspect command
2024-08-05 08:38:19,038.038 DEBUG [ThreadPoolExecutor-4_2] [inspector.py:34] - Sending registered inspect command
2024-08-05 08:38:19,040.040 DEBUG [ThreadPoolExecutor-4_3] [inspector.py:34] - Sending scheduled inspect command
2024-08-05 08:38:19,043.043 DEBUG [ThreadPoolExecutor-4_4] [inspector.py:34] - Sending active inspect command
2024-08-05 08:38:19,045.045 DEBUG [ThreadPoolExecutor-4_5] [inspector.py:34] - Sending reserved inspect command
2024-08-05 08:38:19,046.046 DEBUG [ThreadPoolExecutor-4_6] [inspector.py:34] - Sending revoked inspect command
2024-08-05 08:38:19,048.048 DEBUG [ThreadPoolExecutor-4_7] [inspector.py:34] - Sending conf inspect command
2024-08-05 08:38:19,053.053 DEBUG [Thread-1] [events.py:182] - Capturing events...
2024-08-05 08:38:19,066.066 INFO [Thread-1] [mixins.py:228] - Connected to redis://:**@localhost:6379/1
^C2024-08-05 08:43:20,311.311 DEBUG [ThreadPoolExecutor-4_4] [inspector.py:41] - Inspect command active took 301.27s to complete
2024-08-05 08:43:20,312.312 DEBUG [ThreadPoolExecutor-4_0] [inspector.py:41] - Inspect command stats took 301.27s to complete
2024-08-05 08:43:20,312.312 DEBUG [ThreadPoolExecutor-4_2] [inspector.py:41] - Inspect command registered took 301.27s to complete
2024-08-05 08:43:20,312.312 DEBUG [ThreadPoolExecutor-4_0] [inspector.py:34] - Sending stats inspect command
2024-08-05 08:43:20,314.314 DEBUG [ThreadPoolExecutor-4_2] [inspector.py:34] - Sending active_queues inspect command
2024-08-05 08:43:20,315.315 WARNING [MainThread] [web.py:1873] - 404 GET /worker/celery%40s107 (10.117.14.71): Unknown worker 'celery@s107'
2024-08-05 08:43:21,401.401 DEBUG [ThreadPoolExecutor-4_3] [inspector.py:41] - Inspect command scheduled took 302.36s to complete
2024-08-05 08:43:21,402.402 DEBUG [ThreadPoolExecutor-4_3] [inspector.py:34] - Sending registered inspect command
2024-08-05 08:43:21,465.465 DEBUG [ThreadPoolExecutor-4_1] [inspector.py:41] - Inspect command active_queues took 302.43s to complete
2024-08-05 08:43:21,465.465 DEBUG [ThreadPoolExecutor-4_1] [inspector.py:34] - Sending scheduled inspect command
2024-08-05 08:43:21,465.465 DEBUG [ThreadPoolExecutor-4_7] [inspector.py:41] - Inspect command conf took 302.42s to complete
2024-08-05 08:43:21,465.465 DEBUG [ThreadPoolExecutor-4_7] [inspector.py:34] - Sending active inspect command
2024-08-05 08:43:21,465.465 DEBUG [ThreadPoolExecutor-4_4] [inspector.py:34] - Sending reserved inspect command
2024-08-05 08:43:21,473.473 DEBUG [ThreadPoolExecutor-4_6] [inspector.py:41] - Inspect command revoked took 302.43s to complete
2024-08-05 08:43:21,473.473 DEBUG [ThreadPoolExecutor-4_6] [inspector.py:34] - Sending revoked inspect command
2024-08-05 08:43:21,473.473 DEBUG [ThreadPoolExecutor-4_5] [inspector.py:41] - Inspect command reserved took 302.43s to complete
2024-08-05 08:43:21,474.474 DEBUG [ThreadPoolExecutor-4_5] [inspector.py:34] - Sending conf inspect command
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/hy/dify/api/venv/lib/python3.10/site-packages/flower/events.py", line 183, in run
    recv.capture(limit=None, timeout=None, wakeup=True)
  File "/home/hy/dify/api/venv/lib/python3.10/site-packages/celery/events/receiver.py", line 91, in capture
    for _ in self.consume(limit=limit, timeout=timeout, wakeup=wakeup):
  File "/home/hy/dify/api/venv/lib/python3.10/site-packages/kombu/mixins.py", line 196, in consume
    conn.drain_events(timeout=safety_interval)
  File "/home/hy/dify/api/venv/lib/python3.10/site-packages/kombu/connection.py", line 341, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/home/hy/dify/api/venv/lib/python3.10/site-packages/kombu/transport/virtual/base.py", line 997, in drain_events
    get(self._deliver, timeout=timeout)
  File "/home/hy/dify/api/venv/lib/python3.10/site-packages/kombu/transport/redis.py", line 588, in get
    events = self.poller.poll(timeout)
  File "/home/hy/dify/api/venv/lib/python3.10/site-packages/kombu/utils/eventio.py", line 281, in poll
    read, write, error = _selectf(
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/home/hy/dify/api/venv/lib/python3.10/site-packages/flower/events.py", line 189, in run
    thread.interrupt_main()
KeyboardInterrupt
2024-08-05 08:43:21,551.551 DEBUG [ThreadPoolExecutor-4_1] [inspector.py:41] - Inspect command scheduled took 0.09s to complete
2024-08-05 08:43:21,555.555 DEBUG [ThreadPoolExecutor-4_2] [inspector.py:41] - Inspect command active_queues took 1.24s to complete
2024-08-05 08:43:21,557.557 DEBUG [ThreadPoolExecutor-4_4] [inspector.py:41] - Inspect command reserved took 0.09s to complete
2024-08-05 08:43:21,557.557 DEBUG [ThreadPoolExecutor-4_0] [inspector.py:41] - Inspect command stats took 1.25s to complete
2024-08-05 08:43:21,559.559 DEBUG [ThreadPoolExecutor-4_3] [inspector.py:41] - Inspect command registered took 0.16s to complete
2024-08-05 08:43:21,560.560 DEBUG [ThreadPoolExecutor-4_7] [inspector.py:41] - Inspect command active took 0.09s to complete
2024-08-05 08:43:21,560.560 DEBUG [ThreadPoolExecutor-4_6] [inspector.py:41] - Inspect command revoked took 0.09s to complete
2024-08-05 08:43:21,561.561 DEBUG [ThreadPoolExecutor-4_5] [inspector.py:41] - Inspect command conf took 0.09s to complete
2024-08-05 08:44:36,625.625 DEBUG [ThreadPoolExecutor-4_6] [inspector.py:34] - Sending stats inspect command
2024-08-05 08:44:36,625.625 DEBUG [ThreadPoolExecutor-4_5] [inspector.py:34] - Sending active_queues inspect command
2024-08-05 08:44:36,626.626 DEBUG [ThreadPoolExecutor-4_1] [inspector.py:34] - Sending registered inspect command
2024-08-05 08:44:36,626.626 DEBUG [ThreadPoolExecutor-4_2] [inspector.py:34] - Sending scheduled inspect command
2024-08-05 08:44:36,626.626 DEBUG [ThreadPoolExecutor-4_4] [inspector.py:34] - Sending active inspect command
2024-08-05 08:44:36,626.626 DEBUG [ThreadPoolExecutor-4_0] [inspector.py:34] - Sending reserved inspect command
2024-08-05 08:44:36,626.626 DEBUG [ThreadPoolExecutor-4_3] [inspector.py:34] - Sending revoked inspect command
2024-08-05 08:44:36,627.627 DEBUG [ThreadPoolExecutor-4_7] [inspector.py:34] - Sending conf inspect command
2024-08-05 08:44:36,662.662 DEBUG [ThreadPoolExecutor-4_3] [inspector.py:41] - Inspect command revoked took 0.04s to complete
2024-08-05 08:44:36,663.663 DEBUG [ThreadPoolExecutor-4_5] [inspector.py:41] - Inspect command active_queues took 0.04s to complete
2024-08-05 08:44:36,667.667 DEBUG [ThreadPoolExecutor-4_0] [inspector.py:41] - Inspect command reserved took 0.04s to complete
2024-08-05 08:44:36,670.670 DEBUG [ThreadPoolExecutor-4_4] [inspector.py:41] - Inspect command active took 0.04s to complete
2024-08-05 08:44:36,673.673 DEBUG [ThreadPoolExecutor-4_2] [inspector.py:41] - Inspect command scheduled took 0.05s to complete
2024-08-05 08:44:36,674.674 DEBUG [ThreadPoolExecutor-4_1] [inspector.py:41] - Inspect command registered took 0.05s to complete
2024-08-05 08:44:36,675.675 DEBUG [ThreadPoolExecutor-4_6] [inspector.py:41] - Inspect command stats took 0.05s to complete
2024-08-05 08:44:36,676.676 DEBUG [ThreadPoolExecutor-4_7] [inspector.py:41] - Inspect command conf took 0.05s to complete

Expected behavior The browser should loads the page normally.

System information Output of python -c 'from flower.utils import bugreport; print(bugreport())' command

flower   -> flower:2.0.1 tornado:6.4.1 humanize:4.10.0
software -> celery:5.3.6 (emerald-rush) kombu:5.3.7 py:3.10.12
            billiard:4.2.0 py-amqp:5.2.0
platform -> system:Linux arch:64bit, ELF
            kernel version:5.15.0-117-generic imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:disabled

deprecated_settings: Non
teocns commented 3 months ago

Turn off gevent monkey patching

shizidushu commented 2 months ago

@teocns The page loads normally. Thanks.