search

PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.97k stars 1.57k forks source link

concurrent.futures._base.CancelledError #12972

Open AleksandrLiadov opened 1 year ago

AleksandrLiadov commented 1 year ago

Hello,

When I try to run more than 20 flows (main flow has SequentialTaskRunner and subflow has DaskTaskRunner), I have this error:

concurrent.futures._base.CancelledError

Crash detected! Execution was interrupted by an unexpected exception: concurrent.futures._base.CancelledError: db2025eaentity_cluster-9-3788828d1ea4420095c2b1a477570787-1
Crash details:
Traceback (most recent call last):
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/prefect_dask/task_runners.py", line 269, in wait
    return await future.result(timeout=timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 494, in wait_for
    return fut.result()
  File "/opt/pysetup/.venv/lib/python3.8/site-packages/distributed/client.py", line 295, in _result
    raise exception
concurrent.futures._base.CancelledError: db2025eaentity_cluster-9-3788828d1ea4420095c2b1a477570787-1

Also it bugs UI, flow is in Completed State, but tasks are crashed.

Screenshot 2023-02-10 at 15 02 57
zanieb commented 1 year ago

Hi @AleksandrLiadov please provide a minimal reproducible example so we can recreate this and investigate the cause.

AleksandrLiadov commented 1 year ago

Hi @madkinsz . For now, I cannot provide a minimal reproducible example. However this error is almost always mixed with one from this issue: https://github.com/PrefectHQ/prefect/issues/12977 So I hope, once you could resolve it, maybe this one'll disappear also.