search

PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.36k stars 1.51k forks source link

Circular Import on Deployment (again) #13014

Open chicago-joe opened 1 year ago

chicago-joe commented 1 year ago

@ahuang11 I'm running into a similar issue as PrefectHQ/prefect-ray#33 :

My env (inside docker container):


<details>
<summary>Package                      Version
---------------------------- -----------
adal                         1.2.7
adlfs                        2023.1.0
aiohttp                      3.8.4
aiohttp-cors                 0.7.0
aiosignal                    1.3.1
aiosqlite                    0.18.0
alembic                      1.9.3
anyio                        3.6.2
appdirs                      1.4.4
apprise                      1.2.1
asgi-lifespan                2.0.0
asttokens                    2.2.1
async-timeout                4.0.2
asyncpg                      0.27.0
attrs                        22.2.0
azure-common                 1.1.28
azure-core                   1.26.3
azure-datalake-store         0.0.52
azure-identity               1.12.0
azure-mgmt-containerinstance 10.0.0
azure-mgmt-core              1.3.2
azure-mgmt-resource          22.0.0
azure-storage-blob           12.14.1
backcall                     0.2.0
beautifulsoup4               4.11.2
bleach                       6.0.0
blessed                      1.20.0
blosc2                       2.0.0
bokeh                        2.4.3
Bottleneck                   1.3.6
cachelib                     0.10.2
cachetools                   5.3.0
certifi                      2022.12.7
cffi                         1.15.1
charset-normalizer           3.0.1
click                        8.1.3
cloudpickle                  2.2.1
colorama                     0.4.6
colorful                     0.5.5
comm                         0.1.2
connectorx                   0.3.1
contourpy                    1.0.7
coolname                     2.2.0
cramjam                      2.6.2
croniter                     1.3.8
cryptography                 39.0.1
cssselect                    1.2.0
cycler                       0.11.0
Cython                       0.29.33
dask                         2023.2.0
dateparser                   1.1.7
debugpy                      1.6.6
decorator                    5.1.1
Deprecated                   1.2.13
distlib                      0.3.6
distributed                  2023.2.0
dnspython                    2.3.0
docker                       6.0.1
et-xmlfile                   1.1.0
exchange-calendars           4.2.5
executing                    1.2.0
fastapi                      0.91.0
fastparquet                  2023.2.0
filelock                     3.9.0
finta                        1.3
fonttools                    4.38.0
frozendict                   2.3.4
frozenlist                   1.3.3
fsspec                       2023.1.0
gevent                       22.10.2
google-api-core              2.11.0
google-auth                  2.16.0
googleapis-common-protos     1.58.0
gpustat                      1.0.0
greenlet                     2.0.2
griffe                       0.25.4
grpcio                       1.51.1
h11                          0.14.0
h2                           4.1.0
h5py                         3.8.0
HeapDict                     1.0.1
howdoi                       2.0.20
hpack                        4.0.0
html5lib                     1.1
httpcore                     0.16.3
httpx                        0.23.3
hyperframe                   6.0.1
idna                         3.4
importlib-metadata           6.0.0
ipykernel                    6.21.2
ipython                      8.10.0
ipython-suggestions          1.0.0
ipywidgets                   8.0.4
isodate                      0.6.1
jedi                         0.18.2
Jinja2                       3.1.2
jsonpatch                    1.32
jsonpointer                  2.3
jsonschema                   4.17.3
jupyter_client               8.0.2
jupyter_core                 5.2.0
jupyterlab-widgets           3.0.5
keep                         2.10.1
kiwisolver                   1.4.4
korean-lunar-calendar        0.3.1
kubernetes                   25.3.0
locket                       1.0.0
lxml                         4.9.2
Mako                         1.2.4
Markdown                     3.4.1
markdown-it-py               2.1.0
MarkupSafe                   2.1.2
matplotlib                   3.6.3
matplotlib-inline            0.1.6
mdurl                        0.1.2
modin                        0.18.1
mongoengine                  0.26.0
msal                         1.21.0
msal-extensions              1.0.0
msgpack                      1.0.4
msrest                       0.7.1
multidict                    6.0.4
multitasking                 0.0.11
nest-asyncio                 1.5.6
numexpr                      2.8.4
numpy                        1.24.2
nvidia-ml-py                 11.495.46
oauthlib                     3.2.2
opencensus                   0.11.1
opencensus-context           0.1.3
openpyxl                     3.1.1
orjson                       3.8.6
packaging                    23.0
pandas                       1.5.3
parallel-ssh                 2.12.0
parso                        0.8.3
partd                        1.3.0
pathlib2                     2.3.7.post1
pathspec                     0.11.0
pendulum                     2.1.2
pexpect                      4.8.0
pickleshare                  0.7.5
Pillow                       9.4.0
pip                          23.0
platformdirs                 3.0.0
portalocker                  2.7.0
prefect                      2.8.0
prefect-alert                0.1.3
prefect-azure                0.2.4
prefect-dask                 0.2.2
prefect-ray                  0.2.2
prefect-shell                0.1.4
prometheus-client            0.13.1
prompt-toolkit               3.0.36
protobuf                     3.20.3
psutil                       5.9.4
ptyprocess                   0.7.0
pure-eval                    0.2.2
py-cpuinfo                   9.0.0
py-spy                       0.3.14
pyarrow                      11.0.0
pyasn1                       0.4.8
pyasn1-modules               0.2.8
pycparser                    2.21
pydantic                     1.10.4
PyGithub                     1.57
Pygments                     2.14.0
PyJWT                        2.6.0
pyluach                      2.1.0
pymongo                      4.3.3
PyNaCl                       1.5.0
pyparsing                    3.0.9
pyquery                      2.0.0
pyrsistent                   0.19.3
pyte                         0.8.1
python-dateutil              2.8.2
python-slugify               8.0.0
pytz                         2022.7.1
pytz-deprecation-shim        0.1.0.post0
pytzdata                     2020.1
PyYAML                       6.0
pyzmq                        25.0.0
QuantStats                   0.0.59
ray                          2.2.0
readchar                     4.0.3
regex                        2022.10.31
requests                     2.28.2
requests-oauthlib            1.3.1
rfc3986                      1.5.0
rich                         13.3.1
rsa                          4.9
scipy                        1.10.0
seaborn                      0.12.2
setuptools                   67.2.0
sidetable                    0.9.1
six                          1.16.0
smart-open                   6.3.0
sniffio                      1.3.0
sortedcontainers             2.4.0
soupsieve                    2.3.2.post1
SQLAlchemy                   1.4.46
ssh-python                   1.0.0
ssh2-python                  1.0.0
stack-data                   0.6.2
starlette                    0.24.0
swifter                      1.3.4
sxmpy                        0.3.7
ta-lib-bin                   0.4.26
tables                       3.8.0
tabulate                     0.9.0
tblib                        1.7.0
terminaltables               3.1.10
text-unidecode               1.3
thefuck                      3.32
toml                         0.10.2
toolz                        0.12.0
tornado                      6.2
tqdm                         4.64.1
traitlets                    5.9.0
typer                        0.7.0
typing_extensions            4.4.0
tzdata                       2022.7
tzlocal                      4.2
urllib3                      1.26.14
uvicorn                      0.20.0
virtualenv                   20.19.0
wcwidth                      0.2.6
webencodings                 0.5.1
websocket-client             1.5.1
websockets                   10.4
wheel                        0.38.4
widgetsnbextension           4.0.5
wrapt                        1.14.1
yarl                         1.8.2
yfinance                     0.2.11
zict                         2.2.0
zipp                         3.13.0
zope.event                   4.6
zope.interface               5.5.2</summary>
</details>

python==3.9.16
ray==2.2.0
prefect-ray


Traceback (most recent call last):
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/engine.py", line 650, in orchestrate_flow_run
    result = await run_sync(flow_call)
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 156, in run_sync_in_interruptible_worker_thread
    tg.start_soon(
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__
    raise exceptions[0]
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 135, in capture_worker_thread_and_result
    result = __fn(*args, **kwargs)
  File "sxm_exporter.py", line 471, in Archive_QueueViewer_Loader
    dfDirs = get_archive_directories.submit(
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/futures.py", line 226, in result
    return sync(
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 267, in sync
    return run_async_from_worker_thread(__async_fn, *args, **kwargs)
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 177, in run_async_from_worker_thread
    return anyio.from_thread.run(call)
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/anyio/from_thread.py", line 49, in run
    return asynclib.run_async_from_thread(func, *args)
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 970, in run_async_from_thread
    return f.result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/futures.py", line 237, in _result
    return await final_state.result(raise_on_failure=raise_on_failure, fetch=True)
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/states.py", line 89, in _get_state_result
    raise await get_state_exception(state)
ray.exceptions.RaySystemError: System error: cannot import name 'FlowRun' from partially initialized module 'prefect.client.schemas' (most likely due to a circular import) (/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/client/schemas.py)
traceback: Traceback (most recent call last):
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/ray/serialization.py", line 332, in deserialize_objects
    obj = self._deserialize_object(data, metadata, object_ref)
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/ray/serialization.py", line 235, in _deserialize_object
    return self._deserialize_msgpack_data(data, metadata_fields)
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/ray/serialization.py", line 190, in _deserialize_msgpack_data
    python_objects = self._deserialize_pickle5_data(pickle5_data)
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/ray/serialization.py", line 180, in _deserialize_pickle5_data
    obj = pickle.loads(in_band)
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/__init__.py", line 25, in <module>
    from prefect.states import State
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/states.py", line 14, in <module>
    from prefect.client.schemas import State as State
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/client/schemas.py", line 6, in <module>
    from prefect.orion import schemas
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/orion/__init__.py", line 1, in <module>
    from . import models
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/orion/models/__init__.py", line 1, in <module>
    from . import (
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/orion/models/block_documents.py", line 13, in <module>
    from prefect.orion import schemas
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/orion/schemas/__init__.py", line 1, in <module>
    from . import states, schedules, core, sorting, filters, responses, actions
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/orion/schemas/states.py", line 13, in <module>
    from prefect.orion.utilities.schemas import DateTimeTZ, IDBaseModel, PrefectBaseModel
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/orion/utilities/schemas.py", line 16, in <module>
    from packaging.version import Version
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/packaging/__init__.py", line 1, in <module>
    from prefect.packaging.docker import DockerPackager
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/packaging/__init__.py", line 1, in <module>
    from prefect.packaging.docker import DockerPackager
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/packaging/docker.py", line 16, in <module>
    from prefect.flows import Flow, load_flow_from_script
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/flows.py", line 36, in <module>
    from prefect.context import PrefectObjectRegistry, registry_from_script
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/context.py", line 33, in <module>
    from prefect.client.orion import OrionClient
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/client/orion.py", line 19, in <module>
    from prefect.client.schemas import FlowRun, OrchestrationResult, TaskRun
ImportError: cannot import name 'FlowRun' from partially initialized module 'prefect.client.schemas' (most likely due to a circular import) (/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/client/schemas.py)
04:02:01 PM

Crash detected! Execution was interrupted by an unexpected exception: ray.exceptions.RaySystemError: System error: cannot import name 'FlowRun' from partially initialized module 'prefect.client.schemas' (most likely due to a circular import) (/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/client/schemas.py)
traceback: Traceback (most recent call last):
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/ray/serialization.py", line 332, in deserialize_objects
    obj = self._deserialize_object(data, metadata, object_ref)
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/ray/serialization.py", line 235, in _deserialize_object
    return self._deserialize_msgpack_data(data, metadata_fields)
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/ray/serialization.py", line 190, in _deserialize_msgpack_data
    python_objects = self._deserialize_pickle5_data(pickle5_data)
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/ray/serialization.py", line 180, in _deserialize_pickle5_data
    obj = pickle.loads(in_band)
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/__init__.py", line 25, in <module>
    from prefect.states import State
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/states.py", line 14, in <module>
    from prefect.client.schemas import State as State
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/client/schemas.py", line 6, in <module>
    from prefect.orion import schemas
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/orion/__init__.py", line 1, in <module>
    from . import models
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/orion/models/__init__.py", line 1, in <module>
    from . import (
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/orion/models/block_documents.py", line 13, in <module>
    from prefect.orion import schemas
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/orion/schemas/__init__.py", line 1, in <module>
    from . import states, schedules, core, sorting, filters, responses, actions
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/orion/schemas/states.py", line 13, in <module>
    from prefect.orion.utilities.schemas import DateTimeTZ, IDBaseModel, PrefectBaseModel
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/orion/utilities/schemas.py", line 16, in <module>
    from packaging.version import Version
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/packaging/__init__.py", line 1, in <module>
    from prefect.packaging.docker import DockerPackager
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/packaging/__init__.py", line 1, in <module>
    from prefect.packaging.docker import DockerPackager
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/packaging/docker.py", line 16, in <module>
    from prefect.flows import Flow, load_flow_from_script
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/flows.py", line 36, in <module>
    from prefect.context import PrefectObjectRegistry, registry_from_script
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/context.py", line 33, in <module>
    from prefect.client.orion import OrionClient
  File "/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/client/orion.py", line 19, in <module>
    from prefect.client.schemas import FlowRun, OrchestrationResult, TaskRun
ImportError: cannot import name 'FlowRun' from partially initialized module 'prefect.client.schemas' (most likely due to a circular import) (/opt/pysetup/.venv/lib/python3.9/site-packages/prefect/client/schemas.py)```
desertaxle commented 1 year ago

Thanks for opening an issue @chicago-joe! Can you please share a short script that you run in this environment which causes this error? We need this so that we can reproduce in our environment and start investigating a fix.