Closed radiatorius1024 closed 10 months ago
Hi @radiatorius1024, which version of prefect-airbyte
are you running (pip freeze | grep prefect-airbyte
)?
With prefect==2.13.7
, we've removed the version constraint on pydantic
, which allows pydantic>=2
to be installed, but that may be incompatible with older versions of prefect-airbyte
. In prefect-airbyte==0.3.0
, we've made the import adjustments to allow you to have either version of pydantic installed. If you aren't on 0.3.0
, can you bump your version and let us know if that works?
Hi. Yes I have heard about the changes so the latest version is used: pip freeze | grep prefect-airbyte -> prefect-airbyte==0.3.0
Thanks @radiatorius1024, I should have also asked: what version of pydantic
do you have installed as well?
Here are all installed libraries (Prefect installed on Docker, but same error with local installation): Python 3.11.6
pip freeze
aiobotocore==2.7.0 aiohttp==3.8.6 aioitertools==0.11.0 aiosignal==1.3.1 aiosqlite==0.19.0 alembic==1.12.0 annotated-types==0.6.0 anyio==3.7.1 apprise==1.6.0 asgi-lifespan==2.1.0 async-timeout==4.0.3 asyncpg==0.28.0 attrs==23.1.0 botocore==1.31.64 cachetools==5.3.1 certifi==2023.7.22 cffi==1.16.0 charset-normalizer==3.3.0 click==8.1.7 cloudpickle==3.0.0 colorama==0.4.6 coolname==2.2.0 croniter==1.4.1 cryptography==41.0.4 dask==2023.10.0 dateparser==1.1.8 distributed==2023.10.0 docker==6.1.3 frozenlist==1.4.0 fsspec==2023.10.0 google-auth==2.23.3 graphviz==0.20.1 greenlet==3.0.0 griffe==0.36.7 h11==0.14.0 h2==4.1.0 hpack==4.0.0 httpcore==0.18.0 httpx==0.25.0 hyperframe==6.0.1 idna==3.4 importlib-metadata==6.8.0 Jinja2==3.1.2 jmespath==1.0.1 jsonpatch==1.33 jsonpointer==2.4 jsonschema==4.19.1 jsonschema-specifications==2023.7.1 kubernetes==28.1.0 locket==1.0.0 Mako==1.2.4 Markdown==3.5 markdown-it-py==3.0.0 MarkupSafe==2.1.3 mdurl==0.1.2 msgpack==1.0.7 multidict==6.0.4 numpy==1.26.1 oauthlib==3.2.2 orjson==3.9.9 packaging==23.2 pandas==2.1.1 partd==1.4.1 pathspec==0.11.2 pendulum==2.1.2 prefect @ file:///opt/prefect/dist/prefect.tar.gz prefect-airbyte==0.3.0 prefect-dask==0.2.5 psutil==5.9.6 pyasn1==0.5.0 pyasn1-modules==0.3.0 pycparser==2.21 pydantic==2.4.2 pydantic_core==2.10.1 Pygments==2.16.1 pyodbc==5.0.1 python-dateutil==2.8.2 python-slugify==8.0.1 pytz==2023.3.post1 pytzdata==2020.1 PyYAML==6.0.1 readchar==4.0.5 referencing==0.30.2 regex==2023.10.3 requests==2.31.0 requests-oauthlib==1.3.1 rich==13.6.0 rpds-py==0.10.6 rsa==4.9 ruamel.yaml==0.17.39 ruamel.yaml.clib==0.2.8 s3fs==2023.10.0 sentry-sdk==1.32.0 six==1.16.0 sniffio==1.3.0 sortedcontainers==2.4.0 SQLAlchemy==2.0.22 starlette==0.27.0 tblib==2.0.0 text-unidecode==1.3 toml==0.10.2 toolz==0.12.0 tornado==6.3.3 typer==0.9.0 typing_extensions==4.8.0 tzdata==2023.3 tzlocal==5.1 urllib3==1.26.18 uvicorn==0.23.2 websocket-client==1.6.4 websockets==11.0.3 wrapt==1.15.0 yarl==1.9.2 zict==3.0.0 zipp==3.17.0
Thank you! We'll dig into this
Hi @radiatorius1024, thanks for the bug report, we believe we have corrected this in the latest prefect
release (2.14.1): https://github.com/PrefectHQ/prefect/releases/tag/2.14.1
Can you let us know if that corrected things for you?
Hi @radiatorius1024, thanks for the bug report, we believe we have corrected this in the latest
prefect
release (2.14.1): https://github.com/PrefectHQ/prefect/releases/tag/2.14.1Can you let us know if that corrected things for you?
Everything works great now, thanks.
Hey @chrisguidry I have been stuck on this same error for a while now.
I am using a docker work-pool to run the flow (and subflow) that's stored locally. I was initially using Prefect Server and Connection blocks to maintain the Airbyte Connection information, but have since adjusted my script to match radiatorius', but I'm still getting the same error.
Could it be the work-pool?
Here are my specs:
Sorry about that, @jloehn! Can you paste the full exception you're seeing with the stack trace so we can see where it's happening for you? It may be the same error but manifesting from a different place.
Not sure why the stack trace is showing python 3.10 as library. I updated to 3.11 and removed all 3.10 libraries. Could that be causing it? Or would it be something else with the worker-pool?
Hmm, if you've updated to python 3.11, is it possible this image itself is outdated? Are you able to easily modify this flow to print out some info from inside the container? Maybe something like:
logger = get_run_logger()
logger.info("prefect version %s", prefect.__version__)
logger.info("prefect_airbyte version %s", prefect_airbyte.__version__)
logger.info("pydantic version %s", pydantic.__version__)
from within the flow so we can see if maybe it's not running the latest versions?
@chrisguidry Thanks for pointing me in the right direction. I kept overlooking the push section of my prefect.yaml.
Nice! Glad to hear it's working!
An error is received when executing flow in Prefect 2.13.7...2.13.8:
from prefect_airbyte.server import AirbyteServer from prefect_airbyte.connections import AirbyteConnection from prefect_airbyte.flows import run_connection_sync
server = AirbyteServer(server_host="192.168.1.126", server_port=8000)
connection = AirbyteConnection( airbyte_server=server, connection_id="b35180dc-6bd1-48b0-b0cd-b99deb6caaaa", status_updates=True )
@flow def extract_data_subflow(): run_connection_sync(airbyte_connection=connection)
pydantic_self.pydantic_validator.validate_python(data, self_instance=__pydantic_self__) TypeError: BaseModel.validate() takes 2 positional arguments but 3 were given