search

airbytehq / PyAirbyte

PyAirbyte brings the power of Airbyte to every Python developer.
https://docs.airbyte.com/pyairbyte
Other
234 stars 41 forks source link

Support for Prefect 3 #365

Open Udit107710 opened 2 months ago

Udit107710 commented 2 months ago

Topic

Make Airbyte compatible with Prefect 3

Relevant information

Since we are blocked by parallelism support we are running Prefect + Pyairbyte and want to upgrade to Prefect 3 but are blocked on the dependency of airbyte-cdk (which is a dependency of pyairbyte) on jsonschema = "~3.2.0" and pendulum = "<3.0.0". Prefect 3 brings a lot of performance improvements to utilise pyairbyte to the fullest. Moreover,

  1. jsonschema 4.0.0 was released on Sep 30, 2021 and not upgrading does not make sense.
  2. pendulum 3.0.0 brings lots of performance improvement with extensions rewritten in Rust

Upgrading these core dependencies ensures compatibility across various other integrations with airbyte.

marcosmarxm commented 2 months ago

@aaronsteers moving this issue t PyAirbyte project

Udit107710 commented 2 months ago

I don't think this issue can be solved by Pyairbyte since pyairbyte does not declare dependency on jsonschema = "~3.2.0" and pendulum = "<3.0.0" These dependencies arise from airbyte-cdk which pyairbyte utilises internally.

aaronsteers commented 2 months ago

@Udit107710 - Thanks for raising this issue. I agree with the intention of bumping jsonschema and pendulum refs. As noted, we'd have to first bump them in the CDK and then we could bump their references here.

The jsonschema library is pretty good with backwards compatibility and support for specific draft versions in each release. I vaguely recall some pendulum breaking changes when I last looked at this, but those are probably minimal...

Udit107710 commented 2 months ago

There are breaking changes on Pendulum and jsonschema I have local changes related to airbyte-cdk. I was able to install the latest versions but that caused multiple tests to fail. The issue with jsonschema caused was not something I was able to solve and have no way forward. The issue with pendulum requires changing the format of datetime across the entire module. I will raise a PR soon on aribyte-cdk and we can review and plan new changes.

Furthermore, I think bumping up the dependency in airbyte-cdk will also require making other airbyte packages like airbyte-ci compatible with the new versions since these must be getting used together.

Udit107710 commented 2 months ago

Draft PR for the changes: https://github.com/airbytehq/airbyte/pull/45677

aaronsteers commented 2 months ago

Logged these for tracking:

aman-immersa commented 1 month ago

@Udit107710 would like to learn more about your usage Pyairbyte with Prefect 3.0 since we're attempting something similar.