search

kedro-org / kedro

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
https://kedro.org
Apache License 2.0
9.53k stars 877 forks source link

Kedro compatibility with fsspec #796

Closed selalamiTF closed 2 years ago

selalamiTF commented 3 years ago

Description

Kedro cannot be installed with DVC because of a dependency conflict over fsspec. I need to install Kedro and DVC on the same environment as they are required by the same app.

Context

I can't find a way today to install these two tools together which is quite inconvenient since I find Data Science projects might benefit a lot from them (similar to what Kedro did with Mlflow).

Possible Implementation

Upgrade fsspec or support a lager version.

Possible Alternatives

I can't find any today. Please feel free to suggest any alternatives

datajoely commented 3 years ago

Hi @selalamiTF fsspec has recently switched from the SemVer to CalVer system - we're aware of the problem and looking into how to resolve it.

selalamiTF commented 3 years ago

Hello @datajoely Thank you for you quick reply ! That's great news ! Do you plan to release the fix in the next release ? Just to get an idea on when to expect this as I hit a dead end in my project because of this problem. Thanks

ignacioparicio commented 3 years ago

Hi @selalamiTF, while we work on this, is downgrading DVC an option for you? DVC upgraded their fsspec requirement out of current Kedro bounds on April 9, so any version released before that (e.g. 2.0.17, released on April 2) should work.

selalamiTF commented 3 years ago

Hello @ignacioparicio. Thanks for the tip but I tried 2.0.17 but I still have the same problem (with Kedro 0.16.6) :/

ignacioparicio commented 3 years ago

Unfortunately back in 0.16.6 Kedro's requirement for fsspec was fsspec>=0.5.1, <0.8 (see this), you would also have to bump your Kedro version :/

DVC==2.0.17 and kedro>=0.17 should work

selalamiTF commented 3 years ago

Great ! I guess I can do that since we are still in a prototyping stage (luckily) and trying to figure out how all this can work together ;). Thank you very much for the help, it's great that you guys are available to support your community !

datajoely commented 3 years ago

Tangentially related - the team at Dolt (another data versioning solution) have recently built a Kedro plugin, perhaps this is something that would be helpful?

https://www.dolthub.com/blog/2021-06-16-kedro-dolt-plugin/

selalamiTF commented 3 years ago

Thanks, I will definitely take a look at this !

ignacioparicio commented 2 years ago

We have upgraded our fsspec requirement from fsspec>=0.5.1, <0.9 to fsspec>=2021.04, <2022.01 (the upper bound to be revised past that date). The change will come into effect with the next Kedro release, be it 0.17.5 or 0.18.0.

I'm closing this issue for now, but please shout if you want it reopen.