search

kedro-org / kedro

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
https://kedro.org
Apache License 2.0
10.03k stars 906 forks source link

Replace `requirements.txt` with `pyproject.toml` for dependency management? #2757

Open amandakys opened 1 year ago

amandakys commented 1 year ago

Introduction

Tracking ticket for discussion about

"Moreover, we should move away from using setup.py and make all Kedro starters use only pyproject.toml and setup.cfg, since this is how the Python ecosystem is evolving."

and

"The setup.cfg, src/setup.py and src/requirements.txt should be replaced with pyproject.toml. pip/setuptools support pyproject.toml out of the box. Would love to have this by default in new projects."

These are comments that came up in #2149 and should be discussed separately to the planned work to strip the project template #2756

astrojuanlu commented 1 year ago

This is blocked on https://github.com/kedro-org/kedro/issues/2519.

merelcht commented 1 month ago

Does it make sense to do this after https://github.com/kedro-org/kedro/issues/4116 ? @astrojuanlu

astrojuanlu commented 1 month ago

Oh, in a sense I wouldn't know what's left for this issue after we address #4116.

Maybe before getting completely rid of requirements.txt I'd wait until the acceptance of PEP 751 (still wip) or the complete takeover of uv.lock...

merelcht commented 1 month ago

We just need to do it for all starters as well.

JoostGevaert commented 1 week ago

+1 for this :)

astrojuanlu commented 1 week ago

PEP 751 is about to undergo another change in scope https://discuss.python.org/t/pep-751-now-with-graphs/69721/106 so I'm still not sure it's a good moment to dump requirements.txt.

We should probably have a fully resolved requirements.txt, as pip-compile or uv compile would do.

JoostGevaert commented 1 week ago

And what about using hatchling (uv's default) instead of setuptools as the build-system?

JoostGevaert commented 1 week ago

Maybe that ☝️ question of mine shouldn't be in this issue? In issues #681 and #4116 I found that I can create a kedro project with uv with kedro-init or copier-kedro. However, I'm still quite new to kedro, so I like being able to start a new project with an example pipeline.