[RFC] Should we migrate do Poetry?

[bitterteriyaki]

Hello, everyone.

I am trying to contribute to this project again, but I am having a hard time because I can't event install the dependencies properly. I tried to use Python 3.12.3 but this projects uses aim package, which is deprecated and removed on Python 3.12. The redis-py documentation tells that this library needs Python 3.7 or higher, so I tried to install Python 3.7.17, but I can't install the dependencies again because the library click needs Python 3.8 or later. Now I am installing Python 3.11 and praying that it works (update: it worked :pray:).

The point is: it is becoming difficult to manage compatibility between versions, and I think that we can bring Poetry to solve this problem. FYI, Poetry is a dependencies manager for Python, better than pip IMO. We can separate dev dependencies from regular dependencies much easier there, and manage different dependencies versions.

I want to bring this conversation. What do you guys think? If there is any impediment to the implementation of this feature, I will be able to listen and give my opinion on it.

Positive points

• It automatically manages venvs
• Better dependency management
• More intuitive segregation of regular dependencies from dev dependencies
• We can eliminate all library configuration (e.g. isort.cfg) files and leave everything in pyproject.toml, so the project will look cleaner
• Lockfiles and hash checking, which can improve security

Negative points

• Everyone will need to install Poetry to work with this library
• Everyone will need to learn new commands, which can delay the development
• Configure everything from scratch can take some time

[gerzse]

Hi @bitterteriyaki,

Indeed, as you saw, currently only Python 3.11 is supported. There is a PR started for supporting 3.12 (https://github.com/redis/redis-py/pull/2979), work will hopefully restart on it soon.

Personally I love Poetry, and I would agree to start using it.

When you say "Everyone will need to install Poetry to work with this library", you mean everyone who wants to contribute to this library, right? To me that's a fair price to pay. Poetry usually just works, it should not be a super big overhead.

There is another PR open that suggests using hatchling (https://github.com/redis/redis-py/pull/2930). I never tried it, was planning to take a look. Do you have experience with it, and how it compares to Poetry?

One worry that I have is that existing automation people might have around setup.py will stop working.

[bitterteriyaki]

Hi @gerzse,

There is another PR open that suggests using hatchling (https://github.com/redis/redis-py/pull/2930)

I've never used this library, but I might take a look!

I'll try to make a PR adding Poetry.

[akx]

existing automation people might have around setup.py will stop working.

What kind of automation do you expect people to have?

[akx]

Btw, see #1658 for previous discussion.

[bitterteriyaki]

existing automation people might have around setup.py will stop working.

What kind of automation do you expect people to have?

Automated builds is one example.

[akx]

Automated builds is one example.

👍 They should always be done with build anyway, for pyproject.toml based projects.

redis-py isn't one, but could be made one with

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

as a first step.

[Wh1isper]

There is another PR open that suggests using hatchling (https://github.com/redis/redis-py/pull/2930). I never tried it, was planning to take a look. Do you have experience with it, and how it compares to Poetry?

Hi, @gerzse! Hope I can be of some help:

I have more experience with hatchling, and a little bit with Poetry. In my opinion, hatchling and Poetry specialize in different things.

For building, hatching follows the latest PEP rules for pure python packages, greatly simplifying the burden of using setup.py. Poetry also provides build commands, and where Poetry is more convenient is in creating virtual environments.

To summarize, if we need to manage multiple independent virtual environments locally and switch between them for development, then Poetry is the preferred option, without Poetry, developers usually need to create and switch venv manually, and additional validation work is required for different optional-dependencies.

If we don't have a lot of venvs to switch between, using hatching alone for wheel building is a clean and fast option.

https://github.com/redis/redis-py/pull/2930 is very appealing to me, thanks! @akx