Switch dependency management to poetry

[thedrow]

Poetry has a lock file and a better workflow than pip-tools which most software developers aren't familiar with or don't have the time to contribute something that requires more than a few lines and minimal effort like in #181.

Poetry on the other hand allows you to edit one file, pyproject.toml, and type poetry update and your lock file is updated. We can even automate generating the lock and pushing it to the branch CI is running on with a little bit of effort.

[tillahoffmann]

Happy to switch to poetry. But let's keep the update process manual--changing dependencies should be a conscious choice.

Are you happy to contribute a PR?

Does poetry support lock files for different python version out of the box or do we still have to manage them on a per-python-version basis?

[endorama]

Hi @tillahoffmann, I stumbled upon this issue and would be happy to contribute.

Does poetry support lock files for different python version out of the box or do we still have to manage them on a per-python-version basis?

I don't think using multiple Python versions at the same time is supported by Poetry, at least not in a way that would help with multiple lock files.

I found this issue relevant for this discussion: https://github.com/python-poetry/poetry/issues/305 and this doc page: https://python-poetry.org/docs/managing-environments/

I'm not a Python expert so I may be missing something obvious here, but why are requirements split by Python version? I would like to understand the reasoning behind as is something I've not seen usually done in other Python projects.

[tillahoffmann]

I'm not a Python expert so I may be missing something obvious here, but why are requirements split by Python version? I would like to understand the reasoning behind as is something I've not seen usually done in other Python projects.

The requirements are split by python version because some packages are not available for all python versions. I don't know whether this is an issue for the current requirements, but it caused some trouble in the past.

[santi]

@kiview Tagging you here for visibility. I noticed you're currently researching options for package management tools in Python, and I wanted to chime in with a strong recommendation for Poetry, a modern Python packaging tool that supports the PEP621/pyproject.toml standard, which is the way modern Python packages store their metadata and setup.

It supports a wide range of commands and tools, such as custom scripts (if you are familiar with the JS/NodeJS ecosystem, think of it like scripts in a package.json). It also has decent enough support for monorepo setups such as this repo. I'm currently managing a private repo with 77 packages depending on each other in a wonderful mesh of chaos, and Poetry makes it possible to automate most of the tasks I would otherwise have had to do manually.

[kiview]

Thanks for providing feedback @santi. Could we also start adopting PEP621 through pip alone? Would this be a logical first step?

[santi]

Yes, that would be a good first step on the way. Convert all the packages into using pyproject.toml for their dependencies and other stuff, but still use piptools to build the final requirements.txts for the different operating systems.

What could also be done at the same time is to use markers in the pyproject.toml files themselves to specify specific dependencies for different platform architectures / OSes, removing the need to have multiple combined requirement.txt files. Note: Don't know for sure if piptools support extracting markers, but it is worth looking into at least.

[balint-backmaker]

Hey, to add to the conversation, I also have a strong preference for poetry for the simple reason that it does both dependency management, as well as packaging and pushing to pypi in one tool. On top of that, it receives frequent updates and fixes, takes a lot of pressure off of the users. :+1:

Other than that, I'd like to point at the fact that while there may be several packages, only one is realistically released under sem-ver. The rest remain in this odd state of 0.0.1rc1 https://pypi.org/search/?q=testcontainers I'm not sure I'd call it a "mono repo" in that sense, but happy to be proven wrong (and maybe we should release them as testcontainers-x properly? @kiview does release drafter support that mono-repo style?

One final thing is I'm a bit skeptical on the platform compilation - why do we need this? Can you point at examples? (I do need to read more of the modules I guess!)

[santi]

@balint-backmaker I am actually not sure if the different platforms/OSes actually require different dependencies anywhere in this repo, I only guess at this because it currently lists dependencies for individual OSes in the requirements directory.

[alexanderankin]

Other than that, I'd like to point at the fact that while there may be several packages, only one is realistically released under sem-ver.

i believe the relevant commit history can be found in pr #290

[alexanderankin]

the matrix of lockfile predates this - imo it should go - it doesn't get shipped to the user, so it doesn't actually create a reproducible build when it matters. seems like a lot of maintenance burden without the most important gain. besides, wouldn't we want the builds to start failing here first and be able to isolate what needs to change to unbreak the new users installing the library?

edit: unless i am missing something

[santi]

Other than that, I'd like to point at the fact that while there may be several packages, only one is realistically released under sem-ver. The rest remain in this odd state of 0.0.1rc1 https://pypi.org/search/?q=testcontainers I'm not sure I'd call it a "mono repo" in that sense, but happy to be proven wrong (and maybe we should release them as testcontainers-x properly?

As all of the potential testcontainers-x packages would require testcontainers, imo they are just extras and should then be installable with the testcontainers[postgres,X] extras specification instead of standalone packages. That would at the same time enforce version consistency between multiple extras/packages.

As for the calling this repo a monorepo, the naming does not really matter, but Poetry supports having multiple packages in the same folder structure within a root directory with its own pyproject.toml, like we have here. So naming does not really matter, but the structure we already have going is supported.

[santi]

the matrix of lockfile predates this - imo it should go - it doesn't get shipped to the user, so it doesn't actually create a reproducible build when it matters. seems like a lot of maintenance burden without the most important gain. besides, wouldn't we want the builds to start failing here first and be able to isolate what needs to change to unbreak the new users installing the library?

Lockfiles are meant for reproducible builds during development, not for reproducible builds for the consumers of the package. As you correctly stated the lockfiles does not get shipped to the user, and that is because as a library developer we need to be lenient when it comes to package version ranges, so that installing the package and its dependencies has the smallest amount of chance of conflicting with the version ranges og other packages. If we shipped the lockfile and it was meant to be used during install, we are basically locking our dependencies for other packages to a single version, which would 100% conflict with other packages if they shipped with lockfiles as well.

As for breaking things here first, it makes for a truly horrible development experience (and would probably scare off a lot of new contributors) if there is a chance that the same commit that ran successfully in our CI / last install suddenly fails for new installs. This is what dependency management tools like Dependabot and Renovate are for - they try upgrading dependencies -> install -> report status in a raised PR. That way we can have complete control of our dependencies and get reproducible builds and install while we at the same time get checks for compatibility with new package versions.

[totallyzen]

Hey folks! Some updates on this subject, #408 will address this issue and I am aiming to close it when that gets merged (only one task left on it)