Duplicated lists of dependencies

[leplatrem]

Our requirements.txt is mainly used as a lock / constraints file. We use it everywhere with pip install -c requirements.txt (except in the docker file, see PR).

pip-compile is not able to generate a constraints file from all mentioned dependencies in pyproject.toml (main + extras).

pip does not support extras dependencies in constraints files (that's why we have strip-extras = true), and does not support hashes when installing the package locally in editable mode (that's why we have generate-hashes = false). We need editable mode to be able to make changes without having to reinstal the package between two tests runs for example.

In #3359, we took this approach:

• main and extra dependencies listed in pyproject.toml
• all sort of dependencies manually combined in requirements.in
• requirements.txt generated using pip-compile.

Ideally we would to avoid this duplication, and have one source of truth for all kinds of dependencies. Then have a constraints/lock file for reproducible builds/tests/installations.

Solution #0: status quo

This duplication is fine. The list of package does not change so often.

Solution #1: Get rid of extra dependencies

Simplify everything, and install all dependencies instead of having monitoring, postgresql, memcached extras.

We would still need a distinction for development dependencies, but that's straightforward and doesn't really require pinning.

Solution #2: split into several *.in files

setuptools is able to read *.in files from pyproject.toml. That's what we do in kinto-attachment

We could have:

• requirements.in
• requirements-postgresql.in
• requirements-memcached.in
• requirements-monitoring.in
• requirements-test.in
• requirements-dev.in

The source of truth for the list of dependencies would be these .in files.

Solution #3: use poetry

poetry has a simple .Piplock feature, and has many features around fine-grained installation of extra dependencies and groups.

But poetry does not support the [project] section (yet).

We would have to move away from setuptools and setuptools-scm, and use a [tools.poetry] section.

[alexcottner]

I'm a big fan of solution 1, keep things as simple as possible. We only publish one package anyway, right? If anybody wants to make a custom build it should still be just as simple to keep doing that.

[leplatrem]

Something to consider, with Solution 1 we can't just use pip install kinto && kinto start to start a quick in-memory storage backend. Because we would require postgresql client libs (ie. compiling psycopg for example). Also in all our repos' CI like kinto-wizard, etc.