Open flying-sheep opened 1 year ago
I really need this feature. Is it planned for the future?
In scikit-image, we've resorted to writing out various requirement.txt
files each time pyproject.toml
is updated.
PSA: It will be possible to extract the build deps (with their transitive requirements) of a project in the next pip-tools release — 3 weeks ago we merged a PR that took like a year to polish: https://github.com/jazzband/pip-tools/pull/1681.
@stefanv I also used to maintain a manually extracted copy of the build deps for use with PIP_CONSTRAINT
in the past, but the new feature is going to cover that need. If you'd like to try it out, test the version from Git. See https://pip-tools.rtfd.io/en/stable/#maximizing-reproducibility for the hints.
@pfmoore this new pip-tools feature extracts both static and dynamic deps using said callbacks FYI.
P.S. As for the Ah, it looks like PEP 735 is attempting to address this.--only-deps
feature — I think it's a can of worms, this would encourage projects to expose things through metadata that are meant to be private. Like, why would an end user (not a contributor) need access to the docs or test deps? Those deps usually need env-dependent pinned constraints rather than follow the normal pkg runtime dep considerations…
PSA: It will be possible to extract the build deps (with their transitive requirements) of a project in the next pip-tools release — 3 weeks ago we merged a PR that took like a year to polish: jazzband/pip-tools#1681.
From reading that PR, it looks like you will have to compile the package in order to extract the build dependencies?
@stefanv no, you can feed it a pyproject.toml
. The PEP 517 hooks are called to augment the statically listed build deps. But then, there's also dependency resolution to get the transitive deps, so that might end up building some of the packages in the tree, if that's what you're asking.
Python developers for years if not decades have followed a very simple deploy process for standalone apps:
For reverse compatibility and to unify the Python development experience, as part of this PR could you please add to pyproject.toml
a [build-system]
setting that disables every other step besides install dependencies? Then we get an even simpler command (pip install .
) and identical behavior exactly like pip install -r
provided. It's important this be codified inside pyproject.toml
so that it's self-documenting how to build the code (without redundantly coding it for CI automation with --only-deps
arguments, or simply without having to remember when installing manually, etc.).
tl;dr: adding a only_deps
flag to [build-system]
is valuable for 3 reasons:
pip install -r
into a venv
deployment flow described abovepyproject.toml
pyproject.toml
--only-deps
flags or side-effecting pip-compile
Similar sentiments expressed: https://stackoverflow.com/questions/74354009/install-dependencies-from-pyproject-toml-but-not-the-package https://stackoverflow.com/questions/77150911/equivalent-to-pip-install-r-requirements-dev-txt-for-pyproject-toml https://github.com/astral-sh/uv/issues/1516 https://github.com/pypa/flit/pull/631 https://github.com/pypa/pip/issues/12425 https://github.com/pypa/pip/issues/2171 https://github.com/pypa/pip/issues/11584
PSA: It will be possible to extract the build deps (with their transitive requirements) of a project in the next pip-tools release — 3 weeks ago we merged a PR that took like a year to polish: jazzband/pip-tools#1681.
FTR that feature has finally been released in pip-tools v7.4.0. So the build deps constraints/lockfile are extractable using something like pip-compile --all-build-deps --output-file=build-constraints.txt --strip-extras pyproject.toml
now. Hopefully, this will provide a workaround for a portion of this feature request.
From reading that PR, it looks like you will have to compile the package in order to extract the build dependencies?
@stefanv I only now realized why you were asking about compiling :laughing: The command pip-compile
is a pip-tools CLI for making constraint files out of input deps, that's what "compiling" means in that context.
Edit: this was just a plea to remember the science use-case, but it added nothing to the discussion in the light of @webknjaz's reply. Sorry!
@matterhorn103 pretty sure that it's pending the acceptance/completion of PEP 735 so that there could be an installer-agnostic way of declaring those. It specifically mentions the scientific projects among the use cases, by the way. Please, check it out, it's an interesting read.
@webknjaz Ah, I should have known better than to comment on such an old thread... Thanks for the reply and the pointers. I also since then found threads like this one discussing the future options and particularly the scientific context.
There are some references in here to reading metadata from pyproject.toml
directly. We do this in uv (when the dependencies are not declared as dynamic, etc.), so just to share from my own experience, the one issue we've run into is that some tools have added non-standard extensions to the project.dependencies
table.
For example, in Hatch, you do:
[project]
dependencies = [
"black @ {root:uri}/black_editable"
]
...to declare a dependency at a path relative to the current project. This is called the Context formatting API.
Similarly, PDM injects a magic ${PROJECT_ROOT}
variable for local dependencies:
[project]
dependencies = [
"sub-package @ file:///${PROJECT_ROOT}/sub-package",
"first @ file:///${PROJECT_ROOT}/first-1.0.0-py2.py3-none-any.whl",
]
I think we actually support that in uv too (it falls out of supporting environment variable expansion in requirements.txt
parsing), though I'd like to get rid of it. I view all of these as quite clearly violating PEP 621 so pip would need to make a decision as to whether they're supported, if it were to start reading from pyproject.toml
directly.
If I remember correctly, the goal of static/dynamic in PEP 621 was to allow for backend interoperability and allow decision making on the metadata without invoking the backend.
I view all of these as quite clearly violating PEP 621
So I share this view, as they prevent these optimizations.
Actually, I think frontends should check that backends honor static metadata in the wheels they build.
I view all of these as quite clearly violating PEP 621
So do I.
https://github.com/pypa/pip/issues/11440#issuecomment-1445119899 is the currently agreed upon user-facing design for this feature.
What's the problem this feature will solve?
In #8049, we identified an use case for installing just the dependencies from
pyproject.toml
.As described in the solution section below
--only-deps=<spec>
would determine all dependencies of<spec>
excluding that package itself and install those without installing the package. It could be used toThis example shows both use cases:
Instead of the solution from #8049, @pradyunsg prefers a solution similar to the one below: https://github.com/pypa/pip/issues/8049#issuecomment-1079882786
Describe the solution you'd like
One of those two, or similar:
(used in the example above)
--only-deps
would work like-r
in that it’s not a flag globally modifying pip’s behavior but a CLI option with one argument that can be specified multiple times. Unlike-r
it accepts a dependency spec and not a path to a file containing dependency specs.Where
pip install <spec>
first installs all dependencies and then (build and) install the package referred to by the spec itself,pip install --only-deps=<spec>
would only install the dependencies.--only-deps
would work like--[no|only]-binary
, in that it requires an argument specifying what package not to install. A placeholder like:requested:
could be used, e.g.:Alternative Solutions
Re-using
-r
instead of adding--only-deps
.I don’t think this is a good idea, since people would be tempted to do
-r pyproject.toml
which would be wrong (Dependency specs including file paths look like like./path/to/pkg[extra1,extra2]
)Making
--only-deps
a global switch modifying pip’s behavior like e.g.--pre
.I have found that global switches like that are dangerous and not very intuitive. To install a dev version of your package, doing
pip install --pre mypkg
seems innocuous but will actually install dev versions ofmypkg
and all its dependencies that have any dev versions. It’s safer to do something likepip install mypkg>=0.1.post0.dev0
to limit dev version installations to one package. Similarly it’s unclear what a--only-deps
switch would apply to. Wouldpip install -r reqs.txt --only-deps
install the dependencies of every package specified in the file but none of those packages?Using e.g. beni to convert PEP 621 dependencies to a requirements.txt.
This works even today but feels like is shouldn’t be necessary as it involves quite a few steps, including writing a file to disk.
Additional context
NA
Code of Conduct