search

astral-sh / uv

An extremely fast Python package and project manager, written in Rust.
https://docs.astral.sh/uv
Apache License 2.0
24.73k stars 714 forks source link

Support for the `pip-compile-multi` use case? #5487

Open mistercrunch opened 3 months ago

mistercrunch commented 3 months ago

For Apache Superset, we have fairly complex dependency management and have to use pip-compile-multi as a result.

pip-compile-multi is pretty much simply pip-compile with support for .in files to reference other .in files, making it such that you can create say a base.in file and build say a ci.in file or dev.in file that references that base file, allowing use to do a bit of composition with dependencies and keeping things DRY.

The main issue with pip-compile-multi are that 1. it's slow, 2. it's not compatible with dependabot and 3. I don't think it's actively maintained

I'm curious whether uv has support for this use case and/or whether it may be on the roadmap.

charliermarsh commented 3 months ago

We support those kinds of -r references in .in files. Is the benefit of pip-compile-multi that the output file also contains a -r, to avoid duplicating the dependencies?

mistercrunch commented 3 months ago

Yes that's exactly it.

mistercrunch commented 3 weeks ago

Hey, so say in the simple/common case where I want to manage a prod and dev setup (as documented here https://docs.astral.sh/uv/pip/compile/#locking-requirements):

So presumably I would do both

# for prod
uv pip compile pyproject.toml -o requirements-dev.txt
# and for dev
uv pip compile pyproject.toml requirements-dev.in -o requirements-dev.txt

That's all dandy, but now how do i guarantee that dev stays aligned with prod for the dependencies that they have in common? Presumably if i run both at the same time, with a little bit of luck they should stay aligned, though that's unless there's a conflict between the two. Say dev may have a sub-dependency that forces an older version of a shared/common dependency.

It's not a simple problem to solve, but it's a common/important use case.

Any way to provide these guarantees using uv currently? Any hope that'd get on the roadmap? Also what would be the best solution?

charliermarsh commented 3 weeks ago

@mistercrunch -- I think the common solution here would be to do something like this:

# For prod...
uv pip compile pyproject.toml -o requirements-prod.txt
# For dev
uv pip compile pyproject.toml requirements-dev.in -c requirements-prod.txt -o requirements-dev.txt

So, you use the output of the first command as a constraint for the second command. Then the versions will be kept in sync.

mistercrunch commented 2 weeks ago

Oh, smart! I should have thought of that. Is it guaranteed to produce matching version for all common libs and fail if somehow it can't find a match? I guess if it's not 100% guaranteed I could write a simple script to assert that common libs match as part of our CI.

adRn-s commented 2 weeks ago

I'm going through some effort myself trying to adopt uv in my workflow. It would be great if there was a uv pip compile-multi -d ./requirements/ available that one could use as drop-in... I was having multiple requirements folders, preparing the requirements.txt for multiple python versions (here) (Yes, in the past I was bitten by the fact that a requirements.txt wouldn't "compile" in a given python version that I wasn't using but thought it was supporting, hence this workflow..).

mistercrunch commented 2 weeks ago

One thing on my side is ensuring my future uv setup is going to work nicely with @dependabot. Related to this I found this here -> https://github.com/dependabot/dependabot-core/issues/10039. In the past, with pip-compile-multi, I had to write my own dependabot replacement to account for our pip-compile-multi setup.

Clearly there's more traction with uv (I think pip-compile-multi is on life-support and the dependabot integration ticket doesn't have traction). So it makes more sense to push for a uv/dependabot integration as opposed to a pip-compile-multi/dependabot integration. From my understanding, in its current state, dependabot is probably going to look at our pyproject.toml in isolation while running a pip-compile behind the scene (it simply doesn't know about our specific needs around pinning multiple package sets)

Ideally the dependency management system would have semantics/conventions about supporting different setups (while keeping common libs on the same version). Ideally uv would understand those semantics and dependabot would integrate deeply enough with uv to support those requirements.

Stepping back, it seems like this should be a very common need and arguably should be in-scope for uv (?) Does anyone know how it works in other mature languages/dependency-management tools? Maybe we could get some inspiration from other ecosystems (?)

mistercrunch commented 2 weeks ago

Taking a deeper look into the PR adding uv support to dependabot (open Pr as of the time of this comment), it looks like it's set up to have dependabot run a simple uv pip compile command. So if we add semantics in uv about supporting "atomic multi-outputs", dependabot should do the right thing. Here there any existing config file or env vars we could use to make it such that uv would run a series of commands (as suggested in the comment above https://github.com/astral-sh/uv/issues/5487#issuecomment-2400895477)?