charliermarsh
commented
9 months ago The originating issue is fixed and will be out in the next release. (That is: pydantic<2.0.0 will no longer allow pydantic=2.0b3.)
Closed erdembanak closed 9 months ago
charliermarsh
commented
9 months ago The originating issue is fixed and will be out in the next release. (That is: pydantic<2.0.0 will no longer allow pydantic=2.0b3.)
llucax
commented
9 months ago @llucax - this feels like a misrepresentation of what’s been said in the thread and even a little antagonistic. The reference to “our design” is with respect to how users opt-in or out of pre-release handling. The fact that “<2.0.0” should not include “2.0.0b3” is acknowledged as a bug.
Hi @charliermarsh, I'm sorry if it came across antagonistic and if I misinterpreted what's been said. Rereading some message I know see you acknowledged the bug but still said there was no intention of fixing it soon in https://github.com/astral-sh/uv/issues/1641#issuecomment-1951412679.
I just wanted to bring to your attention that if this was not addressed it sadly meant that the tool was not really a drop-in replacement of pip for the many cases where projects make pre-releases, and since this is not even in control of the user, it means it can't be reliable used ever, otherwise it might broke unpredictably if any dependency decides to make a pre-release.
In any case, thanks for addressing it so fast finally and for the great tool! :heart:
Czaki
commented
9 months ago Hi @charliermarsh, I'm sorry if it came across antagonistic and if I misinterpreted what's been said. Rereading some message I know see you acknowledged the bug but still said there was no intention of fixing it soon in #1641 (comment).
Please test first. During the discussion, there was invented workaround for this problem and everything should work as expected.
Marenz
commented
9 months ago We tested the newest release and the behavior is def. better.
However, it still differs and fails from pip when an indirect dependency is a prerelease:
uv pip install "frequenz-client-base @ git+https://github.com/frequenz-floss/frequenz-client-base-python.git@
57d3093" --reinstall
Updated https://github.com/frequenz-floss/frequenz-client-base-python.git (57d3093) × No solution found when resolving dependencies:
╰─▶ Because only frequenz-channels<1.0.0b2 is available and frequenz-client-base==0.0.post31+g57d3093 depends on frequenz-channels>=1.0.0b2, we can conclude that
frequenz-client-base==0.0.post31+g57d3093 cannot be used.
And because only frequenz-client-base==0.0.post31+g57d3093 is available and you require frequenz-client-base, we can conclude that the requirements are unsatisfiable.
hint: frequenz-channels was requested with a pre-release marker (e.g., frequenz-channels>=1.0.0b2), but pre-releases weren't enabled (try: `--prerelease=allow`)I had to use --reinstall otherwise it wouldn't say any error at all if I had it installed before, that was a bit surprising.
charliermarsh
commented
9 months ago Thanks -- that's intentional and documented: pre-releases need to be enabled globally or on a per-package basis. Did you try enabling pre-releases per the error message?
Marenz
commented
9 months ago Well, sure, with that flag it works... With pip this works without any extra flag though and if the idea is still that it is a drop-in replacement for it, then this is preventing this unfortunately.
llucax
commented
9 months ago Also it is not easy to pass extra flags when using other tools like nox, or is there any configuration file or env variable that could be used to pass options too?
llucax
commented
9 months ago Also, if I understand correctly, using --prerelease=allow will select pre-releases whenever they are available, even if they are not used explicitly as part of the version requirements (like if an indirect dependency x >= 2.0 in its requirements, and there is a x has 2 versions: 1.0.0 and 2.0.0-beta1 release, I don't want the pre-release to be selected, but if the indirect dependency were x >= 2.0.0-beta1, then it should work. This is what pip does, as far as my experience goes.
charliermarsh
commented
9 months ago You're not required to use --prerelease=allow to enable prereleases for a package. If you add x >= 2.0.0-beta1 as a direct dependency, it will enable pre-releases for x. You just need some way to signal to uv upfront (not as a transitive dependency) that prereleases should be enabled for x.
llucax
commented
9 months ago That's not ideal either, my project doesn't use x explicitly, so if my direct dependency stops using x, or uses a different version, my dependency will be incorrect. That's exposing implementation details of dependencies I should not care about.
llucax
commented
9 months ago What would be the issue with replicating pip's behavior?
Marenz
commented
9 months ago If this default behavior shouldn't be changed at all, it would be nice to have as alternative an environment variable PIP_PRERELEASE_MODE or whatever name fits best, that would switch on the same behavior that pip follows.
That way we can also easily integrate it in build tools by just setting the ENV.
charliermarsh
commented
9 months ago Yeah, we can add an environment variable. I'll take a look at it now.
charliermarsh
commented
9 months ago That's not ideal either, my project doesn't use x explicitly, so if my direct dependency stops using x, or uses a different version, my dependency will be incorrect. That's exposing implementation details of dependencies I should not care about.
For what it's worth, I think it's reasonable to ask you to care that your package relies on unreleased dependencies. I don't expect you to agree based on your comment, but it's a very reasonable position IMO.
What would be the issue with replicating pip's behavior?
The core challenge is that you open the door to changing the "available versions" for a package over the course of resolution. And the spec does not cover "dependency resolution"; it's focused on evaluating a single version identifier, which is really different. Even in pip, there's an extensive discussion around what the "correct" behavior is, what the spec implies, and the bugs that pip has today in pre-release handling:
notatallshaw
commented
9 months ago That's not ideal either, my project doesn't use x explicitly, so if my direct dependency stops using x, or uses a different version, my dependency will be incorrect. That's exposing implementation details of dependencies I should not care about.
For what it's worth, I think it's reasonable to ask you to care that your package relies on unreleased dependencies. I don't expect you to agree based on your comment, but it's a very reasonable position IMO.
A workaround to this is to pass in the constraint: somepackage >= 0.0a0
This solves the problem that it does not force a direct dependency on this package and solves the problem that it lets uv know upfront to allow prereleases on this package.
notatallshaw
commented
9 months ago I would also like to mention here pip's behavior is neither consistent nor intentional, the order of how pip compares prerelease requirements affects whether it will accept prereleases or not: https://github.com/pypa/pip/issues/12471
So for uv to match this behavior bug for bug, uv would need to:
Otherwise uv would forever either not select prerelease when pip does or selecting prereleases when pip doesn't.
llucax
commented
9 months ago Hi Charlie, thanks a lot for your reply and time again.
That's not ideal either, my project doesn't use x explicitly, so if my direct dependency stops using x, or uses a different version, my dependency will be incorrect. That's exposing implementation details of dependencies I should not care about.
For what it's worth, I think it's reasonable to ask you to care that your package relies on unreleased dependencies. I don't expect you to agree based on your comment, but it's a very reasonable position IMO.
Yeah, we strongly disagree here. I can give more arguments why I think this is not reasonable, I'm guessing you already thought about those and I will not say anything new to you, but just in case:
I can't see any justification in why should I care at all about which dependencies my dependencies use as long as they are not direct dependencies to my project too, those being pre-releases or not. Once I chose a dependency, I decided to trust it, after all I'm executing arbitrary code from them. Why should I care if the maintainer of one of those dependencies decided the best course of action is to use a pre-release in their dependencies? How would I be better than them to judge if I might not even know what those dependencies are for or follow their development? I will just be copy&pasting dependencies to my project's dependencies. I don't see a point in doing that.
This also adds a lot of maintenance cost for me, as now I need to track all transitional dependencies and litter my direct dependencies. Not only that, now how my dependencies evolve is not transparent to me anymore, as if they change those dependencies I need to update mine or risk my project to be broken or use outdated dependencies.
Is really hard for me to see any advantages in the approach you are proposing.
What would be the issue with replicating pip's behavior?
The core challenge is that you open the door to changing the "available versions" for a package over the course of resolution. And the spec does not cover "dependency resolution"; it's focused on evaluating a single version identifier, which is really different. Even in pip, there's an extensive discussion around what the "correct" behavior is, what the spec implies, and the bugs that pip has today in pre-release handling:
- Pre-release not selected when remotely available pre-releases for version specifiers is the only versions that satisfies the version specifier pypa/pip#12469
- How do I exclude Pip selecting any pre-release? pypa/pip#12470
- Pip inconsistent on whether it looks up a pre-release based on the operator of the specifier pypa/pip#12472
OK, I understand pip has bugs too, and there might be cases where if something is a bug or not might even be debatable, or there might be cases when one behavior is preferred to other.
I actually think it is very nice that uv gives more control over how pre-releases could be selected, that's awesome. My point here is if the idea is that uv is a drop-in replacement, one of these modes should be "be like pip". Otherwise it is not.
And I guess this is the part I don't understand very well and find a bit contradictory. I could totally understand if you said "oh, yes, we are not doing the same as pip here but we won't fix it because it's too hard and we have other more important stuff to work on", that I can totally understand, but my impression is, at least for this issue in particular, that this is not considered an issue in uv and thus there is no intention to fix it (where "fix it" doesn't mean is the right behavior nor is following the specs, "fix it" means behave like pip).
If I'm correct about this, which also I think is totally fine, is your tool and I'm still very grateful you open source it and are sharing it with the world, I just think you should probably not say it is a drop-in replacement for pip because this is a bit misleading, and can cause frustration and waste users time, as well as yours, and even maybe even pip maintainers time, as they will inevitably have to deal with the differences if users start thinking they are interchangeable.
The main reason why we decided to try uv is because the first highlight in the README says "⚖️ Drop-in replacement for common pip, pip-tools, and virtualenv commands.". We can't do the jump and say our projects only work with uv and don't care if it doesn't work with pip, we need to be compatible with pip. While writing this I just realized there is a new document highlighting the differences with pip, so thanks for that too, I guess it will help to clarify and set more realistic expectations.
notatallshaw
commented
9 months ago I can't speak for astral or their communication, but given you replied to me, I will point out by your definition of drop in replacement then new versions of pip are not a drop in replacement for old versions of pip.
For example pip 20.2 can install plenty of things because of it's limited resolver compared to pip 20.3, equally the resolver behavior has changed significantly between 20.3 and 24.0, meaning different requirements will 1) Resolve or not resolve depending on what pip version you have, and 2) Produce different solutions.
I think you are vastly underestimating the challenges of package resolution, from what I've seen so far uv works in all but unusual edge cases and is usually multiple times faster, that's pretty amazing and will save a lot of devs a lot of time over the long run.
Anyway, I tested my workaround which solves not needing --prerelease=allow and not adding a direct dependency on to a transitive dependency.
Here is the failing example
$ echo "frequenz-client-base @ git+https://github.com/frequenz-floss/frequenz-client-base-python.git@57d3093" | uv pip compile -
Updated https://github.com/frequenz-floss/frequenz-client-base-python.git (57d3093)
× No solution found when resolving dependencies:
╰─▶ Because only frequenz-channels<1.0.0b2 is available and frequenz-client-base==0.1.0.post21+g57d3093 depends on frequenz-channels>=1.0.0b2, we can conclude
that frequenz-client-base==0.1.0.post21+g57d3093 cannot be used.
And because only frequenz-client-base==0.1.0.post21+g57d3093 is available and you require frequenz-client-base, we can conclude that the requirements are
unsatisfiable.
hint: frequenz-channels was requested with a pre-release marker (e.g., frequenz-channels>=1.0.0b2), but pre-releases weren't enabled (try:
`--prerelease=allow`)But by adding the constraint with a prerelease marker then one can mark this specific package as allowing prerelease without adding it as a requirement to install:
$ echo "frequenz-channels >= 0.0a0" > constraint.txt
$ echo "frequenz-client-base @ git+https://github.com/frequenz-floss/frequenz-client-base-python.git@57d3093" | uv pip compile - --constraint constraint.txt
Updated https://github.com/frequenz-floss/frequenz-client-base-python.git (57d3093
Resolved 11 packages in 361ms
# This file was autogenerated by uv via the following command:
# uv pip compile - --constraint constraint.txt
anyio==4.3.0
# via watchfiles
frequenz-channels==1.0.0rc1
# via frequenz-client-base
frequenz-client-base @ git+https://github.com/frequenz-floss/frequenz-client-base-python.git@57d3093640f2def2b976a738fcbb9605a02e3576
grpcio==1.62.0
# via
# frequenz-client-base
# grpcio-tools
grpcio-tools==1.62.0
# via frequenz-client-base
idna==3.6
# via anyio
protobuf==4.25.3
# via grpcio-tools
setuptools==69.1.1
# via grpcio-tools
sniffio==1.3.1
# via anyio
typing-extensions==4.10.0
# via
# frequenz-channels
# frequenz-client-base
watchfiles==0.21.0
# via frequenz-channelsI think you are vastly underestimating the challenges of package resolution,
Trying not to repeat myself too much here. I do understand it is a hard problem, and again if the reason not to allow this case is "it's too hard", that is totally reasonable for me. But it doesn't seem to be the case here.
from what I've seen so far uv works in all but unusual edge cases and is usually multiple times faster, that's pretty amazing and will save a lot of devs a lot of time over the long run.
I also agree that it is unrealistic to pretend that uv behaves like pip in every obscure and niche corner case, but I honestly don't think this case (an indirect dependency using explicitly a pre-release in their dependencies) is an obscure corner case, it seems like a pretty reasonable case.
For example pip 20.2 can install plenty of things because of it's limited resolver compared to pip 20.3, equally the resolver behavior has changed significantly between 20.3 and 24.0, meaning different requirements will 1) Resolve or not resolve depending on what pip version you have, and 2) Produce different solutions.
This for me falls into the basket of "pip has bugs too". There is also another difference, following breaking changes in pip is just advancing in time. We can say "I'm compatible with pip 24.0 or newer", but if uv diverges from pip, you are creating a fork. The problem is not leniar anymore.
But by adding the constraint with a prerelease marker then one can mark this specific package as allowing prerelease without adding it as a requirement to install:
Thanks, but this workaround still requires manual intervention every time a dependency starts or stops using a pre-release (if I don't "disable" accepting pre-release for that dependency after my dependencies stopped using them, then I will still accept pre-releases for it even when it is not necessary for my project, which I don't want to allow), which is an unacceptable maintenance burden for me (and it also requires passing more options to UV, which complicate integration with other tools).
notatallshaw
commented
9 months ago This for me falls into the basket of "pip has bugs too".
No, it's just different choices about how to resolve, and there isn't a spec on this so there's nothing to test it against being a bug or not. (FYI, I hope we can make at least some limited progress on this by sharing tests between Python installers using packse that astral have kindly published).
In general anyway, there are actual bugs when a solution can't be found, or is found, but there does exist a solution, or doesn't, under the design resolver choices made.
There is also another difference, following breaking changes in pip is just advancing in time. We can say "I'm compatible with pip 24.0 or newer", but if uv diverges from pip, you are creating a fork. The problem is not leniar anymore.
Pip makes no guarantee of any consistency of it's resolution from one version to the next. So to say you're compatible with exact resolution choices on more than a single version would be impossible, as pip can't say that for itself.
FYI, there is now a pip compatibility page in uv's documentation that explains the known differences. I will remind people, this project is less than a month publicly announced, identifying issues against a broader ecosystem takes time, and to expect it upfront before there's been a chance to encounter these issues is unreasonable.
Thanks, but this workaround still requires manual intervention every time a dependency starts or stops using a pre-release
Yeah, that's the trade off of this workaround compared to allowing all prereleaes. But solves both previously raised concerns about other solutions.
And for me this is a feature, I would prefer to not have unexpected prereleases in my resolution solution unless I specifically request them.
llucax
commented
8 months ago Thanks for your reply and detailed explanation. Once more I totally understand this is a very new project and again I'm super grateful that it is published and it's great. I can also understand there is only a limited amount of resources and not everything can get implemented. My point still stands that it is a shame that is not compatible with pip in what I think is a reasonable use case (but apparently you don't) where pip does the right thing.
I can understand the current approach is a feature for you but it is a bug for other users (at least one, me 😀), so it would be great if you can add a pre-release resolution mode that supports it, and that will make the tool a bit more compatible with pip.
Anyway I don't want to waste more of your time, I think I said everything I had to say to try to convince you there is value in supporting this user care so I'll stop now.
Thanks again for the great tool and for taking the time to reply to my comments!
Using uv 0.1.3, when trying to install pydantic<2.0 with prerelease allow, uv installs pydantic==2.0b3. According to pubgrub, 2.0b3 shouldn't be contained in the set:
Running
uv pip install --prerelease=allow "pydantic<2.0" --verboseprovides the below output:Pubgrub documentation also says that this is a difficult problem; therefore the expected behaviour of the --prerelease=allow should be documented if this is the desired output.