pydantic<2.0.0 installs pydantic=2.0b3 when prereleases are allowed

erdembanak commented 9 months ago

Using uv 0.1.3, when trying to install pydantic<2.0 with prerelease allow, uv installs pydantic==2.0b3. According to pubgrub, 2.0b3 shouldn't be contained in the set:

"For example, "2.0.0-beta" is meant to exist previous to version "2.0.0". Yet, it is not supposed to be contained in the set described by 1.0.0 <= v < 2.0.0, and only within sets where one of the bounds contains a pre-release marker such as 2.0.0-alpha <= v < 2.0.0".

Running uv pip install --prerelease=allow "pydantic<2.0" --verbose provides the below output:

 uv::requirements::from_source source=pydantic<2.0
    0.002072s DEBUG uv_interpreter::virtual_env Found a virtualenv through VIRTUAL_ENV at: /home/erdemb/.venv
    0.002191s DEBUG uv_interpreter::interpreter Using cached markers for: /home/erdemb/.venv/bin/python
    0.002207s DEBUG uv::commands::pip_install Using Python 3.10.12 environment at /home/erdemb/.venv/bin/python
 uv_client::flat_index::from_entries 
 uv_resolver::resolver::solve 
      0.003246s   0ms DEBUG uv_resolver::resolver Solving with target Python version 3.10.12
   uv_resolver::resolver::choose_version package=root
   uv_resolver::resolver::get_dependencies package=root, version=0a0.dev0
        0.003287s   0ms DEBUG uv_resolver::resolver Adding direct dependency: pydantic<2.0
   uv_resolver::resolver::choose_version package=pydantic
     uv_resolver::resolver::package_wait package_name=pydantic
 uv_resolver::resolver::process_request request=Versions pydantic
   uv_client::registry_client::simple_api package=pydantic
     uv_client::cached_client::get_cacheable 
       uv_client::cached_client::read_and_parse_cache file=/home/erdemb/.cache/uv/simple-v1/pypi/pydantic.rkyv
 uv_resolver::resolver::process_request request=Prefetch pydantic <2.0
          0.004614s   1ms  WARN uv_client::cached_client Broken cache entry at /home/erdemb/.cache/uv/simple-v1/pypi/pydantic.rkyv, removing: failed to open file `/home/erdemb/.cache/uv/simple-v1/pypi/pydantic.rkyv`
          0.004832s   1ms DEBUG uv_client::cached_client No cache entry for: https://pypi.org/simple/pydantic/
       uv_client::cached_client::fresh_request url="https://pypi.org/simple/pydantic/"
       uv_client::cached_client::new_cache file=/home/erdemb/.cache/uv/simple-v1/pypi/pydantic.rkyv
       uv_client::registry_client::parse_simple_api package=pydantic
 uv_resolver::version_map::from_metadata 
   uv_distribution::distribution_database::get_or_build_wheel_metadata dist=pydantic==2.0b3
     uv_client::registry_client::wheel_metadata built_dist=pydantic==2.0b3
       uv_client::cached_client::get_serde 
         uv_client::cached_client::get_cacheable 
           uv_client::cached_client::read_and_parse_cache file=/home/erdemb/.cache/uv/wheels-v0/pypi/pydantic/pydantic-2.0b3-py3-none-any.msgpack
        0.967483s 964ms DEBUG uv_resolver::resolver Searching for a compatible version of pydantic (<2.0)
        0.967501s 964ms DEBUG uv_resolver::resolver Selecting: pydantic==2.0b3 (pydantic-2.0b3-py3-none-any.whl)
   uv_resolver::resolver::get_dependencies package=pydantic, version=2.0b3
     uv_resolver::resolver::distributions_wait package_id=pydantic-2.0b3
              0.967626s   0ms DEBUG uv_client::cached_client Found fresh response for: https://files.pythonhosted.org/packages/af/ce/dabc6cd90abaf903d84a329f29a5ce34643fd66ac2a78f3eb1c668bd4fb3/pydantic-2.0b3-py3-none-any.whl
        0.967715s   0ms DEBUG uv_resolver::resolver Adding transitive dependency: annotated-types>=0.4.0
        0.967749s   0ms DEBUG uv_resolver::resolver Adding transitive dependency: pydantic-core==0.39.0
        0.967804s   0ms DEBUG uv_resolver::resolver Adding transitive dependency: typing-extensions>=4.6.1
   uv_resolver::resolver::choose_version package=annotated-types
     uv_resolver::resolver::package_wait package_name=annotated-types
 uv_resolver::resolver::process_request request=Versions annotated-types
   uv_client::registry_client::simple_api package=annotated-types
     uv_client::cached_client::get_cacheable 
       uv_client::cached_client::read_and_parse_cache file=/home/erdemb/.cache/uv/simple-v1/pypi/annotated-types.rkyv
 uv_resolver::resolver::process_request request=Versions pydantic-core
   uv_client::registry_client::simple_api package=pydantic-core
     uv_client::cached_client::get_cacheable 
       uv_client::cached_client::read_and_parse_cache file=/home/erdemb/.cache/uv/simple-v1/pypi/pydantic-core.rkyv
 uv_resolver::resolver::process_request request=Versions typing-extensions
   uv_client::registry_client::simple_api package=typing-extensions
     uv_client::cached_client::get_cacheable 
       uv_client::cached_client::read_and_parse_cache file=/home/erdemb/.cache/uv/simple-v1/pypi/typing-extensions.rkyv
 uv_resolver::resolver::process_request request=Prefetch typing-extensions >=4.6.1
 uv_resolver::resolver::process_request request=Prefetch pydantic-core ==0.39.0
 uv_resolver::resolver::process_request request=Prefetch annotated-types >=0.4.0
          0.967990s   0ms  WARN uv_client::cached_client Broken cache entry at /home/erdemb/.cache/uv/simple-v1/pypi/annotated-types.rkyv, removing: failed to open file `/home/erdemb/.cache/uv/simple-v1/pypi/annotated-types.rkyv`
          0.968012s   0ms  WARN uv_client::cached_client Broken cache entry at /home/erdemb/.cache/uv/simple-v1/pypi/pydantic-core.rkyv, removing: failed to open file `/home/erdemb/.cache/uv/simple-v1/pypi/pydantic-core.rkyv`
          0.969678s   1ms  WARN uv_client::cached_client Broken cache entry at /home/erdemb/.cache/uv/simple-v1/pypi/typing-extensions.rkyv, removing: failed to open file `/home/erdemb/.cache/uv/simple-v1/pypi/typing-extensions.rkyv`
          0.969754s   1ms DEBUG uv_client::cached_client No cache entry for: https://pypi.org/simple/pydantic-core/
       uv_client::cached_client::fresh_request url="https://pypi.org/simple/pydantic-core/"
          0.969908s   2ms DEBUG uv_client::cached_client No cache entry for: https://pypi.org/simple/annotated-types/
       uv_client::cached_client::fresh_request url="https://pypi.org/simple/annotated-types/"
          0.969967s   2ms DEBUG uv_client::cached_client No cache entry for: https://pypi.org/simple/typing-extensions/
       uv_client::cached_client::fresh_request url="https://pypi.org/simple/typing-extensions/"
       uv_client::cached_client::new_cache file=/home/erdemb/.cache/uv/simple-v1/pypi/pydantic-core.rkyv
       uv_client::registry_client::parse_simple_api package=pydantic-core
       uv_client::cached_client::new_cache file=/home/erdemb/.cache/uv/simple-v1/pypi/typing-extensions.rkyv
       uv_client::registry_client::parse_simple_api package=typing-extensions
 uv_resolver::version_map::from_metadata 
   uv_distribution::distribution_database::get_or_build_wheel_metadata dist=typing-extensions==4.10.0rc1
     uv_client::registry_client::wheel_metadata built_dist=typing-extensions==4.10.0rc1
       uv_client::cached_client::get_serde 
         uv_client::cached_client::get_cacheable 
           uv_client::cached_client::read_and_parse_cache file=/home/erdemb/.cache/uv/wheels-v0/pypi/typing-extensions/typing_extensions-4.10.0rc1-py3-none-any.msgpack
              1.050399s   0ms DEBUG uv_client::cached_client Found fresh response for: https://files.pythonhosted.org/packages/d4/b2/4980568cd3814002f1c04b542e2dd52faa0cec11bd5bda99e43eefe9a808/typing_extensions-4.10.0rc1-py3-none-any.whl
       uv_client::cached_client::new_cache file=/home/erdemb/.cache/uv/simple-v1/pypi/annotated-types.rkyv
       uv_client::registry_client::parse_simple_api package=annotated-types
 uv_resolver::version_map::from_metadata 
   uv_distribution::distribution_database::get_or_build_wheel_metadata dist=annotated-types==0.6.0
     uv_client::registry_client::wheel_metadata built_dist=annotated-types==0.6.0
       uv_client::cached_client::get_serde 
         uv_client::cached_client::get_cacheable 
           uv_client::cached_client::read_and_parse_cache file=/home/erdemb/.cache/uv/wheels-v0/pypi/annotated-types/annotated_types-0.6.0-py3-none-any.msgpack
        1.184443s 216ms DEBUG uv_resolver::resolver Searching for a compatible version of annotated-types (>=0.4.0)
        1.184461s 216ms DEBUG uv_resolver::resolver Selecting: annotated-types==0.6.0 (annotated_types-0.6.0-py3-none-any.whl)
   uv_resolver::resolver::get_dependencies package=annotated-types, version=0.6.0
     uv_resolver::resolver::distributions_wait package_id=annotated-types-0.6.0
              1.184758s   0ms DEBUG uv_client::cached_client Found fresh response for: https://files.pythonhosted.org/packages/28/78/d31230046e58c207284c6b2c4e8d96e6d3cb4e52354721b944d3e1ee4aa5/annotated_types-0.6.0-py3-none-any.whl
   uv_resolver::resolver::choose_version package=pydantic-core
     uv_resolver::resolver::package_wait package_name=pydantic-core
 uv_resolver::version_map::from_metadata 
   uv_distribution::distribution_database::get_or_build_wheel_metadata dist=pydantic-core==0.39.0
     uv_client::registry_client::wheel_metadata built_dist=pydantic-core==0.39.0
       uv_client::cached_client::get_serde 
         uv_client::cached_client::get_cacheable 
           uv_client::cached_client::read_and_parse_cache file=/home/erdemb/.cache/uv/wheels-v0/pypi/pydantic-core/pydantic_core-0.39.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.msgpack
        6.767120s   5s  DEBUG uv_resolver::resolver Searching for a compatible version of pydantic-core (==0.39.0)
        6.767138s   5s  DEBUG uv_resolver::resolver Selecting: pydantic-core==0.39.0 (pydantic_core-0.39.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl)
   uv_resolver::resolver::get_dependencies package=pydantic-core, version=0.39.0
     uv_resolver::resolver::distributions_wait package_id=pydantic-core-0.39.0
              6.767260s   0ms DEBUG uv_client::cached_client Found fresh response for: https://files.pythonhosted.org/packages/10/15/82a7866bd5660dd4251c87ca23b01b8ba495ea0b7013bb8ee884dfce0311/pydantic_core-0.39.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
        6.767349s   0ms DEBUG uv_resolver::resolver Adding transitive dependency: typing-extensions*
   uv_resolver::resolver::choose_version package=typing-extensions
     uv_resolver::resolver::package_wait package_name=typing-extensions
        6.767401s   0ms DEBUG uv_resolver::resolver Searching for a compatible version of typing-extensions (>=4.6.1)
        6.767406s   0ms DEBUG uv_resolver::resolver Selecting: typing-extensions==4.10.0rc1 (typing_extensions-4.10.0rc1-py3-none-any.whl)
   uv_resolver::resolver::get_dependencies package=typing-extensions, version=4.10.0rc1
     uv_resolver::resolver::distributions_wait package_id=typing-extensions-4.10.0rc1
Resolved 4 packages in 6.76s
    6.767756s DEBUG uv_installer::plan Requirement already cached: annotated-types==0.6.0
    6.767789s DEBUG uv_installer::plan Requirement already cached: pydantic==2.0b3
    6.767807s DEBUG uv_installer::plan Requirement already cached: pydantic-core==0.39.0
    6.767828s DEBUG uv_installer::plan Requirement already cached: typing-extensions==4.10.0rc1
 uv_installer::installer::install num_wheels=4
Installed 4 packages in 14ms
 + annotated-types==0.6.0
 + pydantic==2.0b3
 + pydantic-core==0.39.0
 + typing-extensions==4.10.0rc1

Pubgrub documentation also says that this is a difficult problem; therefore the expected behaviour of the --prerelease=allow should be documented if this is the desired output.

notatallshaw commented 9 months ago

How Pip behaves with exclusive operators only:

Does not install pydantic prerelease: pip install --dry-run --pre "pydantic < 2.0" Does install pydantic prerelease: pip install --dry-run --pre "pydantic < 2.0rc1"

How Pip behaves with inclusive operators on June 18th 2023 (when there was only pre-releases and no final 2.0, once the final release happened inclusive operators install 2.0 final, and exclusive operators are the same):

Does install pydantic prerelease: pip install --dry-run --pre "pydantic <= 2.0" Does install pydantic prerelease: pip install --dry-run --pre "pydantic <= 2.0rc1"

The relevant parts of the Python spec:

Some recent discussion:

IMO the Python spec leads to ambigious edge cases, I think both uv's and Pip's behavior here can be interpreted to be part of the spec.

mpizenberg commented 9 months ago

According to pubgrub

@erdembanak what pubgrub doc states in that page is that the semantic meaning of "<2.00" is different than its mathematic, which makes solving difficult with solvers that cannot alter the knowledge it gathers while solving depending on context. For this reason, the resolver has to take decisions that are context independent. In the case of uv, as stated in their readme https://github.com/astral-sh/uv?tab=readme-ov-file#pre-release-handling, they choose the two rules:

If the package is a direct dependency, and its version markers include a pre-release specifier (e.g., flask>=2.0.0rc1).
If all published versions of a package are pre-releases.

But if you --prerelease=allow then it allows pre-releases for all dependencies. In your case, 2.0b3 is mathematically < 2.0 and you allowed pre-release for all, so it seems normal that it installs the prerelease.

dimbleby commented 9 months ago

I do not think the spec is ambiguous

The exclusive ordered comparison <V MUST NOT allow a pre-release of the specified version unless the specified version is itself a pre-release.

(bolded capital letters from the original)

mpizenberg commented 9 months ago

Also in the spec: https://packaging.python.org/en/latest/specifications/version-specifiers/#handling-of-pre-releases

Dependency resolution tools SHOULD also allow users to request the following alternative behaviours:

accepting pre-releases for all version specifiers

excluding pre-releases for all version specifiers (reporting an error or warning if a pre-release is already installed locally, or if a pre-release is the only way to satisfy a particular specifier)

dimbleby commented 9 months ago

I suggest that the way to read this so that the spec is consistent is: the section about handling pre-releases is discussing the behaviour for those specifiers that only implicitly disallow pre-releases.

Exclusive ordered comparisons very clearly and very explicitly always disallow pre-releases.

notatallshaw commented 9 months ago

I do not think the spec is ambiguous

The exclusive ordered comparison <V MUST NOT allow a pre-release of the specified version unless the specified version is itself a pre-release.

What's ambiguous is when the resolver specifies the prerelease flag, does that implicitly mean all versions include prerelease?

If it does the uv is following the spec and pip is not.

If it doesn't what does it even mean to include a prerelease flag?

dimbleby commented 9 months ago

If it doesn't what does it even mean to include a prerelease flag?

see the previous comment.

I believe that the section about handling pre-releases is describing an option for eg the inclusive ordered comparisons

for those, the comparison operator in itself does not exclude pre-releases
but pre-releases are nevertheless in general implicitly excluded
an "allow-prereleases" flag is intended to overrule that general implicit exclusion
such a flag is not intended to overrule the basic meaning of a comparison eg the exclusive ordered comparison - regardless of any general implicit rules - always explicitly excludes prereleases

Eg ==2.0.0 does not accept 2.0b3 just because "allow-prereleases" is set: the version 2.0b3 does not satisfy the meaning of ==2.0.0. Similar reasoning holds for <2.0.0

notatallshaw commented 9 months ago

That's certainly a way to interpret the spec, but it doesn't explicitly state it, so one can also interpret how uv has implemented it.

dimbleby commented 9 months ago

I certainly agree the spec could be clearer, but I think you have to work quite hard to justify ignoring

The exclusive ordered comparison <V MUST NOT allow a pre-release of the specified version unless the specified version is itself a pre-release.

notatallshaw commented 9 months ago

You only have to read the next sentence and take prerelease flag to mean that prerelease versions are included.

It takes only the work of correlating two things together.

notatallshaw commented 9 months ago

In any case, I do agree your interpretation is reasonable and it's what pip does. So probably uv maintainers want to follow what pip does when what pip does makes sense?

mpizenberg commented 9 months ago

Let's not fight over this. Let's wait for what the uv team has to say.

charliermarsh commented 9 months ago

Hmm... I think my honest read of the spec is that even with --prerelease=allow, the specifier <2.0.0 should not include 2.0b3, but that it should allow 1.9b3 based on "a pre-release of the specified version".

That being said, I think we're unlikely to implement it (at least not soon), since (1) it breaks fundamental properties of version solving, (2) makes the solver much harder to implement and reason about, and (3) frankly, may be even more confusing for users in some cases. In other words, I just doubt it will be high-priority enough for us to support it in the near future.

charliermarsh commented 9 months ago

I suppose I'd also add (in our defense) that I wouldn't be surprised if there is no Python resolver that is fully spec compliant on pre-releases. (I can't say this definitively, I haven't looked enough.) In particular, the last clause here is very hard to satisfy, since it means you have to expand the set of candidate packages as you proceed with a resolution:

Pre-releases of any kind, including developmental releases, are implicitly excluded from all version specifiers, unless they are already present on the system, explicitly requested by the user, or if the only available version that satisfies the version specifier is a pre-release.

(I'm also not sure how that clause is intended to be interpreted. For example, say you have releases for Pydantic at 1.8.0, 1.9.0a, and 2.1.0 (but no other releases). And then you have one package that requests Pydantic at <2.0.0, and another that requests Pydantic at !=1.8.0. What should happen? Both markers can be individually satisfied without a pre-release, but taken together, they can only be satisfied with a pre-release.)

notatallshaw commented 9 months ago

In particular, the last clause here is very hard to satisfy, since it means you have to expand the set of candidate packages as you proceed with a resolution:

Pre-releases of any kind, including developmental releases, are implicitly excluded from all version specifiers, unless they are already present on the system, explicitly requested by the user, or if the only available version that satisfies the version specifier is a pre-release.

This is discussed, at some length, in the Python thread I linked.

I believe Poetry does attempt to follow this spec, but fails under more complicated circumstances (e.g. when backtracking on transitive dependencies), and there is an open issue on Poetry side for this.

Pip does not to follow it, as linked in the various issues I posted. One of the pip maintainers characterizes the rules pip follows in the linked Python thread pretty early on. Which I believe is the approach rip is following for this.

charliermarsh commented 9 months ago

(Thanks, I swear I did read all of the pip issues you linked, but I didn’t make it through the entire Python thread yet.)

dimbleby commented 9 months ago

I wouldn't be surprised if there is no Python resolver that is fully spec compliant

For what it's worth I don't know any other solver that considers 2.0b3 to be compatible with <2.0.0, though I'm sure they all have bugs!

... What should happen? ...

it can get even more confusing than that! What if the solver goes down a path such that dependency foo can only be satisfied with a pre-release; but - if only it had been omniscient - there was another path available such that foo did not need a pre-release? What a mess!

notatallshaw commented 9 months ago

Yeah, I'm sorry the discussion is so fragmented.

Honestly I got pretty frustrated with that topic, in particular it makes it extremely difficult to try and apply some static analysis resolver optimizations I wanted to try out, because "what is the right answer" appears to both involve a lot of contextual state about what the resolver has exhausted so far and actually be pretty ambiguous in relationship to the spec.

I am looking at packse, and may try and add some test cases that everyone can actually agree on, and then go from there. But I can't promise anything soon.

erdembanak commented 9 months ago

Thanks for the comments.

Hmm... I think my honest read of the spec is that even with --prerelease=allow, the specifier <2.0.0 should not include 2.0b3, but that it should allow 1.9b3 based on "a pre-release of the specified version". ,

I agree with this understanding. Let me tell you how I run into this issue (I wasn't just messing around). This became a problem for me when I am working with an easily reproducible requirements file:

azure-cli==2.46.0 pydantic<2.0.0

If I allow prerelease, pydantic is installed as beta and if I don't, I can't install the packages due to azure-cli depending on a prerelease:

  × No solution found when resolving dependencies:
  ╰─▶ Because there is no version of azure-mgmt-sql==4.0.0b8 and azure-cli==2.46.0 depends on azure-mgmt-sql==4.0.0b8, we can conclude that
      azure-cli==2.46.0 cannot be used.
      And because you require azure-cli==2.46.0, we can conclude that the requirements are unsatisfiable.

      hint: azure-mgmt-sql was requested with a pre-release marker (e.g., azure-mgmt-sql==4.0.0b8), but pre-releases weren't enabled (try:
      `--prerelease=allow`)

Even without --pre marker, pip installs azure-cli; so this is not a problem in pip. I am currently using overrides file (giving pydantic==1.x.x in there) to overcome this problem (the pydantic constraint is not coming from my requirement file); but it is just a workaround and makes migrating to uv from pip a bit harder than it should be.

notatallshaw commented 9 months ago

Hmmm, the problem of "I want prereleases from one part of my requirements but not the other part of my requirements" is not supported by pip either, it just happens to be that the behavior pip has chose works out for you.

Unless I'm missing something fundamental about the soundness of the behavior I'm sure one could come up with another example where it would not work for pip but it would work for uv. But pip is so pervasive that everyone checks their use case works for it and if not changes their use case.

If you're interested in a workaround you can split this into multiple commands with constraints, I actually do this with pip constantly to keep my environment in sync, so this workflow will work with both uv and pip:

$ echo "azure-cli==2.46.0" | uv pip compile --prerelease=allow  - > requirements.txt
$ echo "pydantic<2.0.0" | uv pip compile --constraint requirements.txt - >> requirements.txt
$ uv pip install --requirement requirements.txt

zanieb commented 9 months ago

If anyone interested, writing some additional test scenarios for this could be really helpful.

dimbleby commented 9 months ago

just happens to be that the behavior pip has chose works out for you

I don't think that's right. The requirement azure-mgmt-sql==4.0.0b8 should find the prerelease regardless of "allow prereleases", per

... explicitly requested by the user, or if the only available version that satisfies the version specifier is a pre-release.

notatallshaw commented 9 months ago

just happens to be that the behavior pip has chose works out for you

I don't think that's right. The requirement azure-mgmt-sql==4.0.0b8 should find the prerelease regardless of "allow prereleases", per

... explicitly requested by the user, or if the only available version that satisfies the version specifier is a pre-release.

My comment about the behavior was about the choice prereleases with inclusive/exclusive operators when the prereleases flag is passed, the part of the spec you quote is highly problematic, Poetry is the only tool I'm aware of that tries to faithfully follow it but it fails to when backtracking, and I'm not even sure it's logically sound to follow when backtracking (at least using any of the satisfiability algorithms any installer currently uses).

Pip certainly doesn't: https://github.com/pypa/pip/issues/12469. And rip copies pip's behavior, at least for now: https://github.com/prefix-dev/rip/issues/118. And I think uv copies this behavior by default also? I've not looked at the logic or tested but it seems to be able to install opentelemetry-exporter-prometheus without changing the preinstall flag.

I know I posted a lot of links earlier and it was a bit much to expect everyone to read them, but this is how pip's current behavior is best characterized in how it "follows" that part of the spec: https://discuss.python.org/t/handling-of-pre-releases-when-backtracking/40505/4

notatallshaw commented 9 months ago

Btw, rather than discussing the ambigious, and/or poorly implemented by everyone, spec, maybe it would make more sense to just report the usability issue that pip can install this requirement, even without the prerelease flag, and uv cannot:

$ pip install --dry-run "azure-cli==2.46.0" "pydantic<2.0"
...
Would install Deprecated-1.2.14 PyGithub-1.59.1 PyJWT-2.8.0 PyMySQL-1.0.3 PyNaCl-1.5.0 PySocks-1.7.1 PyYAML-6.0.1 Pygments-2.17.2 adal-1.2.7 antlr4-python3-runtime-4.9.3 applicationinsights-0.11.10 argcomplete-2.1.2 azure-appconfiguration-1.1.1 azure-batch-13.0.0 azure-cli-2.46.0 azure-cli-core-2.46.0 azure-cli-telemetry-1.0.8 azure-common-1.1.28 azure-core-1.30.0 azure-cosmos-3.2.0 azure-data-tables-12.4.0 azure-datalake-store-0.0.53 azure-graphrbac-0.60.0 azure-keyvault-1.1.0 azure-keyvault-administration-4.0.0b3 azure-keyvault-keys-4.8.0b2 azure-loganalytics-0.1.1 azure-mgmt-advisor-9.0.0 azure-mgmt-apimanagement-3.0.0 azure-mgmt-appconfiguration-2.2.0 azure-mgmt-applicationinsights-1.0.0 azure-mgmt-authorization-3.0.0 azure-mgmt-batch-17.0.0 azure-mgmt-batchai-7.0.0b1 azure-mgmt-billing-6.0.0 azure-mgmt-botservice-2.0.0 azure-mgmt-cdn-12.0.0 azure-mgmt-cognitiveservices-13.3.0 azure-mgmt-compute-29.1.0 azure-mgmt-consumption-2.0.0 azure-mgmt-containerinstance-10.1.0 azure-mgmt-containerregistry-10.1.0 azure-mgmt-containerservice-21.2.0 azure-mgmt-core-1.4.0 azure-mgmt-cosmosdb-9.0.0 azure-mgmt-databoxedge-1.0.0 azure-mgmt-datalake-analytics-0.2.1 azure-mgmt-datalake-nspkg-3.0.1 azure-mgmt-datalake-store-0.5.0 azure-mgmt-datamigration-10.0.0 azure-mgmt-devtestlabs-4.0.0 azure-mgmt-dns-8.0.0 azure-mgmt-eventgrid-10.2.0b2 azure-mgmt-eventhub-10.1.0 azure-mgmt-extendedlocation-1.0.0b2 azure-mgmt-hdinsight-9.0.0 azure-mgmt-imagebuilder-1.1.0 azure-mgmt-iotcentral-10.0.0b2 azure-mgmt-iothub-2.3.0 azure-mgmt-iothubprovisioningservices-1.1.0 azure-mgmt-keyvault-10.1.0 azure-mgmt-kusto-0.3.0 azure-mgmt-loganalytics-13.0.0b4 azure-mgmt-managedservices-1.0.0 azure-mgmt-managementgroups-1.0.0 azure-mgmt-maps-2.0.0 azure-mgmt-marketplaceordering-1.1.0 azure-mgmt-media-9.0.0 azure-mgmt-monitor-5.0.1 azure-mgmt-msi-7.0.0 azure-mgmt-netapp-9.0.1 azure-mgmt-network-21.0.1 azure-mgmt-nspkg-3.0.2 azure-mgmt-policyinsights-1.1.0b4 azure-mgmt-privatedns-1.0.0 azure-mgmt-rdbms-10.2.0b14 azure-mgmt-recoveryservices-2.2.0 azure-mgmt-recoveryservicesbackup-5.1.0 azure-mgmt-redhatopenshift-1.2.0 azure-mgmt-redis-14.1.0 azure-mgmt-relay-0.1.0 azure-mgmt-resource-21.1.0b1 azure-mgmt-search-8.0.0 azure-mgmt-security-3.0.0 azure-mgmt-servicebus-8.2.0 azure-mgmt-servicefabric-1.0.0 azure-mgmt-servicefabricmanagedclusters-1.0.0 azure-mgmt-servicelinker-1.2.0b1 azure-mgmt-signalr-1.1.0 azure-mgmt-sql-4.0.0b8 azure-mgmt-sqlvirtualmachine-1.0.0b5 azure-mgmt-storage-21.0.0 azure-mgmt-synapse-2.1.0b5 azure-mgmt-trafficmanager-1.0.0 azure-mgmt-web-7.0.0 azure-multiapi-storage-1.0.0 azure-nspkg-3.0.2 azure-storage-common-1.4.2 azure-synapse-accesscontrol-0.5.0 azure-synapse-artifacts-0.15.0 azure-synapse-managedprivateendpoints-0.4.0 azure-synapse-spark-0.2.0 bcrypt-4.1.2 certifi-2024.2.2 cffi-1.16.0 chardet-3.0.4 charset-normalizer-3.3.2 colorama-0.4.6 cryptography-40.0.2 distro-1.9.0 fabric-2.7.1 humanfriendly-10.0 idna-3.6 invoke-1.7.3 isodate-0.6.1 javaproperties-0.5.2 jmespath-1.0.1 jsondiff-2.0.0 knack-0.10.1 msal-1.20.0 msal-extensions-1.0.0 msrest-0.7.1 msrestazure-0.6.4 oauthlib-3.2.2 packaging-23.2 paramiko-3.4.0 pathlib2-2.3.7.post1 pkginfo-1.9.6 portalocker-2.8.2 psutil-5.9.8 pyOpenSSL-23.2.0 pycparser-2.21 pydantic-1.10.14 python-dateutil-2.8.2 requests-2.31.0 requests-oauthlib-1.3.1 scp-0.13.6 semver-2.13.0 six-1.16.0 sshtunnel-0.1.5 tabulate-0.9.0 typing_extensions-4.9.0 urllib3-2.2.1 websocket-client-1.3.3 wrapt-1.16.0 xmltodict-0.13.0

$ printf "azure-cli==2.46.0\npydantic<2.0" | uv pip compile -
  × No solution found when resolving dependencies:
  ╰─▶ Because there is no version of azure-mgmt-sql==4.0.0b8 and azure-cli==2.46.0 depends on azure-mgmt-sql==4.0.0b8, we can conclude that
      azure-cli==2.46.0 cannot be used.
      And because you require azure-cli==2.46.0, we can conclude that the requirements are unsatisfiable.

      hint: azure-mgmt-sql was requested with a pre-release marker (e.g., azure-mgmt-sql==4.0.0b8), but pre-releases weren't enabled (try:
      `--prerelease=allow`)

erdembanak-invent commented 9 months ago

We can just say uv pip install azure-cli==2.46.0 is not working while pip install azure-cli==2.46.0 is working.

dimbleby commented 9 months ago

So far as I know everyone tries to follow the spec, of course some are more successful than others, and in different ways.

All of the installers that I tried - pip / poetry / pdm - successfully install azure-cli==2.46.0 without extra flags, and all of them agree that 2.0b3 does not satisfy <2.0.0.

100% agree that as a user experience this issue is best reported as uv pip install azure-cli==2.46.0 does not succeed. Referring to the spec is supposed to be a useful way of investigating where this is going wrong.

charliermarsh commented 9 months ago

So, that uv pip install azure-cli==2.46.0 does not succeed is known in our design and documented. The requirement we have is that we need to know which packages are "allowed" to have pre-releases in advance. Without this, correctness becomes extremely difficult.

In the case above, we need to know that you're allowing pre-releases for azure-mgmt-sql. You can do this with:

uv pip install azure-cli==2.46.0 azure-mgmt-sql==4.0.0b8

In uv, the use of the pre-release marker in the direct dependency tells uv to allow pre-releases for that dependency. You can see this in the hint provided above upon failure.

Unfortunately, tt seems like Azure publishes, like, everything as a pre-release?

erdembanak commented 9 months ago

Just before I have seen your reply, I have created a separate issue for this; sorry @charliermarsh . Yes, this is the expected behavior from the documentation, just wanted to open a separate issue to make sure the discussion doesn't get lost in here; but you already replied. Thanks.

charliermarsh commented 9 months ago

No prob. It's fair game to petition that we change the behavior, but those are the rules we settled on for now as-implemented, to make the problem tractable.

erdembanak commented 9 months ago

It looks like to make sure I am not installing something as prerelease when there is a release, I need to do something like:

uv pip install azure-cli==2.46.0 azure-mgmt-sql==4.0.0b8 azure-keyvault-keys==4.8.0b2 azure-mgmt-synapse==2.1.0b5 azure-mgmt-servicelinker==1.2.0b1 azure-mgmt-batchai==7.0.0b1 azure-mgmt-extendedlocation==1.0.0b2 azure-mgmt-loganalytics==13.0.0b4 azure-mgmt-eventgrid==10.2.0b2 azure-mgmt-sqlvirtualmachine==1.0.0b5 azure-keyvault-administration==4.0.0b3 azure-mgmt-resource==21.1.0b1 azure-mgmt-rdbms==10.2.0b6 azure-mgmt-policyinsights==1.1.0b2 azure-mgmt-iotcentral==10.0.0b1

(some of them were >= in the requirements but I converted to ==)

ofek commented 9 months ago

Poetry is the only tool I'm aware of that tries to faithfully follow it but it fails to when backtracking

That was fixed recently.

Marenz commented 9 months ago

Same problem, different example:

python311 ❯ uv pip install --prerelease=allow "h3 >= 3, <4"
Resolved 1 package in 2ms
Installed 1 package in 17ms
 + h3==4.0.0b2

Czaki commented 9 months ago

@charliermarsh couldnt you just interpret package<n as package<n.a0?

charliermarsh commented 9 months ago

@Czaki -- Perhaps... I need to test it out. It may also have an adverse effect on error messages.

llucax commented 9 months ago

It is a shame that unless this is fixed, uv can't really be used as a drop-in replacement for pip, as pip does the right thing.

llucax commented 9 months ago

And if it were really the case that pip is implementing the specs wrongly (which I don't think is the case, actually not including a v4 pre-release if the version specifier is <4 is the only sensible thing to do and what the specs seem to say), and your goal is to be compatible with pip, then IMHO you should ignore the specs if they contradict what pip does and your goal is to be a drop-in replacement for pip. These seems contradictory:

mpizenberg commented 9 months ago

@Czaki -- Perhaps... I need to test it out. It may also have an adverse effect on error messages.

@charliermarsh could it be possible to store the actual original constraint in an incompat coming from dependencies? Maybe even store the package it's from? This way you could have the <4a0 constraint in the solver, but store the <4 and the package this is coming from for error messages.

charliermarsh commented 9 months ago

@llucax - this feels like a misrepresentation of what’s been said in the thread and even a little antagonistic. The reference to “our design” is with respect to how users opt-in or out of pre-release handling. The fact that “<2.0.0” should not include “2.0.0b3” is acknowledged as a bug.

charliermarsh commented 9 months ago

I'm gonna try to get the package<n.a0 to work. Let's see how it plays out.

notatallshaw commented 9 months ago

I would just like to restate my thanks again to @charliermarsh and the astral team for providing another excellent peice of open source software.

uv has been public for just 1 week, issues like this take time to find the "best" solution and come with maturity of a product as more users have time to use it and different use cases are discovered.

I think it would be important to take any information from here about the feasibility of implementing the spec and bring it back to the wider packaging community. I can assure others that pip is not following all parts of the prerelease spec (e.g. https://github.com/pypa/pip/issues/12470, https://github.com/pypa/pip/issues/12469, https://github.com/pypa/pip/issues/12471, https://github.com/pypa/packaging/issues/766). Once @charliermarsh has had time to see what is feasible for uv I plan to make a post on the Python discussion board detailing how various different installers implement/interpret the different parts of the prerelease spec.

your goal is to be compatible with pip,

This brings up something I find genuinly unclear, is that a stated goal of the uv project? The blog post says that uv aims for a compatible API, but does not talk about behavior, and actually there are a lot of differnces in the API, does this mean more that any feature in the pip API there should be some equivelent option in the uv pip API even if not called the same?

I would assume that it is a non-goal to have exact quirk/bug compatibility with pip?

dimbleby commented 9 months ago

this feels like a misrepresentation of what’s been said in the thread and even a little antagonistic. The reference to “our design” is with respect to how users opt-in or out of pre-release handling. The fact that “<2.0.0” should not include “2.0.0b3” is acknowledged as a bug.

I think you may be talking past one another, at least somewhat.

For better or worse, there are two bugs discussed in this thread, and the "our design" comment was I think talking about the other one: namely that uv has chosen not to implement the part of the spec that requires a pre-release to be selected - without special flags - when that is the only way to satisfy the given constraint.

while I agree that the tone was "even a little antagonistic", I share some of the surprise!

charliermarsh commented 9 months ago

Thanks fair, thanks @dimbleby!

namely that uv has chosen not to implement the part of the spec that requires a pre-release to be chosen - without special flags - when that is the only way to satisfy the given constraint.

Yeah, it's correct that we don't do this. Ultimately I'm not yet sold on supporting it, in part because I think it's very hard to get completely right (I'd rather have something that is limited, but predictable and under user control), and in part because the spec is insufficient -- the spec is built around a world of satisfying a single version comparison, but dependency resolution isn't actually covered. Per Paul Moore:

I’ve already mentioned on the pip issue that the spec doesn’t cover how things work when multiple specifiers need to be satisfied at once. That’s a spec issue.

Similarly, my understanding is that it definitively was not the intention of the spec (per one of its original authors) to support this kind of behavior (to the degree that it matters).

dimbleby commented 9 months ago

fwiw if we can get into the business of rewriting specs I would be fully behind a proposal that abolished all of dev releases, alpha releases, beta releases, post releases, local releases, dev releases of post releases of local releases, and so on. The whole thing is a minefield.

charliermarsh commented 9 months ago

It's funny because it's something I just never thought about as a Python user.

Czaki commented 9 months ago

fwiw if we can get into the business of rewriting specs I would be fully behind a proposal that abolished all of dev releases, alpha releases, beta releases, post releases, local releases, dev releases of post releases of local releases, and so on. The whole thing is a minefield.

Having option of RC is really usefull when you are library maintainer. At first it is helpful to ask for tests most active part of users.

It also allow for automatic problem detection using tox --pre -e some_env cron job on CI.

notatallshaw commented 9 months ago

fwiw if we can get into the business of rewriting specs I would be fully behind a proposal that abolished all of dev releases, alpha releases, beta releases, post releases, local releases, dev releases of post releases of local releases, and so on. The whole thing is a minefield.

Epochs are what blew me away when I went through the spec recently.

It boggles the mind to think 1!1.1.1.a1.post1.dev1+local1 is a valid version number

mitsuhiko commented 9 months ago

I can thankfully a bit impartial here. I love the existence of uv, but I have a bit of a distance to it as more of a user of it than as a developer.

I always expected uv to be mostly pip compatible but not inherit necessarily every behavior of pip. And that not because pip is necessarily wrong, but because I absolutely loathe complexity and pip has a lot of it. If a 90% match can be accomplished with a lot less complexity I am a happy camper. Reducing complexity on the long run always results in a better user experience.

Without a doubt there are lots of packages on pypi today that probably require a different behavior than uv has today. However I would try to find a pragmatic solution and to adjust the spec, rather than clinging to either spec or pip.

</My 2 cents :)>

ofek commented 9 months ago

I find it odd to consider changing the spec as a valid course of action when people are expressing need for that behavior in addition to the fact that the maintainers are saying that this is a known bug with which they are interested in accommodating. I have encountered this exact use case at work.

mitsuhiko commented 9 months ago

It's not entirely odd because the creation of that very spec has over time changed the packaging ecosystem in a good way. But that has meant that some of the motivating elements for compatibility with the pre PEP 440 world are no longer quite as needed. See also the post by dstufft on it: https://discuss.python.org/t/handling-of-pre-releases-when-backtracking/40505/20

So I do think that it would be in the interest of the ecosystem to consider adjusting the spec if that results in better user experience and less complex systems.

erdembanak-invent commented 9 months ago

As it had been said before; there are 2 discussions in here, and the original one is definitely a bug (installing 2.ob3 when user requested <2.0.0). The interesting discussion is about the second one, how to accommodate prereleases. I am still in favor of pip's behavior since the ecosystem is currently built around that. There are 2 points in my argument: 1 -uv is not a drop in replacement for pip since I needed to override some packages when I wanted to try it for production. I have to explicitly define the versions of packages which I have to install prereleases (due to some requirements). I need to list around 10 azure prereleases to use one package 3 - Current rule feels shaky. If I understood correctly, there are two conditions for using a prerelease version: "if the prerelease is explicity wanted or the library only has prerelease versions". In my current life, we have many internal libraries. Assume we have a library which requires the library xyz with the requirement files xyz = 0.1.btx. If library xyxz does not release a version, I'm OK with installing it as xyz = 0.1.btx with "uv". When they release the first version, uv will not be installing that version since there is a release version of the library. It means that, I won't be able to use a past version of my library with uv (since it depended on a prerelease version and there is a release now and I'm not allowing all prerelease versions). This means that older versions of my library which is depending on a prerelease version of another library won't be installed.

I would favor prohibiting prereleases (since it is a shaky concept); but currently using uv in production means adding too many libraries in a stricted way. As a regular user, I do not want to allow prereleases; but when there are many libraries OK with that, there is not much to do for me. Btw, I would favor a course of action which changes the specs, instead of talking about it in here.

astral-sh / uv

pydantic<2.0.0 installs pydantic=2.0b3 when prereleases are allowed #1641