search

pypa / pip

The Python package installer
https://pip.pypa.io/
MIT License
9.47k stars 3.01k forks source link

Excluding packages when extra is specified #8686

Open DEKHTIARJonathan opened 4 years ago

DEKHTIARJonathan commented 4 years ago

I have found a dependency resolving issue in PIP, here is how to reproduce. Nor the "old" or the "2020-resolver" is able to fix the issue. However they both fails in different ways.

This is a very problematic for us. When do you think it could be resolved ?

Environment:

pip list

Package    Version
---------- -------
pip        20.2
setuptools 49.2.0
wheel      0.34.2

Script to reproduce

#!/usr/bin/env python

"""The setup script."""

from setuptools import setup

# We ask to install is-sorted by pinned version (points to 0.0.1: https://pypi.org/project/is-sorted/0.0.1/)
install_requires = ["is-sorted==0.0.1; extra != 'env_A' and extra != 'env_B'"]

extras_require = {
    "env_a": ["random-string"],  # a random package
    "env_b": ["get-random"],  # a random package
}

# We ask to install is-sorted by range (currently points to 0.0.2: https://pypi.org/project/is-sorted/0.0.2/)
extras_require = {key: val + ["is-sorted>=0.0.1,<0.1"] for key, val in extras_require.items()}

setup(
    name='test_pkg',
    version='1.0.0',
    install_requires=install_requires,
    extras_require=extras_require,
)

What shall be expected

What happens in practice ?

0. Let's build the wheel: python setup.py bdist_wheel

$ python setup.py bdist_wheel

running bdist_wheel
running build
installing to build/bdist.linux-x86_64/wheel
running install
running install_egg_info
running egg_info
writing test_pkg.egg-info/PKG-INFO
writing dependency_links to test_pkg.egg-info/dependency_links.txt
writing requirements to test_pkg.egg-info/requires.txt
writing top-level names to test_pkg.egg-info/top_level.txt
reading manifest file 'test_pkg.egg-info/SOURCES.txt'
writing manifest file 'test_pkg.egg-info/SOURCES.txt'
Copying test_pkg.egg-info to build/bdist.linux-x86_64/wheel/test_pkg-1.0.0-py3.6.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/test_pkg-1.0.0.dist-info/WHEEL
creating 'dist/test_pkg-1.0.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'test_pkg-1.0.0.dist-info/METADATA'
adding 'test_pkg-1.0.0.dist-info/WHEEL'
adding 'test_pkg-1.0.0.dist-info/top_level.txt'
adding 'test_pkg-1.0.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel

If we look at the file METADATA, everything seems OK:

Requires-Dist: is-sorted (==0.0.1) ; extra != "env_a" and extra != "env_b"
Provides-Extra: env_a
Requires-Dist: random-string ; extra == 'env_a'
Requires-Dist: is-sorted (<0.1,>=0.0.1) ; extra == 'env_a'
Provides-Extra: env_b
Requires-Dist: get-random ; extra == 'env_b'
Requires-Dist: is-sorted (<0.1,>=0.0.1) ; extra == 'env_b'

1. Install with NO extra require

$ pip install "dist/test_pkg-1.0.0-py3-none-any.whl"

Processing ./dist/test_pkg-1.0.0-py3-none-any.whl
Collecting is-sorted==0.0.1; extra != "env_a" and extra != "env_b"
  Downloading is_sorted-0.0.1-py3-none-any.whl (2.8 kB)
Installing collected packages: is-sorted, test-pkg
Successfully installed is-sorted-0.0.1 test-pkg-1.0.0

Conclusion: Everything is good.

2. Install with env_a extra require

$ pip install "dist/test_pkg-1.0.0-py3-none-any.whl[env_a]"

Processing ./dist/test_pkg-1.0.0-py3-none-any.whl
  Ignoring is-sorted: markers 'extra != "env_a" and extra != "env_b"' don't match your environment
Collecting is-sorted<0.1,>=0.0.1; extra == "env_a"
  Downloading is_sorted-0.0.2-py3-none-any.whl (3.3 kB)
Collecting random-string; extra == "env_a"
  Downloading random-string-1.00.tar.gz (996 bytes)
Building wheels for collected packages: random-string
  Building wheel for random-string (setup.py) ... done
  Created wheel for random-string: filename=random_string-1.0-py3-none-any.whl size=1897 sha256=14603b692b5977dfe8f4bc8d4c3ea54a796b5b4c86760c0bb97b21f3f1aaf2cb
  Stored in directory: /root/.cache/pip/wheels/47/2d/fe/13b8d8df913e51efdd1c8a66632a2dd0b54abb101c2c9ac399
Successfully built random-string
Installing collected packages: is-sorted, random-string, test-pkg
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

test-pkg 1.0.0 requires is-sorted==0.0.1; extra != "env_a" and extra != "env_b", but you'll have is-sorted 0.0.2 which is incompatible.
Successfully installed is-sorted-0.0.2 random-string-1.0 test-pkg-1.0.0

Conclusion: A scary "red" warning for no reason: is-sorted-0.0.2 random-string-1.0 test-pkg-1.0.0 shall be the correct set of dependencies

3. Install with env_b extra require

$ pip install "dist/test_pkg-1.0.0-py3-none-any.whl[env_b]"

Processing ./dist/test_pkg-1.0.0-py3-none-any.whl
  Ignoring is-sorted: markers 'extra != "env_a" and extra != "env_b"' don't match your environment
Collecting get-random; extra == "env_b"
  Downloading get_random-0.1.4.tar.gz (2.5 kB)
Collecting is-sorted<0.1,>=0.0.1; extra == "env_b"
  Downloading is_sorted-0.0.2-py3-none-any.whl (3.3 kB)
Building wheels for collected packages: get-random
  Building wheel for get-random (setup.py) ... done
  Created wheel for get-random: filename=get_random-0.1.4-py3-none-any.whl size=2516 sha256=5fdc7d9551dcf4e377d839c938865dc5053745d1e67aaad9cfd418fa67e28848
  Stored in directory: /root/.cache/pip/wheels/ec/01/58/0ee893b104a22036186016ffaf29e1aca6a343c11341677a4c
Successfully built get-random
Installing collected packages: get-random, is-sorted, test-pkg
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

test-pkg 1.0.0 requires is-sorted==0.0.1; extra != "env_a" and extra != "env_b", but you'll have is-sorted 0.0.2 which is incompatible.
Successfully installed get-random-0.1.4 is-sorted-0.0.2 test-pkg-1.0.0

Conclusion: A scary "red" warning for no reason: get-random-0.1.4 is-sorted-0.0.2 test-pkg-1.0.0 shall be the correct set of dependencies

4. Install with env_a && env_b extra require

$ pip install "dist/test_pkg-1.0.0-py3-none-any.whl[env_a,env_b]"

Processing ./dist/test_pkg-1.0.0-py3-none-any.whl
  Ignoring is-sorted: markers 'extra != "env_a" and extra != "env_b"' don't match your environment
Collecting is-sorted<0.1,>=0.0.1; extra == "env_a"
  Downloading is_sorted-0.0.2-py3-none-any.whl (3.3 kB)
Collecting random-string; extra == "env_a"
  Downloading random-string-1.00.tar.gz (996 bytes)
Collecting get-random; extra == "env_b"
  Downloading get_random-0.1.4.tar.gz (2.5 kB)
Building wheels for collected packages: random-string, get-random
  Building wheel for random-string (setup.py) ... done
  Created wheel for random-string: filename=random_string-1.0-py3-none-any.whl size=1897 sha256=6a38795c861b315d94332cc3b381c7441fc83c54dd8a4db8b34cd43f638dca33
  Stored in directory: /root/.cache/pip/wheels/47/2d/fe/13b8d8df913e51efdd1c8a66632a2dd0b54abb101c2c9ac399
  Building wheel for get-random (setup.py) ... done
  Created wheel for get-random: filename=get_random-0.1.4-py3-none-any.whl size=2516 sha256=aa4677747ad67d64a16789e1611e3ca781800d380b8798bde2466aa6f005e4ba
  Stored in directory: /root/.cache/pip/wheels/ec/01/58/0ee893b104a22036186016ffaf29e1aca6a343c11341677a4c
Successfully built random-string get-random
Installing collected packages: is-sorted, random-string, get-random, test-pkg
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

test-pkg 1.0.0 requires is-sorted==0.0.1; extra != "env_a" and extra != "env_b", but you'll have is-sorted 0.0.2 which is incompatible.
Successfully installed get-random-0.1.4 is-sorted-0.0.2 random-string-1.0 test-pkg-1.0.0

Conclusion: A scary "red" warning for no reason: get-random-0.1.4 is-sorted-0.0.2 random-string-1.0 test-pkg-1.0.0 shall be the correct set of dependencies

How about with --use-feature=2020-resolver ?

Now it's even worse, we don't have a warning anymore because the dependency resolving is wrong.

1. Install with NO extra require and --use-feature=2020-resolver

$ pip install "dist/test_pkg-1.0.0-py3-none-any.whl" --use-feature=2020-resolver

Processing ./dist/test_pkg-1.0.0-py3-none-any.whl
Collecting is-sorted==0.0.1
  Downloading is_sorted-0.0.1-py3-none-any.whl (2.8 kB)
Installing collected packages: is-sorted, test-pkg
Successfully installed is-sorted-0.0.1 test-pkg-1.0.0

Conclusion: Everything is good.

2. Install with env_a extra require and --use-feature=2020-resolver

$ pip install "dist/test_pkg-1.0.0-py3-none-any.whl[env_a]"  --use-feature=2020-resolver

Processing ./dist/test_pkg-1.0.0-py3-none-any.whl
Ignoring is-sorted: markers 'extra != "env_a" and extra != "env_b"' don't match your environment
Collecting random-string
  Downloading random-string-1.00.tar.gz (996 bytes)
Collecting is-sorted<0.1,>=0.0.1
  Downloading is_sorted-0.0.1-py3-none-any.whl (2.8 kB)
Building wheels for collected packages: random-string
  Building wheel for random-string (setup.py) ... done
  Created wheel for random-string: filename=random_string-1.0-py3-none-any.whl size=1897 sha256=f9032cea18e5f1c5af61ad3041c16722150c1360aee6ac34c7134e03c30577be
  Stored in directory: /root/.cache/pip/wheels/47/2d/fe/13b8d8df913e51efdd1c8a66632a2dd0b54abb101c2c9ac399
Successfully built random-string
Installing collected packages: is-sorted, test-pkg, random-string
Successfully installed is-sorted-0.0.1 random-string-1.0 test-pkg-1.0.0

Conclusion: No error anymore, however it installs is-sorted-0.0.1 and should install: is-sorted-0.0.2

3. Install with env_b extra require and --use-feature=2020-resolver

$ pip install "dist/test_pkg-1.0.0-py3-none-any.whl[env_b]"  --use-feature=2020-resolver

Processing ./dist/test_pkg-1.0.0-py3-none-any.whl
Ignoring is-sorted: markers 'extra != "env_a" and extra != "env_b"' don't match your environment
Collecting is-sorted<0.1,>=0.0.1
  Downloading is_sorted-0.0.1-py3-none-any.whl (2.8 kB)
Collecting get-random
  Downloading get_random-0.1.4.tar.gz (2.5 kB)
Building wheels for collected packages: get-random
  Building wheel for get-random (setup.py) ... done
  Created wheel for get-random: filename=get_random-0.1.4-py3-none-any.whl size=2516 sha256=abca081d35ed6ae4bb11f1f36da294b25bad3b0dd7bb3293bc6912ec2e54f81c
  Stored in directory: /root/.cache/pip/wheels/ec/01/58/0ee893b104a22036186016ffaf29e1aca6a343c11341677a4c
Successfully built get-random
Installing collected packages: is-sorted, test-pkg, get-random
Successfully installed get-random-0.1.4 is-sorted-0.0.1 test-pkg-1.0.0

Conclusion: No error anymore, however it installs is-sorted-0.0.1 and should install: is-sorted-0.0.2

4. Install with env_a && env_b extra require and --use-feature=2020-resolver:

$ pip install "dist/test_pkg-1.0.0-py3-none-any.whl[env_a,env_b]"  --use-feature=2020-resolver

Processing ./dist/test_pkg-1.0.0-py3-none-any.whl
Ignoring is-sorted: markers 'extra != "env_a" and extra != "env_b"' don't match your environment
Collecting random-string
  Downloading random-string-1.00.tar.gz (996 bytes)
Collecting is-sorted<0.1,>=0.0.1
  Downloading is_sorted-0.0.1-py3-none-any.whl (2.8 kB)
Collecting get-random
  Downloading get_random-0.1.4.tar.gz (2.5 kB)
Building wheels for collected packages: random-string, get-random
  Building wheel for random-string (setup.py) ... done
  Created wheel for random-string: filename=random_string-1.0-py3-none-any.whl size=1897 sha256=6ba41eea0fce2aa432afc47ad7bd5e4aa89c5e1992c0a7dacfb27c3885838dbb
  Stored in directory: /root/.cache/pip/wheels/47/2d/fe/13b8d8df913e51efdd1c8a66632a2dd0b54abb101c2c9ac399
  Building wheel for get-random (setup.py) ... done
  Created wheel for get-random: filename=get_random-0.1.4-py3-none-any.whl size=2516 sha256=2eacde1bfa69a76ad369ea1e085e812c9302e2bd866c8882f148f772948faf50
  Stored in directory: /root/.cache/pip/wheels/ec/01/58/0ee893b104a22036186016ffaf29e1aca6a343c11341677a4c
Successfully built random-string get-random
Installing collected packages: is-sorted, test-pkg, random-string, get-random
Successfully installed get-random-0.1.4 is-sorted-0.0.1 random-string-1.0 test-pkg-1.0.0

Conclusion: No error anymore, however it installs is-sorted-0.0.1 and should install: is-sorted-0.0.2

chrahunt commented 4 years ago

Thank you for the thorough report!

It looks like there are a few issues here:

  1. In the conflict checking that occurs when using the old resolver, which leads to:

    ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

test-pkg 1.0.0 requires is-sorted==0.0.1; extra != "env_a" and extra != "env_b", but you'll have is-sorted 0.0.2 which is incompatible.

    pip doesn't seem to take extras into account.

  2. pip doesn't normalize extras. When trying to reproduce with your current script (which has extra != "env_A" and extra != "env_B") I did not get the Ignoring is-sorted... message. I'm not sure if this is something we should be doing.

  3. Even with the script updated to use env_a and env_b, pip isn't picking is-sorted 0.0.2 even though it is available as a wheel, matches the platform, and none of the other projects seem to have a dependency like is-sorted==0.0.1

I don't think the last issue is related to extras, because the line

Ignoring is-sorted: markers 'extra != "env_a" and extra != "env_b"' don't match your environment

indicates that we ignored that dependency. Technically picking is-sorted 0.0.1 is valid because the matching lines have is-sorted<0.1,>=0.0.1, but we should be preferring the later version.

uranusjr commented 4 years ago

2. pip doesn't normalize extras. When trying to reproduce with your current script (which has extra != "env_A" and extra != "env_B") I did not get the Ignoring is-sorted... message. I'm not sure if this is something we should be doing.

I remember having a discussion with someone about this a while ago. setuptools does not normalise extras when building distributions, and pip either when installing them. But pkg_resources does when checking dependencies, and with entirely undocumented and unspecified rules (AFAIK). This is a much larger problem than pip’s dependency resolver.

chrahunt commented 4 years ago

@uranusjr as far as I can tell, the situation we get into is, given dist/test_pkg-1.0.0-py3-none-any.whl[env_a]:

  1. we create a LinkCandidate for e.g. dist/test_pkg-1.0.0-py3-none-any.whl
  2. we create an ExtrasCandidate that represents dist/test_pkg-1.0.0-py3-none-any.whl[env_a]
  3. when PipProvider.find_matches gets invoked by resolvelib, it is with a criterion constructed from all of the candidates' dependencies. Since we don't filter LinkCandidate dependencies, it includes the one with ==0.0.1. We merge that with the other criteria here and pass it down into PackageFinder, which only yields links with ==0.0.1 because that's the most specific

So we should probably exclude dependencies with extras around here?

uranusjr commented 4 years ago

There are more to this, since the provider currently assumes an extra-ed requirement automatically has all the same dependencies as its extra-less counterpart. But yes, the code you linked would be one of the places that need to change.

Tagging @pfmoore since I imagine you’d have much feeling about this 😉

DEKHTIARJonathan commented 4 years ago

Glad that you all managed to reproduce and locate the issue. Let me know if I can help 👍

uranusjr commented 4 years ago

Another more personal note: I am hesitant to classify this as a pip bug. Although the described usage is well within the specification, I suspect it is against the intention of the extras feature, based on my interaction with people designing it. If that’s the case, this would classify better as a feature request instead, and the resolution may be that we need to fix the specification to disallow the usage, instead of introducing the feature to support it.

DEKHTIARJonathan commented 4 years ago

@uranusjr i'm not sure to understand what you imply. It is "well within the specifications" but it is not a bug? How could this be? It's either something that should work or something that shouldn't work. If it's within the specs then it should be considered a bug IMHO.

There is no other way to make a "default" extra require if no extra is specified (which is eventually what I try to do)... We could go around this limitation in this specific case if we could register the None extra environment. But it's impossible...

And even if that was doable it technically doesn't fix all cases that this issue illustrates.

uranusjr commented 4 years ago

The extras feature is carried over from the pre-specification era of Python packaging, and specified after the fact. It is important to consider the “spirit of the law” when dealing with discrepancies here, since the specification is as suspect to bugs as the implementation.

The lack of a way to define a “default extra” is actually a long-standing problem in Python packaging. This would not be the case if the extras feature was designed initially with this usage in mind. Therefore, the fact that specification creators are still looking for a way to specify default extras indicates that the extras feature was, in fact, not designed to support this usage. The solution may be that we should design a way to specify default extras (which is generally agreed to be needed), instead of stretching a feature beyond its intended purpose simply because the specification allows it.

DEKHTIARJonathan commented 4 years ago

It's been 3 years (in one month) that this issue is opened : https://github.com/pypa/setuptools/issues/1139

Do you guys have any idea when you plan on coming up with a fix? I understand your point, but saying "it's important to consider the spirit of the law" and not providing any solution for 3 years isn't something that helps to accept that aforesaid "spirit".

Pip dependency resolver is currently being refactored and rewritten. I think it's a great time to address the question in whatever way you feel comfortable. Or at the very least provide a workaround and unblock everyone...

uranusjr commented 4 years ago

Please understand that no-one in either pip, setuptools, or Python packaging in general are obliged to provide functionalities that you need. The lack of movement indicates that people who are willing to do things (mostly without return, I want to add) do no care enough about it, and those want the feature do no care enough to actually get involved in the work. Feel free to help out if you do care enough to, but simply asking people to do things they don’t want wouldn’t get you very far, I’m afraid.

DEKHTIARJonathan commented 4 years ago

@uranusjr sorry if I was misunderstood, to quote myself

"Glad that you all managed to reproduce and locate the issue. Let me know if I can help 👍"

I'm all for getting my hands dirty and pushing/implementing this. I already contributes to numerous open source projects and would be glad to join PyPA devs. I may need some directions in how you guys you want to approach the problem. If I understand the situation correctly, it's less a technical problem and more agreeing on how to address the issue.

Do you have a preferred mean of communication within PyPA? A slack or Discord maybe? Please let me know how I can get in touch with the right people and agree on how you all want the issue to be addressed. It may require writing a PEP. I'm fairly new to all of this, so any guidance is appreciated.

uranusjr commented 4 years ago

I think the misunderstanding is the communication channel is not as well-known as I anticipated, sorry. PyPA does have multiple communication channels, as listed on pypa.io. The Discourse forum is the most active these days, but feel free to choose any one you feel the most comfortable with.

DEKHTIARJonathan commented 4 years ago

Sweet. I'm all to use the forum, that way the discussion is public and easily readable/accessible by anyone.

Do you know if there's any on going thread on this topic (I try to avoid creating a duplicate)?

I would appreciate if you could send me an email : contact@jonathandekhtiar.eu that way if I have any question I can reach out to you 😊 Thanks buddy 👍

DEKHTIARJonathan commented 4 years ago

@uranusjr @chrahunt https://discuss.python.org/t/adding-a-default-extra-require-environment/4898

pradyunsg commented 4 years ago

Chris's comment (https://github.com/pypa/pip/issues/8686#issuecomment-667757703) does sounds like a reasonable summary and I don't think we came up with any concensus regarding what to do toward changing behaviors around this area (beyond it's broken and needs fixing).

In other words, I can't figure out what we've decided to do to be able to close this issue. :)

fermulator commented 3 years ago

We hit this in homeassistant upgrades and have no idea what to do yet :) https://community.home-assistant.io/t/upgrade-ha-0-114-2-from-0-107-7/219412

Jayanth1812 commented 3 years ago

Hi, I am getting the following errors: ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts. We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

(tensorflow) C:\tensorflow\models\research\object_detection>python train.py --train_dir=training/ --pipeline_config_path=training/faster_rcnn_inception_v2_pets.config --logtostderr Traceback (most recent call last): File "train.py", line 54, in from object_detection.builders import model_builder File "C:\tensorflow\models\research\object_detection\builders\model_builder.py", line 66, in from object_detection.models import ssd_efficientnet_bifpn_feature_extractor as ssd_efficientnet_bifpn File "C:\tensorflow\models\research\object_detection\models\ssd_efficientnet_bifpn_feature_extractor.py", line 33, in from official.vision.image_classification.efficientnet import efficientnet_model File "C:\tensorflow\models\official\vision\image_classification\efficientnet\efficientnet_model.py", line 35, in from official.modeling import tf_utils File "C:\tensorflow\models\official\modeling\tf_utils.py", line 25, in from official.modeling import activations File "C:\tensorflow\models\official\modeling\activations__init__.py", line 16, in from official.modeling.activations.gelu import gelu File "C:\tensorflow\models\official\modeling\activations\gelu.py", line 20, in @tf.keras.utils.register_keras_serializable(package='Text') AttributeError: module 'tensorflow_core.keras.utils' has no attribute 'register_keras_serializable'

uranusjr commented 3 years ago

@Jayanth1812 This is a Tensorflow issue, not related to pip.

Jayanth1812 commented 3 years ago

I tried installing Tensorflow of different version, but it doesn't work for me. So, I re-installed to tensorflow-2.0. While installing I am getting error like this: ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts. We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default. tf-models-official 2.3.0 requires tensorflow>=2.3.0, but you'll have tensorflow 2.0 which is incompatible.

If I install Tensorflow 2.3.0 then also it is not supporting, What should I do now?

Jayanth1812 commented 3 years ago

@uranusjr Which version of tensor flow should I install?

pfmoore commented 3 years ago

@Jayanth1812 See here for some advice. For anything more detailed, you'll need to ask the Tensorflow project, as the pip developers don't have any information about why tensorflow packages have the dependencies that they claim.

pfmoore commented 3 years ago

@Jayanth1812 Please stop spamming this issue, you need to find a tensorflow support list to get help.

zhiltsov-max commented 3 years ago

Hi, I ran into a similar problem, which can be described with a simplified example from the starting message.

Let's have a setup.py like this:

from setuptools import setup

# Add a dependency on a package, which must be excluded if the extra is specified
install_requires = ["is-sorted; extra != 'env_a'"]

extras_require = {
    "env_a": [] # just list an extra. Can have any deps, but it's not important.
}

setup(
    name='test_pkg',
    version='1.0.0',
    install_requires=install_requires,
    extras_require=extras_require,
)

Then we create a distribution (since we can't just call pip install .[env_a], which will fail)

python setup.py bdist_wheel

And try to install it:

$ pip list

Package    Version
---------- -------
pip        21.2.3
setuptools 57.4.0
wheel      0.37.0

$ pip install dist/test_pkg-1.0.0-py3-none-any.whl[env_a]

Processing ./dist/test_pkg-1.0.0-py3-none-any.whl
Ignoring is-sorted: markers 'extra != "env_a"' don't match your environment
Collecting is-sorted==0.0.1
  Using cached is_sorted-0.0.1-py3-none-any.whl (2.8 kB)
Installing collected packages: is-sorted, test-pkg
Successfully installed is-sorted-0.0.1 test-pkg-1.0.0

$ pip list

Package    Version
---------- -------
is-sorted  0.0.1
pip        21.2.3
setuptools 57.4.0
test-pkg   1.0.0
wheel      0.37.0

And we get the ignored package installed regardless of [env_a] being specified.

In my case, the package to be ignored is opencv-python-headless, which might be replaced by regular opencv-python in an extra.

uranusjr commented 3 years ago

@zhiltsov-max Please respond (and try to push forward the discussion) in https://discuss.python.org/t/4898. This is a specification issue (PEP 440 does not properly specify what this should be handled) and not something we can just fix in pip.

zhiltsov-max commented 3 years ago

@uranusjr, thanks for replying! I've seen that discussion almost a year ago, when I first bumped into variable dependencies, but I workarounded the problem with an environment variable and --no-binary. From what I've seen, in the discussion few approaches were proposed, somebody suggested implementations, core team members reacted and, apparently, no actions were taken. The topic starter and participants have already done a big work in making the idea of default extra real, and I appreciate it, but I'm not sure how can I help with pushing it forward. I'm open for participating.

The problem I reported looks just like a bug, because pip correctly reports an ignored dependency, and then installs it. If dependencies have conflicts between extras or base, they will just be installed all together.

uranusjr commented 3 years ago

Unfortunately this is not pip bug, but a design issue. The reason pip does not do what you want is not because pip interprets extra != "env_a" incorrectly, but that expression does not mean what you think it does (if it does, pip would have done what you want).

When you install a package with extras, say foo[a,b], you are actually requesting extras, not an extra singular. The way this is broken down, as specified in PEP 508, is to break a package’s dependencies into parts: those included when no extras are specified, those included only if you want the a extra, and those included only if you want the b extra. When foo[a,b] is requested, all three are combined together.

Now if you extend the concept to when only one extra is requested, say foo[a], you are still not requesting one extra, but a list of extras of length 1, and the package’s dependencies are broken into two: those included when no extras are specified, and those only if you want the a extra. With this design, bar; extra != 'a' goes into the former group—because when no extras are requested, the expression evaluates to True. So when those parts are installed, bar will still end up in the environment.

I hope this clears up why the current PEP 508 markers is confusing in this aspect and why pip seems to interpret it wrong. Again, pip actually interprets it correctly as designed; its the design that gives extra != value a semantic meaning that you do not expect (and that semantic meaning means there’s currently no way to achieve what you want). So this should not be fixed by pip because pip should not deviate from the standard; you need to fix the standard.