Fail dependency resolution when a specified extra is invalid, instead of installing the base package without the invalid extra

Gallaecio commented 3 years ago

Reproducible with pip 21.0.1 and master (af60df514397b8f63182e77ea84aaafa2d5f9c93).

What did you want to do?

Install the latest version of Twisted[http2] (which as of Twisted 21.2.0 requires h2 >=3.0, <4.0), while enforcing the requirement h2>=3.2.0:

pip install --upgrade 'Twisted[http2]' 'h2>=3.2.0'

Output

What the command does (wrong, I believe) is to:

Install a Twisted package without the http2 extra even defined. I believe the command should rather fail before doing that. If I ask Twisted[http2] to be installed, I should not get Twisted installed instead.
Backtrack so far in Twisted history, when the latest version of Twisted and the previous version of h2 are compatible. This may be related to https://github.com/pypa/pip/issues/9187 or https://github.com/pypa/pip/issues/9215 .

(venv) [adrian@vento temporal]$ pip install --upgrade 'Twisted[http2]' 'h2>=3.2.0'
Requirement already satisfied: Twisted[http2] in ./venv/lib/python3.9/site-packages (21.2.0)
Requirement already satisfied: h2>=3.2.0 in ./venv/lib/python3.9/site-packages (3.2.0)
Collecting h2>=3.2.0
  Using cached h2-4.0.0-py3-none-any.whl (57 kB)
Collecting hyperframe<7,>=6.0
  Using cached hyperframe-6.0.0-py3-none-any.whl (11 kB)
Collecting hpack<5,>=4.0
  Using cached hpack-4.0.0-py3-none-any.whl (32 kB)
Requirement already satisfied: constantly>=15.1 in ./venv/lib/python3.9/site-packages (from Twisted[http2]) (15.1.0)
Requirement already satisfied: hyperlink>=17.1.1 in ./venv/lib/python3.9/site-packages (from Twisted[http2]) (21.0.0)
Requirement already satisfied: Automat>=0.8.0 in ./venv/lib/python3.9/site-packages (from Twisted[http2]) (20.2.0)
Requirement already satisfied: incremental>=16.10.1 in ./venv/lib/python3.9/site-packages (from Twisted[http2]) (21.3.0)
Requirement already satisfied: zope.interface>=4.4.2 in ./venv/lib/python3.9/site-packages (from Twisted[http2]) (5.3.0)
Requirement already satisfied: attrs>=19.2.0 in ./venv/lib/python3.9/site-packages (from Twisted[http2]) (20.3.0)
Requirement already satisfied: priority<2.0,>=1.1.0 in ./venv/lib/python3.9/site-packages (from Twisted[http2]) (1.3.0)
Collecting h2>=3.2.0
  Using cached h2-3.2.0-py2.py3-none-any.whl (65 kB)
INFO: pip is looking at multiple versions of twisted to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of twisted[http2] to determine which version is compatible with other requirements. This could take a while.
Collecting Twisted[http2]
  Using cached Twisted-21.2.0-py3-none-any.whl (3.1 MB)
  Using cached Twisted-20.3.0-cp39-cp39-linux_x86_64.whl
  Using cached Twisted-19.10.0.tar.bz2 (3.1 MB)
  Using cached Twisted-19.7.0.tar.bz2 (3.1 MB)
  Using cached Twisted-19.2.1-cp39-cp39-linux_x86_64.whl
  Using cached Twisted-19.2.0.tar.bz2 (3.1 MB)
  Using cached Twisted-18.9.0.tar.bz2 (3.1 MB)
  Using cached Twisted-18.7.0.tar.bz2 (3.1 MB)
  Using cached Twisted-18.4.0.tar.bz2 (3.0 MB)
  Using cached Twisted-17.9.0.tar.bz2 (3.0 MB)
  Using cached Twisted-17.5.0.tar.bz2 (3.0 MB)
  Using cached Twisted-17.1.0.tar.bz2 (3.0 MB)
  Using cached Twisted-16.6.0.tar.bz2 (3.0 MB)
  Using cached Twisted-16.5.0.tar.bz2 (3.0 MB)
  Using cached Twisted-16.4.1.tar.bz2 (3.0 MB)
  Using cached Twisted-16.4.0.tar.bz2 (3.0 MB)
  Using cached Twisted-16.3.2.tar.bz2 (2.9 MB)
  Using cached Twisted-16.3.1.tar.bz2 (2.9 MB)
  Using cached Twisted-16.3.0.tar.bz2 (2.9 MB)
  Using cached Twisted-16.2.0.tar.bz2 (2.9 MB)
  Using cached Twisted-16.1.1.tar.bz2 (2.9 MB)
  Using cached Twisted-16.1.0.tar.bz2 (2.9 MB)
  Using cached Twisted-16.0.0.tar.bz2 (3.2 MB)
  Using cached Twisted-15.5.0.tar.bz2 (3.1 MB)
  Using cached Twisted-15.4.0.tar.bz2 (3.1 MB)
  Using cached Twisted-15.3.0.tar.bz2 (4.4 MB)
  Using cached Twisted-15.2.1.tar.bz2 (4.6 MB)
  Using cached Twisted-15.2.0.tar.bz2 (4.6 MB)
  Using cached Twisted-15.1.0.tar.bz2 (4.5 MB)
  Using cached Twisted-15.0.0.tar.bz2 (4.4 MB)
  Using cached Twisted-14.0.2.tar.bz2 (4.3 MB)
  Using cached Twisted-14.0.1.tar.bz2 (4.3 MB)
  Using cached Twisted-14.0.0.tar.bz2 (4.3 MB)
  Using cached Twisted-13.2.0.tar.bz2 (2.7 MB)
  Using cached Twisted-13.1.0.tar.bz2 (2.7 MB)
  Using cached Twisted-13.0.0.tar.bz2 (2.6 MB)
  Using cached Twisted-12.3.0.tar.bz2 (2.6 MB)
  Using cached Twisted-12.2.0.tar.bz2 (2.9 MB)
  Using cached Twisted-12.1.0.tar.bz2 (2.8 MB)
    ERROR: Command errored out with exit status 1:
     command: /home/adrian/temporal/venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0hw4x1jt/twisted_16f46140f26648a2b2f1e4fb26738fc1/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0hw4x1jt/twisted_16f46140f26648a2b2f1e4fb26738fc1/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-gd9jg2de                                                                                                                                                                                                            
         cwd: /tmp/pip-install-0hw4x1jt/twisted_16f46140f26648a2b2f1e4fb26738fc1/                                                                                                                                                                                            
    Complete output (6 lines):                                                                                                                                                                                                                                               
    Traceback (most recent call last):                                                                                                                                                                                                                                       
      File "<string>", line 1, in <module>                                                                                                                                                                                                                                   
      File "/tmp/pip-install-0hw4x1jt/twisted_16f46140f26648a2b2f1e4fb26738fc1/setup.py", line 89                                                                                                                                                                            
        print """You seem to be running a very old version of setuptools.                                                                                                                                                                                                    
              ^                                                                                                                                                                                                                                                              
    SyntaxError: Missing parentheses in call to 'print'. Did you mean print("""You seem to be running a very old version of setuptools.)?                                                                                                                                    
    ----------------------------------------                                                                                                                                                                                                                                 
WARNING: Discarding https://files.pythonhosted.org/packages/0c/3e/55d4812013c740c5b9bef073e73286558e1c7b07443f3ff4759f888e1782/Twisted-12.1.0.tar.bz2#sha256=033339af16463069413bc916a9d2d72eb13ed6da6c9b24da2aa630cd437b1cf5 (from https://pypi.org/simple/twisted/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.                                                                                                                                                                           
  Using cached Twisted-12.0.0.tar.bz2 (2.8 MB)
    ERROR: Command errored out with exit status 1:
     command: /home/adrian/temporal/venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0hw4x1jt/twisted_a4baab3373b442278dcbb3bd8bb7e512/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0hw4x1jt/twisted_a4baab3373b442278dcbb3bd8bb7e512/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-6mt6o6n2                                                                                                                                                                                                            
         cwd: /tmp/pip-install-0hw4x1jt/twisted_a4baab3373b442278dcbb3bd8bb7e512/                                                                                                                                                                                            
    Complete output (6 lines):                                                                                                                                                                                                                                               
    Traceback (most recent call last):                                                                                                                                                                                                                                       
      File "<string>", line 1, in <module>                                                                                                                                                                                                                                   
      File "/tmp/pip-install-0hw4x1jt/twisted_a4baab3373b442278dcbb3bd8bb7e512/setup.py", line 89                                                                                                                                                                            
        print """You seem to be running a very old version of setuptools.                                                                                                                                                                                                    
              ^                                                                                                                                                                                                                                                              
    SyntaxError: Missing parentheses in call to 'print'. Did you mean print("""You seem to be running a very old version of setuptools.)?                                                                                                                                    
    ----------------------------------------                                                                                                                                                                                                                                 
WARNING: Discarding https://files.pythonhosted.org/packages/04/6f/ff8d5b822bed8d7fc23518e71d1d3eb70e0f348a4045e48a52258bd29493/Twisted-12.0.0.tar.bz2#sha256=e5561c9714583fd052ea9e0b10331609a719b3878f9d2593e4c57eab0ba0effd (from https://pypi.org/simple/twisted/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.                                                                                                                                                                           
  Using cached Twisted-11.1.0.tar.bz2 (2.8 MB)
    ERROR: Command errored out with exit status 1:
     command: /home/adrian/temporal/venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0hw4x1jt/twisted_aaf3153ff2134b1c8d47d7e3f056a677/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0hw4x1jt/twisted_aaf3153ff2134b1c8d47d7e3f056a677/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-stxczdb2                                                                                                                                                                                                            
         cwd: /tmp/pip-install-0hw4x1jt/twisted_aaf3153ff2134b1c8d47d7e3f056a677/                                                                                                                                                                                            
    Complete output (6 lines):                                                                                                                                                                                                                                               
    Traceback (most recent call last):                                                                                                                                                                                                                                       
      File "<string>", line 1, in <module>                                                                                                                                                                                                                                   
      File "/tmp/pip-install-0hw4x1jt/twisted_aaf3153ff2134b1c8d47d7e3f056a677/setup.py", line 90                                                                                                                                                                            
        print """You seem to be running a very old version of setuptools.                                                                                                                                                                                                    
              ^                                                                                                                                                                                                                                                              
    SyntaxError: Missing parentheses in call to 'print'. Did you mean print("""You seem to be running a very old version of setuptools.)?                                                                                                                                    
    ----------------------------------------                                                                                                                                                                                                                                 
WARNING: Discarding https://files.pythonhosted.org/packages/c7/82/c71021c15625960e11b32cdba7c93bf9cdf79b9fe4f0a2dcde3a97ffcad3/Twisted-11.1.0.tar.bz2#sha256=7fbb4529aaa3547708bb0a84377107db1e6ae7a6a9940bb8d8b05976c2754f15 (from https://pypi.org/simple/twisted/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.                                                                                                                                                                           
  Using cached Twisted-11.0.0.tar.bz2 (2.7 MB)
    ERROR: Command errored out with exit status 1:
     command: /home/adrian/temporal/venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0hw4x1jt/twisted_5ad7f06d51ea416ead64d7920b0e12ad/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0hw4x1jt/twisted_5ad7f06d51ea416ead64d7920b0e12ad/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-1q_ylg2f                                                                                                                                                                                                            
         cwd: /tmp/pip-install-0hw4x1jt/twisted_5ad7f06d51ea416ead64d7920b0e12ad/                                                                                                                                                                                            
    Complete output (6 lines):                                                                                                                                                                                                                                               
    Traceback (most recent call last):                                                                                                                                                                                                                                       
      File "<string>", line 1, in <module>                                                                                                                                                                                                                                   
      File "/tmp/pip-install-0hw4x1jt/twisted_5ad7f06d51ea416ead64d7920b0e12ad/setup.py", line 89                                                                                                                                                                            
        print """You seem to be running a very old version of setuptools.                                                                                                                                                                                                    
              ^                                                                                                                                                                                                                                                              
    SyntaxError: Missing parentheses in call to 'print'. Did you mean print("""You seem to be running a very old version of setuptools.)?                                                                                                                                    
    ----------------------------------------                                                                                                                                                                                                                                 
WARNING: Discarding https://files.pythonhosted.org/packages/a1/78/179889e5cdb8f887595bc714dc277aff6ab46029ded5595098420aed7613/Twisted-11.0.0.tar.bz2#sha256=dd91254a4c946f51c4248590ca31b496ad3e4fd1b5a9fbc3a82c39a41131a2e2 (from https://pypi.org/simple/twisted/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.                                                                                                                                                                           
  Using cached Twisted-10.2.0.tar.bz2 (2.7 MB)
    ERROR: Command errored out with exit status 1:
     command: /home/adrian/temporal/venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0hw4x1jt/twisted_c561d7425d3142c49072f052f4c95c23/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0hw4x1jt/twisted_c561d7425d3142c49072f052f4c95c23/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-bygxgexo                                                                                                                                                                                                            
         cwd: /tmp/pip-install-0hw4x1jt/twisted_c561d7425d3142c49072f052f4c95c23/                                                                                                                                                                                            
    Complete output (6 lines):                                                                                                                                                                                                                                               
    Traceback (most recent call last):                                                                                                                                                                                                                                       
      File "<string>", line 1, in <module>                                                                                                                                                                                                                                   
      File "/tmp/pip-install-0hw4x1jt/twisted_c561d7425d3142c49072f052f4c95c23/setup.py", line 84                                                                                                                                                                            
        print """You seem to be running a very old version of setuptools.                                                                                                                                                                                                    
              ^                                                                                                                                                                                                                                                              
    SyntaxError: Missing parentheses in call to 'print'. Did you mean print("""You seem to be running a very old version of setuptools.)?                                                                                                                                    
    ----------------------------------------                                                                                                                                                                                                                                 
WARNING: Discarding https://files.pythonhosted.org/packages/f9/80/50b40d787ee26af3062eb83b9a57fa3bdb5e0417f6a3047fffdbd09de6d9/Twisted-10.2.0.tar.bz2#sha256=562ed61c18aa72da99c23fb19c2c101d178995eb3a78ab3c09560a613e180c84 (from https://pypi.org/simple/twisted/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.                                                                                                                                                                           
  Using cached Twisted-10.1.0.tar.bz2 (2.6 MB)
    ERROR: Command errored out with exit status 1:
     command: /home/adrian/temporal/venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0hw4x1jt/twisted_0c4aa198e02c4efaa01cd16d8f73541a/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0hw4x1jt/twisted_0c4aa198e02c4efaa01cd16d8f73541a/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-w6m0cjzi                                                                                                                                                                                                            
         cwd: /tmp/pip-install-0hw4x1jt/twisted_0c4aa198e02c4efaa01cd16d8f73541a/                                                                                                                                                                                            
    Complete output (6 lines):                                                                                                                                                                                                                                               
    Traceback (most recent call last):                                                                                                                                                                                                                                       
      File "<string>", line 1, in <module>                                                                                                                                                                                                                                   
      File "/tmp/pip-install-0hw4x1jt/twisted_0c4aa198e02c4efaa01cd16d8f73541a/setup.py", line 84                                                                                                                                                                            
        print """You seem to be running a very old version of setuptools.                                                                                                                                                                                                    
              ^                                                                                                                                                                                                                                                              
    SyntaxError: Missing parentheses in call to 'print'. Did you mean print("""You seem to be running a very old version of setuptools.)?                                                                                                                                    
    ----------------------------------------                                                                                                                                                                                                                                 
WARNING: Discarding https://files.pythonhosted.org/packages/00/ea/6e50396d390aedf4af6e5ceff74a8885ae0bae741c5e22f72e5ec19832ff/Twisted-10.1.0.tar.bz2#sha256=eda6e0e9e5ef6f6c19ab75bcb094f83a12ee25fe589fbcddf946e8a655c8070b (from https://pypi.org/simple/twisted/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.                                                                                                                                                                           
  Using cached Twisted-10.0.0.tar.bz2 (2.6 MB)
    ERROR: Command errored out with exit status 1:
     command: /home/adrian/temporal/venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0hw4x1jt/twisted_de2619e11fde4aab84abf13d95ef13e6/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0hw4x1jt/twisted_de2619e11fde4aab84abf13d95ef13e6/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-lik8t1rj                                                                                                                                                                                                            
         cwd: /tmp/pip-install-0hw4x1jt/twisted_de2619e11fde4aab84abf13d95ef13e6/                                                                                                                                                                                            
    Complete output (6 lines):                                                                                                                                                                                                                                               
    Traceback (most recent call last):                                                                                                                                                                                                                                       
      File "<string>", line 1, in <module>                                                                                                                                                                                                                                   
      File "/tmp/pip-install-0hw4x1jt/twisted_de2619e11fde4aab84abf13d95ef13e6/setup.py", line 84                                                                                                                                                                            
        print """You seem to be running a very old version of setuptools.                                                                                                                                                                                                    
              ^                                                                                                                                                                                                                                                              
    SyntaxError: Missing parentheses in call to 'print'. Did you mean print("""You seem to be running a very old version of setuptools.)?                                                                                                                                    
    ----------------------------------------                                                                                                                                                                                                                                 
WARNING: Discarding https://files.pythonhosted.org/packages/a8/17/f425d2160b92f0c9c69964bece02eab4fc8afb45a7d9a6532c07d81d0610/Twisted-10.0.0.tar.bz2#sha256=db118c709989ba3377a989daa7566e430addbd6f187b37c8568269def8be78b1 (from https://pypi.org/simple/twisted/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.                                                                                                                                                                           
  Using cached Twisted-9.0.0.tar.bz2 (2.5 MB)
    ERROR: Command errored out with exit status 1:
     command: /home/adrian/temporal/venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0hw4x1jt/twisted_f7122596a4a549aa8a4414357aa0947f/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0hw4x1jt/twisted_f7122596a4a549aa8a4414357aa0947f/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-wuprzx6j                                                                                                                                                                                                            
         cwd: /tmp/pip-install-0hw4x1jt/twisted_f7122596a4a549aa8a4414357aa0947f/                                                                                                                                                                                            
    Complete output (6 lines):                                                                                                                                                                                                                                               
    Traceback (most recent call last):                                                                                                                                                                                                                                       
      File "<string>", line 1, in <module>                                                                                                                                                                                                                                   
      File "/tmp/pip-install-0hw4x1jt/twisted_f7122596a4a549aa8a4414357aa0947f/setup.py", line 84                                                                                                                                                                            
        print """You seem to be running a very old version of setuptools.                                                                                                                                                                                                    
              ^                                                                                                                                                                                                                                                              
    SyntaxError: Missing parentheses in call to 'print'. Did you mean print("""You seem to be running a very old version of setuptools.)?                                                                                                                                    
    ----------------------------------------                                                                                                                                                                                                                                 
WARNING: Discarding https://files.pythonhosted.org/packages/3b/b0/c4cafea9ab2e31aa15556eddc151ba987e130cb347b753d991aa2ace768a/Twisted-9.0.0.tar.bz2#sha256=5711c2f70ce415cfd8d47da2e2387fba36df4dfbf59efba4fc0ba2ef889f6ca8 (from https://pypi.org/simple/twisted/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached Twisted-2.1.0.tar.bz2 (1.1 MB)
    ERROR: Command errored out with exit status 1:
     command: /home/adrian/temporal/venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0hw4x1jt/twisted_85968e4cd3024feda72e8ac378eee3ca/setup.py'"'"'; __file__='"'"'/tmp/pip-install-0hw4x1jt/twisted_85968e4cd3024feda72e8ac378eee3ca/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-b8os6nfi
         cwd: /tmp/pip-install-0hw4x1jt/twisted_85968e4cd3024feda72e8ac378eee3ca/
    Complete output (6 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-0hw4x1jt/twisted_85968e4cd3024feda72e8ac378eee3ca/setup.py", line 37
        print "Compiler not found, skipping C extensions."
              ^
    SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Compiler not found, skipping C extensions.")?
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/7d/11/6f06a890249c36076d531c94468b0d250da87ce31c95e71149726b094001/Twisted-2.1.0.tar.bz2#sha256=a34e6fb06797c3de1e2c4037b373e633e7265e336b84a61d4f7ea8a7f0f07579 (from https://pypi.org/simple/twisted/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Collecting PyHamcrest!=1.10.0,>=1.9.0
  Using cached PyHamcrest-2.0.2-py3-none-any.whl (52 kB)
INFO: pip is looking at multiple versions of twisted to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of twisted[http2] to determine which version is compatible with other requirements. This could take a while.
WARNING: twisted 16.3.2 does not provide the extra 'http2'
Requirement already satisfied: setuptools in ./venv/lib/python3.9/site-packages (from zope.interface>=4.4.2->Twisted[http2]) (49.2.1)
Using legacy 'setup.py install' for Twisted, since package 'wheel' is not installed.
Installing collected packages: Twisted, hyperframe, hpack, h2
  Attempting uninstall: Twisted
    Found existing installation: Twisted 21.2.0
    Uninstalling Twisted-21.2.0:
      Successfully uninstalled Twisted-21.2.0
    Running setup.py install for Twisted ... done
  Attempting uninstall: hyperframe
    Found existing installation: hyperframe 5.2.0
    Uninstalling hyperframe-5.2.0:
      Successfully uninstalled hyperframe-5.2.0
  Attempting uninstall: hpack
    Found existing installation: hpack 3.0.0
    Uninstalling hpack-3.0.0:
      Successfully uninstalled hpack-3.0.0
  Attempting uninstall: h2
    Found existing installation: h2 3.2.0
    Uninstalling h2-3.2.0:
      Successfully uninstalled h2-3.2.0
Successfully installed Twisted-16.3.2 h2-4.0.0 hpack-4.0.0 hyperframe-6.0.0

Workaround

The desired results are obtained if the upper h2 version limit from Twisted[http2], <4, is copied into the command-line level h2 requirement:

pip install --upgrade 'Twisted[http2]' 'h2>=3.2.0,<4'

Additional information

Based on https://github.com/scrapy/scrapy/pull/5066

uranusjr commented 3 years ago

Another workaround is to set h2>=3.2.0 as a constraint instead. This is because the resolver’s current logic to “prefer” a chosen package version is by 1. whether the package is user-specified, and 2. alphabetically, so h2 is preferred and twisted backtracks in this case. By specifying h2 as a constraint, twisted will be preferred.

I have some improvements in mind that may help with this[1], but there’s no actual code yet and will have to wait until 21.2 (at least). I’ll try to remember to ping here when I have something to test.

[1]: Basically adding an additional sort criteria to prefer keeping user-specified packages that are not also transitive requirements of other packages, so twisted can be preferred over h1 because h1 is also depended by twisted.

uranusjr commented 3 years ago

Looking at the logic, untimately this is where things go against your expectation:

WARNING: twisted 16.3.2 does not provide the extra 'http2'

But pip is historically very lenient about extras and would happily drop any invalid values. We could potentially enforce that instead, but that would affect a lot of people since there are packages out there (mostly old and less updated stuff) with invalid extras, and it would be extremely frustrating for their users if pip start failing to install those.

I guess a middle-ground solution would be to enforce extras supplied by the user, i.e. in your case twisted would fail to install because your supplied http2 extra cannot be installed. But if a package transitively depends on foo[a] and extra a doesn’t actually exist on foo, that extra will continue to be dropped, since this is outside the user’s control—unless the user also specifies foo[a] from the command line, in which case pip should error out.

The problem is, this is already pretty not trivial to explain, and the implementation would be even less straightforward. But I guess that is something doable (and hopefully reasonable). As always, contributions are always welcomed.

Gallaecio commented 3 years ago

[enforcing valid extras] would affect a lot of people since there are packages out there (mostly old and less updated stuff) with invalid extras. […]. I guess a middle-ground solution would be to enforce extras supplied by the user […]

What about enforcing extras if they are found to be valid? For example, when pip downloads the latest valid Twisted version here, and finds that extra is defined in it, it could as a consequence consider that extra a valid extra (i.e. same as user-defined) and stop backtracking if a version is found that does not define it.

Just a thought, though. I realize this may make the implementation even more complex, and maybe create other problems.

uranusjr commented 3 years ago

It’s difficult to qualify what is “found to be valid” though. Say a package foo was published with an invalid extra bar[ext], but that ext extra was added to a later version of bar. Does ext count as a valid bar extra now when you install foo? Does the answer change depending on whether the later bar version is installable on the target machine?

The most difficult thing about designing dependency resolution logic is the same dependency specification may change meaning over time. Therefore it is usually a good idea to distinguish between user input and transitive dependencies, because the former is guaranteed to express an accurate user intent, which the latter may not.

meejah commented 1 year ago

(I found this issue while searching before filing a feature request.)

The core of what @Gallaecio is asking for makes a lot of sense to me. Currently, if I say: pip install foo[bar] and the "bar" extra does not exist, a warning is printed out (yay!) but then immediately gets pushed out of the way while a bunch of things install ... at first glance, this appears successful!

But what actually happened is that I get foo installed .. which isn't the foo[bar] I asked for.

I think pip should error out in this case (i.e. the "warning" becomes an error). Ideally it would also print out which extras are available for the package you asked for. That is, something like: The "bar" extra is not available in package "foo", which has the following extras: quux, baz, zinga

(I realize the above doesn't necessarily help with the wider issue of what to do about extras declared as dependencies or transitive dependencies -- but I think the CLI user-experience would be better for at least "command-line-specified installation targets")

Maybe this makes sense as a separate feature bug, but it seems highly related to this one...

jaraco commented 4 months ago

To add some color to this discussion, I'm working on a build/test system that aims to move a lot of the project configuration into tooling and infrastructure. It assumes that every project under test is going to declare any test dependencies under the test extra and installs them when setting up to run tests.

Some projects, however, will have no project-specific test dependencies. In this case, the project gets built without any Provides-Extra: test in the metadata. I'm leaning toward having every project supply the test extra even when there are no dependencies, to suppress the warning under the current pip install technique but also satisfy a future where installers are more strict about installing them, such as this issue is proposing.

If you have any concerns about that approach, please comment in https://github.com/coherent-oss/coherent.test/issues/5.

pypa / pip

Fail dependency resolution when a specified extra is invalid, instead of installing the base package without the invalid extra #9736