search

pypa / pip

The Python package installer
https://pip.pypa.io/
MIT License
9.53k stars 3.03k forks source link

dependency resolution behaves differently with requirements.txt vs listing on the command-line #10544

Closed stefansjs closed 3 years ago

stefansjs commented 3 years ago

Description

When doing pip install -r requirements.txt the resolver notices a version conflict and fails quickly, whereas listing the same requirements on the command line causes an extended un-resolved backtracking. pip install -r failed in 20s on my machine, while the pip install command is still running after an hour so far.

Expected behavior

pip install -r and pip install behave the same with respect to the resolver

pip version

21.2.4

Python version

3.9

OS

Mac OS 10.15.6

How to Reproduce

  1. pip install -r requirements.txt
  2. pip install 'numpy>1.18' -e .

requirements.txt contains:

-e .
numpy>1.19

setup.py contains

from setuptools import setup
setup(
          install_requires=[
              "configparser>=3.5",
              "kafka>=1.3.5; python_version<'3.7'",
              "kafka-python>=1.4.3; python_version>='3.7'",
              "confluent-kafka[avro]==1.3.0",
              "fastavro",  # Whatever's needed by confluent-kafka[avro]
              'avro-python3==1.9.2.1',  # confluent-kafka[avro] defines a dependency on 1.9.2, but only 1.9.2.1 works
              "isort>=4",

              "scipy>=1.2.1, <=1.4.1; python_version<'3.7'",
              "scipy",
              "numpy>=1.17; python_version<'3.9'",
              "numpy>1.17.2; python_version>='3.9'",
              "teamcity-messages>=1.17",
              "vertica_python>=0.9.1",  # Literally any version since we only need imports to succeed
              "mido>=1.2.8",
              "requests<=2.23.0",  # We've tested with up to 2.23.0, but snowflake-connector-python requires less than that :'(
              "python-Levenshtein>=0.12",
              "aubio==0.4.9",
              "future>=0.17",
              "six>=1.11",
              "datasketch>=1.5",
              "jams>=0.3",
              "python-slugify>=3",
          ],
)

Output

$ pip install -r requirements.txt
Looking in indexes: https://pypi.org/simple, http://sfw.sf.smle.co:8080/sfw/pip
Looking in links: file:///Users/stefansullivan/code/test_pip_resolution.
Obtaining file:///Users/stefansullivan/code/test_pip_resolution (from -r requirements.in (line 3))
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Obtaining file:///<my project dir> (from -r requirements.in (line 4))
Requirement already satisfied: bokeh in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from -r requirements.in (line 5)) (2.4.0)
Requirement already satisfied: matplotlib in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from -r requirements.in (line 6)) (3.4.3)
Requirement already satisfied: pandas in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from -r requirements.in (line 7)) (1.3.3)
Requirement already satisfied: brigit in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from -r requirements.in (line 8)) (1.3.0)
Requirement already satisfied: tqdm in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from -r requirements.in (line 9)) (4.62.3)
Requirement already satisfied: numpy>1.18 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from -r requirements.in (line 10)) (1.21.2)
Collecting confluent-kafka[avro]==1.3.0
  Using cached confluent_kafka-1.3.0-cp39-cp39-macosx_10_15_x86_64.whl
Collecting mido<=1.2.9,>=1.2.8
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/mido/mido-1.2.9-py2.py3-none-any.whl (52 kB)
Collecting isort==4.3.21
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/isort/isort-4.3.21-py2.py3-none-any.whl (42 kB)
Collecting datasketch==1.5.1
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/datasketch/datasketch-1.5.1-py2.py3-none-any.whl (73 kB)
Collecting kafka==1.3.5
  Using cached kafka-1.3.5-py2.py3-none-any.whl (207 kB)
Collecting requests<=2.23.0,==2.22.0
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/requests/requests-2.22.0-py2.py3-none-any.whl (57 kB)
Collecting jams==0.3.4
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/jams/jams-0.3.4-py3-none-any.whl (64 kB)
Collecting avro-python3==1.9.2.1
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/avro-python3/avro_python3-1.9.2.1-py3-none-any.whl (43 kB)
Collecting configparser==3.5.0
  Using cached configparser-3.5.0-py3-none-any.whl
Collecting scipy<=1.4.1,>=1.2.1
  Using cached scipy-1.4.1.tar.gz (24.6 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Collecting vertica-python<=0.10.2,>=0.9.1
  Using cached vertica_python-0.10.2-py2.py3-none-any.whl (152 kB)
Collecting python-Levenshtein==0.12.0
  Using cached python_Levenshtein-0.12.0-cp39-cp39-macosx_10_15_x86_64.whl
Collecting aubio==0.4.9
  Using cached aubio-0.4.9+smule.1-cp39-cp39-macosx_10_15_x86_64.whl

The conflict is caused by:
    The user requested numpy>1.18
    unknown 0.1 depends on numpy==1.17.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

$ pip install 'numpy>1.18' -e .
Looking in indexes: https://pypi.org/simple, http://sfw.sf.smle.co:8080/sfw/pip
Looking in links: .
Obtaining file:///Users/stefansullivan/code/test_pip_resolution
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Obtaining file:///<my project dir>
Requirement already satisfied: bokeh in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (2.4.0)
Requirement already satisfied: matplotlib in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (3.4.3)
Requirement already satisfied: pandas in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (1.3.3)
Requirement already satisfied: brigit in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (1.3.0)
Requirement already satisfied: tqdm in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (4.62.3)
Requirement already satisfied: numpy>1.18 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (1.21.2)
Requirement already satisfied: pillow>=7.1.0 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from bokeh) (8.3.2)
Requirement already satisfied: Jinja2>=2.9 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from bokeh) (3.0.1)
Requirement already satisfied: tornado>=5.1 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from bokeh) (6.1)
Requirement already satisfied: packaging>=16.8 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from bokeh) (21.0)
Requirement already satisfied: PyYAML>=3.10 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from bokeh) (5.4.1)
Requirement already satisfied: typing-extensions>=3.10.0 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from bokeh) (3.10.0.2)
Requirement already satisfied: kiwisolver>=1.0.1 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from matplotlib) (1.3.2)
Requirement already satisfied: python-dateutil>=2.7 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from matplotlib) (2.8.2)
Requirement already satisfied: pyparsing>=2.2.1 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from matplotlib) (2.4.7)
Requirement already satisfied: cycler>=0.10 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from matplotlib) (0.10.0)
Requirement already satisfied: pytz>=2017.3 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from pandas) (2021.1)
Requirement already satisfied: log-colorizer in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from brigit) (1.8.6)
Collecting configparser==3.5.0
  Using cached configparser-3.5.0-py3-none-any.whl
Collecting datasketch==1.5.1
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/datasketch/datasketch-1.5.1-py2.py3-none-any.whl (73 kB)
Collecting teamcity-messages==1.17
  Using cached teamcity_messages-1.17-py3-none-any.whl
Collecting requests<=2.23.0,==2.22.0
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/requests/requests-2.22.0-py2.py3-none-any.whl (57 kB)
Collecting isort==4.3.21
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/isort/isort-4.3.21-py2.py3-none-any.whl (42 kB)
Collecting six<=1.14.0,>=1.11.0
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
INFO: pip is looking at multiple versions of numpy to determine which version is compatible with other requirements. This could take a while.
Collecting numpy>1.18
  Using cached numpy-1.21.2-cp39-cp39-macosx_10_9_x86_64.whl (17.0 MB)
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
  Using cached numpy-1.21.1-cp39-cp39-macosx_10_9_x86_64.whl (17.0 MB)
  Using cached numpy-1.21.0-cp39-cp39-macosx_10_9_x86_64.whl (16.9 MB)
  Using cached numpy-1.20.3-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
  Using cached numpy-1.20.2-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
  Using cached numpy-1.20.1-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
  Using cached numpy-1.20.0-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
INFO: pip is looking at multiple versions of numpy to determine which version is compatible with other requirements. This could take a while.
  Using cached numpy-1.19.5-cp39-cp39-macosx_10_9_x86_64.whl (15.6 MB)
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
  Using cached numpy-1.19.4-cp39-cp39-macosx_10_9_x86_64.whl (15.4 MB)
  Using cached numpy-1.19.3-cp39-cp39-macosx_10_9_x86_64.whl (15.9 MB)
  Using cached numpy-1.19.2.zip (7.3 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  Using cached numpy-1.19.1.zip (7.3 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Using cached numpy-1.19.0.zip (7.3 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Using cached numpy-1.18.5.zip (5.4 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  Using cached numpy-1.18.4.zip (5.4 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  Using cached numpy-1.18.3.zip (5.4 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  Using cached numpy-1.18.2.zip (5.4 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  Using cached numpy-1.18.1.zip (5.4 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
INFO: pip is looking at multiple versions of tqdm to determine which version is compatible with other requirements. This could take a while.
Collecting tqdm
  Using cached tqdm-4.62.3-py2.py3-none-any.whl (76 kB)
  Using cached tqdm-4.62.2-py2.py3-none-any.whl (76 kB)
  Using cached tqdm-4.62.1-py2.py3-none-any.whl (76 kB)
  Using cached tqdm-4.62.0-py2.py3-none-any.whl (76 kB)
  Using cached tqdm-4.61.2-py2.py3-none-any.whl (76 kB)
  Using cached tqdm-4.61.1-py2.py3-none-any.whl (75 kB)
  Using cached tqdm-4.61.0-py2.py3-none-any.whl (75 kB)
INFO: pip is looking at multiple versions of tqdm to determine which version is compatible with other requirements. This could take a while.
  Using cached tqdm-4.60.0-py2.py3-none-any.whl (75 kB)
  Using cached tqdm-4.59.0-py2.py3-none-any.whl (74 kB)
  Using cached tqdm-4.58.0-py2.py3-none-any.whl (73 kB)
  Using cached tqdm-4.57.0-py2.py3-none-any.whl (72 kB)
  Using cached tqdm-4.56.2-py2.py3-none-any.whl (72 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Using cached tqdm-4.56.1-py2.py3-none-any.whl (72 kB)
  Using cached tqdm-4.56.0-py2.py3-none-any.whl (72 kB)
  Using cached tqdm-4.55.2-py2.py3-none-any.whl (68 kB)
  Using cached tqdm-4.55.1-py2.py3-none-any.whl (68 kB)
  Using cached tqdm-4.55.0-py2.py3-none-any.whl (68 kB)
  Using cached tqdm-4.54.1-py2.py3-none-any.whl (69 kB)
  Using cached tqdm-4.54.0-py2.py3-none-any.whl (69 kB)
  Using cached tqdm-4.53.0-py2.py3-none-any.whl (70 kB)
  Using cached tqdm-4.52.0-py2.py3-none-any.whl (71 kB)
  Using cached tqdm-4.51.0-py2.py3-none-any.whl (70 kB)
  Using cached tqdm-4.50.2-py2.py3-none-any.whl (70 kB)
  Using cached tqdm-4.50.1-py2.py3-none-any.whl (70 kB)
  Using cached tqdm-4.50.0-py2.py3-none-any.whl (70 kB)
  Using cached tqdm-4.49.0-py2.py3-none-any.whl (69 kB)
  Using cached tqdm-4.48.2-py2.py3-none-any.whl (68 kB)
  Using cached tqdm-4.48.1-py2.py3-none-any.whl (68 kB)
  Using cached tqdm-4.48.0-py2.py3-none-any.whl (67 kB)
  Using cached tqdm-4.47.0-py2.py3-none-any.whl (66 kB)
  Using cached tqdm-4.46.1-py2.py3-none-any.whl (63 kB)
  Using cached tqdm-4.46.0-py2.py3-none-any.whl (63 kB)
  Using cached tqdm-4.45.0-py2.py3-none-any.whl (60 kB)
  Using cached tqdm-4.44.1-py2.py3-none-any.whl (60 kB)
  Using cached tqdm-4.44.0-py2.py3-none-any.whl (60 kB)
  Using cached tqdm-4.43.0-py2.py3-none-any.whl (59 kB)
  Using cached tqdm-4.42.1-py2.py3-none-any.whl (59 kB)
  Using cached tqdm-4.42.0-py2.py3-none-any.whl (59 kB)
  Using cached tqdm-4.41.1-py2.py3-none-any.whl (56 kB)
  Using cached tqdm-4.41.0-py2.py3-none-any.whl (56 kB)
  Using cached tqdm-4.40.2-py2.py3-none-any.whl (55 kB)
  Using cached tqdm-4.40.1-py2.py3-none-any.whl (55 kB)
  Using cached tqdm-4.40.0-py2.py3-none-any.whl (54 kB)
  Using cached tqdm-4.39.0-py2.py3-none-any.whl (53 kB)
  Using cached tqdm-4.38.0-py2.py3-none-any.whl (53 kB)
  Using cached tqdm-4.37.0-py2.py3-none-any.whl (53 kB)
  Using cached tqdm-4.36.1-py2.py3-none-any.whl (52 kB)
  Using cached tqdm-4.36.0-py2.py3-none-any.whl (52 kB)
  Using cached tqdm-4.35.0-py2.py3-none-any.whl (50 kB)
  Using cached tqdm-4.34.0-py2.py3-none-any.whl (50 kB)
  Using cached tqdm-4.33.0-py2.py3-none-any.whl (50 kB)
  Using cached tqdm-4.32.2-py2.py3-none-any.whl (50 kB)
  Using cached tqdm-4.32.1-py2.py3-none-any.whl (49 kB)
  Using cached tqdm-4.32.0-py2.py3-none-any.whl (49 kB)
  Using cached tqdm-4.31.1-py2.py3-none-any.whl (48 kB)
  Using cached tqdm-4.31.0-py2.py3-none-any.whl (48 kB)
  Using cached tqdm-4.30.0-py2.py3-none-any.whl (47 kB)
  Using cached tqdm-4.29.1-py2.py3-none-any.whl (46 kB)
  Using cached tqdm-4.29.0-py2.py3-none-any.whl (46 kB)
  Using cached tqdm-4.28.1-py2.py3-none-any.whl (45 kB)
  Using cached tqdm-4.28.0-py2.py3-none-any.whl (43 kB)
  Using cached tqdm-4.27.0-py2.py3-none-any.whl (44 kB)
  Using cached tqdm-4.26.0-py2.py3-none-any.whl (43 kB)
  Using cached tqdm-4.25.0-py2.py3-none-any.whl (43 kB)
  Using cached tqdm-4.24.0-py2.py3-none-any.whl (43 kB)
  Using cached tqdm-4.23.4-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.23.3-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.23.2-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.23.1-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.23.0-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.22.0-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.21.0-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.20.0-py2.py3-none-any.whl (41 kB)
  Using cached tqdm-4.19.9-py2.py3-none-any.whl (52 kB)
  Using cached tqdm-4.19.8-py2.py3-none-any.whl (52 kB)
  Using cached tqdm-4.19.7-py2.py3-none-any.whl (52 kB)
  Using cached tqdm-4.19.6-py2.py3-none-any.whl (52 kB)
  Using cached tqdm-4.19.5-py2.py3-none-any.whl (51 kB)
  Using cached tqdm-4.19.4-py2.py3-none-any.whl (50 kB)
  Using cached tqdm-4.19.2-py2.py3-none-any.whl (50 kB)
  Using cached tqdm-4.19.1.post1-py2.py3-none-any.whl (51 kB)
  Using cached tqdm-4.19.1-py2.py3-none-any.whl (50 kB)
  Using cached tqdm-4.18.0-py2.py3-none-any.whl (48 kB)
  Using cached tqdm-4.17.1-py2.py3-none-any.whl (47 kB)
  Using cached tqdm-4.17.0-py2.py3-none-any.whl (47 kB)
  Using cached tqdm-4.16.0-py2.py3-none-any.whl (47 kB)
  Using cached tqdm-4.15.0-py2.py3-none-any.whl (46 kB)
  Using cached tqdm-4.14.0-py2.py3-none-any.whl (46 kB)
  Using cached tqdm-4.13.0-py2.py3-none-any.whl (46 kB)
  Using cached tqdm-4.12.0-py2.py3-none-any.whl (46 kB)
  Using cached tqdm-4.11.2-py2.py3-none-any.whl (46 kB)
  Using cached tqdm-4.11.1-py2.py3-none-any.whl (45 kB)
  Using cached tqdm-4.11.0-py2.py3-none-any.whl (45 kB)
  Using cached tqdm-4.10.0-py2.py3-none-any.whl (43 kB)
  Using cached tqdm-4.9.0-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.8.4-py2.py3-none-any.whl (39 kB)
  Using cached tqdm-4.8.3-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.8.2-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.8.1-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.7.6-py2.py3-none-any.whl (40 kB)
  Using cached tqdm-4.7.4-py2.py3-none-any.whl (40 kB)
  Using cached tqdm-4.7.2-py2.py3-none-any.whl (40 kB)
  Using cached tqdm-4.7.1-py2.py3-none-any.whl (39 kB)
  Using cached tqdm-4.7.0-py2.py3-none-any.whl (39 kB)
  Using cached tqdm-4.6.2-py2.py3-none-any.whl (38 kB)
  Using cached tqdm-4.6.1-py2.py3-none-any.whl (38 kB)
  Using cached tqdm-4.5.2-py2.py3-none-any.whl (34 kB)
  Using cached tqdm-4.5.0-py2.py3-none-any.whl (34 kB)
  Using cached tqdm-4.4.3-py2.py3-none-any.whl (33 kB)
  Using cached tqdm-4.4.1-py2.py3-none-any.whl (32 kB)
  Using cached tqdm-4.4.0-py2.py3-none-any.whl (32 kB)
  Using cached tqdm-4.1.0-py2.py3-none-any.whl (29 kB)
  Using cached tqdm-3.8.0-py2.py3-none-any.whl (28 kB)
  Using cached tqdm-3.7.1-py2.py3-none-any.whl (28 kB)
  Using cached tqdm-3.7.0-py2.py3-none-any.whl (28 kB)
  Using cached tqdm-3.4.0-py2.py3-none-any.whl (26 kB)
  Using cached tqdm-3.1.4-py2.py3-none-any.whl (19 kB)
  Using cached tqdm-3.1.3-py2.py3-none-any.whl (19 kB)
  Using cached tqdm-2.2.4-py2.py3-none-any.whl (17 kB)
  Using cached tqdm-2.2.3-py2.py3-none-any.whl (17 kB)
  Using cached tqdm-2.0.0-py2.py3-none-any.whl (16 kB)
  Using cached tqdm-1.0.tar.gz (1.8 kB)
INFO: pip is looking at multiple versions of brigit to determine which version is compatible with other requirements. This could take a while.
Collecting brigit
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/brigit/brigit-1.3.0-py2.py3-none-any.whl (8.6 kB)
  Using cached brigit-1.2.tar.gz (2.5 kB)
  Using cached brigit-1.1.tar.gz (2.5 kB)
  Using cached brigit-1.0.tar.gz (2.5 kB)
INFO: pip is looking at multiple versions of pandas to determine which version is compatible with other requirements. This could take a while.
Collecting pandas
  Using cached pandas-1.3.3-cp39-cp39-macosx_10_9_x86_64.whl (11.6 MB)
INFO: pip is looking at multiple versions of brigit to determine which version is compatible with other requirements. This could take a while.
  Using cached pandas-1.3.2-cp39-cp39-macosx_10_9_x86_64.whl (11.6 MB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Using cached pandas-1.3.1-cp39-cp39-macosx_10_9_x86_64.whl (11.3 MB)
  Using cached pandas-1.3.0-cp39-cp39-macosx_10_9_x86_64.whl (11.6 MB)
  Using cached pandas-1.2.5-cp39-cp39-macosx_10_9_x86_64.whl (10.7 MB)
  Using cached pandas-1.2.4-cp39-cp39-macosx_10_9_x86_64.whl (10.7 MB)
  Using cached pandas-1.2.3-cp39-cp39-macosx_10_9_x86_64.whl (10.7 MB)
INFO: pip is looking at multiple versions of pandas to determine which version is compatible with other requirements. This could take a while.
  Using cached pandas-1.2.2-cp39-cp39-macosx_10_9_x86_64.whl (10.7 MB)
  Using cached pandas-1.2.1-cp39-cp39-macosx_10_9_x86_64.whl (10.7 MB)
  Using cached pandas-1.2.0-cp39-cp39-macosx_10_9_x86_64.whl (10.7 MB)
  Using cached pandas-1.1.5-cp39-cp39-macosx_10_9_x86_64.whl (10.3 MB)
  Using cached pandas-1.1.4-cp39-cp39-macosx_10_9_x86_64.whl (10.3 MB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Using cached pandas-1.1.3-cp39-cp39-macosx_10_9_x86_64.whl (10.3 MB)
  Using cached pandas-1.1.2.tar.gz (5.2 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  Using cached pandas-1.1.1.tar.gz (5.2 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  Using cached pandas-1.1.0.tar.gz (5.2 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  Using cached pandas-1.0.5.tar.gz (5.0 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  Using cached pandas-1.0.4.tar.gz (5.0 MB)
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'error'
  Using cached pandas-1.0.3.tar.gz (5.0 MB)
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'error'
  Using cached pandas-1.0.2.tar.gz (5.0 MB)
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'error'
  Using cached pandas-1.0.1.tar.gz (4.9 MB)
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'error'
  Using cached pandas-1.0.0.tar.gz (4.8 MB)
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'error'
  Using cached pandas-0.25.3.tar.gz (12.6 MB)
  Using cached pandas-0.25.2.tar.gz (12.6 MB)
  Using cached pandas-0.25.1.tar.gz (12.6 MB)
  Using cached pandas-0.25.0.tar.gz (12.6 MB)
  Using cached pandas-0.24.2.tar.gz (11.8 MB)
  Using cached pandas-0.24.1.tar.gz (11.8 MB)
  Using cached pandas-0.24.0.tar.gz (11.8 MB)
  Using cached pandas-0.23.4.tar.gz (10.5 MB)
  Using cached pandas-0.23.3.tar.gz (10.5 MB)
  Using cached pandas-0.23.2.tar.gz (10.0 MB)
  Using cached pandas-0.23.1.tar.gz (13.1 MB)
  Using cached pandas-0.23.0.tar.gz (13.1 MB)
  Using cached pandas-0.22.0.tar.gz (11.3 MB)
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'error'
  Using cached pandas-0.21.1.tar.gz (11.3 MB)
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'error'
  Using cached pandas-0.21.0.tar.gz (11.3 MB)
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'error'
  Using cached pandas-0.20.3.tar.gz (10.4 MB)
  Using cached pandas-0.20.2.tar.gz (10.3 MB)
  Using cached pandas-0.20.1.tar.gz (10.3 MB)
  Using cached pandas-0.20.0.tar.gz (10.3 MB)
  Using cached pandas-0.19.2.tar.gz (9.2 MB)

Code of Conduct

notatallshaw commented 3 years ago

The resolver since 21.2 attempts to prefer the order in which you give it requirements.

In your requirements file example your order is effectively:

{your non-numpy requirements}
{your numpy requirement}

In your command line example your order is effectively:

{your numpy requirement}
{your non-numpy requirements}

Because in the former example numpy is attempted to be pinned last and in the latter example numpy is attempted to pinned first they are going to have very different dependency resolution behavior, especially if your requirements need to be backtracked. You should be able to confirm this by swapping the order you give the requirements in either of your examples.

I am hoping that the performance in both cases should be acceptable once this lands: https://github.com/pypa/pip/issues/10479 but I don't have time to test right now. I'll see if I can test tomorrow evening.

stefansjs commented 3 years ago

So that's an interesting detail that I wasn't aware of. However, I can be quite certain that the order is not the culprit here. I've redone the test using only bash invocation differences and I can see the behavior is quite different. With a -r command it takes about 40s, and with command-line requirements it takes 3m40s.

3 minutes might not seem like an insane amount of time to wait (IMHO it is, but that's a different discussion), but this is actually a reduced example from what I was doing which was installing many more packages. They should behave the same when provided with a requirements file vs command-line arguments, especially if order is important.

It appears that when presented on the command-line, the resolver starts doing backtracking on the numpy package, whereas the requirements file fails much more quickly.

Here's another invocation that does a better job of showing the difference where I simply invoke

pip install -r <(echo "-e .\nnumpy>1.18")
pip install -e . "numpy>1.18"
$ time pip install -r <(echo "-e .\nnumpy>1.18")
Looking in indexes: https://pypi.org/simple, http://sfw.sf.smle.co:8080/sfw/pip
Obtaining file:///Users/stefansullivan/code/test_pip_resolution (from -r /dev/fd/11 (line 1))
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: numpy>1.18 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from -r /dev/fd/11 (line 2)) (1.21.2)
Collecting cysignals==1.6.5
  Using cached cysignals-1.6.5-cp39-cp39-macosx_10_15_x86_64.whl
Collecting python-Levenshtein==0.12.0
  Using cached python_Levenshtein-0.12.0-cp39-cp39-macosx_10_15_x86_64.whl
Requirement already satisfied: aubio==0.4.9 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from unknown==0.1->-r /dev/fd/11 (line 1)) (0.4.9+smule.1)
Requirement already satisfied: fastavro in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from unknown==0.1->-r /dev/fd/11 (line 1)) (1.4.5)
Collecting configparser==3.5.0
  Using cached configparser-3.5.0-py3-none-any.whl
Collecting scipy<=1.4.1,>=1.2.1
  Using cached scipy-1.4.1.tar.gz (24.6 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting isort==4.3.21
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/isort/isort-4.3.21-py2.py3-none-any.whl (42 kB)
     |████████████████████████████████| 42 kB 923 kB/s 
Collecting requests<=2.23.0,==2.22.0
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/requests/requests-2.22.0-py2.py3-none-any.whl (57 kB)
     |████████████████████████████████| 57 kB 824 kB/s 
ERROR: Cannot install -r /dev/fd/11 (line 1) and numpy>1.18 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested numpy>1.18
    unknown 0.1 depends on numpy==1.17.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
pip install -r <(echo "-e .\nnumpy>1.18")  28.69s user 4.88s system 85% cpu 39.083 total

$ time pip install -e . "numpy>1.18"
Looking in indexes: https://pypi.org/simple, http://sfw.sf.smle.co:8080/sfw/pip
Obtaining file:///Users/stefansullivan/code/test_pip_resolution
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: numpy>1.18 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (1.21.2)
Collecting teamcity-messages==1.17
  Using cached teamcity_messages-1.17-py3-none-any.whl
INFO: pip is looking at multiple versions of numpy to determine which version is compatible with other requirements. This could take a while.
Collecting numpy>1.18
  Using cached numpy-1.21.2-cp39-cp39-macosx_10_9_x86_64.whl (17.0 MB)
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
  Using cached numpy-1.21.1-cp39-cp39-macosx_10_9_x86_64.whl (17.0 MB)
  Using cached numpy-1.21.0-cp39-cp39-macosx_10_9_x86_64.whl (16.9 MB)
  Using cached numpy-1.20.3-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
  Using cached numpy-1.20.2-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
  Using cached numpy-1.20.1-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
  Using cached numpy-1.20.0-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
INFO: pip is looking at multiple versions of numpy to determine which version is compatible with other requirements. This could take a while.
  Using cached numpy-1.19.5-cp39-cp39-macosx_10_9_x86_64.whl (15.6 MB)
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
  Using cached numpy-1.19.4-cp39-cp39-macosx_10_9_x86_64.whl (15.4 MB)
  Using cached numpy-1.19.3-cp39-cp39-macosx_10_9_x86_64.whl (15.9 MB)
  Using cached numpy-1.19.2.zip (7.3 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Using cached numpy-1.19.1.zip (7.3 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Using cached numpy-1.19.0.zip (7.3 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Using cached numpy-1.18.5.zip (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Using cached numpy-1.18.4.zip (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Using cached numpy-1.18.3.zip (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Using cached numpy-1.18.2.zip (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Using cached numpy-1.18.1.zip (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
ERROR: Cannot install numpy>1.18 and unknown==0.1 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested numpy>1.18
    unknown 0.1 depends on numpy==1.17.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
pip install -e . "numpy>1.18"  164.19s user 44.28s system 94% cpu 3:40.53 total
notatallshaw commented 3 years ago

I can't reproduce your issue but I don't have an OS X device to test on and I don't know what or how you've configured this second index "http://sfw.sf.smle.co:8080/sfw/pip" which definitely appears to be contributing to the backtracking.

If I could reproduce the issue I would attempt to look at what is getting ordered different when self._get_preference is called: https://github.com/pypa/pip/blob/21.2.4/src/pip/_vendor/resolvelib/resolvers.py#L365 which ultimately calls this logic: https://github.com/pypa/pip/blob/21.2.4/src/pip/_internal/resolution/resolvelib/provider.py#L69 . Debugging why the resolver does something is unfortunately a little tricky.

stefansjs commented 3 years ago

It's an apache http server with vanilla directory listings as a simple repository server. What's your hypothesis about that index? I don't see it contributing to the longer run from the log I posted.

If I run the command again without the extra index server in my pip.conf, I still see the behavior

$ time pip install -e . "numpy>1.18"       
Obtaining file:///Users/stefansullivan/code/test_pip_resolution
Requirement already satisfied: numpy>1.18 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (1.21.2)
Requirement already satisfied: configparser>=3.5 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (5.0.2)
Requirement already satisfied: confluent-kafka[avro]==1.3.0 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.3.0)
Requirement already satisfied: fastavro in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.4.5)
Requirement already satisfied: avro-python3==1.9.2.1 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.9.2.1)
Requirement already satisfied: isort>=4 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (5.9.3)
Requirement already satisfied: scipy in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.7.1)
Requirement already satisfied: teamcity-messages>=1.17 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.29)
Requirement already satisfied: vertica_python>=0.9.1 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.0.1)
Requirement already satisfied: mido>=1.2.8 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.2.10)
Requirement already satisfied: requests<=2.23.0 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (2.23.0)
Requirement already satisfied: python-Levenshtein>=0.12 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (0.12.2)
Requirement already satisfied: aubio==0.4.9 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (0.4.9+smule.1)
Requirement already satisfied: future>=0.17 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (0.18.2)
Requirement already satisfied: six>=1.11 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.16.0)
Requirement already satisfied: datasketch>=1.5 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.5.3)
Requirement already satisfied: jams>=0.3 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (0.3.4)
Requirement already satisfied: python-slugify>=3 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (5.0.2)
Requirement already satisfied: kafka-python>=1.4.3 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (2.0.2)
Requirement already satisfied: mir-eval>=0.5 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from jams>=0.3->UNKNOWN==0.0.0) (0.6)
Requirement already satisfied: sortedcontainers>=2.0.0 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from jams>=0.3->UNKNOWN==0.0.0) (2.4.0)
Requirement already satisfied: decorator in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from jams>=0.3->UNKNOWN==0.0.0) (5.1.0)
Requirement already satisfied: jsonschema>=3.0.0 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from jams>=0.3->UNKNOWN==0.0.0) (3.2.0)
Requirement already satisfied: pandas in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from jams>=0.3->UNKNOWN==0.0.0) (1.3.3)
Requirement already satisfied: attrs>=17.4.0 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from jsonschema>=3.0.0->jams>=0.3->UNKNOWN==0.0.0) (21.2.0)
Requirement already satisfied: setuptools in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from jsonschema>=3.0.0->jams>=0.3->UNKNOWN==0.0.0) (57.4.0)
Requirement already satisfied: pyrsistent>=0.14.0 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from jsonschema>=3.0.0->jams>=0.3->UNKNOWN==0.0.0) (0.18.0)
Requirement already satisfied: text-unidecode>=1.3 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from python-slugify>=3->UNKNOWN==0.0.0) (1.3)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from requests<=2.23.0->UNKNOWN==0.0.0) (1.25.11)
Requirement already satisfied: idna<3,>=2.5 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from requests<=2.23.0->UNKNOWN==0.0.0) (2.10)
Requirement already satisfied: chardet<4,>=3.0.2 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from requests<=2.23.0->UNKNOWN==0.0.0) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from requests<=2.23.0->UNKNOWN==0.0.0) (2020.12.5)
Requirement already satisfied: python-dateutil>=1.5 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from vertica_python>=0.9.1->UNKNOWN==0.0.0) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from pandas->jams>=0.3->UNKNOWN==0.0.0) (2021.1)
Installing collected packages: UNKNOWN
  Running setup.py develop for UNKNOWN
Successfully installed UNKNOWN-0.0.0
pip install -e . "numpy>1.18"  3.78s user 0.79s system 88% cpu 5.163 total
(mir39) ~/code/test_pip_resolution % time pip install -e . "numpy>1.18"
Obtaining file:///Users/stefansullivan/code/test_pip_resolution
Requirement already satisfied: numpy>1.18 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (1.21.2)
Collecting cysignals==1.6.5
  Using cached cysignals-1.6.5-cp39-cp39-macosx_10_15_x86_64.whl
Collecting configparser==3.5.0
  Using cached configparser-3.5.0.tar.gz (39 kB)
Collecting kafka==1.3.5
  Using cached kafka-1.3.5-py2.py3-none-any.whl (207 kB)
Requirement already satisfied: confluent-kafka[avro]==1.3.0 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.3.0)
Requirement already satisfied: fastavro in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.4.5)
Requirement already satisfied: avro-python3==1.9.2.1 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.9.2.1)
Collecting isort==4.3.21
  Using cached isort-4.3.21-py2.py3-none-any.whl (42 kB)
INFO: pip is looking at multiple versions of numpy to determine which version is compatible with other requirements. This could take a while.
Collecting numpy>1.18
  Using cached numpy-1.21.2-cp39-cp39-macosx_10_9_x86_64.whl (17.0 MB)
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
  Using cached numpy-1.21.1-cp39-cp39-macosx_10_9_x86_64.whl (17.0 MB)
  Using cached numpy-1.21.0-cp39-cp39-macosx_10_9_x86_64.whl (16.9 MB)
  Using cached numpy-1.20.3-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
  Using cached numpy-1.20.2-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
  Using cached numpy-1.20.1-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
  Using cached numpy-1.20.0-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
INFO: pip is looking at multiple versions of numpy to determine which version is compatible with other requirements. This could take a while.
  Using cached numpy-1.19.5-cp39-cp39-macosx_10_9_x86_64.whl (15.6 MB)
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
  Using cached numpy-1.19.4-cp39-cp39-macosx_10_9_x86_64.whl (15.4 MB)
  Using cached numpy-1.19.3-cp39-cp39-macosx_10_9_x86_64.whl (15.9 MB)
  Using cached numpy-1.19.2.zip (7.3 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Using cached numpy-1.19.1.zip (7.3 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Using cached numpy-1.19.0.zip (7.3 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Using cached numpy-1.18.5.zip (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Using cached numpy-1.18.4.zip (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Using cached numpy-1.18.3.zip (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Using cached numpy-1.18.2.zip (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Using cached numpy-1.18.1.zip (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
ERROR: Cannot install numpy>1.18 and unknown==0.0.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested numpy>1.18
    unknown 0.0.0 depends on numpy==1.17.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
pip install -e . "numpy>1.18"  341.89s user 91.06s system 93% cpu 7:44.40 total
stefansjs commented 3 years ago

I took a quick look (without debugging). It looks to me like the difference in behavior here https://github.com/pypa/pip/blob/21.2.4/src/pip/_internal/cli/req_command.py#L363 vs https://github.com/pypa/pip/blob/21.2.4/src/pip/_internal/cli/req_command.py#L387 which calls out to https://github.com/pypa/pip/blob/21.2.4/src/pip/_internal/req/constructors.py#L430 is notable. In one case pip does 


        for req in args:
            req_to_add = install_req_from_line(
                req,
                None,
                isolated=options.isolated_mode,
                use_pep517=options.use_pep517,
                user_supplied=True,
            )
            requirements.append(req_to_add)

        for req in options.editables:
            req_to_add = install_req_from_editable(
                req,
                user_supplied=True,
                isolated=options.isolated_mode,
                use_pep517=options.use_pep517,
            )
            requirements.append(req_to_add)

and in the other, it calls install_req_from_parsed_requirements (I presume line-by-line without reordering them), which itself does:

    if parsed_req.is_editable:
        req = install_req_from_editable(
            parsed_req.requirement,
            comes_from=parsed_req.comes_from,
            use_pep517=use_pep517,
            constraint=parsed_req.constraint,
            isolated=isolated,
            user_supplied=user_supplied,
        )

    else:
        req = install_req_from_line(
            parsed_req.requirement,
            comes_from=parsed_req.comes_from,
            use_pep517=use_pep517,
            isolated=isolated,
            options=parsed_req.options,
            constraint=parsed_req.constraint,
            line_source=parsed_req.line_source,
            user_supplied=user_supplied,
        )

IIUC, it looks like passing arguments on the command line orders non-editables before editables, whereas the requirements file is parsed line by line and added in the order in the file. That of course assumes that the downstream resolver depends on order and doesn't do any further reordering.

If I'm right, I see a few solutions

  1. one solution is to modify the requirements parser to put editables after non-editables. That seems like it's extremely likely to break somebody, but I don't know to what degree requirements.txt ordering is a guaranteed API. It's probably the simplest fix.
Index: src/pip/_internal/req/req_file.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/pip/_internal/req/req_file.py b/src/pip/_internal/req/req_file.py
--- a/src/pip/_internal/req/req_file.py (revision 0981e071bee8ac3ca2497813fa0863005033c6c2)
+++ b/src/pip/_internal/req/req_file.py (date 1633472570176)
@@ -328,6 +328,8 @@
     def _parse_and_recurse(
         self, filename: str, constraint: bool
     ) -> Iterator[ParsedLine]:
+        editable_requirements = []
+        
         for line in self._parse_file(filename, constraint):
             if not line.is_requirement and (
                 line.opts.requirements or line.opts.constraints
@@ -353,8 +355,12 @@
                     )

                 yield from self._parse_and_recurse(req_path, nested_constraint)
+            elif line.is_editable:
+                editable_requirements.append(line)
             else:
                 yield line
+                
+        yield from editable_requirements

     def _parse_file(self, filename: str, constraint: bool) -> Iterator[ParsedLine]:
         _, content = get_file_content(filename, self._session)

  1. Alternatively, it seems like either doing some more complex handling of arguments ordering together with options would be necessary. Given the use of argparse, this seems a little more involved but not insurmountable. You'd just have to find the order of args and -e opts in sys.argv and reorder the args + options.editables according to their command-line order.

  2. Finally, (still assuming that I'm right about the underlying bug), a third option would be to have the install command inspect the order of requirements and re-order editables according to whatever ordering is most correct.

What's the correct ordering of requirements? Is it by command-line order, or should editables always be last? Is it considered ok to have requirements.txt install differently than command-line invocations re: editable installs? Is it considered a guarantee that requirements.txt ordering implies a specific resolution ordering, and does that same guarantee apply to command-line invocations?

For my mental model, it seems like they should be the same. But I completely understand the engineering decision to not offer it as a guarantee. It also seems like this may have been a difference for years, but now that resolvers routinely take minutes and hours to find/fail to find installation candidates, suddenly the route to failure is relevant. Not sure if this is going to end up causing other differences in resolution.

uranusjr commented 3 years ago

Assuming your analysis is correct (I'm quite confident it is, but didn't actually check, and nobody should trust my memory on this kind of stuff 🙂), ordering requirements by command line order is the most sensible solution. There is no "correct" ordering because everything could be editable or not depending on the situation (conceptually editable-ness is completely orthogontal to dependency resolution), so we can only use an ordering that surprises the least people.

stefansjs commented 3 years ago

Also, I made a repository to make testing a little easier https://github.com/stefansjs/test_pip_install_differences.

pfmoore commented 3 years ago

I suspect that at least part of the problem here is that on the command line, -e file is an option rather than an argument. So it will be collected differently by our argument parsing code, and any "original" order of interspersed options and arguments might be hard to recover. Also, needing to retain that order may constrain us when we finally get around to moving to a newer, better, command line parsing library.

Unfortunately, the decision to use an option to signal an editable install is long-established, and would be very hard to change at this point.

(Please check before acting on the above, this is purely from memory and I'm not that familiar with our option parsing code).

uranusjr commented 3 years ago

FWIW that's my understanding from memory as well. The only reasonable-ish way would be to manually parse sys.argv and "reconstruct" ordering (I think this is what's hinted in https://github.com/pypa/pip/issues/10544#issuecomment-934984007?)

stefansjs commented 3 years ago

So I was curious about the same thing. Obviously doing a secondary custom parsing of sys.argv, when you have a module whose whole purpose is to parse sys.argv, is error prone. BUT, I think there may be some ways to use optparse (or future parsers) to do something desirable. The most direct of which is to have the -e flag append to the same requirements variable. There's at least a couple of ways to supply the extra information, but storing some extra state through a callback seems reasonably straight forward (to me)

Index: src/pip/_internal/cli/cmdoptions.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/pip/_internal/cli/cmdoptions.py b/src/pip/_internal/cli/cmdoptions.py
--- a/src/pip/_internal/cli/cmdoptions.py   (revision 0981e071bee8ac3ca2497813fa0863005033c6c2)
+++ b/src/pip/_internal/cli/cmdoptions.py   (date 1633544838206)
@@ -430,9 +430,9 @@
     return Option(
         "-e",
         "--editable",
-        dest="editables",
+        dest="requirements",
         action="append",
-        default=[],
+        callback=optparse_editable_callback,
         metavar="path/url",
         help=(
             "Install a project in editable mode (i.e. setuptools "
@@ -440,6 +440,12 @@
         ),
     )

+def optparse_editable_callback(option, opt, value, parser):
+    if not hasattr(parser.values, 'editables'):
+        setattr(parser.values, 'editables', set())
+
+    parser.values.editable.add(value)
+

 def _handle_src(option: Option, opt_str: str, value: str, parser: OptionParser) -> None:
     value = os.path.abspath(value)

Then my thought was to modify the loop that loops over requirements then editables to react on editables differently

Index: src/pip/_internal/cli/req_command.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/pip/_internal/cli/req_command.py b/src/pip/_internal/cli/req_command.py
--- a/src/pip/_internal/cli/req_command.py  (revision 0981e071bee8ac3ca2497813fa0863005033c6c2)
+++ b/src/pip/_internal/cli/req_command.py  (date 1633485496843)
@@ -382,15 +382,7 @@
                 isolated=options.isolated_mode,
                 use_pep517=options.use_pep517,
                 user_supplied=True,
-            )
-            requirements.append(req_to_add)
-
-        for req in options.editables:
-            req_to_add = install_req_from_editable(
-                req,
-                user_supplied=True,
-                isolated=options.isolated_mode,
-                use_pep517=options.use_pep517,
+                editable=req in options.editables,
             )
             requirements.append(req_to_add)

If it's looking like a reasonable approach, I can start to put together a PR. The code changes themselves will be quick to write and share; we could discuss the approaches and any alternatives. Making the thing unit tested and ready for merge would take a little longer.

pfmoore commented 3 years ago

Honestly, I'm -1 on reconstructing the command line on general principle. It constrains how we develop our command line interface in future. Any of the other options would be better IMO, if we decided we wanted to do anything here.

stefansjs commented 3 years ago

Right, I'm not sure if we're in agreement or not. I think I agree that reconstructing the command-line from parsed options/arguments is a Bad Idea ™️. Anything that involves looking at sys.argv smells like a misuse of a parsing library to me.

Are you saying you disagree with the dest='requirements' implementation I proposed? Or were you just clarifying that we shouldn't be tempted by the "figure out the original order for ourselves" rabbit hole?

pfmoore commented 3 years ago

Personally, I feel that any attempt to preserve the order of normal requirements and -e is problematic. The logic of "options and positional arguments are independent" is common, and I'm concerned that not following that will be confusing for users. Furthermore, we have a long-outstanding intention to switch our option parsing from optparse to "something more modern" (probably click) and when we do that, I'd rather not have non-standard semantics that could potentially be difficult to port to a new library.

But I'm happy to hear what the other pip developers say here - I don't have a strong opinion, so if someone else does, I'll defer to them.

uranusjr commented 3 years ago

Relying on callbacks has an additional issue of relying on the argument parsing library calling them sequentially. I don't think either optparse or argparse guarantees that, and it's not unimaginable for either of them to implement some sort of optimisation that breaks this. So assuming we don't do manual sys.argv parsing (it seems like the case), this is probably not possible unless we switch to a parsing library that natively guarantees to preserve argument ordering.

stefansjs commented 3 years ago

Ok, it really sounds like order of command-line arguments shouldn't be relied upon, and requirements.txt should be considered more reliable. I'm going to close this since it seems the behavior does differ in a way that's expected-ish. Does anybody disagree?