Python 3.7 pip does not support multiple indices

[RossBoylan]

This problem is likely to be an issue only for package developers. And it has a ready solution: upgrade pip.

Background

Before general release, packages are uploaded to test.pypi.org. The archive must be specified explicitly during a pip install, but the naive approach fails because it will look for all required packages there, and some are not present. To get the installation to work, one must specify the regular pypi as an alternate.

Clearly not an issue for a general users who install from pypi.

Problem

The version of pip with Python 3.7 is quite old and doesn't handle the 2 repository case properly. At least that's the case on Debian buster (=Debian 10 = old-stable). The title of this issue reflects my guess this is a general Python 3.7 issue. Also, it might be specific to virtual environments.

ross@barley:~/UCSF/Kornak$ python3 -m venv testEnv
ross@barley:~/UCSF/Kornak$ source testEnv/bin/activate
(testEnv) ross@barley:~/UCSF/Kornak$ python --version
Python 3.7.3
(testEnv) ross@barley:~/UCSF/Kornak$ date; time python -m pip install  --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ bifs
Tue 21 Sep 2021 09:07:26 PM PDT
Looking in indexes: https://test.pypi.org/simple/, https://pypi.org/simple/
Collecting bifs
Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://pypi.org/simple/bifs/

real    0m2.435s
user    0m0.677s
sys 0m0.024s

(testEnv) ross@barley:~/UCSF/Kornak$ date; time python -m pip install  --index-url https://test.pypi.org/simple --extra-index-url https://pypi.org/simple bifs  # no trailing / on URLs
Tue 21 Sep 2021 09:08:40 PM PDT
Looking in indexes: https://test.pypi.org/simple, https://pypi.org/simple
Collecting bifs
Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://pypi.org/simple/bifs/

real    0m1.501s
user    0m0.666s
sys 0m0.044s
# next is an attempt to pre-cache bifs
# it also illustrates why naive use of test.pypi as only "index" doesn't work
(testEnv) ross@barley:~/UCSF/Kornak$ date; time python -m pip install  --index-url https://test.pypi.org/simple/  bifs
Tue 21 Sep 2021 09:12:27 PM PDT
Looking in indexes: https://test.pypi.org/simple/
Collecting bifs
  Downloading https://test-files.pythonhosted.org/packages/90/cf/0d2148966c28ca066ce34d7f96d9200acc2a19958e93ec5944acdab9c4e9/bifs-0.9.1-py3-none-any.whl
Collecting nibabel (from bifs)
Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://test.pypi.org/simple/nibabel/

real    0m1.606s
user    0m0.667s
sys 0m0.064s
# try original command, hoping it will grab bifs from the cache.
# it doesn't.
(testEnv) ross@barley:~/UCSF/Kornak$ date; time python -m pip install  --index-url https://test.pypi.org/simple --extra-index-url https://pypi.org/simple bifs
Tue 21 Sep 2021 09:12:46 PM PDT
Looking in indexes: https://test.pypi.org/simple, https://pypi.org/simple
Collecting bifs
Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://pypi.org/simple/bifs/

real    0m2.096s
user    0m0.630s
sys 0m0.072s

Solution

Upgrade pip.

(testEnv) ross@barley:~/UCSF/Kornak$ python -m pip install --upgrade pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/ca/31/b88ef447d595963c01060998cb329251648acf4a067721b0452c45527eb8/pip-21.2.4-py3-none-any.whl (1.6MB)
    100% |████████████████████████████████| 1.6MB 514kB/s 
Installing collected packages: pip
  Found existing installation: pip 18.1
    Uninstalling pip-18.1:
      Successfully uninstalled pip-18.1
Successfully installed pip-21.2.4

At this point the pip install with 2 repositories works.