Pip Donwloads Different Wheels from the same Package [pytorch nightly]

[FabianSchuetze]

Description

Thanks for the work on this repo.

I am trying to install the nightlies of pytorch, torchvision, and torchtext with pip. Pip downloads several packages with slightly different versions. This continues until the space on the mounted disk is exhausted.

The output is the follwoing (before I aboreted the process):

#10 [6/7] RUN pip3 install --pre torch torchvision torchtext --index-url https://download.pytorch.org/whl/nightly/cu121 --upgrade

And

#10 1.028 Looking in indexes: https://download.pytorch.org/whl/nightly/cu121
#10 1.635 Collecting torch
#10 1.932   Downloading https://download.pytorch.org/whl/nightly/cu121/torch-2.3.0.dev20240217%2Bcu121-cp310-cp310-linux_x86_64.whl (770.8 MB)
#10 31.41      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 770.8/770.8 MB 854.7 kB/s eta 0:00:00
#10 31.98 Collecting torchvision
#10 32.00   Downloading https://download.pytorch.org/whl/nightly/cu121/torchvision-0.18.0.dev20240216%2Bcu121-cp310-cp310-linux_x86_64.whl (7.1 MB)
#10 32.08      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.1/7.1 MB 91.8 MB/s eta 0:00:00
#10 32.35 Collecting torchtext
#10 32.47   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20240115%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 32.65      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 11.6 MB/s eta 0:00:00
.... [SOME OTHER PACKAGES...]
#10 87.28 Collecting torch
#10 87.31   Downloading https://download.pytorch.org/whl/nightly/cu121/torch-2.3.0.dev20240216%2Bcu121-cp310-cp310-linux_x86_64.whl (770.7 MB)
#10 107.0      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 770.7/770.7 MB 763.5 kB/s eta 0:00:00
#10 107.3 Collecting requests
#10 107.3   Downloading https://download.pytorch.org/whl/nightly/requests-2.28.1-py3-none-any.whl (62 kB)
#10 107.3      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 KB 10.0 MB/s eta 0:00:00
#10 107.3 Collecting torchtext
#10 107.6   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20240114%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 107.8      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 11.5 MB/s eta 0:00:00
#10 108.0   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20240113%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 108.2      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 11.5 MB/s eta 0:00:00
#10 108.4   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20240110%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 108.6      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 12.8 MB/s eta 0:00:00
#10 108.9   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20240109%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 109.1      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 11.5 MB/s eta 0:00:00
#10 109.3   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20240108%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 109.5      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 12.5 MB/s eta 0:00:00
#10 109.7 Collecting tqdm
#10 109.9   Downloading https://download.pytorch.org/whl/nightly/tqdm-4.64.1-py2.py3-none-any.whl (78 kB)
#10 110.3      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.5/78.5 KB 165.4 kB/s eta 0:00:00
#10 110.3 Collecting torchtext
#10 110.6   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20240107%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 111.4      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 2.4 MB/s eta 0:00:00
#10 111.6   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20240106%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 111.8      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 13.3 MB/s eta 0:00:00
#10 112.0   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20240105%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 112.2      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 12.7 MB/s eta 0:00:00
#10 112.5 Collecting torchdata==0.7.1.dev20240105
#10 112.8   Downloading https://download.pytorch.org/whl/nightly/torchdata-0.7.1.dev20240105-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB)
#10 113.0      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 21.4 MB/s eta 0:00:00
#10 113.0 Collecting torchtext
#10 113.3   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20240104%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 113.4      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 11.8 MB/s eta 0:00:00
#10 113.5 Collecting torchdata==0.7.1.dev20240104
#10 113.8   Downloading https://download.pytorch.org/whl/nightly/torchdata-0.7.1.dev20240104-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB)
#10 114.0      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 21.6 MB/s eta 0:00:00
#10 114.0 Collecting torchtext
#10 114.3   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20240103%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 114.5      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 11.4 MB/s eta 0:00:00
#10 114.7   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20240102%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 114.9      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 13.3 MB/s eta 0:00:00
#10 114.9 Collecting torchdata==0.7.1.dev20240102
#10 115.2   Downloading https://download.pytorch.org/whl/nightly/torchdata-0.7.1.dev20240102-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB)
#10 115.4      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 24.6 MB/s eta 0:00:00
#10 115.4 Collecting torchtext
#10 115.6   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20240101%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 115.8      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 11.7 MB/s eta 0:00:00
#10 115.8 Collecting torchdata==0.7.1.dev20240101
#10 116.1   Downloading https://download.pytorch.org/whl/nightly/torchdata-0.7.1.dev20240101-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB)
#10 116.3      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 24.5 MB/s eta 0:00:00
#10 116.3 Collecting torchtext
#10 116.6   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20231231%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 116.7      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 12.1 MB/s eta 0:00:00
#10 117.0   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20231230%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 117.2      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 13.8 MB/s eta 0:00:00
#10 117.4   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20231229%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 117.6      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 12.9 MB/s eta 0:00:00
#10 117.6 Collecting torchdata==0.7.1.dev20231229
#10 118.0   Downloading https://download.pytorch.org/whl/nightly/torchdata-0.7.1.dev20231229-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB)
#10 118.2      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 21.9 MB/s eta 0:00:00
#10 118.2 Collecting torchtext
#10 118.4   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20231228%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 118.6      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 11.2 MB/s eta 0:00:00
#10 118.6 Collecting torchdata==0.7.1.dev20231228
#10 118.8   Downloading https://download.pytorch.org/whl/nightly/torchdata-0.7.1.dev20231228-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB)
#10 119.0      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 21.4 MB/s eta 0:00:00
#10 119.0 Collecting torchtext
#10 119.3   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20231227%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 120.0      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 2.8 MB/s eta 0:00:00
#10 120.0 Collecting torchdata==0.7.1.dev20231227
#10 120.2   Downloading https://download.pytorch.org/whl/nightly/torchdata-0.7.1.dev20231227-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB)
#10 120.4      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 22.0 MB/s eta 0:00:00
#10 120.5 Collecting torchtext
#10 120.7   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20231226%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 120.8      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 11.8 MB/s eta 0:00:00
#10 120.9 Collecting torchdata==0.7.1.dev20231226
#10 121.0   Downloading https://download.pytorch.org/whl/nightly/torchdata-0.7.1.dev20231226-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB)
#10 121.2      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 39.4 MB/s eta 0:00:00
#10 121.2 Collecting torchtext
#10 121.3   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20231225%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 121.5      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 11.6 MB/s eta 0:00:00
#10 121.5 Collecting torchdata==0.7.1.dev20231225
#10 121.6   Downloading https://download.pytorch.org/whl/nightly/torchdata-0.7.1.dev20231225-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB)
#10 121.7      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 47.1 MB/s eta 0:00:00
#10 121.7 Collecting torchtext
#10 122.0   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20231224%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 122.1      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 11.8 MB/s eta 0:00:00
#10 122.2 Collecting torchdata==0.7.1.dev20231223
#10 122.5   Downloading https://download.pytorch.org/whl/nightly/torchdata-0.7.1.dev20231223-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB)
#10 122.7      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 22.4 MB/s eta 0:00:00
#10 122.7 Collecting torchtext
#10 122.9   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20231223%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 123.1      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 13.7 MB/s eta 0:00:00
#10 123.3   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20231222%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 123.5      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 12.3 MB/s eta 0:00:00
#10 123.7   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20231221%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 123.9      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 12.1 MB/s eta 0:00:00
#10 124.2   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20231220%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 124.3      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 11.8 MB/s eta 0:00:00
#10 124.5   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20231219%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 125.5      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 2.2 MB/s eta 0:00:00
#10 125.7   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20231218%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 125.9      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 11.3 MB/s eta 0:00:00
#10 126.1   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20231217%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 126.1      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 51.6 MB/s eta 0:00:00
#10 126.2   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20231216%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 126.2      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 74.3 MB/s eta 0:00:00
#10 126.2 Collecting torchdata==0.7.1.dev20231216
#10 126.5   Downloading https://download.pytorch.org/whl/nightly/torchdata-0.7.1.dev20231216-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB)
#10 126.7      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 22.4 MB/s eta 0:00:00
#10 126.7 Collecting torchtext
#10 127.0   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.17.0.dev20231215%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 127.1      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 12.3 MB/s eta 0:00:00
#10 127.4   Downloading https://download.pytorch.org/whl/nightly/torchtext-0.16.0.dev20231010%2Bcpu-cp310-cp310-linux_x86_64.whl (2.0 MB)
#10 127.9      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 4.1 MB/s eta 0:00:00
#10 127.9 INFO: pip is looking at multiple versions of torch to determine which version is compatible with other requirements. This could take a while.
#10 127.9 INFO: pip is looking at multiple versions of torchvision to determine which version is compatible with other requirements. This could take a while.
#10 127.9 Collecting torchvision
#10 128.2   Downloading https://download.pytorch.org/whl/nightly/cu121/torchvision-0.18.0.dev20240215%2Bcu121-cp310-cp310-linux_x86_64.whl (7.1 MB)
#10 128.4      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.1/7.1 MB 32.5 MB/s eta 0:00:00
#10 128.4 Collecting torch
#10 128.7   Downloading https://download.pytorch.org/whl/nightly/cu121/torch-2.3.0.dev20240215%2Bcu121-cp310-cp310-linux_x86_64.whl (770.8 MB)
#10 159.6      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 770.8/770.8 MB 896.2 kB/s eta 0:00:00
#10 159.9 Collecting torchvision
#10 160.2   Downloading https://download.pytorch.org/whl/nightly/cu121/torchvision-0.18.0.dev20240214%2Bcu121-cp310-cp310-linux_x86_64.whl (7.1 MB)
#10 160.4      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.1/7.1 MB 31.6 MB/s eta 0:00:00
#10 160.4 Collecting torch
#10 160.7   Downloading https://download.pytorch.org/whl/nightly/cu121/torch-2.3.0.dev20240214%2Bcu121-cp310-cp310-linux_x86_64.whl (770.4 MB)

I think this issue is related to #11840 .

Expected behavior

Download only one wheel of each package.

pip version

24.0

Python version

3.10.12

OS

ubuntu 22.04

How to Reproduce

pip3 install --pre torch torchvision torchtext --index-url https://download.pytorch.org/whl/nightly/cu121 --upgrade

Output

No response

Code of Conduct

• [X] I agree to follow the PSF Code of Conduct.

[pfmoore]

Pip is downloading different versions trying to find one that satisfies the various constraints the packages you're installing put on what versions will work. This is normal behaviour of the resolver. As noted in the linked issue, if the pytorch index implemented PEP 658 that might help reduce the amount of data downloaded, but that's out of pip's control (and quite likely out of your control as well).

You'll either need to determine and install the exact versions you need (maybe using pip install --dry-run --report on a machine with enough disk space to determine what those versions are) or provide additional temporary space for pip to do the resolution.

[FabianSchuetze]

Thanks for the comment, I will raise the issue in the pytorch repo then and close it here.