facebookresearch / pytorch3d

PyTorch3D is FAIR's library of reusable components for deep learning with 3D data
https://pytorch3d.org/
Other
8.66k stars 1.3k forks source link

Conflicts during conda installation with Python 3.8, cudatoolkit=10.2, Ubuntu 18.04 (minimum working examples provided) #1538

Closed DanielTakeshi closed 1 year ago

DanielTakeshi commented 1 year ago

The Problem

I am running into problems with installing PyTorch3D. I am looking at the instructions:

https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md

Due to my existing code restrictions, I have to use Python 3.8 but that should be supported. In addition, I am using cudatoolkit=10.2 for my pytorch installations which is again a function of the machines that I am using. I am trying different pytorch versions, downgrading to 1.10 (as shown in my minimum working examples). Unfortunately when I install pytorch3d I am running into the dreaded "Found conflicts" error. This has been reported before:

See below in my Minimum Working Examples for an example output of this error.

Minimum Working Examples

Here are a set of minimum working examples. They are modeled after the one in this post.

To clarify, I tried each of these MWEs one by one in sequence. They all resulted in the same error at the end. After I tried each attempt, I removed the conda env with conda env remove -n seuss and then did the next attempt. All are done on an Ubuntu 18.04 machine.

Attempt 1

conda create --name seuss python=3.8 -y && conda activate seuss
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=10.2 -c pytorch -y
conda install -c fvcore -c iopath -c conda-forge fvcore iopath -y
conda install pytorch3d -c pytorch3d -y

Attempt 2 (downgrade torch to 1.12):

conda create --name seuss python=3.8 -y && conda activate seuss
conda install pytorch==1.12.0 torchvision==0.13.0 torchaudio==0.12.0 cudatoolkit=10.2 -c pytorch -y
conda install -c fvcore -c iopath -c conda-forge fvcore iopath -y
conda install pytorch3d -c pytorch3d -y

Attempt 3 (downgrade torch to 1.11):

conda create --name seuss python=3.8 -y && conda activate seuss
conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=10.2 -c pytorch -y
conda install -c fvcore -c iopath -c conda-forge fvcore iopath -y
conda install pytorch3d -c pytorch3d -y

Attempt 4 (downgrade torch to 1.10):

conda create --name seuss python=3.8 -y && conda activate seuss
conda install pytorch==1.10.0 torchvision==0.11.0 torchaudio==0.10.0 cudatoolkit=10.2 -c pytorch -y
conda install -c fvcore -c iopath -c conda-forge fvcore iopath -y
conda install pytorch3d -c pytorch3d -y

Here is an example output from the 4th attempt with the error message: https://pastebin.com/raw/DK6deHam (it was too long to paste into GitHub).

Other Information

After the 4th attempt above, I tried doing conda list and checking to see if I can use torch:

(seuss) seita@marvin:~ $ conda list
# packages in environment at /home/seita/miniconda3/envs/seuss:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
blas                      1.0                         mkl  
bzip2                     1.0.8                h7b6447c_0  
ca-certificates           2023.5.7             hbcca054_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
cudatoolkit               10.2.89              hfd86e86_1  
ffmpeg                    4.3                  hf484d3e_0    pytorch
freetype                  2.12.1               h4a9f257_0  
fvcore                    0.1.5.post20210915            py38    fvcore
giflib                    5.2.1                h5eee18b_3  
gmp                       6.2.1                h295c915_3  
gnutls                    3.6.15               he1e5248_0  
intel-openmp              2023.1.0         hdb19cb5_46305  
iopath                    0.1.9                      py38    iopath
jpeg                      9e                   h5eee18b_1  
lame                      3.100                h7b6447c_0  
lcms2                     2.12                 h3be6417_0  
ld_impl_linux-64          2.38                 h1181459_1  
lerc                      3.0                  h295c915_0  
libdeflate                1.17                 h5eee18b_0  
libffi                    3.4.4                h6a678d5_0  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libiconv                  1.16                 h7f8727e_2  
libidn2                   2.3.2                h7f8727e_0  
libpng                    1.6.39               h5eee18b_0  
libstdcxx-ng              11.2.0               h1234567_1  
libtasn1                  4.19.0               h5eee18b_0  
libtiff                   4.5.0                h6a678d5_2  
libunistring              0.9.10               h27cfd23_0  
libuv                     1.44.2               h5eee18b_0  
libwebp                   1.2.4                h11a3e52_1  
libwebp-base              1.2.4                h5eee18b_1  
lz4-c                     1.9.4                h6a678d5_0  
mkl                       2023.1.0         h6d00ec8_46342  
mkl-service               2.4.0            py38h5eee18b_1  
mkl_fft                   1.3.6            py38h417a72b_1  
mkl_random                1.2.2            py38h417a72b_1  
ncurses                   6.4                  h6a678d5_0  
nettle                    3.7.3                hbbd107a_1  
numpy                     1.24.3           py38hf6e8229_1  
numpy-base                1.24.3           py38h060ed82_1  
openh264                  2.1.1                h4ff587b_0  
openssl                   1.1.1t               h7f8727e_0  
pillow                    9.4.0            py38h6a678d5_0  
pip                       23.0.1           py38h06a4308_0  
portalocker               1.4.0                      py_0    conda-forge
python                    3.8.16               h7a1cb2a_3  
pytorch                   1.10.0          py3.8_cuda10.2_cudnn7.6.5_0    pytorch
pytorch-mutex             1.0                        cuda    pytorch
pyyaml                    5.1.2            py38h516909a_1    conda-forge
readline                  8.2                  h5eee18b_0  
setuptools                66.0.0           py38h06a4308_0  
sqlite                    3.41.2               h5eee18b_0  
tabulate                  0.9.0              pyhd8ed1ab_1    conda-forge
tbb                       2021.8.0             hdb19cb5_0  
termcolor                 2.3.0              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h1ccaba5_0  
torchaudio                0.10.0               py38_cu102    pytorch
torchvision               0.11.0               py38_cu102    pytorch
tqdm                      4.65.0             pyhd8ed1ab_1    conda-forge
typing_extensions         4.5.0            py38h06a4308_0  
wheel                     0.38.4           py38h06a4308_0  
xz                        5.4.2                h5eee18b_0  
yacs                      0.1.8              pyhd8ed1ab_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zlib                      1.2.13               h5eee18b_0  
zstd                      1.5.5                hc292b87_0  
(seuss) seita@marvin:~ $ python
Python 3.8.16 (default, Mar  2 2023, 03:21:46) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.10.0'
>>> exit()
(seuss) seita@marvin:~ $ 

I do not notice anything wrong here.

Here is conda info:

(seuss) seita@marvin:~ $ conda info

     active environment : seuss
    active env location : /home/seita/miniconda3/envs/seuss
            shell level : 1
       user config file : /home/seita/.condarc
 populated config files : /home/seita/.condarc
          conda version : 4.13.0
    conda-build version : not installed
         python version : 3.9.5.final.0
       virtual packages : __cuda=12.0=0
                          __linux=4.15.0=0
                          __glibc=2.27=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /home/seita/miniconda3  (writable)
      conda av data dir : /home/seita/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/seita/miniconda3/pkgs
                          /home/seita/.conda/pkgs
       envs directories : /home/seita/miniconda3/envs
                          /home/seita/.conda/envs
               platform : linux-64
             user-agent : conda/4.13.0 requests/2.29.0 CPython/3.9.5 Linux/4.15.0-208-generic ubuntu/18.04.6 glibc/2.27
                UID:GID : 1002:1002
             netrc file : /home/seita/.netrc
           offline mode : False

(seuss) seita@marvin:~ $ 

Thus, I am unsure what to do now other than to keep trying different combinations. Is there a compatibility matrix of versions of python, torch, cudatoolkit, and pytorch3d, ideally ones that work with Python 3.8?

For Reference, Python 3.7

Using Python 3.7, and an older pytorch3d version, this will NOT run into an installation error:

conda create --name seuss python=3.7 -y && conda activate seuss
conda install pytorch==1.9.0 torchvision==0.10.0 torchaudio==0.9.0 cudatoolkit=10.2 -c pytorch -y
conda install -c fvcore -c iopath -c conda-forge fvcore iopath -y
conda install -c pytorch3d 'pytorch3d<=0.7.1' -y

I am wondering if there is something related to recent versions of pytorch3d that might result in incompatibilities?

bottler commented 1 year ago

I don't understand what is wrong with your attempt #1. Here's something similar which works for me today

conda create -n seuss python=3.8 -y
conda activate seuss
conda install pytorch==1.12.1 torchvision cudatoolkit=10.2 -c pytorch -c conda-forge -y
conda install -c fvcore -c iopath -c conda-forge fvcore iopath -y
conda install pytorch3d=0.7.4 -c pytorch3d -y

The resulting conda list is as follows, which may help you ask for more specific versions.

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
_openmp_mutex             5.1                       1_gnu
blas                      1.0                         mkl
brotlipy                  0.7.0           py38h0a891b7_1004    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2023.5.7             hbcca054_0    conda-forge
certifi                   2023.5.7         py38h06a4308_0
cffi                      1.15.0           py38h3931269_0    conda-forge
charset-normalizer        3.1.0              pyhd8ed1ab_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
cryptography              3.4.8            py38h3e25421_1    conda-forge
cudatoolkit               10.2.89             h713d32c_10    conda-forge
ffmpeg                    4.3                  hf484d3e_0    pytorch
freetype                  2.10.4               h0708190_1    conda-forge
fvcore                    0.1.5.post20210915            py38    fvcore
giflib                    5.2.1                h36c2ea0_2    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
gnutls                    3.6.13               h85f3911_1    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
intel-openmp              2021.4.0          h06a4308_3561
iopath                    0.1.9                      py38    iopath
jpeg                      9e                   h166bdaf_1    conda-forge
lame                      3.100             h7f98852_1001    conda-forge
lcms2                     2.12                 h3be6417_0
ld_impl_linux-64          2.38                 h1181459_1
lerc                      3.0                  h295c915_0
libdeflate                1.17                 h5eee18b_0
libffi                    3.4.4                h6a678d5_0
libgcc-ng                 11.2.0               h1234567_1
libgomp                   11.2.0               h1234567_1
libiconv                  1.17                 h166bdaf_0    conda-forge
libpng                    1.6.39               h5eee18b_0
libstdcxx-ng              11.2.0               h1234567_1
libtiff                   4.5.0                h6a678d5_2
libwebp                   1.2.4                h11a3e52_1
libwebp-base              1.2.4                h5eee18b_1
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
mkl                       2021.4.0           h06a4308_640
mkl-service               2.4.0            py38h95df7f1_0    conda-forge
mkl_fft                   1.3.1            py38hd3c417c_0
mkl_random                1.2.2            py38h1abd341_0    conda-forge
ncurses                   6.4                  h6a678d5_0
nettle                    3.6                  he412f7d_0    conda-forge
numpy                     1.24.3           py38h14f4228_0
numpy-base                1.24.3           py38h31eccc5_0
openh264                  2.1.1                h4ff587b_0
openssl                   1.1.1t               h7f8727e_0
pillow                    9.4.0            py38h6a678d5_0
pip                       23.0.1           py38h06a4308_0
portalocker               2.7.0            py38h578d9bd_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyopenssl                 20.0.1             pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.8.16               h7a1cb2a_3
python_abi                3.8                      2_cp38    conda-forge
pytorch                   1.12.1          py3.8_cuda10.2_cudnn7.6.5_0    pytorch
pytorch-mutex             1.0                        cuda    pytorch
pytorch3d                 0.7.4           py38_cu102_pyt1121    pytorch3d
pyyaml                    6.0              py38h0a891b7_4    conda-forge
readline                  8.2                  h5eee18b_0
requests                  2.29.0             pyhd8ed1ab_0    conda-forge
setuptools                66.0.0           py38h06a4308_0
six                       1.16.0             pyh6c4a22f_0    conda-forge
sqlite                    3.41.2               h5eee18b_0
tabulate                  0.9.0              pyhd8ed1ab_1    conda-forge
termcolor                 2.3.0              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h1ccaba5_0
torchvision               0.13.1               py38_cu102    pytorch
tqdm                      4.65.0             pyhd8ed1ab_1    conda-forge
typing_extensions         4.5.0              pyha770c72_0    conda-forge
urllib3                   1.26.15            pyhd8ed1ab_0    conda-forge
wheel                     0.38.4           py38h06a4308_0
xz                        5.4.2                h5eee18b_0
yacs                      0.1.8              pyhd8ed1ab_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zlib                      1.2.13               h5eee18b_0
zstd                      1.5.2                ha4553b6_0
DanielTakeshi commented 1 year ago

Thanks @bottler and I apologize for my delayed response in following up on this. I investigated more and found some interesting things about installing (pytorch 1.12.1 seems OK but 2.0.1 seems to be giving some errors):

First Part (Reconcile Differences in Our Installation, PyTorch 1.12.1)

I tried this again which is my first attempt in my original post:

conda create --name seuss python=3.8 -y && conda activate seuss
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=10.2 -c pytorch -y
conda install -c fvcore -c iopath -c conda-forge fvcore iopath -y
conda install pytorch3d -c pytorch3d -y

Unfortunately it still did not work on the Ubuntu 18.04 machine I'm testing.

The conda list right after I had killed the last command. ``` (seuss) seita@starship:~ $ conda list # packages in environment at /home/seita/miniconda3/envs/seuss: # # Name Version Build Channel _libgcc_mutex 0.1 main _openmp_mutex 5.1 1_gnu blas 1.0 mkl brotlipy 0.7.0 py38h27cfd23_1003 bzip2 1.0.8 h7b6447c_0 ca-certificates 2023.7.22 hbcca054_0 conda-forge certifi 2023.7.22 pyhd8ed1ab_0 conda-forge cffi 1.15.1 py38h5eee18b_3 charset-normalizer 2.0.4 pyhd3eb1b0_0 colorama 0.4.6 pyhd8ed1ab_0 conda-forge cryptography 41.0.2 py38h22a60cf_0 cudatoolkit 10.2.89 hfd86e86_1 ffmpeg 4.3 hf484d3e_0 pytorch freetype 2.12.1 h4a9f257_0 fvcore 0.1.5.post20210915 py38 fvcore giflib 5.2.1 h5eee18b_3 gmp 6.2.1 h295c915_3 gnutls 3.6.15 he1e5248_0 idna 3.4 py38h06a4308_0 intel-openmp 2023.1.0 hdb19cb5_46305 iopath 0.1.9 py38 iopath jpeg 9e h5eee18b_1 lame 3.100 h7b6447c_0 lcms2 2.12 h3be6417_0 ld_impl_linux-64 2.38 h1181459_1 lerc 3.0 h295c915_0 libdeflate 1.17 h5eee18b_0 libffi 3.4.4 h6a678d5_0 libgcc-ng 11.2.0 h1234567_1 libgomp 11.2.0 h1234567_1 libiconv 1.16 h7f8727e_2 libidn2 2.3.4 h5eee18b_0 libpng 1.6.39 h5eee18b_0 libstdcxx-ng 11.2.0 h1234567_1 libtasn1 4.19.0 h5eee18b_0 libtiff 4.5.0 h6a678d5_2 libunistring 0.9.10 h27cfd23_0 libwebp 1.2.4 h11a3e52_1 libwebp-base 1.2.4 h5eee18b_1 lz4-c 1.9.4 h6a678d5_0 mkl 2023.1.0 h213fc3f_46343 mkl-service 2.4.0 py38h5eee18b_1 mkl_fft 1.3.6 py38h417a72b_1 mkl_random 1.2.2 py38h417a72b_1 ncurses 6.4 h6a678d5_0 nettle 3.7.3 hbbd107a_1 numpy 1.24.3 py38hf6e8229_1 numpy-base 1.24.3 py38h060ed82_1 openh264 2.1.1 h4ff587b_0 openssl 3.0.10 h7f8727e_0 pillow 9.4.0 py38h6a678d5_0 pip 23.2.1 py38h06a4308_0 portalocker 1.4.0 py_0 conda-forge pycparser 2.21 pyhd3eb1b0_0 pyopenssl 23.2.0 py38h06a4308_0 pysocks 1.7.1 py38h06a4308_0 python 3.8.17 h955ad1f_0 pytorch 1.12.1 py3.8_cuda10.2_cudnn7.6.5_0 pytorch pytorch-mutex 1.0 cuda pytorch pyyaml 5.1.2 py38h516909a_1 conda-forge readline 8.2 h5eee18b_0 requests 2.31.0 py38h06a4308_0 setuptools 68.0.0 py38h06a4308_0 sqlite 3.41.2 h5eee18b_0 tabulate 0.9.0 pyhd8ed1ab_1 conda-forge tbb 2021.8.0 hdb19cb5_0 termcolor 2.3.0 pyhd8ed1ab_0 conda-forge tk 8.6.12 h1ccaba5_0 torchaudio 0.12.1 py38_cu102 pytorch torchvision 0.13.1 py38_cu102 pytorch tqdm 4.66.0 pyhd8ed1ab_0 conda-forge typing_extensions 4.7.1 py38h06a4308_0 urllib3 1.26.16 py38h06a4308_0 wheel 0.38.4 py38h06a4308_0 xz 5.4.2 h5eee18b_0 yacs 0.1.8 pyhd8ed1ab_0 conda-forge yaml 0.2.5 h7f98852_2 conda-forge zlib 1.2.13 h5eee18b_0 zstd 1.5.5 hc292b87_0 (seuss) seita@starship:~ $ ```

But when I delete the seuss env and then try your commands:

conda create -n seuss python=3.8 -y && conda activate seuss
conda install pytorch==1.12.1 torchvision cudatoolkit=10.2 -c pytorch -c conda-forge -y
conda install -c fvcore -c iopath -c conda-forge fvcore iopath -y
conda install pytorch3d=0.7.4 -c pytorch3d -y

Suddenly the last command will finish correctly and not complain about conflicts. Here's the paste bin which shows my sequence of commands: https://pastebin.com/raw/dYYgXNDR This starts from my removal of the old conda environment that failed and then I ran your commands.

The only differences are:

Would any of these cause issues that you are aware of?

Second Part (Try PyTorch 2.0.1)

But will this also work for the newer version of PyTorch? The installation said that it supports PyTorch 2.0.1. I try the sequence of commands:

conda create -n seuss python=3.8 -y && conda activate seuss
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
conda install -c fvcore -c iopath -c conda-forge fvcore iopath -y
conda install pytorch3d=0.7.4 -c pytorch3d -y

I am running into package conflicts. When you run the last installation, do you run into the same issue?

The actual error I get (fortunately it's not too long): ``` (seuss) seita@starship:~ $ conda install pytorch3d=0.7.4 -c pytorch3d -y Collecting package metadata (current_repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source. Collecting package metadata (repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Solving environment: \ Found conflicts! Looking for incompatible packages. This can take several minutes. Press CTRL-C to abort. failed UnsatisfiableError: The following specifications were found to be incompatible with the existing python installation in your environment: Specifications: - pytorch3d=0.7.4 -> python[version='>=2.7,<2.8.0a0|>=3.11,<3.12.0a0|>=3.7,<3.8.0a0|>=3.6,<3.7.0a0|>=3.5,<3.6.0a0'] Your python: python=3.8 If python is on the left-most side of the chain, that's the version you've asked for. When python appears to the right, that indicates that the thing on the left is somehow not available for the python version you are constrained to. Note that conda will not change your python version to a different minor version unless you explicitly specify that. The following specifications were found to be incompatible with your system: - feature:/linux-64::__glibc==2.27=0 - feature:|@/linux-64::__glibc==2.27=0 - pytorch3d=0.7.4 -> torchvision[version='>=0.5'] -> __glibc[version='>=2.17,<3.0.a0'] Your installed version is: 2.27 (seuss) seita@starship:~ $ ```

Also tried other PyTorch versions and it is giving similar errors.

PyTorch 2.0:

conda create -n seuss python=3.8 -y && conda activate seuss
conda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.7 -c pytorch -c nvidia -y
conda install -c fvcore -c iopath -c conda-forge fvcore iopath -y
conda install pytorch3d=0.7.4 -c pytorch3d -y

PyTorch 1.13:

conda create -n seuss python=3.8 -y && conda activate seuss
conda install pytorch=1.13.0 torchvision pytorch-cuda=11.6 -c pytorch -c nvidia -y
conda install -c fvcore -c iopath -c conda-forge fvcore iopath -y
conda install pytorch3d=0.7.4 -c pytorch3d -y

Both sets of the above commands result in conflicts when I try and install the pytorch3d command.

Thanks again for checking this.

bottler commented 1 year ago

torchvision and torchaudio probably won't matter. If pytorch installs without conda-forge that's ok too: it doesn't for some sets of versions. Note newer instructions replace cudatoolkit with pytorch-cuda, adding -c nvidia. That might be needed for older versions now as well.

For the second part, exactly the right build https://anaconda.org/pytorch3d/pytorch3d/0.7.4/download/linux-64/pytorch3d-0.7.4-py38_cu117_pyt201.tar.bz2 is available. Maybe you could download it directly and conda install the file?

DanielTakeshi commented 1 year ago

Thanks @bottler I realized that the code I needed in pytorch3d can be "ripped out" of the code base as it only relies on pure pytorch functions, so I gave up figuring out how to install the full conda package and just copied the methods in pytorch3d that I needed.