search

facebookresearch / pytorch3d

PyTorch3D is FAIR's library of reusable components for deep learning with 3D data
https://pytorch3d.org/
Other
8.69k stars 1.3k forks source link

ERROR: Could not build wheels for pytorch3d, which is required to install pyproject.toml-based projects | Suggested methods in Issue #1734 did not help #1770

Closed KomputerMaster64 closed 5 months ago

KomputerMaster64 commented 5 months ago

The suggested methods in Issue #1734 did not resolve my error i.e. installing nvcc and g++ did NOT result in the successful installation of PyTorch3D.

(Thank you for sharing the issue #1734 and the possible resolution)

I am facing similar issue while installing PyTorch3D using the stable and the default build, using the command pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable" and pip install "git+https://github.com/facebookresearch/pytorch3d.git"

I am using an NVIDIA RTX A4500 and an Intel Xeon Silver CPU (details about the hardware have been mentioned at the very end)

I am getting the main error as:

ERROR: Could not build wheels for pytorch3d, which is required to install pyproject.toml-based projects

Note: Given below are the outputs for PyTorch3D installation command, and the following commands: conda info, conda list, g++ --version, nvcc --version, lscpu, and nvidia-smi.

Here is the output for the command pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable":

Note: The output was too long with recurrent errors so I have added the last few lines of the output

(foundationpose) USER@MACHINE:~$ pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable"
Collecting git+https://github.com/facebookresearch/pytorch3d.git@stable
  Cloning https://github.com/facebookresearch/pytorch3d.git (to revision stable) to /tmp/pip-req-build-v44npxdv
  Running command git clone --filter=blob:none --quiet https://github.com/facebookresearch/pytorch3d.git /tmp/pip-req-build-v44npxdv
  Running command git checkout -q f34104cf6ebefacd7b7e07955ee7aaa823e616ac
  Resolved https://github.com/facebookresearch/pytorch3d.git to commit f34104cf6ebefacd7b7e07955ee7aaa823e616ac
  Preparing metadata (setup.py) ... done
Collecting fvcore (from pytorch3d==0.7.6)
  Using cached fvcore-0.1.5.post20221221-py3-none-any.whl
Collecting iopath (from pytorch3d==0.7.6)
  Using cached iopath-0.1.10-py3-none-any.whl
Requirement already satisfied: numpy in ./miniconda3/envs/foundationpose/lib/python3.8/site-packages (from fvcore->pytorch3d==0.7.6) (1.24.1)
Collecting yacs>=0.1.6 (from fvcore->pytorch3d==0.7.6)
  Using cached yacs-0.1.8-py3-none-any.whl.metadata (639 bytes)
Collecting pyyaml>=5.1 (from fvcore->pytorch3d==0.7.6)
  Using cached PyYAML-6.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
Collecting tqdm (from fvcore->pytorch3d==0.7.6)
  Using cached tqdm-4.66.2-py3-none-any.whl.metadata (57 kB)
Collecting termcolor>=1.1 (from fvcore->pytorch3d==0.7.6)
  Using cached termcolor-2.4.0-py3-none-any.whl.metadata (6.1 kB)
Requirement already satisfied: Pillow in ./miniconda3/envs/foundationpose/lib/python3.8/site-packages (from fvcore->pytorch3d==0.7.6) (10.2.0)
Collecting tabulate (from fvcore->pytorch3d==0.7.6)
  Using cached tabulate-0.9.0-py3-none-any.whl.metadata (34 kB)
Requirement already satisfied: typing-extensions in ./miniconda3/envs/foundationpose/lib/python3.8/site-packages (from iopath->pytorch3d==0.7.6) (4.8.0)
Collecting portalocker (from iopath->pytorch3d==0.7.6)
  Using cached portalocker-2.8.2-py3-none-any.whl.metadata (8.5 kB)
Using cached PyYAML-6.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (736 kB)
Using cached termcolor-2.4.0-py3-none-any.whl (7.7 kB)
Using cached yacs-0.1.8-py3-none-any.whl (14 kB)
Using cached portalocker-2.8.2-py3-none-any.whl (17 kB)
Using cached tabulate-0.9.0-py3-none-any.whl (35 kB)
Using cached tqdm-4.66.2-py3-none-any.whl (78 kB)
Building wheels for collected packages: pytorch3d
  Building wheel for pytorch3d (setup.py) ... 

[ Removed the lines ] ... [ Removed the lines ]

[ Removed the lines ] ... [ Removed the lines ]

[ Removed the lines ] ... [ Removed the lines ]

[ Removed the lines ] ... [ Removed the lines ]

or, T=c10::impl::ScalarTypeToCPPTypeT<c10::ScalarType::Float>, minN=1L, maxN=8L, curN=1L, minM=1L, maxM=4L, curM=1L, Args=<size_t, size_t, c10::impl::ScalarTypeToCPPTypeT<c10::ScalarType::Float> *, c10::impl::ScalarTypeToCPPTypeT<c10::ScalarType::Float> *, int64_t *, int64_t *, c10::impl::ScalarTypeToCPPTypeT<c10::ScalarType::Float> *, int64_t *, int64_t, int64_t, int64_t, int>]"
      (345): here
                  instantiation of "void DispatchKernel2D<Kernel,T,minN,maxN,minM,maxM,Args...>(int64_t, int64_t, Args...) [with Kernel=KNearestNeighborKernelV3Functor, T=c10::impl::ScalarTypeToCPPTypeT<c10::ScalarType::Float>, minN=1L, maxN=8L, minM=1L, maxM=4L, Args=<size_t, size_t, c10::impl::ScalarTypeToCPPTypeT<c10::ScalarType::Float> *, c10::impl::ScalarTypeToCPPTypeT<c10::ScalarType::Float> *, int64_t *, int64_t *, c10::impl::ScalarTypeToCPPTypeT<c10::ScalarType::Float> *, int64_t *, int64_t, int64_t, int64_t, int>]"
      /tmp/pip-req-build-a42lrpsm/pytorch3d/csrc/knn/knn.cu(433): here

      ninja: build stopped: subcommand failed.
      Traceback (most recent call last):
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1893, in _run_ninja_build
          subprocess.run(
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/subprocess.py", line 516, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

      The above exception was the direct cause of the following exception:

      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-req-build-a42lrpsm/setup.py", line 144, in <module>
          setup(
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/setuptools/dist.py", line 989, in run_command
          super().run_command(command)
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 364, in run
          self.run_command("build")
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/setuptools/dist.py", line 989, in run_command
          super().run_command(command)
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/setuptools/_distutils/command/build.py", line 131, in run
          self.run_command(cmd_name)
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/setuptools/dist.py", line 989, in run_command
          super().run_command(command)
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 88, in run
          _build_ext.run(self)
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
          self.build_extensions()
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 843, in build_extensions
          build_ext.build_extensions(self)
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
          self._build_extensions_serial()
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
          self.build_extension(ext)
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 249, in build_extension
          _build_ext.build_extension(self, ext)
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 548, in build_extension
          objects = self.compiler.compile(
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 658, in unix_wrap_ninja_compile
          _write_ninja_file_and_compile_objects(
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1574, in _write_ninja_file_and_compile_objects
          _run_ninja_build(
        File "/home/USER/miniconda3/envs/foundationpose/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1909, in _run_ninja_build
          raise RuntimeError(message) from e
      RuntimeError: Error compiling objects for extension
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pytorch3d
  Running setup.py clean for pytorch3d
Failed to build pytorch3d
ERROR: Could not build wheels for pytorch3d, which is required to install pyproject.toml-based projects

Here is the output for the command conda info:

(foundationpose) USER@MACHINE:~$ conda info

     active environment : foundationpose
    active env location : /home/USER/miniconda3/envs/foundationpose
            shell level : 2
       user config file : /home/USER/.condarc
 populated config files : /home/USER/.condarc
          conda version : 24.1.2
    conda-build version : not installed
         python version : 3.11.5.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=icelake
                          __conda=24.1.2=0
                          __cuda=12.4=0
                          __glibc=2.31=0
                          __linux=5.15.0=0
                          __unix=0=0
       base environment : /home/USER/miniconda3  (writable)
      conda av data dir : /home/USER/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/USER/miniconda3/pkgs
                          /home/USER/.conda/pkgs
       envs directories : /home/USER/miniconda3/envs
                          /home/USER/.conda/envs
               platform : linux-64
             user-agent : conda/24.1.2 requests/2.31.0 CPython/3.11.5 Linux/5.15.0-101-generic ubuntu/20.04.6 glibc/2.31 solver/libmamba conda-libmamba-solver/23.12.0 libmambapy/1.5.3
                UID:GID : 1003:1003
             netrc file : None
           offline mode : False

Here is the output for the command conda list:

(foundationpose) USER@MACHINE:~$ conda list
# packages in environment at /home/USER/miniconda3/envs/foundationpose:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
ca-certificates           2024.3.11            h06a4308_0  
certifi                   2022.12.7                pypi_0    pypi
cmake                     3.25.0                   pypi_0    pypi
filelock                  3.9.0                    pypi_0    pypi
idna                      3.4                      pypi_0    pypi
jinja2                    3.1.2                    pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1  
libffi                    3.4.4                h6a678d5_0  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libstdcxx-ng              11.2.0               h1234567_1  
lit                       15.0.7                   pypi_0    pypi
markupsafe                2.1.3                    pypi_0    pypi
mpmath                    1.3.0                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0  
networkx                  3.0                      pypi_0    pypi
numpy                     1.24.1                   pypi_0    pypi
openssl                   3.0.13               h7f8727e_0  
pillow                    10.2.0                   pypi_0    pypi
pip                       23.3.1           py38h06a4308_0  
pycryptodomex             3.20.0                   pypi_0    pypi
python                    3.8.19               h955ad1f_0  
readline                  8.2                  h5eee18b_0  
scipy                     1.5.0                    pypi_0    pypi
setuptools                68.2.2           py38h06a4308_0  
sqlite                    3.41.2               h5eee18b_0  
sympy                     1.12                     pypi_0    pypi
tk                        8.6.12               h1ccaba5_0  
torch                     2.0.0+cu118              pypi_0    pypi
torchaudio                2.0.1+cu118              pypi_0    pypi
torchvision               0.15.1+cu118             pypi_0    pypi
triton                    2.0.0                    pypi_0    pypi
typing-extensions         4.8.0                    pypi_0    pypi
wheel                     0.41.2           py38h06a4308_0  
xz                        5.4.6                h5eee18b_0  
zlib                      1.2.13               h5eee18b_0

Here is the output for the command g++ --version:

(foundationpose) USER@MACHINE:~$ g++ --version
g++ (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Here is the output for the command nvcc --version:

(foundationpose) USER@MACHINE:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Mar_21_19:15:46_PDT_2021
Cuda compilation tools, release 11.3, V11.3.58
Build cuda_11.3.r11.3/compiler.29745058_0

Extra Information about the CPU and the GPU

Intel CPU Information

USER@MACHINE:~$ lscpu
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Byte Order:                         Little Endian
Address sizes:                      46 bits physical, 57 bits virtual
CPU(s):                             64
On-line CPU(s) list:                0-63
Thread(s) per core:                 2
Core(s) per socket:                 16
Socket(s):                          2
NUMA node(s):                       2
Vendor ID:                          GenuineIntel
CPU family:                         6
Model:                              106
Model name:                         Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz
Stepping:                           6
CPU MHz:                            3400.000
CPU max MHz:                        3400.0000
CPU min MHz:                        800.0000
BogoMIPS:                           4800.00
Virtualization:                     VT-x
L1d cache:                          1.5 MiB
L1i cache:                          1 MiB
L2 cache:                           40 MiB
L3 cache:                           48 MiB
NUMA node0 CPU(s):                  0-15,32-47
NUMA node1 CPU(s):                  16-31,48-63
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and secco
                                    mp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitizat
                                    ion
Vulnerability Spectre v2:           Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-e
                                    IBRS SW sequence
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
                                     pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
                                     pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_goo
                                    d nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes6
                                    4 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pc
                                    id dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes x
                                    save avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ca
                                    t_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enha
                                    nced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_ad
                                    just bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq 
                                    rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_
                                    ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_o
                                    ccup_llc cqm_mbm_total cqm_mbm_local split_lock_detect wbnoinvd d
                                    therm ida arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfn
                                    i vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq 
                                    la57 rdpid fsrm md_clear pconfig flush_l1d arch_capabilities

NVIDIA GPU Information

USER@MACHINE:~$ nvidia-smi
Sat Mar 30 15:48:06 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.67                 Driver Version: 550.67         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A4500               Off |   00000000:4B:00.0  On |                  Off |
| 30%   46C    P8             26W /  200W |     660MiB /  20470MiB |      3%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      1923      G   /usr/lib/xorg/Xorg                            147MiB |
|    0   N/A  N/A      2386      G   /usr/bin/gnome-shell                          387MiB |
|    0   N/A  N/A     42174      G   ...99,262144 --variations-seed-version        115MiB |
+-----------------------------------------------------------------------------------------+
bottler commented 5 months ago

Let's focus on this problem here, not elsewhere.

I think we need to look at something you've removed in the output pasted above. "ERROR: Could not build wheels for pytorch3d, which is required to install pyproject.toml-based projects" is just a generic error from a build failure. It isn't the specific part. The important, specific part ends with

or, T=c10::impl::ScalarTypeToCPPTypeT<c10::ScalarType::Float>, minN=1L, maxN=8L, curN=1L, minM=1L, maxM=4L, curM=1L, Args=<size_t, size_t, c10::impl::ScalarTypeToCPPTypeT<c10::ScalarType::Float> *, c10::impl::ScalarTypeToCPPTypeT<c10::ScalarType::Float> *, int64_t *, int64_t *, c10::impl::ScalarTypeToCPPTypeT<c10::ScalarType::Float> *, int64_t *, int64_t, int64_t, int64_t, int>]"
      (345): here
                  instantiation of "void DispatchKernel2D<Kernel,T,minN,maxN,minM,maxM,Args...>(int64_t, int64_t, Args...) [with Kernel=KNearestNeighborKernelV3Functor, T=c10::impl::ScalarTypeToCPPTypeT<c10::ScalarType::Float>, minN=1L, maxN=8L, minM=1L, maxM=4L, Args=<size_t, size_t, c10::impl::ScalarTypeToCPPTypeT<c10::ScalarType::Float> *, c10::impl::ScalarTypeToCPPTypeT<c10::ScalarType::Float> *, int64_t *, int64_t *, c10::impl::ScalarTypeToCPPTypeT<c10::ScalarType::Float> *, int64_t *, int64_t, int64_t, int64_t, int>]"
      /tmp/pip-req-build-a42lrpsm/pytorch3d/csrc/knn/knn.cu(433): here

      ninja: build stopped: subcommand failed.

We don't know if the beginning of this was an error or a warning.

I see two possibilities here. (1) There was a compiler error, perhaps ending with the nearest neighbor stuff here, or perhaps coming from somewhere else, which caused the failure. (2) The parallel compilation ran out of memory and died. If you pasted everything in the output (here or using pastebin or similar) we could tell.

You could also try building with MAX_JOBS=1 set in the environment, which would usually avoid any running out of memory.

KomputerMaster64 commented 5 months ago

Thank you for the response. I will go ovre the two suggestions (compiler error with nearest neighbour; MAX_JOBS=1).

__ I think that the official PyTorch3D guide suggests using Python 3.8, 3.9, and 3.10 here. I will try with Python version 3.9 (currently using Python 3.8.19)

__ image

__ I have also attached both the stderr and stdout (output) in a file foundationpose_log_2024.04.02_01.txt as a Google Drive Link (File size too big: 25 MB are allowed, 37 MB were attempted to upload.)

bottler commented 5 months ago

There's no compilation error in that log: you have an out-of-memory problem. And "Python 3.8" is meant to include 3.8.19 etc.. There's no important difference between python versions for this - switching to 3.9 won't change anything.

KomputerMaster64 commented 5 months ago

Thank you for the resposne. I was able to do the set-up foundationpose.

SubhamBurnwal-BiswaGames commented 4 months ago

@KomputerMaster64 what do you mean by foundationpose? What changes did you do?