Problems building cucim-c11 when running setup.py

conorhughmcfadden commented 3 months ago

I am following the setup procedure on BioHPC. Did "module load gpu_prepare", loaded CUDA, etc. For some reason when running setup.py it threw an error:

Collecting cucim-cu11 (from u-Segment3D==0.1.0)
  Using cached cucim_cu11-24.6.0.tar.gz (3.2 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

Preparing metadata (pyproject.toml) did not run successfully.
  \u2502 exit code: 1
  \u2570\u2500> [56 lines of output]
        File "/tmp/pip-build-env-rvv_3hr_/overlay/lib/python3.11/site-packages/nvidia_stub/wheel.py", line 147, in download_wheel
          return download_manual(wheel_directory, distribution, version)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-rvv_3hr_/overlay/lib/python3.11/site-packages/nvidia_stub/wheel.py", line 114, in download_manual
          raise RuntimeError(f"Didn't find wheel for {distribution} {version}")
      Traceback (most recent call last):
        File "/tmp/pip-build-env-rvv_3hr_/overlay/lib/python3.11/site-packages/nvidia_stub/wheel.py", line 147, in download_wheel
          return download_manual(wheel_directory, distribution, version)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-rvv_3hr_/overlay/lib/python3.11/site-packages/nvidia_stub/wheel.py", line 114, in download_manual
          raise RuntimeError(f"Didn't find wheel for {distribution} {version}")
      RuntimeError: Didn't find wheel for cucim-cu11 24.6.0

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/cm/shared/apps/python/3.11.x-mamba/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/cm/shared/apps/python/3.11.x-mamba/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/cm/shared/apps/python/3.11.x-mamba/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
          whl_basename = backend.build_wheel(metadata_directory, config_settings)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-rvv_3hr_/overlay/lib/python3.11/site-packages/nvidia_stub/buildapi.py", line 29, in build_wheel
          return download_wheel(pathlib.Path(wheel_directory), config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-rvv_3hr_/overlay/lib/python3.11/site-packages/nvidia_stub/wheel.py", line 149, in download_wheel
          report_install_failure(distribution, version, exception_context)
        File "/tmp/pip-build-env-rvv_3hr_/overlay/lib/python3.11/site-packages/nvidia_stub/error.py", line 63, in report_install_failure
          raise InstallFailedError(
      nvidia_stub.error.InstallFailedError:
      *******************************************************************************

      The installation of cucim-cu11 for version 24.6.0 failed.

      This is a special placeholder package which downloads a real wheel package
      from https://pypi.nvidia.com. If https://pypi.nvidia.com is not reachable, we
      cannot download the real wheel file to install.

      You might try installing this package via

  $ pip install --extra-index-url https://pypi.nvidia.com cucim-cu11
  ```

  Here is some debug information about your platform to include in any bug
  report:

  Python Version: CPython 3.11.6
  Operating System: Linux 3.10.0-1160.88.1.el7.x86_64
  CPU Architecture: x86_64
  Driver Version: 550.54
  CUDA Version: 12.4

  *******************************************************************************

  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed

Encountered error while generating package metadata. \u2570\u2500> See above for output.

note: This is an issue with the package mentioned above, not pip. hint: See above for details.


Note that it still thinks I am using CUDA 12.4, but I did have the correct runtime CUDA loaded:

(u_Segment3D_env) [s438161@NucleusC019 u-segment3D]$ nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Wed_Sep_21_10:33:58_PDT_2022 Cuda compilation tools, release 11.8, V11.8.89 Build cuda_11.8.r11.8/compiler.31833905_0


However, the version recognized by nvidia-smi is still 12.4,

+-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 69490 G /usr/bin/X 66MiB | | 0 N/A N/A 69581 G /usr/bin/gnome-shell 84MiB | +-----------------------------------------------------------------------------------------+


Python version is correct,

(u_Segment3D_env) [s438161@NucleusC019 u-segment3D]$ python --version Python 3.11.6


I managed to workaround by installing cucim-c11 directly from wheel, but this may not be the best solution:

(u_Segment3D_env) [s438161@NucleusC019 u-segment3D]$ wget https://pypi.nvidia.com/cucim-cu11/cucim_cu11-24.4.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (u_Segment3D_env) [s438161@NucleusC019 u-segment3D]$ pip install cucim_cu11-24.4.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl



This installed successfully, and I was able to run setup.py the rest of the way. Please disregard if this is a system specific issue to me, but I feel like this is something others may run into.

- Conor

fyz11 commented 3 months ago

Thanks Conor. This is good to know as a fix ! Seems pip has issues with pulling the Python 3.11 versions with cucim.

The environment created with python-3.9 and cudnn should work fine though :

conda create -n u_Segment3D_env python=3.9 cudatoolkit=11.8. cudnn==8. -c anaconda

please do make sure to then activate the env:

conda activate u_Segment3D_env

before

pip install .

conorhughmcfadden commented 3 months ago

Update: I recreated my conda environment, making sure its python 3.9. After googling, I learned you can have previous conda envs running which conflict with eachother. I ran "conda deactivate" multiple times, reactivated my env, and...

(/work/bioinformatics/s438161/conda/u_Segment3D_env) [s438161@NucleusC022 u-segment3D]$ python --version
Python 3.9.18

The version is correct in my env! However...

(/work/bioinformatics/s438161/conda/u_Segment3D_env) [s438161@NucleusC022 u-segment3D]$ pip install .
Processing /endosome/archive/bioinformatics/Danuser_lab/Fiolka/LabMembers/Conor/python/u-segment3D
  Preparing metadata (setup.py) ... done
Collecting cellpose (from u-Segment3D==0.1.0)
  Using cached cellpose-3.0.9-py3-none-any.whl.metadata (24 kB)
Collecting cupy-cuda11x (from u-Segment3D==0.1.0)
  Using cached cupy_cuda11x-13.2.0-cp39-cp39-manylinux2014_x86_64.whl.metadata (2.7 kB)
Collecting cucim-cu11 (from u-Segment3D==0.1.0)
  Using cached cucim_cu11-24.6.0.tar.gz (3.2 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  � Preparing metadata (pyproject.toml) did not run successfully.
  \u2502 exit code: 1
  \u2570\u2500> [51 lines of output]
        File "/tmp/pip-build-env-u5v58x16/overlay/lib/python3.9/site-packages/nvidia_stub/wheel.py", line 147, in download_wheel
          return download_manual(wheel_directory, distribution, version)
        File "/tmp/pip-build-env-u5v58x16/overlay/lib/python3.9/site-packages/nvidia_stub/wheel.py", line 114, in download_manual
          raise RuntimeError(f"Didn't find wheel for {distribution} {version}")
      Traceback (most recent call last):
        File "/tmp/pip-build-env-u5v58x16/overlay/lib/python3.9/site-packages/nvidia_stub/wheel.py", line 147, in download_wheel
          return download_manual(wheel_directory, distribution, version)
        File "/tmp/pip-build-env-u5v58x16/overlay/lib/python3.9/site-packages/nvidia_stub/wheel.py", line 114, in download_manual
          raise RuntimeError(f"Didn't find wheel for {distribution} {version}")
      RuntimeError: Didn't find wheel for cucim-cu11 24.6.0

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/work/bioinformatics/s438161/conda/u_Segment3D_env/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/work/bioinformatics/s438161/conda/u_Segment3D_env/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/work/bioinformatics/s438161/conda/u_Segment3D_env/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
          whl_basename = backend.build_wheel(metadata_directory, config_settings)
        File "/tmp/pip-build-env-u5v58x16/overlay/lib/python3.9/site-packages/nvidia_stub/buildapi.py", line 29, in build_wheel
          return download_wheel(pathlib.Path(wheel_directory), config_settings)
        File "/tmp/pip-build-env-u5v58x16/overlay/lib/python3.9/site-packages/nvidia_stub/wheel.py", line 149, in download_wheel
          report_install_failure(distribution, version, exception_context)
        File "/tmp/pip-build-env-u5v58x16/overlay/lib/python3.9/site-packages/nvidia_stub/error.py", line 63, in report_install_failure
          raise InstallFailedError(
      nvidia_stub.error.InstallFailedError:
      *******************************************************************************

      The installation of cucim-cu11 for version 24.6.0 failed.

      This is a special placeholder package which downloads a real wheel package
      from https://pypi.nvidia.com. If https://pypi.nvidia.com is not reachable, we
      cannot download the real wheel file to install.

      You might try installing this package via

  $ pip install --extra-index-url https://pypi.nvidia.com cucim-cu11
  ```

  Here is some debug information about your platform to include in any bug
  report:

  Python Version: CPython 3.9.18
  Operating System: Linux 3.10.0-1160.88.1.el7.x86_64
  CPU Architecture: x86_64
  Driver Version: 550.54
  CUDA Version: 12.4

  *******************************************************************************

  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed

� Encountered error while generating package metadata. \u2570\u2500> See above for output.

note: This is an issue with the package mentioned above, not pip. hint: See above for details.



I really think this is a CUDA issue and not a python issue. Note that cucim still thinks I have CUDA 12.4. Again, I can solve this by installing from wheel directly, but it's not ideal. Very strange.

**NOTE:** I had to use the cp39 wheel to get it to work this time, further indicating that I'm now using the right version...

[https://pypi.nvidia.com/cucim-cu11/cucim_cu11-24.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl](url)

JennyZouUTSW commented 3 months ago

Encountered same issue on a HPC cluster.

conda create -n u_Segment3D_env python=3.9 cudatoolkit=11.8.* cudnn==8.* -c anaconda
conda activate u_Segment3D_env
module load gcc/8.3.0
cd root/of/the/u-segment3D/repository
pip install . # failed with same error above

Conor's solution worked well with cp39 wheel. Thanks!

wget https://pypi.nvidia.com/cucim-cu11/cucim_cu11-24.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
pip install cucim_cu11-24.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
pip install . # passed

DanuserLab commented 2 months ago

The Readme file has been updated with "Errors we have encountered" and solutions. This ticket is closed.

DanuserLab / u-segment3D

Problems building cucim-c11 when running setup.py #1