MIC-DKFZ / nnDetection

nnDetection is a self-configuring framework for 3D (volumetric) medical object detection which can be applied to new data sets without manual intervention. It includes guides for 12 data sets that were used to develop and evaluate the performance of the proposed method.
Apache License 2.0
541 stars 93 forks source link

problem when install on RTX3090 #220

Closed ztt0821 closed 8 months ago

ztt0821 commented 8 months ago

Hi, i found the following problem when I install on RTX3090 gpu.

Running setup.py develop for nndet error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [55 lines of output]
    Build csrc
    Building with sys.version_info(major=3, minor=8, micro=18, releaselevel='final', serial=0)
    main_file [PosixPath('/media/ttzhang/T7/nnDetection/nndet/csrc/ops.cpp')]
    source_cpu []
    source_cuda [PosixPath('/media/ttzhang/T7/nnDetection/nndet/csrc/cuda/nms.cu')]
    Adding CUDA csrc to build
    CUDA ARCH None
    running develop
    running egg_info
    writing nndet.egg-info/PKG-INFO
    writing dependency_links to nndet.egg-info/dependency_links.txt
    writing entry points to nndet.egg-info/entry_points.txt
    writing requirements to nndet.egg-info/requires.txt
    writing top-level names to nndet.egg-info/top_level.txt
    reading manifest file 'nndet.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/setuptools/dist.py:314: InformationOnly: Normalizing 'v0.1' to '0.1'
      self.metadata.version = self._normalize_version(
    /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/setuptools/command/develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated.
    !!

            ********************************************************************************
            Please avoid running ``setup.py`` and ``easy_install``.
            Instead, use pypa/build, pypa/installer or other
            standards-based tools.

            See https://github.com/pypa/setuptools/issues/917 for details.
            ********************************************************************************

    !!
      easy_install.initialize_options(self)
    /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
    !!

            ********************************************************************************
            Please avoid running ``setup.py`` directly.
            Instead, use pypa/build, pypa/installer or other
            standards-based tools.

            See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
            ********************************************************************************

    !!
      self.initialize_options()
    /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/torch/utils/cpp_extension.py:476: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
      warnings.warn(msg.format('we could not find ninja.'))
    warning: no files found matching 'LICENSE'
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '__pycache__' found anywhere in distribution
    writing manifest file 'nndet.egg-info/SOURCES.txt'
    running build_ext
    building 'nndet._C' extension
    /usr/bin/nvcc -DWITH_CUDA -I/media/ttzhang/T7/nnDetection/nndet/csrc -I/home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/torch/include -I/home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/torch/include/TH -I/home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/torch/include/THC -I/home/ttzhang/anaconda3/envs/nndetection/include/python3.8 -c /media/ttzhang/T7/nnDetection/nndet/csrc/cuda/nms.cu -o build/temp.linux-x86_64-cpython-38/media/ttzhang/T7/nnDetection/nndet/csrc/cuda/nms.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++14
    ptxas fatal   : Value 'sm_86' is not defined for option 'gpu-name'
    error: command '/usr/bin/nvcc' failed with exit code 255
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

error: subprocess-exited-with-error

× python setup.py develop did not run successfully. │ exit code: 1 ╰─> [55 lines of output] Build csrc Building with sys.version_info(major=3, minor=8, micro=18, releaselevel='final', serial=0) main_file [PosixPath('/media/ttzhang/T7/nnDetection/nndet/csrc/ops.cpp')] source_cpu [] source_cuda [PosixPath('/media/ttzhang/T7/nnDetection/nndet/csrc/cuda/nms.cu')] Adding CUDA csrc to build CUDA ARCH None running develop running egg_info writing nndet.egg-info/PKG-INFO writing dependency_links to nndet.egg-info/dependency_links.txt writing entry points to nndet.egg-info/entry_points.txt writing requirements to nndet.egg-info/requires.txt writing top-level names to nndet.egg-info/top_level.txt reading manifest file 'nndet.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/setuptools/dist.py:314: InformationOnly: Normalizing 'v0.1' to '0.1' self.metadata.version = self._normalize_version( /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/setuptools/command/develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated. !!

        ********************************************************************************
        Please avoid running ``setup.py`` and ``easy_install``.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://github.com/pypa/setuptools/issues/917 for details.
        ********************************************************************************

!!
  easy_install.initialize_options(self)
/home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
/home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/torch/utils/cpp_extension.py:476: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
  warnings.warn(msg.format('we could not find ninja.'))
warning: no files found matching 'LICENSE'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
writing manifest file 'nndet.egg-info/SOURCES.txt'
running build_ext
building 'nndet._C' extension
/usr/bin/nvcc -DWITH_CUDA -I/media/ttzhang/T7/nnDetection/nndet/csrc -I/home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/torch/include -I/home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/torch/include/TH -I/home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/torch/include/THC -I/home/ttzhang/anaconda3/envs/nndetection/include/python3.8 -c /media/ttzhang/T7/nnDetection/nndet/csrc/cuda/nms.cu -o build/temp.linux-x86_64-cpython-38/media/ttzhang/T7/nnDetection/nndet/csrc/cuda/nms.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++14
ptxas fatal   : Value 'sm_86' is not defined for option 'gpu-name'
error: command '/usr/bin/nvcc' failed with exit code 255
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

My nvcc version is 11.7 Screenshot from 2024-01-08 16-32-25

My GCC is 7.5

mibaumgartner commented 8 months ago

Dear @ztt0821 ,

can you please install with pip install -e . -v to obtain more information since the exact error is not shown. We have successfully run nnDetection on RTX3090's in the past.

Best, Michael

ztt0821 commented 8 months ago

Hi, the following is the information:

(nndetection) ttzhang@kate-Precision-5820-Tower:/media/ttzhang/T7/nnDetection$ pip install -e . -v Using pip 23.3.1 from /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/pip (python 3.8) Obtaining file:///media/ttzhang/T7/nnDetection Running command python setup.py egg_info Build csrc Building with sys.version_info(major=3, minor=8, micro=18, releaselevel='final', serial=0) main_file [PosixPath('/media/ttzhang/T7/nnDetection/nndet/csrc/ops.cpp')] source_cpu [] source_cuda [PosixPath('/media/ttzhang/T7/nnDetection/nndet/csrc/cuda/nms.cu')] Adding CUDA csrc to build CUDA ARCH None running egg_info creating /tmp/pip-pip-egg-info-l1xy7gyk/nndet.egg-info writing /tmp/pip-pip-egg-info-l1xy7gyk/nndet.egg-info/PKG-INFO writing dependency_links to /tmp/pip-pip-egg-info-l1xy7gyk/nndet.egg-info/dependency_links.txt writing entry points to /tmp/pip-pip-egg-info-l1xy7gyk/nndet.egg-info/entry_points.txt writing requirements to /tmp/pip-pip-egg-info-l1xy7gyk/nndet.egg-info/requires.txt writing top-level names to /tmp/pip-pip-egg-info-l1xy7gyk/nndet.egg-info/top_level.txt writing manifest file '/tmp/pip-pip-egg-info-l1xy7gyk/nndet.egg-info/SOURCES.txt' reading manifest file '/tmp/pip-pip-egg-info-l1xy7gyk/nndet.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/setuptools/dist.py:314: InformationOnly: Normalizing 'v0.1' to '0.1' self.metadata.version = self._normalize_version( /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/torch/utils/cpp_extension.py:476: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend. warnings.warn(msg.format('we could not find ninja.')) warning: no files found matching 'LICENSE' warning: no previously-included files matching '.pyc' found anywhere in distribution warning: no previously-included files matching 'pycache' found anywhere in distribution writing manifest file '/tmp/pip-pip-egg-info-l1xy7gyk/nndet.egg-info/SOURCES.txt' Preparing metadata (setup.py) ... done Requirement already satisfied: pytorch_lightning<=1.4.2,>=1.3.1 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nndet==0.1) (1.4.2) Requirement already satisfied: batchgenerators>=0.24 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nndet==0.1) (0.25) Requirement already satisfied: nnunet==1.7.1 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nndet==0.1) (1.7.1) Requirement already satisfied: scipy in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nndet==0.1) (1.10.1) Requirement already satisfied: scikit-learn in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nndet==0.1) (1.3.2) Requirement already satisfied: scikit-image>=0.14 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nndet==0.1) (0.21.0) Requirement already satisfied: pandas>=0.8.1 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nndet==0.1) (2.0.3) Requirement already satisfied: PyYAML!=5.4.,>=5.1 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nndet==0.1) (6.0.1) Requirement already satisfied: nevergrad<=0.4.2 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nndet==0.1) (0.4.2) Requirement already satisfied: dicom2nifti in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nndet==0.1) (2.4.9) Requirement already satisfied: medpy in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nndet==0.1) (0.4.0) Requirement already satisfied: SimpleITK<2.1.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nndet==0.1) (2.0.2) Requirement already satisfied: tqdm in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nndet==0.1) (4.66.1) Requirement already satisfied: loguru in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nndet==0.1) (0.7.2) Requirement already satisfied: hydra-core>=1.1.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nndet==0.1) (1.3.2) Requirement already satisfied: mlflow in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nndet==0.1) (2.9.2) Requirement already satisfied: GitPython in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nndet==0.1) (3.1.40) Requirement already satisfied: matplotlib in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nndet==0.1) (3.7.4) Requirement already satisfied: seaborn in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nndet==0.1) (0.13.1) Requirement already satisfied: torchmetrics<=0.7.3,>=0.7.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nndet==0.1) (0.7.3) Requirement already satisfied: torch>1.10.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nnunet==1.7.1->nndet==0.1) (1.13.0) Requirement already satisfied: numpy in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nnunet==1.7.1->nndet==0.1) (1.24.3) Requirement already satisfied: requests in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nnunet==1.7.1->nndet==0.1) (2.31.0) Requirement already satisfied: nibabel in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nnunet==1.7.1->nndet==0.1) (5.2.0) Requirement already satisfied: tifffile in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nnunet==1.7.1->nndet==0.1) (2023.7.10) Requirement already satisfied: pillow>=7.1.2 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from batchgenerators>=0.24->nndet==0.1) (10.0.1) Requirement already satisfied: future in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from batchgenerators>=0.24->nndet==0.1) (0.18.3) Requirement already satisfied: unittest2 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from batchgenerators>=0.24->nndet==0.1) (1.1.0) Requirement already satisfied: threadpoolctl in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from batchgenerators>=0.24->nndet==0.1) (3.2.0) Requirement already satisfied: omegaconf<2.4,>=2.2 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from hydra-core>=1.1.0->nndet==0.1) (2.3.0) Requirement already satisfied: antlr4-python3-runtime==4.9.* in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from hydra-core>=1.1.0->nndet==0.1) (4.9.3) Requirement already satisfied: packaging in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from hydra-core>=1.1.0->nndet==0.1) (23.2) Requirement already satisfied: importlib-resources in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from hydra-core>=1.1.0->nndet==0.1) (6.1.1) Requirement already satisfied: cma>=2.6.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nevergrad<=0.4.2->nndet==0.1) (3.3.0) Requirement already satisfied: bayesian-optimization>=1.1.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nevergrad<=0.4.2->nndet==0.1) (1.4.3) Requirement already satisfied: typing-extensions>=3.6.6 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from nevergrad<=0.4.2->nndet==0.1) (4.7.1) Requirement already satisfied: python-dateutil>=2.8.2 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from pandas>=0.8.1->nndet==0.1) (2.8.2) Requirement already satisfied: pytz>=2020.1 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from pandas>=0.8.1->nndet==0.1) (2023.3.post1) Requirement already satisfied: tzdata>=2022.1 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from pandas>=0.8.1->nndet==0.1) (2023.4) Requirement already satisfied: fsspec!=2021.06.0,>=2021.05.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from fsspec[http]!=2021.06.0,>=2021.05.0->pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (2023.12.2) Requirement already satisfied: tensorboard>=2.2.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (2.14.0) Requirement already satisfied: pyDeprecate==0.3.1 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (0.3.1) Requirement already satisfied: networkx>=2.8 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from scikit-image>=0.14->nndet==0.1) (3.1) Requirement already satisfied: imageio>=2.27 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from scikit-image>=0.14->nndet==0.1) (2.33.1) Requirement already satisfied: PyWavelets>=1.1.1 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from scikit-image>=0.14->nndet==0.1) (1.4.1) Requirement already satisfied: lazy_loader>=0.2 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from scikit-image>=0.14->nndet==0.1) (0.3) Requirement already satisfied: pydicom>=2.2.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from dicom2nifti->nndet==0.1) (2.4.4) Requirement already satisfied: python-gdcm in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from dicom2nifti->nndet==0.1) (3.0.22) Requirement already satisfied: gitdb<5,>=4.0.1 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from GitPython->nndet==0.1) (4.0.11) Requirement already satisfied: contourpy>=1.0.1 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from matplotlib->nndet==0.1) (1.1.1) Requirement already satisfied: cycler>=0.10 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from matplotlib->nndet==0.1) (0.12.1) Requirement already satisfied: fonttools>=4.22.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from matplotlib->nndet==0.1) (4.47.0) Requirement already satisfied: kiwisolver>=1.0.1 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from matplotlib->nndet==0.1) (1.4.5) Requirement already satisfied: pyparsing>=2.3.1 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from matplotlib->nndet==0.1) (3.1.1) Requirement already satisfied: click<9,>=7.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from mlflow->nndet==0.1) (8.1.7) Requirement already satisfied: cloudpickle<4 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from mlflow->nndet==0.1) (3.0.0) Requirement already satisfied: databricks-cli<1,>=0.8.7 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from mlflow->nndet==0.1) (0.18.0) Requirement already satisfied: entrypoints<1 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from mlflow->nndet==0.1) (0.4) Requirement already satisfied: protobuf<5,>=3.12.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from mlflow->nndet==0.1) (4.25.1) Requirement already satisfied: importlib-metadata!=4.7.0,<8,>=3.7.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from mlflow->nndet==0.1) (7.0.1) Requirement already satisfied: sqlparse<1,>=0.4.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from mlflow->nndet==0.1) (0.4.4) Requirement already satisfied: alembic!=1.10.0,<2 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from mlflow->nndet==0.1) (1.13.1) Requirement already satisfied: docker<7,>=4.0.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from mlflow->nndet==0.1) (6.1.3) Requirement already satisfied: Flask<4 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from mlflow->nndet==0.1) (3.0.0) Requirement already satisfied: querystring-parser<2 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from mlflow->nndet==0.1) (1.2.4) Requirement already satisfied: sqlalchemy<3,>=1.4.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from mlflow->nndet==0.1) (2.0.25) Requirement already satisfied: pyarrow<15,>=4.0.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from mlflow->nndet==0.1) (14.0.2) Requirement already satisfied: markdown<4,>=3.3 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from mlflow->nndet==0.1) (3.5.1) Requirement already satisfied: gunicorn<22 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from mlflow->nndet==0.1) (21.2.0) Requirement already satisfied: Jinja2<4,>=2.11 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from mlflow->nndet==0.1) (3.1.2) Requirement already satisfied: joblib>=1.1.1 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from scikit-learn->nndet==0.1) (1.3.2) Requirement already satisfied: Mako in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from alembic!=1.10.0,<2->mlflow->nndet==0.1) (1.3.0) Requirement already satisfied: colorama>=0.4.6 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from bayesian-optimization>=1.1.0->nevergrad<=0.4.2->nndet==0.1) (0.4.6) Requirement already satisfied: pyjwt>=1.7.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from databricks-cli<1,>=0.8.7->mlflow->nndet==0.1) (2.8.0) Requirement already satisfied: oauthlib>=3.1.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from databricks-cli<1,>=0.8.7->mlflow->nndet==0.1) (3.2.2) Requirement already satisfied: tabulate>=0.7.7 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from databricks-cli<1,>=0.8.7->mlflow->nndet==0.1) (0.9.0) Requirement already satisfied: six>=1.10.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from databricks-cli<1,>=0.8.7->mlflow->nndet==0.1) (1.16.0) Requirement already satisfied: urllib3<3,>=1.26.7 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from databricks-cli<1,>=0.8.7->mlflow->nndet==0.1) (1.26.18) Requirement already satisfied: websocket-client>=0.32.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from docker<7,>=4.0.0->mlflow->nndet==0.1) (1.7.0) Requirement already satisfied: Werkzeug>=3.0.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from Flask<4->mlflow->nndet==0.1) (3.0.1) Requirement already satisfied: itsdangerous>=2.1.2 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from Flask<4->mlflow->nndet==0.1) (2.1.2) Requirement already satisfied: blinker>=1.6.2 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from Flask<4->mlflow->nndet==0.1) (1.7.0) Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from fsspec[http]!=2021.06.0,>=2021.05.0->pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (3.9.1) Requirement already satisfied: smmap<6,>=3.0.1 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from gitdb<5,>=4.0.1->GitPython->nndet==0.1) (5.0.1) Requirement already satisfied: zipp>=0.5 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from importlib-metadata!=4.7.0,<8,>=3.7.0->mlflow->nndet==0.1) (3.17.0) Requirement already satisfied: MarkupSafe>=2.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from Jinja2<4,>=2.11->mlflow->nndet==0.1) (2.1.3) Requirement already satisfied: charset-normalizer<4,>=2 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from requests->nnunet==1.7.1->nndet==0.1) (2.0.4) Requirement already satisfied: idna<4,>=2.5 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from requests->nnunet==1.7.1->nndet==0.1) (3.4) Requirement already satisfied: certifi>=2017.4.17 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from requests->nnunet==1.7.1->nndet==0.1) (2023.11.17) Requirement already satisfied: greenlet!=0.4.17 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from sqlalchemy<3,>=1.4.0->mlflow->nndet==0.1) (3.0.3) Requirement already satisfied: absl-py>=0.4 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from tensorboard>=2.2.0->pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (2.0.0) Requirement already satisfied: grpcio>=1.48.2 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from tensorboard>=2.2.0->pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (1.60.0) Requirement already satisfied: google-auth<3,>=1.6.3 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from tensorboard>=2.2.0->pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (2.26.1) Requirement already satisfied: google-auth-oauthlib<1.1,>=0.5 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from tensorboard>=2.2.0->pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (1.0.0) Requirement already satisfied: setuptools>=41.0.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from tensorboard>=2.2.0->pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (68.2.2) Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from tensorboard>=2.2.0->pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (0.7.2) Requirement already satisfied: wheel>=0.26 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from tensorboard>=2.2.0->pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (0.41.2) Collecting argparse (from unittest2->batchgenerators>=0.24->nndet==0.1) Using cached argparse-1.4.0-py2.py3-none-any.whl (23 kB) Requirement already satisfied: traceback2 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from unittest2->batchgenerators>=0.24->nndet==0.1) (1.4.0) Requirement already satisfied: attrs>=17.3.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (23.2.0) Requirement already satisfied: multidict<7.0,>=4.5 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (6.0.4) Requirement already satisfied: yarl<2.0,>=1.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (1.9.4) Requirement already satisfied: frozenlist>=1.1.1 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (1.4.1) Requirement already satisfied: aiosignal>=1.1.2 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (1.3.1) Requirement already satisfied: async-timeout<5.0,>=4.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (4.0.3) Requirement already satisfied: cachetools<6.0,>=2.0.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (5.3.2) Requirement already satisfied: pyasn1-modules>=0.2.1 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (0.3.0) Requirement already satisfied: rsa<5,>=3.1.4 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (4.9) Requirement already satisfied: requests-oauthlib>=0.7.0 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from google-auth-oauthlib<1.1,>=0.5->tensorboard>=2.2.0->pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (1.3.1) Requirement already satisfied: linecache2 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from traceback2->unittest2->batchgenerators>=0.24->nndet==0.1) (1.0.0) Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch_lightning<=1.4.2,>=1.3.1->nndet==0.1) (0.5.1) Installing collected packages: argparse, nndet Running setup.py develop for nndet Running command python setup.py develop Build csrc Building with sys.version_info(major=3, minor=8, micro=18, releaselevel='final', serial=0) main_file [PosixPath('/media/ttzhang/T7/nnDetection/nndet/csrc/ops.cpp')] source_cpu [] source_cuda [PosixPath('/media/ttzhang/T7/nnDetection/nndet/csrc/cuda/nms.cu')] Adding CUDA csrc to build CUDA ARCH None running develop running egg_info writing nndet.egg-info/PKG-INFO writing dependency_links to nndet.egg-info/dependency_links.txt writing entry points to nndet.egg-info/entry_points.txt writing requirements to nndet.egg-info/requires.txt writing top-level names to nndet.egg-info/top_level.txt reading manifest file 'nndet.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/setuptools/dist.py:314: InformationOnly: Normalizing 'v0.1' to '0.1' self.metadata.version = self._normalize_version( /home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/setuptools/command/develop.py:40: EasyInstallDeprecationWarning: easy_install command is deprecated. !!

        ********************************************************************************
        Please avoid running ``setup.py`` and ``easy_install``.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://github.com/pypa/setuptools/issues/917 for details.
        ********************************************************************************

!!
  easy_install.initialize_options(self)
/home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
/home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/torch/utils/cpp_extension.py:476: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
  warnings.warn(msg.format('we could not find ninja.'))
warning: no files found matching 'LICENSE'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
writing manifest file 'nndet.egg-info/SOURCES.txt'
running build_ext
building 'nndet._C' extension
/usr/bin/nvcc -DWITH_CUDA -I/media/ttzhang/T7/nnDetection/nndet/csrc -I/home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/torch/include -I/home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/torch/include/TH -I/home/ttzhang/anaconda3/envs/nndetection/lib/python3.8/site-packages/torch/include/THC -I/home/ttzhang/anaconda3/envs/nndetection/include/python3.8 -c /media/ttzhang/T7/nnDetection/nndet/csrc/cuda/nms.cu -o build/temp.linux-x86_64-cpython-38/media/ttzhang/T7/nnDetection/nndet/csrc/cuda/nms.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++14
ptxas fatal   : Value 'sm_86' is not defined for option 'gpu-name'
error: command '/usr/bin/nvcc' failed with exit code 255
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /home/ttzhang/anaconda3/envs/nndetection/bin/python -c '
exec(compile('"'"''"'"''"'"'
# This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
#
# - It imports setuptools before invoking setup.py, to enable projects that directly
#   import from `distutils.core` to work with newer packaging standards.
# - It provides a clear error message when setuptools is not installed.
# - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
#   setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
#     manifest_maker: standard file '"'"'-c'"'"' not found".
# - It generates a shim setup.py, for handling setup.cfg-only projects.
import os, sys, tokenize

try:
    import setuptools
except ImportError as error:
    print(
        "ERROR: Can not execute `setup.py` since setuptools is not available in "
        "the build environment.",
        file=sys.stderr,
    )
    sys.exit(1)

__file__ = %r
sys.argv[0] = __file__

if os.path.exists(__file__):
    filename = __file__
    with tokenize.open(__file__) as f:
        setup_py_code = f.read()
else:
    filename = "<auto-generated setuptools caller>"
    setup_py_code = "from setuptools import setup; setup()"

exec(compile(setup_py_code, filename, "exec"))
'"'"''"'"''"'"' % ('"'"'/media/ttzhang/T7/nnDetection/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' develop --no-deps
cwd: /media/ttzhang/T7/nnDetection/

error: subprocess-exited-with-error

× python setup.py develop did not run successfully. │ exit code: 1 ╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

mibaumgartner commented 8 months ago

The error indicates some problem where the CUDA version seems to be too old.

Can you please run python scripts/utils.py to collect the environment as it is seen by nnDet.

As pointed out in the very end of https://github.com/tensorflow/tensorflow/issues/45590 please make suer that CUDA_HOME and LD_LIBRARY_PATH point to the correct positions.

Please also check nvcc --help which should print the SM_86 architecture. (https://askubuntu.com/questions/1301714/nvcc-fatal-value-sm-86-is-not-defined-for-option-gpu-architecture)

ztt0821 commented 8 months ago

Hi, this is the output of ----- PyTorch Information ----- PyTorch Version: 1.13.0 PyTorch Debug: False PyTorch CUDA: 11.7 PyTorch Backend cudnn: 8500 PyTorch CUDA Arch List: ['sm_37', 'sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'compute_37'] PyTorch Current Device Capability: (8, 6) PyTorch CUDA available: True

----- System Information ----- System NVCC: nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Tue_May__3_18:49:52_PDT_2022 Cuda compilation tools, release 11.7, V11.7.64 Build cuda_11.7.r11.7/compiler.31294372_0

System Arch List: None System OMP_NUM_THREADS: None System CUDA_HOME is None: True System CPU Count: 8 Python Version: 3.8.18 (default, Sep 11 2023, 13:40:15) [GCC 11.2.0]

----- nnDetection Information ----- det_num_threads None det_data is set False det_models is set False

ztt0821 commented 8 months ago

My cuda version is 11.7, it supports sm_86 --gpu-architecture (-arch)
Specify the name of the class of NVIDIA 'virtual' GPU architecture for which the CUDA input files must be compiled. With the exception as described for the shorthand below, the architecture specified with this option must be a 'virtual' architecture (such as compute_50). Normally, this option alone does not trigger assembly of the generated PTX for a 'real' architecture (that is the role of nvcc option '--gpu-code', see below); rather, its purpose is to control preprocessing and compilation of the input to PTX. For convenience, in case of simple nvcc compilations, the following shorthand is supported. If no value for option '--gpu-code' is specified, then the value of this option defaults to the value of '--gpu-architecture'. In this situation, as only exception to the description above, the value specified for '--gpu-architecture' may be a 'real' architecture (such as a sm_50), in which case nvcc uses the specified 'real' architecture and its closest 'virtual' architecture as effective architecture values. For example, 'nvcc --gpu-architecture=sm_50' is equivalent to 'nvcc --gpu-architecture=compute_50 --gpu-code=sm_50,compute50'. -arch=all build for all supported architectures (sm), and add PTX for the highest major architecture to the generated code. -arch=all-major build for just supported major versions (sm_0), plus the earliest supported, and add PTX for the highest major architecture to the generated code. -arch=native build for all architectures (sm_*) on the current system Note: -arch=native, -arch=all, -arch=all-major cannot be used with the -code option, but can be used with -gencode options Note: the values compute_30, compute_32, compute_35, compute_37, compute_50, sm_30, sm_32, sm_35, sm_37 and sm_50 are deprecated and may be removed in a future release. Allowed values for this option: 'all','all-major','compute_35','compute_37', 'compute_50','compute_52','compute_53','compute_60','compute_61','compute_62', 'compute_70','compute_72','compute_75','compute_80','compute_86','compute_87', 'lto_35','lto_37','lto_50','lto_52','lto_53','lto_60','lto_61','lto_62', 'lto_70','lto_72','lto_75','lto_80','lto_86','lto_87','native','sm_35','sm_37', 'sm_50','sm_52','sm_53','sm_60','sm_61','sm_62','sm_70','sm_72','sm_75', 'sm_80','sm_86','sm_87'.

ztt0821 commented 8 months ago

Hi, I have solved the problem. It's the CUDA problem. I need to use the sudo su, then open the .bashrc and CUDA_HOME and LD_LIBRARY_PATH in it. The path is /usr/bin/cuda-11.7 not /usr/bin/cuda

ztt0821 commented 8 months ago

Thank you very much for your help!