search

masadcv / FastGeodis

Fast Implementation of Generalised Geodesic Distance Transform for CPU (OpenMP) and GPU (CUDA)
https://fastgeodis.readthedocs.io
BSD 3-Clause "New" or "Revised" License
90 stars 14 forks source link

[BUG] Not compiled with CUDA support. #35

Closed joshmyersdean closed 1 year ago

joshmyersdean commented 2 years ago

Describe the bug Thank you for the work! FastGeodis is not being installed with CUDA support no matter the install method.

To Reproduce Steps to reproduce the behavior:

  1. pip install FastGeodis --no-build-isolation
  2. See below example.
  3. See error below.

example

import FastGeodis as fd
import torch
device = torch.device('cuda')
a = torch.rand(1,1,512,512).to(device)
b = torch.ones_like(a).to(device)
fd.generalised_geodesic2d(a, b, 256, 0., 1)

error

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [1], in <cell line: 6>()
      4 a = torch.rand(1,1,512,512).to(device)
      5 b = torch.ones_like(a).to(device)
----> 6 fd.generalised_geodesic2d(a, b, 256, 0., 1)

File /opt/conda/envs/deepselect/lib/python3.8/site-packages/FastGeodis/__init__.py:62, in generalised_geodesic2d(image, softmask, v, lamb, iter)
     36 def generalised_geodesic2d(
     37     image: torch.Tensor, 
     38     softmask: torch.Tensor, 
   (...)
     41     iter: int = 2
     42 ):
     43     r"""Computes Generalised Geodesic Distance using FastGeodis raster scanning.
     44     For more details on generalised geodesic distance, check the following reference:
     45 
   (...)
     60         torch.Tensor with distance transform
     61     """
---> 62     return FastGeodisCpp.generalised_geodesic2d(
     63         image, softmask, v, lamb, 1 - lamb, iter
     64     )

RuntimeError: Not compiled with CUDA support.

Expected behavior Distance map calculation on GPU.

Desktop (please complete the following information):

Thank you for the amazing work!

joshmyersdean commented 2 years ago

Here are the install notes as well - 

$ pip install FastGeodis --no-build-isolation                                                                                                                                                                                                                                                                                                                            
Collecting FastGeodis                                                                                                                                                                                                                                                                                                                                                                                                              
  Downloading FastGeodis-1.0.0rc9.tar.gz (22 kB)                                                                                                                                                                                                                                                                                                                                                                                   
  Preparing metadata (pyproject.toml) ... done                                                                                                                                                                                                                                                                                                                                                                                     
Requirement already satisfied: torch>=1.5.0 in /opt/conda/envs/deepselect/lib/python3.8/site-packages (from FastGeodis) (1.12.0+cu113)                                                                                                                                                                                                                                                                                             
Requirement already satisfied: typing-extensions in /opt/conda/envs/deepselect/lib/python3.8/site-packages (from torch>=1.5.0->FastGeodis) (4.3.0)                                                                                                                                                                                                                                                                                 
Building wheels for collected packages: FastGeodis                                                                                                                                                                                                                                                                                                                                                                                 
  Building wheel for FastGeodis (pyproject.toml) ... done                                                                                                                                                                                                                                                                                                                                                                          
  Created wheel for FastGeodis: filename=FastGeodis-1.0.0rc9-cp38-cp38-linux_x86_64.whl size=101116 sha256=8825fec901214cdf267ffb2e596ead31d2dd8afd616f781a56da43ca231eaf3e                                                                                                                                                                                                                                                        
  Stored in directory: /home/jmyersdean/.cache/pip/wheels/6e/62/df/61076661a371b1c14d8fb3a8ff5f218b4d3e215f8845394384                                                                                                                                                                                                                                                                                                              
Successfully built FastGeodis                                                                           
Installing collected packages: FastGeodis                                                               
Successfully installed FastGeodis-1.0.0rc9
masadcv commented 2 years ago

Hi,

Many thanks for submitting a bug report for this issue. From your initial information, it seems like something on your system is preventing CUDA compilation of the library. I need more information to debug and help you fix this issue. Do you have an NVCC compiler installed as well? If so, what is the output of:

$ nvcc --version

Can you also try the following commands and copy the output of the compilation:

$ git clone https://github.com/masadcv/FastGeodis
$ cd FastGeodis
$ python3 setup.py build
joshmyersdean commented 1 year ago

Thank you for the quick response!

I did not have nvcc prior as I was using the cuda toolkit installed in conda. I attempted to install nvcc but borked my CUDA versions in the process.

The output of the latter block is:

setup.py with torch 1.12.0+cu113
BUILD_CPP=True, BUILD_CUDA=False, TORCH_VERSION=11200.
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.8
creating build/lib.linux-x86_64-3.8/FastGeodis
copying FastGeodis/__init__.py -> build/lib.linux-x86_64-3.8/FastGeodis
running build_ext
building 'FastGeodisCpp' extension
creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/FastGeodis
gcc -pthread -B /opt/conda/envs/deepselect/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DAT_PARALLEL_OPENMP=1 -IFastGeodis -I/opt/conda/envs/deepselect/lib/python3.8/site-packages/torch/include -I/opt/conda/envs/deepselect/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/envs/deepselect/lib/python3.8/site-packages/torch/include/TH -I/opt/conda/envs/deepselect/lib/python3.8/site-packages/torch/include/THC -I/opt/conda/envs/deepselect/include/python3.8 -c FastGeodis/fastgeodis.cpp -o build/temp.linux-x86_64-3.8/FastGeodis/fastgeodis.o -fopenmp -g0 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1013\" -DTORCH_EXTENSION_NAME=FastGeodisCpp -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
gcc -pthread -B /opt/conda/envs/deepselect/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DAT_PARALLEL_OPENMP=1 -IFastGeodis -I/opt/conda/envs/deepselect/lib/python3.8/site-packages/torch/include -I/opt/conda/envs/deepselect/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/envs/deepselect/lib/python3.8/site-packages/torch/include/TH -I/opt/conda/envs/deepselect/lib/python3.8/site-packages/torch/include/THC -I/opt/conda/envs/deepselect/include/python3.8 -c FastGeodis/fastgeodis_cpu.cpp -o build/temp.linux-x86_64-3.8/FastGeodis/fastgeodis_cpu.o -fopenmp -g0 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1013\" -DTORCH_EXTENSION_NAME=FastGeodisCpp -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
gcc -pthread -B /opt/conda/envs/deepselect/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DAT_PARALLEL_OPENMP=1 -IFastGeodis -I/opt/conda/envs/deepselect/lib/python3.8/site-packages/torch/include -I/opt/conda/envs/deepselect/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/envs/deepselect/lib/python3.8/site-packages/torch/include/TH -I/opt/conda/envs/deepselect/lib/python3.8/site-packages/torch/include/THC -I/opt/conda/envs/deepselect/include/python3.8 -c FastGeodis/geodis_fastmarch.cpp -o build/temp.linux-x86_64-3.8/FastGeodis/geodis_fastmarch.o -fopenmp -g0 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1013\" -DTORCH_EXTENSION_NAME=FastGeodisCpp -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
gcc -pthread -B /opt/conda/envs/deepselect/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DAT_PARALLEL_OPENMP=1 -IFastGeodis -I/opt/conda/envs/deepselect/lib/python3.8/site-packages/torch/include -I/opt/conda/envs/deepselect/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/envs/deepselect/lib/python3.8/site-packages/torch/include/TH -I/opt/conda/envs/deepselect/lib/python3.8/site-packages/torch/include/THC -I/opt/conda/envs/deepselect/include/python3.8 -c FastGeodis/geodis_toivanen.cpp -o build/temp.linux-x86_64-3.8/FastGeodis/geodis_toivanen.o -fopenmp -g0 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1013\" -DTORCH_EXTENSION_NAME=FastGeodisCpp -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
g++ -pthread -B /opt/conda/envs/deepselect/compiler_compat -Wl,--sysroot=/ -pthread -shared -B /opt/conda/envs/deepselect/compiler_compat -L/opt/conda/envs/deepselect/lib -Wl,-rpath=/opt/conda/envs/deepselect/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.8/FastGeodis/fastgeodis.o build/temp.linux-x86_64-3.8/FastGeodis/fastgeodis_cpu.o build/temp.linux-x86_64-3.8/FastGeodis/geodis_fastmarch.o build/temp.linux-x86_64-3.8/FastGeodis/geodis_toivanen.o -L/opt/conda/envs/deepselect/lib/python3.8/site-packages/torch/lib -lc10 -ltorch -ltorch_cpu -ltorch_python -o build/lib.linux-x86_64-3.8/FastGeodisCpp.cpython-38-x86_64-linux-gnu.so -fopenmp

It appears the BUILD_CUDA flag is false?

masadcv commented 1 year ago

Thank you for the quick response!

I did not have nvcc prior as I was using the cuda toolkit installed in conda. I attempted to install nvcc but borked my CUDA versions in the process.

Ooops, if it is a driver issue, maybe this guide can help you fix it: https://github.com/luiscarlosgph/how-to/tree/main/nvidia-driver

Another option to get around such nvcc/nvidia tooling issues is to use a ngc pytorch docker image from here: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch

The output of the latter block is:

setup.py with torch 1.12.0+cu113
BUILD_CPP=True, BUILD_CUDA=False, TORCH_VERSION=11200.
...

It appears the BUILD_CUDA flag is false?

It seems that BUILD_CUDA is indeed false. This is set in the following line in setup script: https://github.com/masadcv/FastGeodis/blob/master/setup.py#L24

It seems the pytorch version picked up for compilation on your machine may only support CPU.

Can you try the following command in the same environment and paste the output here:

python3 -c "import torch; print(torch.cuda.is_available())"

It should print true if torch with CUDA support is picked up.

Additionally, can you paste the output of the following:

pip3 list
masadcv commented 1 year ago

Also if you havent already, can you add cuda folder to your path variables. For example, I have the following set on my test machine:

$ export | grep cuda
declare -x LD_LIBRARY_PATH="/usr/local/cuda-11.3/lib64:"
declare -x PATH="/usr/local/cuda-11.3/bin:"
masadcv commented 1 year ago

I also have some how-to to setup CUDA Toolkit (https://github.com/masadcv/SettingUpUbuntu#setting-up-cuda-toolkit) and the right gcc compiler for nvcc (https://github.com/masadcv/SettingUpUbuntu#setting-up-gcc-8-for-cuda-102) - in case this could help you setup nvcc/cuda...

joshmyersdean commented 1 year ago

Thank you so much for the guidance! I do have CUDA support with my PyTorch install but I may be missing some path variables as you pointed out. I believe this is a me problem and not a software problem so I'll go ahead and close this issue.

Thank you again, great package!

masadcv commented 1 year ago

Many thanks @joshmyersdean ! I am glad I could be of help! Do share any suggestions/feature requests that you come across for this... All the best!