Building with CMake fails to compile the GPU programs

[yuankailiu]

I am trying to use CMake to install a featured branch of ISCE2 from @CunrenLiang with GPUs functioning for topsApp.py.

Instead of using Scons compiler to use GPU, I use CMake to build ISCE here, following the guidelines from GitHub provided by @yunjunz. Summarized shortly as below:

# Download source code
cd ~/apps
mkdir isce2; cd isce2
mkdir build install src; cd src
git clone https://github.com/CunrenLiang/isce2#isce2

# load CUDA and GNU compilers module pre-installed on server
module load cuda/10.1
module load /home/geomod/apps/rhel7/modules/gcc/7.3.1

# generate build system for ISCE-2 (use pre-installed GCC-7.3.1)
cd ~/apps/isce2/build
export GCC_BIN=/net/kraken/home1/geomod/apps/rhel7/gcc7/bin
cmake ~/apps/isce2/src/isce2 -DCMAKE_INSTALL_PREFIX=~/apps/isce2/install -DCMAKE_CUDA_FLAGS="-arch=sm_60" -DCMAKE_PREFIX_PATH=${CONDA_PREFIX} -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=${GCC_BIN}/gcc -DCMAKE_CXX_COMPILER=${GCC_BIN}/g++ -DCMAKE_Fortran_COMPILER=${GCC_BIN}/gfortran

# compile and install
make -j 16
make install

Now I am able to run topsApp.py without problems, but the GPU module is not loaded properly. So the application is still running CPU instead of GPU in certain steps.

I did the following checks in the python3 environment:

import isce, isceobj
from zerodop.GPUtopozero.GPUtopozero import PyTopozero
from zerodop.GPUgeo2rdr.GPUgeo2rdr import PyGeo2rdr

Importing isce and isceobj works fine, with normal output: This is the Open Source version of ISCE...

But importing from zerodop doesn't work. So CMake did not correctly compile the GPU programs.

ModuleNotFoundError: No module named 'zerodop'

My CMakeLists didn’t include the cuda modules under zerodop:

my CMakeLists while building isce (click)

``` -- The C compiler identification is GNU 7.3.1 -- The CXX compiler identification is GNU 7.3.1 -- The Fortran compiler identification is GNU 7.3.1 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /net/kraken/home1/geomod/apps/rhel7/gcc7/bin/gcc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /net/kraken/home1/geomod/apps/rhel7/gcc7/bin/g++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Detecting Fortran compiler ABI info -- Detecting Fortran compiler ABI info - done -- Check for working Fortran compiler: /net/kraken/home1/geomod/apps/rhel7/gcc7/bin/gfortran - skipped -- Checking whether /net/kraken/home1/geomod/apps/rhel7/gcc7/bin/gfortran supports Fortran 90 -- Checking whether /net/kraken/home1/geomod/apps/rhel7/gcc7/bin/gfortran supports Fortran 90 - yes -- Looking for a CUDA compiler -- Looking for a CUDA compiler - /usr/local/cuda-10.1/bin/nvcc -- The CUDA compiler identification is NVIDIA 10.1.243 -- Detecting CUDA compiler ABI info -- Detecting CUDA compiler ABI info - done -- Check for working CUDA compiler: /usr/local/cuda-10.1/bin/nvcc - skipped -- Detecting CUDA compile features -- Detecting CUDA compile features - done -- Found CUDAToolkit: /usr/local/cuda-10.1/include (found version "10.1.243") -- Looking for pthread.h -- Looking for pthread.h - found -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE -- Found Python: /home/ykliu/apps/Anaconda/envs/insar/bin/python3.8 (found suitable version "3.8.6", minimum required is "3.5") found components: Interpreter Development NumPy Development.Module Development.Embed -- Found PkgConfig: /usr/bin/pkg-config (found version "0.27.1") -- Found FFTW: /home/ykliu/apps/Anaconda/envs/insar/lib/libfftw3f.so;/home/ykliu/apps/Anaconda/envs/insar/lib/libfftw3f_threads.so;/home/ykliu/apps/Anaconda/envs/insar/lib/libfftw3f_omp.so;/home/ykliu/apps/Anaconda/envs/insar/lib/libfftw3.so;/home/ykliu/apps/Anaconda/envs/insar/lib/libfftw3_threads.so;/home/ykliu/apps/Anaconda/envs/insar/lib/libfftw3_omp.so;/home/ykliu/apps/Anaconda/envs/insar/lib/libfftw3l.so;/home/ykliu/apps/Anaconda/envs/insar/lib/libfftw3l_threads.so;/home/ykliu/apps/Anaconda/envs/insar/lib/libfftw3l_omp.so -- Found Motif: /home/ykliu/apps/Anaconda/envs/insar/lib/libXm.so -- Found OpenMP_C: -fopenmp (found version "4.5") -- Found OpenMP_CXX: -fopenmp (found version "4.5") -- Found OpenMP_Fortran: -fopenmp (found version "4.5") -- Found OpenMP: TRUE (found version "4.5") found components: C CXX Fortran -- Found OpenCV: /home/ykliu/apps/Anaconda/envs/insar (found version "4.5.0") found components: core highgui imgproc -- Found GDAL: /home/ykliu/apps/Anaconda/envs/insar/lib/libgdal.so (found version "3.1.4") -- Found X11: /home/ykliu/apps/Anaconda/envs/insar/include found components: Xau Xt -- Looking for XOpenDisplay in /home/ykliu/apps/Anaconda/envs/insar/lib/libX11.so;/home/ykliu/apps/Anaconda/envs/insar/lib/libXext.so -- Looking for XOpenDisplay in /home/ykliu/apps/Anaconda/envs/insar/lib/libX11.so;/home/ykliu/apps/Anaconda/envs/insar/lib/libXext.so - found -- Looking for gethostbyname -- Looking for gethostbyname - found -- Looking for connect -- Looking for connect - found -- Looking for remove -- Looking for remove - found -- Looking for shmat -- Looking for shmat - found -- Looking for IceConnectionNumber in ICE -- Looking for IceConnectionNumber in ICE - found -- Found Cython: /home/ykliu/apps/Anaconda/envs/insar/bin/python3.8;-m;cython -- Found Python: /home/ykliu/apps/Anaconda/envs/insar/include/python3.8 (found version "3.8.6") found components: Development Development.Module Development.Embed -- Found Cython: 0.29.21 -- Performing Test C_FNO_COMMON -- Performing Test C_FNO_COMMON - Success -- Performing Test CXX_FNO_COMMON -- Performing Test CXX_FNO_COMMON - Success -- Configuring done -- Generating done -- Build files have been written to: /home/ykliu/apps/isce2/build ```

Given that a certain version of CUDA only supports certain versions of GNU compilers (the guidence from @lijun99), I've also tried a different version of GNU compilers for CUDA 10.1 (GNU<=7.3 installed from conda): conda install gcc_linux-64=7.3.0 gxx_linux-64=7.3.0 gfortran_linux-64=7.3.0 But it gives the same outcome.

Is it possible to include this feature in future updates to enable CMake to compile GPU programs?

I am not familiar with many things here. So if once find anything not clear, please just chime in or ask me for further explanation, info, etc.

Thank you for any comments and feedback!

[rtburns-jpl]

Hi, thanks for the detailed bug report. I'm not sure what went wrong, because the cmake output looks correct. Could you post your CMakeFiles/CMakeError.log and CMakeCache.txt from your build directory? Also, maybe you can run this python script and tell me what it says about your installation:

import isce
import sys
print(isce.__file__)
print(sys.path)

[lijun99]

Ryan, could you please add CMakefiles for GPUtopozero and GPUresampslc, under https://github.com/isce-framework/isce2/tree/main/components/zerodop?

[rtburns-jpl]

Oh you're right, those are missing - should be an easy fix. But if that's the issue here, I'm surprised that python said zerodop is missing and not GPUtopozero.

[lijun99]

I can load zerodop. It might be a path-setting problem. I can help him with that later.

from zerodop.GPUtopozero.GPUtopozero import PyTopozero Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'zerodop.GPUtopozero'

[yuankailiu]

Sorry for the late response.

print(isce.__file__)

/home/ykliu/apps/isce2/install_cunrenl/packages/isce/__init__.py

print(sys.path)

['/home/ykliu/apps/Anaconda/envs/insar/bin', '/home/ykliu/apps/isce2/install_cunrenl/packages/isce/library', '/home/ykliu/apps/isce2/install_cunrenl/packages/isce/components', '/home/ykliu/apps/isce2/install_cunrenl/packages/isce/applications', '/home/ykliu/apps/isce2/install_cunrenl/packages/isce', '/home/ykliu', '/home/ykliu/apps/MintPy', '/home/ykliu/apps/PyAPS', '/home/ykliu/apps/PySolid', '/home/ykliu/apps/ARIA-tools/tools', '/home/ykliu/apps/isce2/install_cunrenl/packages', '/home/ykliu/apps/isce2/src/isce2/contrib/stack', '/home/ykliu/apps/FRInGE/install/python', '/home/ykliu/apps/Anaconda/envs/insar/lib/python38.zip', '/home/ykliu/apps/Anaconda/envs/insar/lib/python3.8', '/home/ykliu/apps/Anaconda/envs/insar/lib/python3.8/lib-dynload', '', '/home/ykliu/apps/Anaconda/envs/insar/lib/python3.8/site-packages', '/home/ykliu/apps/Anaconda/envs/insar/lib/python3.8/site-packages/IPython/extensions', '/home/ykliu/.ipython']

Hope this clears things up if this is just my problem with path settings.

CMakeCache.txt

CMakeError.log

[yuankailiu]

Thanks! That works for me now.

@lijun99 further told me that there is some control inside the CUDA code, which needs to be enabled to run on GPU.

The control is set on SCons script. We will also need to add it to the CMake configuration.

So, for the below the CMake List files under isce2/components/components/zerodop:

GPUtopozero/CMakeLists.txt
GPUgeo2rdr/CMakeLists.txt
GPUresampslc/CMakeLists.txt       # this module is not called though

we need to add this line at the very beginning:

add_definitions(-DGPU_ACC_ENABLED)

[lijun99]

The "modern" way seems to be target_compile_definitions. I have created a PR #231 accordingly. Please check.

[yuankailiu]

Thanks for all your updates.