search

conda-forge / nvcc-feedstock

A conda-smithy repository for nvcc.
BSD 3-Clause "New" or "Revised" License
12 stars 23 forks source link

Link error with ppc64le, but not sbsa arm #101

Closed carterbox closed 1 year ago

carterbox commented 1 year ago

Solution to issue cannot be found in the documentation.

Issue

When cross compiling on linux for ppc and arm, I get a linking error that the linker is trying to use the x86 libcuart.so instead of the powerpc one.

2023-09-28T19:39:52.2159854Z FAILED: lib/libmagma.so 
2023-09-28T19:39:52.2247552Z : && $BUILD_PREFIX/bin/powerpc64le-conda-linux-gnu-c++ -fPIC -fvisibility-inlines-hidden -fmessage-length=0 -mcpu=power8 -mtune=power8 -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/libmagma-2.7.1 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -isystem /usr/local/cuda/targets/ppc64le-linux/include -std=c++11 -DADD_ -fopenmp -Wall -Wno-unused-function -O3 -DNDEBUG  -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--allow-shlib-undefined -Wl,-rpath,$PREFIX/lib -Wl,-rpath-link,$PREFIX/lib -L$PREFIX/lib -shared -Wl,-soname,libmagma.so -o lib/libmagma.so CMakeFiles/magma.dir/control/magma_f77.cpp.o CMakeFiles/magma.dir/control/abs.cpp.o CMakeFiles/magma.dir/control/affinity.cpp.o
...
CMakeFiles/magma.dir/magmablas/dposv_irgmres_kernels.cu.o CMakeFiles/magma.dir/magmablas/sposv_irgmres_kernels.cu.o -L/usr/local/cuda/targets/ppc64le-linux/lib/stubs   -L/usr/local/cuda/targets/ppc64le-linux/lib -Wl,-rpath,/usr/local/cuda/lib64:  $PREFIX/lib/liblapack.so  $PREFIX/lib/libblas.so  /usr/local/cuda/lib64/libcudart.so  /usr/local/cuda/lib64/libcublas.so  /usr/local/cuda/lib64/libcusparse.so  /usr/local/cuda/lib64/libcublasLt.so  /usr/local/cuda/lib64/libculibos.a  -lcudadevrt  -lcudart_static  -lrt  -lpthread  -ldl && :
2023-09-28T19:39:52.2297538Z $BUILD_PREFIX/bin/../lib/gcc/powerpc64le-conda-linux-gnu/11.4.0/../../../../powerpc64le-conda-linux-gnu/bin/ld: /usr/local/cuda/lib64/libcudart.so: error adding symbols: file in wrong format
2023-09-28T19:39:52.2297827Z collect2: error: ld returned 1 exit status
2023-09-28T19:39:52.2298009Z ninja: build stopped: subcommand failed.

The same command doesn't fail for ARM

$BUILD_PREFIX/bin/aarch64-conda-linux-gnu-c++ -fPIC -fvisibility-inlines-hidden -fmessage-length=0 -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/libmagma-2.7.1 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -isystem /usr/local/cuda/targets/sbsa-linux/include -std=c++11 -DADD_ -fopenmp -Wall -Wno-unused-function -O3 -DNDEBUG  -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--allow-shlib-undefined -Wl,-rpath,$PREFIX/lib -Wl,-rpath-link,$PREFIX/lib -L$PREFIX/lib -shared -Wl,-soname,libmagma.so -o lib/libmagma.so CMakeFiles/magma.dir/control/magma_f77.cpp.o CMakeFiles/magma.dir/control/abs.cpp.o CMakeFiles/magma.dir/control/affinity.cpp.o CMakeFiles/magma.dir/control/auxiliary.cpp.o 
...
CMakeFiles/magma.dir/magmablas/cposv_irgmres_kernels.cu.o CMakeFiles/magma.dir/magmablas/dposv_irgmres_kernels.cu.o CMakeFiles/magma.dir/magmablas/sposv_irgmres_kernels.cu.o -L/usr/local/cuda/targets/sbsa-linux/lib/stubs   -L/usr/local/cuda/targets/sbsa-linux/lib -Wl,-rpath,/usr/local/cuda/targets/sbsa-linux/lib:  $PREFIX/lib/liblapack.so  $PREFIX/lib/libblas.so  /usr/local/cuda/targets/sbsa-linux/lib/libcudart.so  /usr/local/cuda/targets/sbsa-linux/lib/libcublas.so  /usr/local/cuda/targets/sbsa-linux/lib/libcusparse.so  /usr/local/cuda/targets/sbsa-linux/lib/libcublasLt.so  /usr/local/cuda/targets/sbsa-linux/lib/libculibos.a  -lcudadevrt  -lcudart_static  -lrt  -lpthread  -ldl && :

It looks like in some instances, /usr/local/cuda/lib64 is being used instead of /usr/local/cuda/targets/.

Installed packages

The following NEW packages will be INSTALLED:
2023-09-28T15:44:11.7472472Z 
2023-09-28T15:44:11.7472928Z     _libgcc_mutex:                             0.1-conda_forge            conda-forge
2023-09-28T15:44:11.7473330Z     _openmp_mutex:                             4.5-2_gnu                  conda-forge
2023-09-28T15:44:11.7473735Z     _sysroot_linux-ppc64le_curr_repodata_hack: 4-h43410cf_13              conda-forge
2023-09-28T15:44:11.7474217Z     binutils_impl_linux-64:                    2.40-hf600244_0            conda-forge
2023-09-28T15:44:11.7474614Z     binutils_impl_linux-ppc64le:               2.40-hef43fe1_0            conda-forge
2023-09-28T15:44:11.7474972Z     binutils_linux-ppc64le:                    2.40-h631d6fd_2            conda-forge
2023-09-28T15:44:11.7475368Z     bzip2:                                     1.0.8-h7f98852_4           conda-forge
2023-09-28T15:44:11.7475760Z     c-ares:                                    1.19.1-hd590300_0          conda-forge
2023-09-28T15:44:11.7476148Z     ca-certificates:                           2023.7.22-hbcca054_0       conda-forge
2023-09-28T15:44:11.7476530Z     cmake:                                     3.27.6-hcfe8598_0          conda-forge
2023-09-28T15:44:11.7476863Z     gcc_impl_linux-64:                         11.4.0-h7aa1c59_2          conda-forge
2023-09-28T15:44:11.7477258Z     gcc_impl_linux-ppc64le:                    11.4.0-h3afab18_2          conda-forge
2023-09-28T15:44:11.7477601Z     gcc_linux-ppc64le:                         11.4.0-haedd7f4_2          conda-forge
2023-09-28T15:44:11.7478213Z     gxx_impl_linux-64:                         11.4.0-h7aa1c59_2          conda-forge
2023-09-28T15:44:11.7478641Z     gxx_impl_linux-ppc64le:                    11.4.0-h3afab18_2          conda-forge
2023-09-28T15:44:11.7478981Z     gxx_linux-ppc64le:                         11.4.0-hf56eef1_2          conda-forge
2023-09-28T15:44:11.7479376Z     kernel-headers_linux-64:                   2.6.32-he073ed8_16         conda-forge
2023-09-28T15:44:11.7479734Z     kernel-headers_linux-ppc64le:              3.10.0-h23d7e6c_13         conda-forge
2023-09-28T15:44:11.7480135Z     keyutils:                                  1.6.1-h166bdaf_0           conda-forge
2023-09-28T15:44:11.7480504Z     krb5:                                      1.21.2-h659d440_0          conda-forge
2023-09-28T15:44:11.7480832Z     ld_impl_linux-64:                          2.40-h41732ed_0            conda-forge
2023-09-28T15:44:11.7481239Z     ld_impl_linux-ppc64le:                     2.40-h2f009d6_0            conda-forge
2023-09-28T15:44:11.7481569Z     libcurl:                                   8.3.0-hca28451_0           conda-forge
2023-09-28T15:44:11.7481949Z     libedit:                                   3.1.20191231-he28a2e2_2    conda-forge
2023-09-28T15:44:11.7482323Z     libev:                                     4.33-h516909a_1            conda-forge
2023-09-28T15:44:11.7482648Z     libexpat:                                  2.5.0-hcb278e6_1           conda-forge
2023-09-28T15:44:11.7483095Z     libffi:                                    3.4.2-h7f98852_5           conda-forge
2023-09-28T15:44:11.7483461Z     libgcc-devel_linux-64:                     11.4.0-h922705a_2          conda-forge
2023-09-28T15:44:11.7483850Z     libgcc-devel_linux-ppc64le:                11.4.0-hd3dd039_2          conda-forge
2023-09-28T15:44:11.7484230Z     libgcc-ng:                                 13.2.0-h807b86a_2          conda-forge
2023-09-28T15:44:11.7484562Z     libgomp:                                   13.2.0-h807b86a_2          conda-forge
2023-09-28T15:44:11.7484939Z     libnghttp2:                                1.52.0-h61bc06f_0          conda-forge
2023-09-28T15:44:11.7485266Z     libnsl:                                    2.0.0-hd590300_1           conda-forge
2023-09-28T15:44:11.7485653Z     libsanitizer:                              11.4.0-h4dcbe23_2          conda-forge
2023-09-28T15:44:11.7485977Z     libsqlite:                                 3.43.0-h2797004_0          conda-forge
2023-09-28T15:44:11.7486443Z     libssh2:                                   1.11.0-h0841786_0          conda-forge
2023-09-28T15:44:11.7486827Z     libstdcxx-devel_linux-64:                  11.4.0-h922705a_2          conda-forge
2023-09-28T15:44:11.7487172Z     libstdcxx-devel_linux-ppc64le:             11.4.0-hd3dd039_2          conda-forge
2023-09-28T15:44:11.7487565Z     libstdcxx-ng:                              13.2.0-h7e041cc_2          conda-forge
2023-09-28T15:44:11.7487960Z     libuuid:                                   2.38.1-h0b41bf4_0          conda-forge
2023-09-28T15:44:11.7488416Z     libuv:                                     1.46.0-hd590300_0          conda-forge
2023-09-28T15:44:11.7488787Z     libzlib:                                   1.2.13-hd590300_5          conda-forge
2023-09-28T15:44:11.7489111Z     make:                                      4.3-hd18ef5c_1             conda-forge
2023-09-28T15:44:11.7489476Z     ncurses:                                   6.4-hcb278e6_0             conda-forge
2023-09-28T15:44:11.7489798Z     ninja:                                     1.11.1-h924138e_0          conda-forge
2023-09-28T15:44:11.7490195Z     nvcc_linux-ppc64le:                        11.8-hb69a355_22           conda-forge
2023-09-28T15:44:11.7490569Z     openssl:                                   3.1.3-hd590300_0           conda-forge
2023-09-28T15:44:11.7490921Z     python:                                    3.10.12-hd12c33a_0_cpython conda-forge
2023-09-28T15:44:11.7491305Z     readline:                                  8.2-h8228510_1             conda-forge
2023-09-28T15:44:11.7491625Z     rhash:                                     1.4.4-hd590300_0           conda-forge
2023-09-28T15:44:11.7683607Z     sed:                                       4.8-he412f7d_0             conda-forge
2023-09-28T15:44:11.7684564Z     sysroot_linux-64:                          2.12-he073ed8_16           conda-forge
2023-09-28T15:44:11.7684954Z     sysroot_linux-ppc64le:                     2.17-h23d7e6c_13           conda-forge
2023-09-28T15:44:11.7685324Z     tk:                                        8.6.13-h2797004_0          conda-forge
2023-09-28T15:44:11.7685672Z     tzdata:                                    2023c-h71feb2d_0           conda-forge
2023-09-28T15:44:11.7686013Z     xz:                                        5.2.6-h166bdaf_0           conda-forge
2023-09-28T15:44:11.7686355Z     zstd:                                      1.5.5-hfc55251_0           conda-forge

Environment info

https://github.com/conda-forge/libmagma-feedstock/pull/8

isuruf commented 1 year ago

The same command doesn't fail for ARM

Those are two different commands. For eg: /usr/local/cuda/lib64/libcudart.so for ppc64le vs /usr/local/cuda/targets/sbsa-linux/lib/libcudart.so for aarch64.

carterbox commented 1 year ago

I think my issue may be related to find_package(CUDAToolkit) picking up the host platform instead of the target. I'm going to try overriding the -DCUDAToolkit_ROOT from /usr/local/cuda to /usr/local/cuda/targets/.

carterbox commented 1 year ago

Probably, this line should be updated so it points to the target prefix. i.e.

-DCUDAToolkit_ROOT="${CUDA_HOME}/targets/ppc64le-linux"