NVIDIA / spark-rapids-jni

RAPIDS Accelerator JNI For Apache Spark
Apache License 2.0
32 stars 60 forks source link

[BUG] libcudf.so being built with dynamic CUDA runtime dependency #2162

Closed jlowe closed 2 weeks ago

jlowe commented 2 weeks ago

The nightly builds are producing a jar that has a libcudf.so with a dynamic CUDA runtime dependency. The CUDA runtime dependency is supposed to be static, and this can cause issues with environments that don't provide the CUDA 11 runtime library. Note the missing libcudart.so.11.0 dependency on libcudf.so below:

$ jar xvf ~/.m2/repository/com/nvidia/spark-rapids-jni/24.08.0-SNAPSHOT/spark-rapids-jni-24.08.0-SNAPSHOT-cuda11.jar amd64/Linux/libcudf.so
 inflated: amd64/Linux/libcudf.so
$ ldd amd64/Linux/libcudf.so
    linux-vdso.so.1 (0x00007ffded5e8000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x0000737afc368000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x0000737afc34c000)
    libnvcomp.so => /home/jlowe/src/spark-rapids/amd64/Linux/libnvcomp.so (0x0000737ad2c00000)
    libnvcomp_gdeflate.so => /home/jlowe/src/spark-rapids/amd64/Linux/libnvcomp_gdeflate.so (0x0000737ad1c00000)
    libnvcomp_bitcomp.so => /home/jlowe/src/spark-rapids/amd64/Linux/libnvcomp_bitcomp.so (0x0000737ad0800000)
    libcudart.so.11.0 => not found
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x0000737afc345000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x0000737afc340000)
    libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x0000737ad0400000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x0000737afc259000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x0000737afc237000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000737ad0000000)
    /lib64/ld-linux-x86-64.so.2 (0x0000737afc384000)
jlowe commented 2 weeks ago

Something is adding /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudart.so to the link libraries in addition to the static CUDA runtime library. So we're both dynamically and statically linking the CUDA runtime libraries.