LD_LIBRARY_PATH may not be well configured in Singurarity image

[ItokawaK]

Hi,

Thank you for your interesting tool.

I got a following error when I executed herro inference in Singurarity.

$ singurarity run --nv herro.sif inference <args>
[00:01:02] Parsed 393084 reads.                                                                                                                                  
[00:00:16] Processing 1/? batch ⠠
[>---------------------------------------] 2/75403                                                                                                               
Could not load library libcudnn_cnn_infer.so.8. Error: libcuda.so: cannot open shared object file: No such file or directory
Aborted

So, I entered into the bash console first, configurered the LD_LIBRARY_PATH environmental variable inside, and executed herro inference as below.

$ singurarity exec --nv herro.sif bash
Singularity> export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.7/compat/
Singularity> herro inference <args>

Then, it worked.

Best

[dominikstanojevic]

Hello,

thank you for reporting the issue.

Unfortunately, I cannot reproduce the problem on the two test machines using a pre-build singularity image. Can you please specify if you're using a pre-build singularity image (or you built it from scratch) and a singularity version?

Best regrads, Dominik

[ItokawaK]

Hi Dominik,

I am using the pre-build image downloaded from aws S3.

Other specs are as follow. OS: Ubuntu 20.04 LTS singularity-ce: version 4.0.2

Thanks, Kentaro

[dominikstanojevic]

Hi Kentaro,

can you run echo $LD_LIBRARY_PATH after singurarity exec --nv herro.sif bash and paste the output?

Moreover, can you paste the output from ls -l /.singularity.d/libs? Thanks.

Best regards, Dominik

[ItokawaK]

Hi Dominik,

Thank you for looking into this issue.

$ singularity exec --nv ~/tools/herro/herro.sif bash
Singularity> echo $LD_LIBRARY_PATH 
/libs/libtorch/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/.singularity.d/libs
Singularity> ls -l /.singularity.d/libs/
-rw-r--r-- 1 root root     80512 Mar  8  2022 libEGL.so
-rw-r--r-- 1 root root     80512 Mar  8  2022 libEGL.so.1
-rw-r--r-- 1 root root   1333232 May 30  2023 libEGL_nvidia.so.0
-rw-r--r-- 1 root root    547152 Mar  8  2022 libGL.so
-rw-r--r-- 1 root root    547152 Mar  8  2022 libGL.so.1
-rw-r--r-- 1 root root     43336 Mar  8  2022 libGLESv1_CM.so
-rw-r--r-- 1 root root     43336 Mar  8  2022 libGLESv1_CM.so.1
-rw-r--r-- 1 root root     67880 May 30  2023 libGLESv1_CM_nvidia.so.1
-rw-r--r-- 1 root root     72008 Mar  8  2022 libGLESv2.so
-rw-r--r-- 1 root root     72008 Mar  8  2022 libGLESv2.so.2
-rw-r--r-- 1 root root    117032 May 30  2023 libGLESv2_nvidia.so.2
-rw-r--r-- 1 root root    141912 Mar  8  2022 libGLX.so
-rw-r--r-- 1 root root    141912 Mar  8  2022 libGLX.so.0
-rw-r--r-- 1 root root   1191352 May 30  2023 libGLX_nvidia.so.0
-rw-r--r-- 1 root root    715200 Mar  8  2022 libGLdispatch.so
-rw-r--r-- 1 root root    715200 Mar  8  2022 libGLdispatch.so.0
-rwxr-xr-x 1 root root     30856 Mar  9  2023 libOpenCL.so
-rwxr-xr-x 1 root root     30856 Mar  9  2023 libOpenCL.so.1
-rw-r--r-- 1 root root    174408 Mar  8  2022 libOpenGL.so
-rw-r--r-- 1 root root    174408 Mar  8  2022 libOpenGL.so.0
-rw-r--r-- 1 root root  29867944 May 30  2023 libcuda.so.1
-rw-r--r-- 1 root root   7781216 May 30  2023 libnvcuvid.so.1
-rw-r--r-- 1 root root    262512 May 30  2023 libnvidia-cfg.so.1
-rw-r--r-- 1 root root  56309912 May 30  2023 libnvidia-compiler.so.525.125.06
-rw-r--r-- 1 root root  42054840 May 30  2023 libnvidia-eglcore.so.525.125.06
-rw-r--r-- 1 root root    198720 May 30  2023 libnvidia-encode.so.1
-rw-r--r-- 1 root root    133816 May 30  2023 libnvidia-fbc.so.1
-rw-r--r-- 1 root root  44282616 May 30  2023 libnvidia-glcore.so.525.125.06
-rw-r--r-- 1 root root    652264 May 30  2023 libnvidia-glsi.so.525.125.06
-rw-r--r-- 1 root root  25056984 May 30  2023 libnvidia-glvkspirv.so.525.125.06
-rw-r--r-- 1 root root   1375912 Jan 19  2022 libnvidia-gtk2.so.470.57.01
-rw-r--r-- 1 root root   1384584 Jan 19  2022 libnvidia-gtk3.so.470.57.01
-rw-r--r-- 1 root root   1827448 May 30  2023 libnvidia-ml.so.1
-rw-r--r-- 1 root root  85957696 May 30  2023 libnvidia-nvvm.so.4
-rw-r--r-- 1 root root  22991312 May 30  2023 libnvidia-opencl.so.1
-rw-r--r-- 1 root root     67584 May 30  2023 libnvidia-opticalflow.so.1
-rw-r--r-- 1 root root  20727192 May 30  2023 libnvidia-ptxjitcompiler.so.1
-rw-r--r-- 1 root root 118080512 May 30  2023 libnvidia-rtcore.so.525.125.06
-rw-r--r-- 1 root root     18528 May 30  2023 libnvidia-tls.so.525.125.06
-rw-r--r-- 1 root root 201324280 May 30  2023 libnvoptix.so.1

Best, Kentatro

[dominikstanojevic]

Hi Kentaro,

I was checking on the third machine (with quite an old CUDA version) but still cannot reproduce the issue. Can you give me more info about your CUDA and NVIDIA driver versions?

Best regards, Dominik

[ItokawaK]

Hi Dominik,

I have only one GPU, NVIDIA RTX A6000.

nvidia-smi told

Driver Version: 525.125.06 CUDA Version: 12.0

I saw same result when I executed nvidia-smi in Singularity container.

Best, Kentaro