search

JuliaGPU / CuArrays.jl

A Curious Cumulation of CUDA Cuisine
https://juliagpu.org/cuda/
Other
280 stars 83 forks source link

`import CuArrays` always fails with CUDA 10.2.89 (but works fine with CUDA 10.0.130 and 10.1.105) #601

Closed DilumAluthge closed 4 years ago

DilumAluthge commented 4 years ago

Summary

I am unable to run import CuArrays with CUDA 10.2.89. However, I am able to successfully run import CuArrays with either CUDA 10.0.130 or CUDA 10.1.105 on the same cluster. (This is an HPC cluster with multiple different versions of CUDA available.)

The error I get looks like this:

┌ Error: CuArrays.jl failed to initialize
│   exception =
│    could not load library "libcublas"
│    libcublas.so: cannot open shared object file: No such file or directory
│    Stacktrace:
│     [1] dlopen(::String, ::UInt32; throw_error::Bool) at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109
│     [2] dlopen at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109 [inlined] (repeats 2 times)
│     [3] (::CuArrays.CUBLAS.var"#509#lookup_fptr#28")() at /users/daluthge/.julia/packages/CUDAapi/wYUAO/src/call.jl:29
│     [4] macro expansion at /users/daluthge/.julia/packages/CUDAapi/wYUAO/src/call.jl:37 [inlined]
│     [5] macro expansion at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/error.jl:65 [inlined]
│     [6] cublasGetProperty at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/libcublas.jl:27 [inlined]
│     [7] cublasGetProperty at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/wrappers.jl:38 [inlined]
│     [8] version() at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/wrappers.jl:42
│     [9] __init__() at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/CuArrays.jl:98
│     [10] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:697
│     [11] _require_from_serialized(::String) at ./loading.jl:748
│     [12] _require(::Base.PkgId) at ./loading.jl:1039
│     [13] require(::Base.PkgId) at ./loading.jl:927
│     [14] require(::Module, ::Symbol) at ./loading.jl:922
│     [15] eval(::Module, ::Any) at ./boot.jl:331
│     [16] eval_user_input(::Any, ::REPL.REPLBackend) at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:118
│     [17] macro expansion at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:150 [inlined]
│     [18] (::REPL.var"#31#32"{REPL.REPLBackend})() at ./task.jl:358
└ @ CuArrays ~/.julia/packages/CuArrays/HE8G6/src/CuArrays.jl:141

How to reproduce

First run these commands in Bash:

export JULIA_CUDA_VERBOSE="true"
export JULIA_DEBUG="all"
rm -rf ~/.julia

Then open Julia and run the following:

julia> versioninfo(verbose = true)

julia> import Pkg

julia> Pkg.add("CuArrays")

julia> import CuArrays

Full output

CUDA 10.2.89: (fails)

Click to expand ```bash $ which nvcc /gpfs/runtime/opt/cuda/10.2/cuda/bin/nvcc $ which nvdisasm /gpfs/runtime/opt/cuda/10.2/cuda/bin/nvdisasm $ nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2019 NVIDIA Corporation Built on Wed_Oct_23_19:24:38_PDT_2019 Cuda compilation tools, release 10.2, V10.2.89 $ nvdisasm --version nvdisasm: NVIDIA (R) CUDA disassembler Copyright (c) 2005-2019 NVIDIA Corporation Built on Wed_Oct_23_19:25:30_PDT_2019 Cuda compilation tools, release 10.2, V10.2.89 ``` ```julia _ _ _ _(_)_ | Documentation: https://docs.julialang.org (_) | (_) (_) | _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help. | | | | | | |/ _` | | | | |_| | | | (_| | | Version 1.5.0-DEV.274 (2020-02-15) _/ |\__'_|_|_|\__'_| | Commit 8eb0f9fefb (8 days old master) |__/ | julia> versioninfo(verbose = true) Julia Version 1.5.0-DEV.274 Commit 8eb0f9fefb (2020-02-15 12:41 UTC) Platform Info: OS: Linux (x86_64-pc-linux-gnu) "Red Hat Enterprise Linux Server release 7.3 (Maipo)" uname: Linux 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64 CPU: Intel(R) Xeon(R) Gold 5122 CPU @ 3.60GHz: speed user nice sys idle irq #1 3601 MHz 260888663 s 209 s 79076485 s 314462620 s 0 s #2 3601 MHz 225435476 s 556 s 66634601 s 362178032 s 0 s #3 3601 MHz 153758996 s 331 s 47119890 s 453453880 s 0 s #4 3601 MHz 133428383 s 355 s 44200083 s 476426058 s 0 s #5 3601 MHz 231463336 s 286 s 61205976 s 362261076 s 0 s #6 3601 MHz 154414432 s 604 s 45161270 s 455708660 s 0 s #7 3601 MHz 104946172 s 268 s 34510105 s 515709887 s 0 s #8 3601 MHz 94884882 s 496 s 31751569 s 528198402 s 0 s Memory: 93.04103088378906 GB (80644.640625 MB free) Uptime: 6.578586e6 sec Load Avg: 1.201171875 1.01953125 0.62353515625 WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-9.0.1 (ORCJIT, skylake) Environment: JULIA_DEBUG = all JULIA_CUDA_VERBOSE = true CPLUS_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include MANPATH = /gpfs/runtime/opt/gcc/8.3/share/man:/gpfs/runtime/opt/python/3.7.4/share/man:/gpfs/runtime/opt/git/2.20.2/share/man:/gpfs/runtime/opt/binutils/2.31/share/man:/gpfs/runtime/opt/intel/2017.0/man/common/man1: TERM = xterm-256color LIBRARY_PATH = /gpfs/runtime/opt/cuda/10.2/cuda/lib64:/gpfs/runtime/opt/cuda/10.2/cuda/lib:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64 CUDA_HOME = /gpfs/runtime/opt/cuda/10.2/cuda LD_LIBRARY_PATH = /gpfs/runtime/opt/cuda/10.2/cuda/lib64:/gpfs/runtime/opt/cuda/10.2/cuda/lib:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64:/gpfs/runtime/opt/java/8u111/jre/lib/amd64 CPATH = /gpfs/runtime/opt/cuda/10.2/cuda/include:/gpfs/runtime/opt/gcc/8.3/include:/gpfs/runtime/opt/python/3.7.4/include:/gpfs/runtime/opt/binutils/2.31/include:/gpfs/runtime/opt/intel/2017.0/mkl/include NLSPATH = /gpfs/runtime/opt/intel/2017.0/lib/intel64/locale/en_US:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64/locale/en_US PATH = /gpfs/runtime/opt/cuda/10.2/cuda/bin:/gpfs/runtime/opt/gcc/8.3/bin:/users/daluthge/bin:/gpfs/runtime/opt/python/3.7.4/bin:/gpfs/runtime/opt/git/2.20.2/bin:/gpfs/runtime/opt/binutils/2.31/bin:/gpfs/runtime/opt/intel/2017.0/bin:/gpfs/runtime/opt/matlab/R2017b/bin:/gpfs/runtime/opt/java/8u111/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/usr/lpp/mmfs/sbin:/opt/ibutils/bin:/gpfs/runtime/bin C_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include LD_RUN_PATH = /gpfs/runtime/opt/cuda/10.2/cuda/lib64:/gpfs/runtime/opt/cuda/10.2/cuda/lib:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64 JAVA_HOME = /gpfs/runtime/opt/java/8u111 MODULEPATH = /gpfs/runtime/modulefiles HOME = /users/daluthge IPP_PATH = /gpfs/runtime/opt/intel/2017.0/ipp MODULEHOME = /gpfs/runtime/pymodules PKG_CONFIG_PATH = /gpfs/runtime/opt/python/3.7.4/lib/pkgconfig QT_PLUGIN_PATH = /usr/lib64/kde4/plugins:/usr/lib/kde4/plugins julia> import Pkg julia> Pkg.add("CuArrays") Cloning default registries into `~/.julia` ######################################################################## 100.0% Added registry `General` to `~/.julia/registries/General` Resolving package versions... Installed Requires ─────────── v1.0.1 Installed Adapt ────────────── v1.0.1 Installed TimerOutputs ─────── v0.5.3 Installed CUDAapi ──────────── v3.1.0 Installed AbstractFFTs ─────── v0.5.0 Installed GPUArrays ────────── v2.0.1 Installed CuArrays ─────────── v1.7.2 Installed CUDAnative ───────── v2.10.2 Installed CEnum ────────────── v0.2.0 Installed OrderedCollections ─ v1.1.0 Installed DataStructures ───── v0.17.9 Installed MacroTools ───────── v0.5.4 Installed BinaryProvider ───── v0.5.8 Installed NNlib ────────────── v0.6.4 Installed CUDAdrv ──────────── v6.0.0 Installed LLVM ─────────────── v1.3.3 Updating `/gpfs_home/daluthge/.julia/environments/v1.5/Project.toml` [3a865a2d] + CuArrays v1.7.2 Updating `/gpfs_home/daluthge/.julia/environments/v1.5/Manifest.toml` [621f4979] + AbstractFFTs v0.5.0 [79e6a3ab] + Adapt v1.0.1 [b99e7846] + BinaryProvider v0.5.8 [fa961155] + CEnum v0.2.0 [3895d2a7] + CUDAapi v3.1.0 [c5f51814] + CUDAdrv v6.0.0 [be33ccc6] + CUDAnative v2.10.2 [3a865a2d] + CuArrays v1.7.2 [864edb3b] + DataStructures v0.17.9 [0c68f7d7] + GPUArrays v2.0.1 [929cbde3] + LLVM v1.3.3 [1914dd2f] + MacroTools v0.5.4 [872c559c] + NNlib v0.6.4 [bac558e1] + OrderedCollections v1.1.0 [ae029012] + Requires v1.0.1 [a759f4b9] + TimerOutputs v0.5.3 [2a0f44e3] + Base64 [8ba89e20] + Distributed [b77e0a4c] + InteractiveUtils [8f399da3] + Libdl [37e2e46d] + LinearAlgebra [56ddb016] + Logging [d6f4376e] + Markdown [de0858da] + Printf [9a3f8284] + Random [ea8e919c] + SHA [9e88b42a] + Serialization [6462fe0b] + Sockets [2f01184e] + SparseArrays [10745b16] + Statistics [8dfed614] + Test [cf7118a7] + UUIDs [4ec0a83e] + Unicode Building NNlib → `~/.julia/packages/NNlib/3krvM/deps/build.log` julia> import CuArrays [ Info: Precompiling CuArrays [3a865a2d-5b23-5a0f-bc46-62713ec82fae] ┌ Debug: Precompiling CUDAapi [3895d2a7-ec45-59b8-82bb-cfc6a382f9b3] └ @ Base loading.jl:1276 ┌ Debug: Precompiling CUDAdrv [c5f51814-7f29-56b8-a69c-e4d8f6be1fde] └ @ Base loading.jl:1276 ┌ Debug: Precompiling CEnum [fa961155-64e5-5f13-b03f-caf6b980ea82] └ @ Base loading.jl:1276 ┌ Debug: Precompiling CUDAnative [be33ccc6-a3ff-5ff2-a52e-74243cff1e17] └ @ Base loading.jl:1276 ┌ Debug: Precompiling LLVM [929cbde3-209d-540e-8aea-75f648917ca0] └ @ Base loading.jl:1276 ┌ Debug: Found LLVM v9.0.1 at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/bin/../lib/libLLVM-9.so with support for AArch64, AMDGPU, ARC, ARM, AVR, BPF, Hexagon, Lanai, MSP430, Mips, NVPTX, PowerPC, RISCV, Sparc, SystemZ, WebAssembly, X86, XCore └ @ LLVM ~/.julia/packages/LLVM/DAnFH/src/LLVM.jl:47 ┌ Debug: Using LLVM.jl wrapper for LLVM v9.0 └ @ LLVM ~/.julia/packages/LLVM/DAnFH/src/LLVM.jl:75 ┌ Debug: Precompiling Adapt [79e6a3ab-5dfb-504d-930d-738a2a938a0e] └ @ Base loading.jl:1276 ┌ Debug: Precompiling TimerOutputs [a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f] └ @ Base loading.jl:1276 ┌ Debug: Precompiling DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] └ @ Base loading.jl:1276 ┌ Debug: Precompiling OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] └ @ Base loading.jl:1276 ┌ Debug: Looking for CUDA toolkit via environment variables CUDA_HOME └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Request to look for binary nvdisasm │ locations = │ 1-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.2/cuda" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Looking for binary nvdisasm │ locations = │ 20-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.2/cuda" │ "/gpfs/runtime/opt/cuda/10.2/cuda/bin" │ "/gpfs/runtime/opt/cuda/10.2/cuda/bin" │ "/gpfs/runtime/opt/gcc/8.3/bin" │ "/users/daluthge/bin" │ "/gpfs/runtime/opt/python/3.7.4/bin" │ "/gpfs/runtime/opt/git/2.20.2/bin" │ "/gpfs/runtime/opt/binutils/2.31/bin" │ ⋮ │ "/usr/bin" │ "/usr/local/sbin" │ "/usr/sbin" │ "/usr/lpp/mmfs/bin" │ "/usr/lpp/mmfs/sbin" │ "/opt/ibutils/bin" │ "/gpfs/runtime/bin" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Found binary nvdisasm at /gpfs/runtime/opt/cuda/10.2/cuda/bin └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:141 ┌ Debug: CUDA toolkit identified as 10.2.89 └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:297 ┌ Debug: Request to look for libdevice │ locations = │ 1-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.2/cuda" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Look for libdevice │ locations = │ 2-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.2/cuda" │ "/gpfs/runtime/opt/cuda/10.2/cuda/nvvm/libdevice" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Found unified device library at /gpfs/runtime/opt/cuda/10.2/cuda/nvvm/libdevice/libdevice.10.bc └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:327 ┌ Debug: Request to look for libcudadevrt │ locations = │ 1-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.2/cuda" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Looking for CUDA device runtime library libcudadevrt.a │ locations = │ 3-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.2/cuda" │ "/gpfs/runtime/opt/cuda/10.2/cuda/lib" │ "/gpfs/runtime/opt/cuda/10.2/cuda/lib64" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Found CUDA device runtime library libcudadevrt.a at /gpfs/runtime/opt/cuda/10.2/cuda/lib64 └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:379 ┌ Debug: Request to look for library nvToolsExt │ locations = │ 1-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.2/cuda" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Looking for library libnvToolsExt.so, libnvToolsExt.so.1, libnvToolsExt.so.1.0 │ locations = │ 4-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.2/cuda" │ "/gpfs/runtime/opt/cuda/10.2/cuda/lib" │ "/gpfs/runtime/opt/cuda/10.2/cuda/lib64" │ "/gpfs/runtime/opt/cuda/10.2/cuda/libx64" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Found library libnvToolsExt.so at /gpfs/runtime/opt/cuda/10.2/cuda/lib64 └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:90 ┌ Debug: Request to look for library cupti │ locations = │ 2-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.2/cuda" │ "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Looking for library libcupti.so, libcupti.so.10, libcupti.so.10.2 │ locations = │ 8-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.2/cuda" │ "/gpfs/runtime/opt/cuda/10.2/cuda/lib" │ "/gpfs/runtime/opt/cuda/10.2/cuda/lib64" │ "/gpfs/runtime/opt/cuda/10.2/cuda/libx64" │ "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI" │ "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI/lib" │ "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI/lib64" │ "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI/libx64" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Found library libcupti.so at /gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI/lib64 └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:90 ┌ Debug: Using LLVM v9.0.1 └ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:170 ┌ Debug: LLVM supports capabilities 2.0, 2.1, 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3 and 6.4 └ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:191 ┌ Debug: Using CUDA driver v10.2.0 and toolkit v10.2.0 └ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:196 ┌ Debug: CUDA driver supports capabilities 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 2.0, 2.1, 2.2, 2.3, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.2, 6.3, 6.4 and 6.5 └ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:213 ┌ Debug: CUDA toolkit supports capabilities 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 2.0, 2.1, 2.2, 2.3, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.2, 6.3, 6.4 and 6.5 └ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:214 ┌ Debug: CUDAnative supports devices 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5; PTX 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3 and 6.4 └ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:174 ┌ Debug: Precompiling GPUArrays [0c68f7d7-f131-5f86-a1c3-88cf8149b2d7] └ @ Base loading.jl:1276 ┌ Debug: Precompiling AbstractFFTs [621f4979-c628-5d54-868e-fcf4e3e8185c] └ @ Base loading.jl:1276 ┌ Debug: Precompiling Requires [ae029012-a4dd-5104-9daa-d747884805df] └ @ Base loading.jl:1276 ┌ Debug: Precompiling MacroTools [1914dd2f-81c6-5fcd-8719-6d5c9610ff09] └ @ Base loading.jl:1276 ┌ Debug: Precompiling NNlib [872c559c-99b0-510c-b3b7-b6c96a88d5cd] └ @ Base loading.jl:1276 ┌ Warning: Incompatibility detected between CUDA and LLVM 8.0+; disabling debug info emission for CUDA kernels └ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:114 ┌ Error: CuArrays.jl failed to initialize │ exception = │ could not load library "libcublas" │ libcublas.so: cannot open shared object file: No such file or directory │ Stacktrace: │ [1] dlopen(::String, ::UInt32; throw_error::Bool) at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109 │ [2] dlopen at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109 [inlined] (repeats 2 times) │ [3] (::CuArrays.CUBLAS.var"#509#lookup_fptr#28")() at /users/daluthge/.julia/packages/CUDAapi/wYUAO/src/call.jl:29 │ [4] macro expansion at /users/daluthge/.julia/packages/CUDAapi/wYUAO/src/call.jl:37 [inlined] │ [5] macro expansion at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/error.jl:65 [inlined] │ [6] cublasGetProperty at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/libcublas.jl:27 [inlined] │ [7] cublasGetProperty at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/wrappers.jl:38 [inlined] │ [8] version() at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/wrappers.jl:42 │ [9] __init__() at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/CuArrays.jl:98 │ [10] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:697 │ [11] _require_from_serialized(::String) at ./loading.jl:748 │ [12] _require(::Base.PkgId) at ./loading.jl:1039 │ [13] require(::Base.PkgId) at ./loading.jl:927 │ [14] require(::Module, ::Symbol) at ./loading.jl:922 │ [15] eval(::Module, ::Any) at ./boot.jl:331 │ [16] eval_user_input(::Any, ::REPL.REPLBackend) at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:118 │ [17] macro expansion at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:150 [inlined] │ [18] (::REPL.var"#31#32"{REPL.REPLBackend})() at ./task.jl:358 └ @ CuArrays ~/.julia/packages/CuArrays/HE8G6/src/CuArrays.jl:141 ```

CUDA 10.1.105: (works fine)

Click to expand ```bash $ which nvcc /gpfs/runtime/opt/cuda/10.1.105/cuda/bin/nvcc $ which nvdisasm /gpfs/runtime/opt/cuda/10.1.105/cuda/bin/nvdisasm $ nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2019 NVIDIA Corporation Built on Fri_Feb__8_19:08:17_PST_2019 Cuda compilation tools, release 10.1, V10.1.105 $ nvdisasm --version nvdisasm: NVIDIA (R) CUDA disassembler Copyright (c) 2005-2019 NVIDIA Corporation Built on Fri_Feb__8_19:08:51_PST_2019 Cuda compilation tools, release 10.1, V10.1.105 ``` ```julia bash-4.2$ julia _ _ _ _(_)_ | Documentation: https://docs.julialang.org (_) | (_) (_) | _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help. | | | | | | |/ _` | | | | |_| | | | (_| | | Version 1.5.0-DEV.274 (2020-02-15) _/ |\__'_|_|_|\__'_| | Commit 8eb0f9fefb (8 days old master) |__/ | julia> versioninfo(verbose = true) Julia Version 1.5.0-DEV.274 Commit 8eb0f9fefb (2020-02-15 12:41 UTC) Platform Info: OS: Linux (x86_64-pc-linux-gnu) "Red Hat Enterprise Linux Server release 7.3 (Maipo)" uname: Linux 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64 CPU: Intel(R) Xeon(R) Gold 5122 CPU @ 3.60GHz: speed user nice sys idle irq #1 3601 MHz 260888239 s 209 s 79076296 s 314453175 s 0 s #2 3601 MHz 225435401 s 556 s 66634561 s 362168060 s 0 s #3 3601 MHz 153758683 s 331 s 47119620 s 453444427 s 0 s #4 3601 MHz 133428321 s 355 s 44199667 s 476416450 s 0 s #5 3601 MHz 231457860 s 286 s 61205722 s 362256718 s 0 s #6 3601 MHz 154411110 s 604 s 45161031 s 455702131 s 0 s #7 3601 MHz 104945705 s 268 s 34510058 s 515700335 s 0 s #8 3601 MHz 94884872 s 496 s 31751407 s 528188490 s 0 s Memory: 93.04103088378906 GB (80663.54296875 MB free) Uptime: 6.578485e6 sec Load Avg: 1.19580078125 0.96240234375 0.56298828125 WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-9.0.1 (ORCJIT, skylake) Environment: JULIA_DEBUG = all JULIA_CUDA_VERBOSE = true CPLUS_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include MANPATH = /gpfs/runtime/opt/python/3.7.4/share/man:/gpfs/runtime/opt/git/2.20.2/share/man:/gpfs/runtime/opt/gcc/8.3/share/man:/gpfs/runtime/opt/binutils/2.31/share/man:/gpfs/runtime/opt/intel/2017.0/man/common/man1: TERM = xterm-256color LIBRARY_PATH = /gpfs/runtime/opt/cuda/10.1.105/cuda/lib64:/gpfs/runtime/opt/cuda/10.1.105/cuda/lib:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64 CUDA_HOME = /gpfs/runtime/opt/cuda/10.1.105/cuda LD_LIBRARY_PATH = /gpfs/runtime/opt/cuda/10.1.105/cuda/lib64:/gpfs/runtime/opt/cuda/10.1.105/cuda/lib:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64:/gpfs/runtime/opt/java/8u111/jre/lib/amd64 CPATH = /gpfs/runtime/opt/cuda/10.1.105/cuda/include:/gpfs/runtime/opt/python/3.7.4/include:/gpfs/runtime/opt/gcc/8.3/include:/gpfs/runtime/opt/binutils/2.31/include:/gpfs/runtime/opt/intel/2017.0/mkl/include NLSPATH = /gpfs/runtime/opt/intel/2017.0/lib/intel64/locale/en_US:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64/locale/en_US PATH = /gpfs/runtime/opt/cuda/10.1.105/cuda/bin:/users/daluthge/bin:/gpfs/runtime/opt/python/3.7.4/bin:/gpfs/runtime/opt/git/2.20.2/bin:/gpfs/runtime/opt/gcc/8.3/bin:/gpfs/runtime/opt/binutils/2.31/bin:/gpfs/runtime/opt/intel/2017.0/bin:/gpfs/runtime/opt/matlab/R2017b/bin:/gpfs/runtime/opt/java/8u111/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/usr/lpp/mmfs/sbin:/opt/ibutils/bin:/gpfs/runtime/bin C_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include LD_RUN_PATH = /gpfs/runtime/opt/cuda/10.1.105/cuda/lib64:/gpfs/runtime/opt/cuda/10.1.105/cuda/lib:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64 JAVA_HOME = /gpfs/runtime/opt/java/8u111 MODULEPATH = /gpfs/runtime/modulefiles HOME = /users/daluthge IPP_PATH = /gpfs/runtime/opt/intel/2017.0/ipp MODULEHOME = /gpfs/runtime/pymodules PKG_CONFIG_PATH = /gpfs/runtime/opt/python/3.7.4/lib/pkgconfig QT_PLUGIN_PATH = /usr/lib64/kde4/plugins:/usr/lib/kde4/plugins julia> import Pkg julia> Pkg.add("CuArrays") Cloning default registries into `~/.julia` ######################################################################## 100.0% Added registry `General` to `~/.julia/registries/General` Resolving package versions... Installed Requires ─────────── v1.0.1 Installed Adapt ────────────── v1.0.1 Installed TimerOutputs ─────── v0.5.3 Installed AbstractFFTs ─────── v0.5.0 Installed CUDAapi ──────────── v3.1.0 Installed GPUArrays ────────── v2.0.1 Installed CuArrays ─────────── v1.7.2 Installed CUDAnative ───────── v2.10.2 Installed CEnum ────────────── v0.2.0 Installed OrderedCollections ─ v1.1.0 Installed DataStructures ───── v0.17.9 Installed MacroTools ───────── v0.5.4 Installed BinaryProvider ───── v0.5.8 Installed NNlib ────────────── v0.6.4 Installed CUDAdrv ──────────── v6.0.0 Installed LLVM ─────────────── v1.3.3 Updating `/gpfs_home/daluthge/.julia/environments/v1.5/Project.toml` [3a865a2d] + CuArrays v1.7.2 Updating `/gpfs_home/daluthge/.julia/environments/v1.5/Manifest.toml` [621f4979] + AbstractFFTs v0.5.0 [79e6a3ab] + Adapt v1.0.1 [b99e7846] + BinaryProvider v0.5.8 [fa961155] + CEnum v0.2.0 [3895d2a7] + CUDAapi v3.1.0 [c5f51814] + CUDAdrv v6.0.0 [be33ccc6] + CUDAnative v2.10.2 [3a865a2d] + CuArrays v1.7.2 [864edb3b] + DataStructures v0.17.9 [0c68f7d7] + GPUArrays v2.0.1 [929cbde3] + LLVM v1.3.3 [1914dd2f] + MacroTools v0.5.4 [872c559c] + NNlib v0.6.4 [bac558e1] + OrderedCollections v1.1.0 [ae029012] + Requires v1.0.1 [a759f4b9] + TimerOutputs v0.5.3 [2a0f44e3] + Base64 [8ba89e20] + Distributed [b77e0a4c] + InteractiveUtils [8f399da3] + Libdl [37e2e46d] + LinearAlgebra [56ddb016] + Logging [d6f4376e] + Markdown [de0858da] + Printf [9a3f8284] + Random [ea8e919c] + SHA [9e88b42a] + Serialization [6462fe0b] + Sockets [2f01184e] + SparseArrays [10745b16] + Statistics [8dfed614] + Test [cf7118a7] + UUIDs [4ec0a83e] + Unicode Building NNlib → `~/.julia/packages/NNlib/3krvM/deps/build.log` julia> import CuArrays [ Info: Precompiling CuArrays [3a865a2d-5b23-5a0f-bc46-62713ec82fae] ┌ Debug: Precompiling CUDAapi [3895d2a7-ec45-59b8-82bb-cfc6a382f9b3] └ @ Base loading.jl:1276 ┌ Debug: Precompiling CUDAdrv [c5f51814-7f29-56b8-a69c-e4d8f6be1fde] └ @ Base loading.jl:1276 ┌ Debug: Precompiling CEnum [fa961155-64e5-5f13-b03f-caf6b980ea82] └ @ Base loading.jl:1276 ┌ Debug: Precompiling CUDAnative [be33ccc6-a3ff-5ff2-a52e-74243cff1e17] └ @ Base loading.jl:1276 ┌ Debug: Precompiling LLVM [929cbde3-209d-540e-8aea-75f648917ca0] └ @ Base loading.jl:1276 ┌ Debug: Found LLVM v9.0.1 at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/bin/../lib/libLLVM-9.so with support for AArch64, AMDGPU, ARC, ARM, AVR, BPF, Hexagon, Lanai, MSP430, Mips, NVPTX, PowerPC, RISCV, Sparc, SystemZ, WebAssembly, X86, XCore └ @ LLVM ~/.julia/packages/LLVM/DAnFH/src/LLVM.jl:47 ┌ Debug: Using LLVM.jl wrapper for LLVM v9.0 └ @ LLVM ~/.julia/packages/LLVM/DAnFH/src/LLVM.jl:75 ┌ Debug: Precompiling Adapt [79e6a3ab-5dfb-504d-930d-738a2a938a0e] └ @ Base loading.jl:1276 ┌ Debug: Precompiling TimerOutputs [a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f] └ @ Base loading.jl:1276 ┌ Debug: Precompiling DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] └ @ Base loading.jl:1276 ┌ Debug: Precompiling OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] └ @ Base loading.jl:1276 ┌ Debug: Looking for CUDA toolkit via environment variables CUDA_HOME └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Request to look for binary nvdisasm │ locations = │ 1-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.1.105/cuda" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Looking for binary nvdisasm │ locations = │ 20-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.1.105/cuda" │ "/gpfs/runtime/opt/cuda/10.1.105/cuda/bin" │ "/gpfs/runtime/opt/cuda/10.1.105/cuda/bin" │ "/users/daluthge/bin" │ "/gpfs/runtime/opt/python/3.7.4/bin" │ "/gpfs/runtime/opt/git/2.20.2/bin" │ "/gpfs/runtime/opt/gcc/8.3/bin" │ "/gpfs/runtime/opt/binutils/2.31/bin" │ ⋮ │ "/usr/bin" │ "/usr/local/sbin" │ "/usr/sbin" │ "/usr/lpp/mmfs/bin" │ "/usr/lpp/mmfs/sbin" │ "/opt/ibutils/bin" │ "/gpfs/runtime/bin" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Found binary nvdisasm at /gpfs/runtime/opt/cuda/10.1.105/cuda/bin └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:141 ┌ Debug: CUDA toolkit identified as 10.1.105 └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:297 ┌ Debug: Request to look for libdevice │ locations = │ 1-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.1.105/cuda" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Look for libdevice │ locations = │ 2-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.1.105/cuda" │ "/gpfs/runtime/opt/cuda/10.1.105/cuda/nvvm/libdevice" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Found unified device library at /gpfs/runtime/opt/cuda/10.1.105/cuda/nvvm/libdevice/libdevice.10.bc └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:327 ┌ Debug: Request to look for libcudadevrt │ locations = │ 1-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.1.105/cuda" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Looking for CUDA device runtime library libcudadevrt.a │ locations = │ 3-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.1.105/cuda" │ "/gpfs/runtime/opt/cuda/10.1.105/cuda/lib" │ "/gpfs/runtime/opt/cuda/10.1.105/cuda/lib64" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Found CUDA device runtime library libcudadevrt.a at /gpfs/runtime/opt/cuda/10.1.105/cuda/lib64 └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:379 ┌ Debug: Request to look for library nvToolsExt │ locations = │ 1-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.1.105/cuda" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Looking for library libnvToolsExt.so, libnvToolsExt.so.1, libnvToolsExt.so.1.0 │ locations = │ 4-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.1.105/cuda" │ "/gpfs/runtime/opt/cuda/10.1.105/cuda/lib" │ "/gpfs/runtime/opt/cuda/10.1.105/cuda/lib64" │ "/gpfs/runtime/opt/cuda/10.1.105/cuda/libx64" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Found library libnvToolsExt.so at /gpfs/runtime/opt/cuda/10.1.105/cuda/lib64 └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:90 ┌ Debug: Request to look for library cupti │ locations = │ 2-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.1.105/cuda" │ "/gpfs/runtime/opt/cuda/10.1.105/cuda/extras/CUPTI" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Looking for library libcupti.so, libcupti.so.10, libcupti.so.10.1 │ locations = │ 8-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.1.105/cuda" │ "/gpfs/runtime/opt/cuda/10.1.105/cuda/lib" │ "/gpfs/runtime/opt/cuda/10.1.105/cuda/lib64" │ "/gpfs/runtime/opt/cuda/10.1.105/cuda/libx64" │ "/gpfs/runtime/opt/cuda/10.1.105/cuda/extras/CUPTI" │ "/gpfs/runtime/opt/cuda/10.1.105/cuda/extras/CUPTI/lib" │ "/gpfs/runtime/opt/cuda/10.1.105/cuda/extras/CUPTI/lib64" │ "/gpfs/runtime/opt/cuda/10.1.105/cuda/extras/CUPTI/libx64" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Found library libcupti.so at /gpfs/runtime/opt/cuda/10.1.105/cuda/extras/CUPTI/lib64 └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:90 ┌ Debug: Using LLVM v9.0.1 └ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:170 ┌ Debug: LLVM supports capabilities 2.0, 2.1, 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3 and 6.4 └ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:191 ┌ Debug: Using CUDA driver v10.2.0 and toolkit v10.1.0 └ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:196 ┌ Debug: CUDA driver supports capabilities 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 2.0, 2.1, 2.2, 2.3, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.2, 6.3, 6.4 and 6.5 └ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:213 ┌ Debug: CUDA toolkit supports capabilities 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 2.0, 2.1, 2.2, 2.3, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.2, 6.3 and 6.4 └ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:214 ┌ Debug: CUDAnative supports devices 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5; PTX 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3 and 6.4 └ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:174 ┌ Debug: Precompiling GPUArrays [0c68f7d7-f131-5f86-a1c3-88cf8149b2d7] └ @ Base loading.jl:1276 ┌ Debug: Precompiling AbstractFFTs [621f4979-c628-5d54-868e-fcf4e3e8185c] └ @ Base loading.jl:1276 ┌ Debug: Precompiling Requires [ae029012-a4dd-5104-9daa-d747884805df] └ @ Base loading.jl:1276 ┌ Debug: Precompiling MacroTools [1914dd2f-81c6-5fcd-8719-6d5c9610ff09] └ @ Base loading.jl:1276 ┌ Debug: Precompiling NNlib [872c559c-99b0-510c-b3b7-b6c96a88d5cd] └ @ Base loading.jl:1276 ┌ Warning: Incompatibility detected between CUDA and LLVM 8.0+; disabling debug info emission for CUDA kernels └ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:114 ```

CUDA 10.0.130: (works fine)

Click to expand ```bash $ which nvcc /gpfs/runtime/opt/cuda/10.0.130/cuda/bin/nvcc $ which nvdisasm /gpfs/runtime/opt/cuda/10.0.130/cuda/bin/nvdisasm $ nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2018 NVIDIA Corporation Built on Sat_Aug_25_21:08:01_CDT_2018 Cuda compilation tools, release 10.0, V10.0.130 $ nvdisasm --version nvdisasm: NVIDIA (R) CUDA disassembler Copyright (c) 2005-2018 NVIDIA Corporation Built on Sat_Aug_25_21:08:11_CDT_2018 Cuda compilation tools, release 10.0, V10.0.130 ``` ```julia bash-4.2$ julia _ _ _ _(_)_ | Documentation: https://docs.julialang.org (_) | (_) (_) | _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help. | | | | | | |/ _` | | | | |_| | | | (_| | | Version 1.5.0-DEV.274 (2020-02-15) _/ |\__'_|_|_|\__'_| | Commit 8eb0f9fefb (8 days old master) |__/ | julia> versioninfo(verbose = true) Julia Version 1.5.0-DEV.274 Commit 8eb0f9fefb (2020-02-15 12:41 UTC) Platform Info: OS: Linux (x86_64-pc-linux-gnu) "Red Hat Enterprise Linux Server release 7.3 (Maipo)" uname: Linux 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64 CPU: Intel(R) Xeon(R) Gold 5122 CPU @ 3.60GHz: speed user nice sys idle irq #1 3601 MHz 260887968 s 209 s 79076153 s 314446557 s 0 s #2 3601 MHz 225435375 s 556 s 66634534 s 362161063 s 0 s #3 3601 MHz 153758324 s 331 s 47119535 s 453437851 s 0 s #4 3601 MHz 133428293 s 355 s 44199325 s 476409768 s 0 s #5 3601 MHz 231454591 s 286 s 61205506 s 362253151 s 0 s #6 3601 MHz 154408009 s 604 s 45160867 s 455698340 s 0 s #7 3601 MHz 104945700 s 268 s 34510038 s 515693325 s 0 s #8 3601 MHz 94884863 s 496 s 31751307 s 528181552 s 0 s Memory: 93.04103088378906 GB (80678.2890625 MB free) Uptime: 6.578414e6 sec Load Avg: 1.1494140625 0.908203125 0.515625 WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-9.0.1 (ORCJIT, skylake) Environment: JULIA_DEBUG = all JULIA_CUDA_VERBOSE = true CPLUS_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include MANPATH = /gpfs/runtime/opt/python/3.7.4/share/man:/gpfs/runtime/opt/git/2.20.2/share/man:/gpfs/runtime/opt/gcc/8.3/share/man:/gpfs/runtime/opt/binutils/2.31/share/man:/gpfs/runtime/opt/intel/2017.0/man/common/man1: TERM = xterm-256color LIBRARY_PATH = /gpfs/runtime/opt/cuda/10.0.130/cuda/lib64:/gpfs/runtime/opt/cuda/10.0.130/cuda/lib:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64 CUDA_HOME = /gpfs/runtime/opt/cuda/10.0.130/cuda LD_LIBRARY_PATH = /gpfs/runtime/opt/cuda/10.0.130/cuda/lib64:/gpfs/runtime/opt/cuda/10.0.130/cuda/lib:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64:/gpfs/runtime/opt/java/8u111/jre/lib/amd64 CPATH = /gpfs/runtime/opt/cuda/10.0.130/cuda/include:/gpfs/runtime/opt/python/3.7.4/include:/gpfs/runtime/opt/gcc/8.3/include:/gpfs/runtime/opt/binutils/2.31/include:/gpfs/runtime/opt/intel/2017.0/mkl/include NLSPATH = /gpfs/runtime/opt/intel/2017.0/lib/intel64/locale/en_US:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64/locale/en_US PATH = /gpfs/runtime/opt/cuda/10.0.130/cuda/bin:/users/daluthge/bin:/gpfs/runtime/opt/python/3.7.4/bin:/gpfs/runtime/opt/git/2.20.2/bin:/gpfs/runtime/opt/gcc/8.3/bin:/gpfs/runtime/opt/binutils/2.31/bin:/gpfs/runtime/opt/intel/2017.0/bin:/gpfs/runtime/opt/matlab/R2017b/bin:/gpfs/runtime/opt/java/8u111/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/usr/lpp/mmfs/sbin:/opt/ibutils/bin:/gpfs/runtime/bin C_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include LD_RUN_PATH = /gpfs/runtime/opt/cuda/10.0.130/cuda/lib64:/gpfs/runtime/opt/cuda/10.0.130/cuda/lib:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64 JAVA_HOME = /gpfs/runtime/opt/java/8u111 MODULEPATH = /gpfs/runtime/modulefiles HOME = /users/daluthge IPP_PATH = /gpfs/runtime/opt/intel/2017.0/ipp MODULEHOME = /gpfs/runtime/pymodules PKG_CONFIG_PATH = /gpfs/runtime/opt/python/3.7.4/lib/pkgconfig QT_PLUGIN_PATH = /usr/lib64/kde4/plugins:/usr/lib/kde4/plugins julia> import Pkg julia> Pkg.add("CuArrays") Cloning default registries into `~/.julia` ######################################################################## 100.0% Added registry `General` to `~/.julia/registries/General` Resolving package versions... Installed TimerOutputs ─────── v0.5.3 Installed Adapt ────────────── v1.0.1 Installed Requires ─────────── v1.0.1 Installed AbstractFFTs ─────── v0.5.0 Installed CUDAapi ──────────── v3.1.0 Installed GPUArrays ────────── v2.0.1 Installed CEnum ────────────── v0.2.0 Installed OrderedCollections ─ v1.1.0 Installed DataStructures ───── v0.17.9 Installed CuArrays ─────────── v1.7.2 Installed MacroTools ───────── v0.5.4 Installed BinaryProvider ───── v0.5.8 Installed CUDAnative ───────── v2.10.2 Installed NNlib ────────────── v0.6.4 Installed CUDAdrv ──────────── v6.0.0 Installed LLVM ─────────────── v1.3.3 Updating `/gpfs_home/daluthge/.julia/environments/v1.5/Project.toml` [3a865a2d] + CuArrays v1.7.2 Updating `/gpfs_home/daluthge/.julia/environments/v1.5/Manifest.toml` [621f4979] + AbstractFFTs v0.5.0 [79e6a3ab] + Adapt v1.0.1 [b99e7846] + BinaryProvider v0.5.8 [fa961155] + CEnum v0.2.0 [3895d2a7] + CUDAapi v3.1.0 [c5f51814] + CUDAdrv v6.0.0 [be33ccc6] + CUDAnative v2.10.2 [3a865a2d] + CuArrays v1.7.2 [864edb3b] + DataStructures v0.17.9 [0c68f7d7] + GPUArrays v2.0.1 [929cbde3] + LLVM v1.3.3 [1914dd2f] + MacroTools v0.5.4 [872c559c] + NNlib v0.6.4 [bac558e1] + OrderedCollections v1.1.0 [ae029012] + Requires v1.0.1 [a759f4b9] + TimerOutputs v0.5.3 [2a0f44e3] + Base64 [8ba89e20] + Distributed [b77e0a4c] + InteractiveUtils [8f399da3] + Libdl [37e2e46d] + LinearAlgebra [56ddb016] + Logging [d6f4376e] + Markdown [de0858da] + Printf [9a3f8284] + Random [ea8e919c] + SHA [9e88b42a] + Serialization [6462fe0b] + Sockets [2f01184e] + SparseArrays [10745b16] + Statistics [8dfed614] + Test [cf7118a7] + UUIDs [4ec0a83e] + Unicode Building NNlib → `~/.julia/packages/NNlib/3krvM/deps/build.log` julia> import CuArrays [ Info: Precompiling CuArrays [3a865a2d-5b23-5a0f-bc46-62713ec82fae] ┌ Debug: Precompiling CUDAapi [3895d2a7-ec45-59b8-82bb-cfc6a382f9b3] └ @ Base loading.jl:1276 ┌ Debug: Precompiling CUDAdrv [c5f51814-7f29-56b8-a69c-e4d8f6be1fde] └ @ Base loading.jl:1276 ┌ Debug: Precompiling CEnum [fa961155-64e5-5f13-b03f-caf6b980ea82] └ @ Base loading.jl:1276 ┌ Debug: Precompiling CUDAnative [be33ccc6-a3ff-5ff2-a52e-74243cff1e17] └ @ Base loading.jl:1276 ┌ Debug: Precompiling LLVM [929cbde3-209d-540e-8aea-75f648917ca0] └ @ Base loading.jl:1276 ┌ Debug: Found LLVM v9.0.1 at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/bin/../lib/libLLVM-9.so with support for AArch64, AMDGPU, ARC, ARM, AVR, BPF, Hexagon, Lanai, MSP430, Mips, NVPTX, PowerPC, RISCV, Sparc, SystemZ, WebAssembly, X86, XCore └ @ LLVM ~/.julia/packages/LLVM/DAnFH/src/LLVM.jl:47 ┌ Debug: Using LLVM.jl wrapper for LLVM v9.0 └ @ LLVM ~/.julia/packages/LLVM/DAnFH/src/LLVM.jl:75 ┌ Debug: Precompiling Adapt [79e6a3ab-5dfb-504d-930d-738a2a938a0e] └ @ Base loading.jl:1276 ┌ Debug: Precompiling TimerOutputs [a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f] └ @ Base loading.jl:1276 ┌ Debug: Precompiling DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] └ @ Base loading.jl:1276 ┌ Debug: Precompiling OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d] └ @ Base loading.jl:1276 ┌ Debug: Looking for CUDA toolkit via environment variables CUDA_HOME └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Request to look for binary nvdisasm │ locations = │ 1-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.0.130/cuda" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Looking for binary nvdisasm │ locations = │ 20-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.0.130/cuda" │ "/gpfs/runtime/opt/cuda/10.0.130/cuda/bin" │ "/gpfs/runtime/opt/cuda/10.0.130/cuda/bin" │ "/users/daluthge/bin" │ "/gpfs/runtime/opt/python/3.7.4/bin" │ "/gpfs/runtime/opt/git/2.20.2/bin" │ "/gpfs/runtime/opt/gcc/8.3/bin" │ "/gpfs/runtime/opt/binutils/2.31/bin" │ ⋮ │ "/usr/bin" │ "/usr/local/sbin" │ "/usr/sbin" │ "/usr/lpp/mmfs/bin" │ "/usr/lpp/mmfs/sbin" │ "/opt/ibutils/bin" │ "/gpfs/runtime/bin" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Found binary nvdisasm at /gpfs/runtime/opt/cuda/10.0.130/cuda/bin └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:141 ┌ Debug: CUDA toolkit identified as 10.0.130 └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:297 ┌ Debug: Request to look for libdevice │ locations = │ 1-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.0.130/cuda" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Look for libdevice │ locations = │ 2-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.0.130/cuda" │ "/gpfs/runtime/opt/cuda/10.0.130/cuda/nvvm/libdevice" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Found unified device library at /gpfs/runtime/opt/cuda/10.0.130/cuda/nvvm/libdevice/libdevice.10.bc └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:327 ┌ Debug: Request to look for libcudadevrt │ locations = │ 1-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.0.130/cuda" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Looking for CUDA device runtime library libcudadevrt.a │ locations = │ 3-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.0.130/cuda" │ "/gpfs/runtime/opt/cuda/10.0.130/cuda/lib" │ "/gpfs/runtime/opt/cuda/10.0.130/cuda/lib64" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Found CUDA device runtime library libcudadevrt.a at /gpfs/runtime/opt/cuda/10.0.130/cuda/lib64 └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:379 ┌ Debug: Request to look for library nvToolsExt │ locations = │ 1-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.0.130/cuda" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Looking for library libnvToolsExt.so, libnvToolsExt.so.1, libnvToolsExt.so.1.0 │ locations = │ 4-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.0.130/cuda" │ "/gpfs/runtime/opt/cuda/10.0.130/cuda/lib" │ "/gpfs/runtime/opt/cuda/10.0.130/cuda/lib64" │ "/gpfs/runtime/opt/cuda/10.0.130/cuda/libx64" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Found library libnvToolsExt.so at /gpfs/runtime/opt/cuda/10.0.130/cuda/lib64 └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:90 ┌ Debug: Request to look for library cupti │ locations = │ 2-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.0.130/cuda" │ "/gpfs/runtime/opt/cuda/10.0.130/cuda/extras/CUPTI" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Looking for library libcupti.so, libcupti.so.10, libcupti.so.10.0 │ locations = │ 8-element Array{String,1}: │ "/gpfs/runtime/opt/cuda/10.0.130/cuda" │ "/gpfs/runtime/opt/cuda/10.0.130/cuda/lib" │ "/gpfs/runtime/opt/cuda/10.0.130/cuda/lib64" │ "/gpfs/runtime/opt/cuda/10.0.130/cuda/libx64" │ "/gpfs/runtime/opt/cuda/10.0.130/cuda/extras/CUPTI" │ "/gpfs/runtime/opt/cuda/10.0.130/cuda/extras/CUPTI/lib" │ "/gpfs/runtime/opt/cuda/10.0.130/cuda/extras/CUPTI/lib64" │ "/gpfs/runtime/opt/cuda/10.0.130/cuda/extras/CUPTI/libx64" └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8 ┌ Debug: Found library libcupti.so at /gpfs/runtime/opt/cuda/10.0.130/cuda/extras/CUPTI/lib64 └ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:90 ┌ Debug: Using LLVM v9.0.1 └ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:170 ┌ Debug: LLVM supports capabilities 2.0, 2.1, 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3 and 6.4 └ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:191 ┌ Debug: Using CUDA driver v10.2.0 and toolkit v10.0.0 └ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:196 ┌ Debug: CUDA driver supports capabilities 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 2.0, 2.1, 2.2, 2.3, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.2, 6.3, 6.4 and 6.5 └ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:213 ┌ Debug: CUDA toolkit supports capabilities 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 2.0, 2.1, 2.2, 2.3, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.2 and 6.3 └ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:214 ┌ Debug: CUDAnative supports devices 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5; PTX 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1 and 6.3 └ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:174 ┌ Debug: Precompiling GPUArrays [0c68f7d7-f131-5f86-a1c3-88cf8149b2d7] └ @ Base loading.jl:1276 ┌ Debug: Precompiling AbstractFFTs [621f4979-c628-5d54-868e-fcf4e3e8185c] └ @ Base loading.jl:1276 ┌ Debug: Precompiling Requires [ae029012-a4dd-5104-9daa-d747884805df] └ @ Base loading.jl:1276 ┌ Debug: Precompiling MacroTools [1914dd2f-81c6-5fcd-8719-6d5c9610ff09] └ @ Base loading.jl:1276 ┌ Debug: Precompiling NNlib [872c559c-99b0-510c-b3b7-b6c96a88d5cd] └ @ Base loading.jl:1276 ┌ Warning: Incompatibility detected between CUDA and LLVM 8.0+; disabling debug info emission for CUDA kernels └ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:114 ```
DilumAluthge commented 4 years ago

cc: @maleadt

maleadt commented 4 years ago

The relevant code is here: https://github.com/JuliaGPU/CuArrays.jl/blob/6feb28f71aad38e60ae438d1b158e13b9f283b58/src/CuArrays.jl#L69-L98

There isn't any mention about searching for libcublas in the debug output, which is strange given how the dlopen subsequently fails. The initial value for libcublas is libcublas, could you show what the output of Libdl.dlopen_e("libcublas") is? Maybe it's nothing and not C_NULL? I've seen BB try to check for both (if dlopen_e in (nothing, C_NULL)).

The difference between CUDA versions is probably caused by libcublas being on the default library search path for some versions and not for others.

DilumAluthge commented 4 years ago

WIth CUDA 10.0.130:

julia> import Libdl

julia> Libdl.dlopen("libcublas")
Ptr{Nothing} @0x00000000013c7360

julia> Libdl.dlopen_e("libcublas")
Ptr{Nothing} @0x00000000013c7360

WIth CUDA 10.1.105:

julia> import Libdl

julia> Libdl.dlopen("libcublas")
Ptr{Nothing} @0x00000000012f0300

julia> Libdl.dlopen_e("libcublas")
Ptr{Nothing} @0x00000000012f0300

WIth CUDA 10.2.89

julia> import Libdl

julia> Libdl.dlopen("libcublas")
ERROR: could not load library "libcublas"
libcublas.so: cannot open shared object file: No such file or directory
Stacktrace:
 [1] dlopen(::String, ::UInt32; throw_error::Bool) at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109
 [2] dlopen at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109 [inlined] (repeats 2 times)
 [3] top-level scope at REPL[2]:1

julia> Libdl.dlopen_e("libcublas")
Ptr{Nothing} @0x0000000000000000

julia> Libdl.dlopen_e("libcublas") == C_NULL
true

julia> Libdl.dlopen_e("libcublas") === C_NULL
true
maleadt commented 4 years ago

Confirms my suspicion, but doesn't explain the lack of debug info. Could you run the following with JULIA_DEBUG=CUDAapi:

find_cuda_library("cublas", CUDAnative.prefix(), [CUDAnative.version()])
DilumAluthge commented 4 years ago 
julia> ENV["JULIA_DEBUG"] = "CUDAapi"
"CUDAapi"

julia> import CUDAapi

julia> import CUDAnative
┌ Warning: Incompatibility detected between CUDA and LLVM 8.0+; disabling debug info emission for CUDA kernels
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:114
┌ Debug: Looking for CUDA toolkit via environment variables CUDA_HOME
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for binary nvdisasm
│   locations =
│    1-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for binary nvdisasm
│   locations =
│    20-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/bin"
│     "/gpfs/runtime/opt/gcc/8.3/bin"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/bin"
│     "/users/daluthge/bin"
│     "/gpfs/runtime/opt/python/3.7.4/bin"
│     "/gpfs/runtime/opt/git/2.20.2/bin"
│     "/gpfs/runtime/opt/binutils/2.31/bin"
│     ⋮
│     "/usr/bin"
│     "/usr/local/sbin"
│     "/usr/sbin"
│     "/usr/lpp/mmfs/bin"
│     "/usr/lpp/mmfs/sbin"
│     "/opt/ibutils/bin"
│     "/gpfs/runtime/bin"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found binary nvdisasm at /gpfs/runtime/opt/cuda/10.2/cuda/bin
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:141
┌ Debug: CUDA toolkit identified as 10.2.89
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:297
┌ Debug: Request to look for libdevice
│   locations =
│    1-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Look for libdevice
│   locations =
│    2-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/nvvm/libdevice"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found unified device library at /gpfs/runtime/opt/cuda/10.2/cuda/nvvm/libdevice/libdevice.10.bc
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:327
┌ Debug: Request to look for libcudadevrt
│   locations =
│    1-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for CUDA device runtime library libcudadevrt.a
│   locations =
│    3-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/lib"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/lib64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found CUDA device runtime library libcudadevrt.a at /gpfs/runtime/opt/cuda/10.2/cuda/lib64
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:379
┌ Debug: Request to look for library nvToolsExt
│   locations =
│    1-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libnvToolsExt.so, libnvToolsExt.so.1, libnvToolsExt.so.1.0
│   locations =
│    4-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/lib"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/lib64"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/libx64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found library libnvToolsExt.so at /gpfs/runtime/opt/cuda/10.2/cuda/lib64
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:90
┌ Debug: Request to look for library cupti
│   locations =
│    2-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcupti.so, libcupti.so.10, libcupti.so.10.2
│   locations =
│    8-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/lib"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/lib64"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/libx64"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI/lib"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI/lib64"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI/libx64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found library libcupti.so at /gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI/lib64
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:90

julia> CUDAnative.prefix()
2-element Array{String,1}:
 "/gpfs/runtime/opt/cuda/10.2/cuda"
 "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI"

julia> CUDAnative.version()
v"10.2.89"

julia> CUDAapi.find_cuda_library("cublas", CUDAnative.prefix(), [CUDAnative.version()])
┌ Debug: Request to look for library cublas
│   locations =
│    2-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcublas.so, libcublas.so.10, libcublas.so.10.2
│   locations =
│    8-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/lib"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/lib64"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/libx64"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI/lib"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI/lib64"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI/libx64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
maleadt commented 4 years ago

So it returns nothing? What's the actual name of libcublas in /gpfs/runtime/opt/cuda/10.2/cuda/lib*?

DilumAluthge commented 4 years ago

Yeah it returns nothing.

$ find /gpfs/runtime/opt/cuda/10.2/ -name "*blas*"
/gpfs/runtime/opt/cuda/10.2/cuda/doc/man/man7/libcublas.7
/gpfs/runtime/opt/cuda/10.2/cuda/doc/man/man7/libcublas.so.7
/gpfs/runtime/opt/cuda/10.2/cuda/doc/html/cublas
/gpfs/runtime/opt/cuda/10.2/cuda/doc/html/cublas/graphics/cublasmg_gemm.jpg
/gpfs/runtime/opt/cuda/10.2/cuda/doc/html/nvblas
/gpfs/runtime/opt/cuda/10.2/cuda/targets/x86_64-linux/include/cublas_api.h
/gpfs/runtime/opt/cuda/10.2/cuda/targets/x86_64-linux/include/nvblas.h
/gpfs/runtime/opt/cuda/10.2/cuda/targets/x86_64-linux/include/cublas.h
/gpfs/runtime/opt/cuda/10.2/cuda/targets/x86_64-linux/include/cublas_v2.h
/gpfs/runtime/opt/cuda/10.2/cuda/targets/x86_64-linux/include/cublasLt.h
/gpfs/runtime/opt/cuda/10.2/cuda/targets/x86_64-linux/include/cublasXt.h
/gpfs/runtime/opt/cuda/10.2/src/include/cublas_api.h
/gpfs/runtime/opt/cuda/10.2/src/include/nvblas.h
/gpfs/runtime/opt/cuda/10.2/src/include/cublas.h
/gpfs/runtime/opt/cuda/10.2/src/include/cublas_v2.h
/gpfs/runtime/opt/cuda/10.2/src/include/cublasLt.h
/gpfs/runtime/opt/cuda/10.2/src/include/cublasXt.h
/gpfs/runtime/opt/cuda/10.2/src/lib64/libcublasLt.so
/gpfs/runtime/opt/cuda/10.2/src/lib64/libcublas.so.10.2.2.89
/gpfs/runtime/opt/cuda/10.2/src/lib64/libnvblas.so.10.2.2.89
/gpfs/runtime/opt/cuda/10.2/src/lib64/libcublasLt.so.10
/gpfs/runtime/opt/cuda/10.2/src/lib64/libcublas_static.a
/gpfs/runtime/opt/cuda/10.2/src/lib64/libcublas.so.10
/gpfs/runtime/opt/cuda/10.2/src/lib64/libcublasLt_static.a
/gpfs/runtime/opt/cuda/10.2/src/lib64/stubs/libcublasLt.so
/gpfs/runtime/opt/cuda/10.2/src/lib64/stubs/libcublas.so
/gpfs/runtime/opt/cuda/10.2/src/lib64/libnvblas.so.10
/gpfs/runtime/opt/cuda/10.2/src/lib64/libcublasLt.so.10.2.2.89
/gpfs/runtime/opt/cuda/10.2/src/lib64/libnvblas.so
/gpfs/runtime/opt/cuda/10.2/src/lib64/libcublas.so

$ find /gpfs/runtime/opt/cuda/10.1.105/ -name "*blas*"
/gpfs/runtime/opt/cuda/10.1.105/cuda/doc/man/man7/libcublas.7
/gpfs/runtime/opt/cuda/10.1.105/cuda/doc/man/man7/libcublas.so.7
/gpfs/runtime/opt/cuda/10.1.105/cuda/doc/html/cublas
/gpfs/runtime/opt/cuda/10.1.105/cuda/doc/html/cublas/graphics/cublasmg_gemm.jpg
/gpfs/runtime/opt/cuda/10.1.105/cuda/doc/html/nvblas
/gpfs/runtime/opt/cuda/10.1.105/cuda/targets/x86_64-linux/include/cublas_api.h
/gpfs/runtime/opt/cuda/10.1.105/cuda/targets/x86_64-linux/include/nvblas.h
/gpfs/runtime/opt/cuda/10.1.105/cuda/targets/x86_64-linux/include/cublas.h
/gpfs/runtime/opt/cuda/10.1.105/cuda/targets/x86_64-linux/include/cublas_v2.h
/gpfs/runtime/opt/cuda/10.1.105/cuda/targets/x86_64-linux/include/cublasLt.h
/gpfs/runtime/opt/cuda/10.1.105/cuda/targets/x86_64-linux/include/cublasXt.h
/gpfs/runtime/opt/cuda/10.1.105/cuda/targets/x86_64-linux/lib/libcublasLt.so
/gpfs/runtime/opt/cuda/10.1.105/cuda/targets/x86_64-linux/lib/libnvblas.so.10.1.0.105
/gpfs/runtime/opt/cuda/10.1.105/cuda/targets/x86_64-linux/lib/libcublasLt.so.10.1.0.105
/gpfs/runtime/opt/cuda/10.1.105/cuda/targets/x86_64-linux/lib/libcublasLt.so.10
/gpfs/runtime/opt/cuda/10.1.105/cuda/targets/x86_64-linux/lib/libcublas_static.a
/gpfs/runtime/opt/cuda/10.1.105/cuda/targets/x86_64-linux/lib/libcublas.so.10
/gpfs/runtime/opt/cuda/10.1.105/cuda/targets/x86_64-linux/lib/libcublasLt_static.a
/gpfs/runtime/opt/cuda/10.1.105/cuda/targets/x86_64-linux/lib/stubs/libcublasLt.so
/gpfs/runtime/opt/cuda/10.1.105/cuda/targets/x86_64-linux/lib/stubs/libcublas.so
/gpfs/runtime/opt/cuda/10.1.105/cuda/targets/x86_64-linux/lib/libnvblas.so.10
/gpfs/runtime/opt/cuda/10.1.105/cuda/targets/x86_64-linux/lib/libcublas.so.10.1.0.105
/gpfs/runtime/opt/cuda/10.1.105/cuda/targets/x86_64-linux/lib/libnvblas.so
/gpfs/runtime/opt/cuda/10.1.105/cuda/targets/x86_64-linux/lib/libcublas.so

$ find /gpfs/runtime/opt/cuda/10.0.130/ -name "*blas*"
/gpfs/runtime/opt/cuda/10.0.130/cuda/include/cublas_api.h
/gpfs/runtime/opt/cuda/10.0.130/cuda/include/nvblas.h
/gpfs/runtime/opt/cuda/10.0.130/cuda/include/cublas.h
/gpfs/runtime/opt/cuda/10.0.130/cuda/include/cublas_v2.h
/gpfs/runtime/opt/cuda/10.0.130/cuda/include/cublasXt.h
/gpfs/runtime/opt/cuda/10.0.130/cuda/lib64/libnvblas.so.10.0
/gpfs/runtime/opt/cuda/10.0.130/cuda/lib64/libcublas_static.a
/gpfs/runtime/opt/cuda/10.0.130/cuda/lib64/stubs/libcublas.so
/gpfs/runtime/opt/cuda/10.0.130/cuda/lib64/libnvblas.so
/gpfs/runtime/opt/cuda/10.0.130/cuda/lib64/libnvblas.so.10.0.130
/gpfs/runtime/opt/cuda/10.0.130/cuda/lib64/libcublas.so.10.0.130
/gpfs/runtime/opt/cuda/10.0.130/cuda/lib64/libcublas.so.10.0
/gpfs/runtime/opt/cuda/10.0.130/cuda/lib64/libcublas.so
/gpfs/runtime/opt/cuda/10.0.130/cuda/doc/man/man7/libcublas.7
/gpfs/runtime/opt/cuda/10.0.130/cuda/doc/man/man7/libcublas.so.7
/gpfs/runtime/opt/cuda/10.0.130/cuda/doc/html/cublas
/gpfs/runtime/opt/cuda/10.0.130/cuda/doc/html/cublas/graphics/cublasmg_gemm.jpg
/gpfs/runtime/opt/cuda/10.0.130/cuda/doc/html/nvblas
/gpfs/runtime/opt/cuda/10.0.130/cuda/pkgconfig/cublas-10.0.pc
DilumAluthge commented 4 years ago

/gpfs/runtime/opt/cuda/10.2/src/lib64/libcublas.so

Why is it in the src directory???

maleadt commented 4 years ago

OK, this is your cluster being set-up all weird. nvdisasm is picked up in /gpfs/runtime/opt/cuda/10.2/cuda/bin, so CUDAapi assumes /gpfs/runtime/opt/cuda/10.2/cuda is the toolkit root. However, the CUBLAS library is located in /gpfs/runtime/opt/cuda/10.2/src/lib64... I haven't seen such a src prefix anywhere, so it's not a common thing. If the admins want to structure it like that, at least there should be some configuration pointing towards it, e.g. adding that directory to the library search path (as apparently was done for CUDA 10.0 and 10.1).

DilumAluthge commented 4 years ago

You have to load the different environments with "modules".

So e.g. when I run module load cuda/10.0.130, it exports the LD_LIBRARY_PATH environment variable to include the /gpfs/runtime/opt/cuda/10.0.130/cuda/lib64 directory.

So presumably the cluster admins need to do something similar for their cuda/10.2 module.

DilumAluthge commented 4 years ago

While I wait for them, is there any way I can point Julia directly to the location of libcublas.so?

maleadt commented 4 years ago

Define LD_LIBRARY_PATH yourself? CUDAapi doesn't support per-library overrides, but picks up whatever is loadable out of the box. Alternatively, if that src directory contains the entire toolkit, you can set CUDA_ROOT to force the prefix to that directory.

DilumAluthge commented 4 years ago

Alternatively, if that src directory contains the entire toolkit, you can set CUDA_ROOT to force the prefix to that directory.

Unfortunately it does not. It only contains the libcublas libraries and some libnvblas libraries.

All the other libraries (e.g. libcudart, libcurand, libcufft, etc.) are in /gpfs/runtime/opt/cuda/10.2/cuda/lib64

Seems like maybe a botched install of CUDA by the cluster admins? It's unclear to me why you would move those specific libraries to a separate location.

DilumAluthge commented 4 years ago

I'll probably just use their CUDA 10.0 or 10.1 install (since those seem to be installed correctly) until they fix it.