Closed boogiewoogit closed 1 year ago
maleadt
commented
1 year ago
┌ Debug: No system CUDA driver found └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:77
If you look at the preceding functionality, we just call find_library, so something must be off with your set-up: https://github.com/JuliaBinaryWrappers/CUDA_Driver_jll.jl/blob/e24faca71ae83b2c7c8b64543fac33baac367e40/src/wrappers/x86_64-linux-gnu.jl#L71-L79
boogiewoogit
commented
1 year ago
┌ Debug: No system CUDA driver found └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:77If you look at the preceding functionality, we just call
find_library, so something must be off with your set-up: https://github.com/JuliaBinaryWrappers/CUDA_Driver_jll.jl/blob/e24faca71ae83b2c7c8b64543fac33baac367e40/src/wrappers/x86_64-linux-gnu.jl#L71-L79
Yes, I checked that, it's what lead me to the documentation of find_library and in consequence me manually setting Base.DL_LOAD_PATH.
What throws me off is the fact that find_library should find libcuda where it doesn't, as indicated by the first checks which are successful, and as showcased in my direct calling of find_library toward the bottom of the post.
maleadt
commented
1 year ago How are you setting DL_LOAD_PATH? It's an internal Julia setting, so doesn't propagate to spawned processes. And Pkg.test does testing in a subprocess.
You probably want to put the directory containing libcuda in the LD_LIBRARY_PATH instead, or otherwise make sure the dynamic loader can find it automatically (e.g. through ld.so.conf.d).
boogiewoogit
commented
1 year ago How are you setting
DL_LOAD_PATH? It's an internal Julia setting, so doesn't propagate to spawned processes. AndPkg.testdoes testing in a subprocess.
In my startup.jl, via push!(Base.DL_LOAD_PATH,"/path/to/libcuda"). The fact that it's subprocesses explains this, thank you for clarifying.
You probably want to put the directory containing
libcudain the LD_LIBRARY_PATH instead, or otherwise make sure the dynamic loader can find it automatically (e.g. throughld.so.conf.d).
Yep, did that and it worked. The reason I didn't is that in NixOS, LD_LIBRARY_PATH doesn't really work as it usually does, so you have to take some extra steps. And just from the documentation, DL_LOAD_PATH did seem to fit the bill while being more direct.
For future reference (and for any NixOS user stumbling upon this issue):
To get the correct LD_LIBRARY_PATH to your julia, you can use this overlay (i.e., add this line to your configuration.nix):
nixpkgs.overlays = [ (final: prev: { julia-bin = prev.julia-bin.overrideAttrs (old: rec { nativeBuildInputs = old.nativeBuildInputs ++ [ pkgs.makeWrapper ]; postFixup = ''wrapProgram "$out/bin/julia" --prefix LD_LIBRARY_PATH : ${lib.makeLibraryPath (with pkgs; [ stdenv.cc.cc linuxPackages.nvidia_x11 ])}''; }); }) ];And to get the correct ELF interpreter to julia artifacts (e.g. ptxas, which CUDA uses), you'll need to have NIX_LD and NIX_LD_LIBRARY_PATH set (see nix-ld), for example globally with these lines in your configuration.nix:
environment.variables = {
NIX_LD = "${pkgs.glibc}/lib/ld-linux-x86-64.so.2";
NIX_LD_LIBRARY_PATH = lib.makeLibraryPath (with pkgs; [ stdenv.cc.cc linuxPackages.nvidia_x11 ]);
};I changed the title to make this more discoverable. :-)
maleadt
commented
1 year ago Thanks for the update!
arunoruto
commented
2 months ago Sorry for answering on a closed issue, but have you tried using nix-ld? By providing your packages to programs.nix-ld.libraries, the paths will be available in $NIX_LD_LIBRARY_PATH. I had a similar problem with oneAPI.jl and fixed it by running LD_LIBRARY_PATH=$NIX_LD_LIBRARY_PATH julia. Maybe julia should also look at NIX_LD_LIBRARY_PATH for libraries by default, but that's just an idea of mine. I am not that well versed in libs.
When testing CUDA, I get both
in the initial checks and then, during the actual testing,
Full REPL output
``` (@v1.9) pkg> test CUDA ┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 12.2.0 └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:103 ┌ Debug: System CUDA driver is recent enough; not using forward-compatible driver └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:121 Testing CUDA ┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 12.2.0 └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:103 ┌ Debug: System CUDA driver is recent enough; not using forward-compatible driver └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:121 Status `/tmp/jl_ckpRkS/Project.toml` ┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 12.2.0 └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:103 ┌ Debug: System CUDA driver is recent enough; not using forward-compatible driver └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:121 [ ------------ collapsed for brevity ------------ ] Status `/tmp/jl_ckpRkS/Manifest.toml` ┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 12.2.0 └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:103 ┌ Debug: System CUDA driver is recent enough; not using forward-compatible driver └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:121 [ ------------ collapsed for brevity ------------ ] ┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 12.2.0 └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:103 ┌ Debug: System CUDA driver is recent enough; not using forward-compatible driver └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:121 Precompiling project... 88 dependencies successfully precompiled in 166 seconds 3 dependencies had warnings during precompilation: ┌ SpecialFunctionsExt [1285c0f1-ff9f-5867-b66e-0f359bcf09ba] │ ┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 12.2.0 │ └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:103 │ ┌ Debug: System CUDA driver is recent enough; not using forward-compatible driver │ └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:121 └ ┌ CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2] │ ┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 12.2.0 │ └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:103 │ ┌ Debug: System CUDA driver is recent enough; not using forward-compatible driver │ └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:121 │ ┌ Debug: Using CUDA_Driver_jll for driver discovery │ └ @ CUDA_Runtime_jll ~/.julia/packages/CUDA_Runtime_jll/GnSRU/.pkg/platform_augmentation.jl:127 │ ┌ Debug: Found CUDA driver at 'libcuda.so.1' │ └ @ CUDA_Runtime_jll ~/.julia/packages/CUDA_Runtime_jll/GnSRU/.pkg/platform_augmentation.jl:154 │ ┌ Debug: CUDA driver version: 12.2.0 │ └ @ CUDA_Runtime_jll ~/.julia/packages/CUDA_Runtime_jll/GnSRU/.pkg/platform_augmentation.jl:217 │ ┌ Debug: Selected CUDA toolkit: 12.2.0 │ └ @ CUDA_Runtime_jll ~/.julia/packages/CUDA_Runtime_jll/GnSRU/.pkg/platform_augmentation.jl:248 └ ┌ CUDA [052768ef-5323-5732-b1bb-66c8b64840ba] │ ┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 12.2.0 │ └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:103 │ ┌ Debug: System CUDA driver is recent enough; not using forward-compatible driver │ └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:121 └ Testing Running tests... ┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 12.2.0 └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:103 ┌ Debug: System CUDA driver is recent enough; not using forward-compatible driver └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:121 ┌ Info: System information: │ CUDA runtime 12.2, artifact installation │ CUDA driver 12.2 │ NVIDIA driver 535.104.5 │ │ CUDA libraries: │ - CUBLAS: 12.2.5 │ - CURAND: 10.3.3 │ - CUFFT: 11.0.8 │ - CUSOLVER: 11.5.2 │ - CUSPARSE: 12.1.2 │ - CUPTI: 20.0.0 │ - NVML: 12.0.0+535.104.5 │ │ Julia packages: │ - CUDA: 5.0.0 │ - CUDA_Driver_jll: 0.6.0+3 │ - CUDA_Runtime_jll: 0.9.2+0 │ │ Toolchain: │ - Julia: 1.9.3 │ - LLVM: 14.0.6 │ - PTX ISA support: 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3, 6.4, 6.5, 7.0, 7.1, 7.2, 7.3, 7.4, 7.5 │ - Device capability support: sm_37, sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80, sm_86 │ │ 1 device: └ 0: NVIDIA GeForce GTX 980 (sm_52, 3.250 GiB / 4.000 GiB available) [ Info: Testing using device 0 (NVIDIA GeForce GTX 980). To change this, specify the `--gpus` argument to the tests, or set the `CUDA_VISIBLE_DEVICES` environment variable. ┌ Debug: cuBLAS (v12.2) function cublasStatus_t cublasGetProperty(libraryPropertyType, int*) called: │ type: type=SOME TYPE; val=0 │ value: type=int; val=POINTER (IN HEX:0x0x7f3d6548a7b0) │ Time: 2023-09-20T12:39:12 elapsed from start 0.216667 minutes or 13.000000 seconds │ Process=93996; Thread=139905249177856; GPU=0; Handle=POINTER (IN HEX:0x(nil)) │ COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3) └ @ CUDA.CUBLAS ~/.julia/packages/CUDA/nbRJk/lib/cublas/CUBLAS.jl:224 [ Info: Running 1 tests in parallel. If this is too many, specify the `--jobs` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable. ┌ Debug: cuBLAS (v12.2) function cublasStatus_t cublasGetProperty(libraryPropertyType, int*) called: │ type: type=SOME TYPE; val=1 │ value: type=int; val=POINTER (IN HEX:0x0x7f3d6548a800) │ Time: 2023-09-20T12:39:12 elapsed from start 0.216667 minutes or 13.000000 seconds │ Process=93996; Thread=139905249177856; GPU=0; Handle=POINTER (IN HEX:0x(nil)) │ COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3) └ @ CUDA.CUBLAS ~/.julia/packages/CUDA/nbRJk/lib/cublas/CUBLAS.jl:224 ┌ Debug: cuBLAS (v12.2) function cublasStatus_t cublasGetProperty(libraryPropertyType, int*) called: │ type: type=SOME TYPE; val=2 │ value: type=int; val=POINTER (IN HEX:0x0x7f3d6548a850) │ Time: 2023-09-20T12:39:12 elapsed from start 0.216667 minutes or 13.000000 seconds │ Process=93996; Thread=139905249177856; GPU=0; Handle=POINTER (IN HEX:0x(nil)) │ COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3) │ └ @ CUDA.CUBLAS ~/.julia/packages/CUDA/nbRJk/lib/cublas/CUBLAS.jl:224 From worker 2: ┌ Debug: No system CUDA driver found From worker 2: └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:77 | | ---------------- GPU ---------------- | ---------------- CPU ---------------- | Test (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB) | GC (s) | GC % | Alloc (MB) | RSS (MB) | core/initialization (2) | failed at 2023-09-20T12:39:32.433 From worker 3: ┌ Debug: No system CUDA driver found From worker 3: └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:77 gpuarrays/indexing scalar (3) | failed at 2023-09-20T12:39:46.592 From worker 4: ┌ Debug: No system CUDA driver found From worker 4: └ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/eDshu/src/wrappers/x86_64-linux-gnu.jl:77 gpuarrays/reductions/reducedim! (4) | failed at 2023-09-20T12:40:00.593 ^C Testing Tests interrupted. Exiting the test process ```
To reproduce
Manifest.toml
``` # This file is machine-generated - editing it directly is not advised julia_version = "1.9.3" manifest_format = "2.0" project_hash = "4d6f52122ea9741175c8c71d00021be0a921f3ad" [[deps.AbstractFFTs]] deps = ["LinearAlgebra"] git-tree-sha1 = "d92ad398961a3ed262d8bf04a1a2b8340f915fef" uuid = "621f4979-c628-5d54-868e-fcf4e3e8185c" version = "1.5.0" [deps.AbstractFFTs.extensions] AbstractFFTsChainRulesCoreExt = "ChainRulesCore" AbstractFFTsTestExt = "Test" [deps.AbstractFFTs.weakdeps] ChainRulesCore = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4" Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40" [[deps.Adapt]] deps = ["LinearAlgebra", "Requires"] git-tree-sha1 = "76289dc51920fdc6e0013c872ba9551d54961c24" uuid = "79e6a3ab-5dfb-504d-930d-738a2a938a0e" version = "3.6.2" weakdeps = ["StaticArrays"] [deps.Adapt.extensions] AdaptStaticArraysExt = "StaticArrays" [[deps.ArgTools]] uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f" version = "1.1.1" [[deps.Artifacts]] uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33" [[deps.Atomix]] deps = ["UnsafeAtomics"] git-tree-sha1 = "c06a868224ecba914baa6942988e2f2aade419be" uuid = "a9b6321e-bd34-4604-b9c9-b65b8de01458" version = "0.1.0" [[deps.BFloat16s]] deps = ["LinearAlgebra", "Printf", "Random", "Test"] git-tree-sha1 = "dbf84058d0a8cbbadee18d25cf606934b22d7c66" uuid = "ab4f0b2a-ad5b-11e8-123f-65d77653426b" version = "0.4.2" [[deps.Base64]] uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f" [[deps.CEnum]] git-tree-sha1 = "eb4cb44a499229b3b8426dcfb5dd85333951ff90" uuid = "fa961155-64e5-5f13-b03f-caf6b980ea82" version = "0.4.2" [[deps.CUDA]] deps = ["AbstractFFTs", "Adapt", "BFloat16s", "CEnum", "CUDA_Driver_jll", "CUDA_Runtime_Discovery", "CUDA_Runtime_jll", "Crayons", "DataFrames", "ExprTools", "GPUArrays", "GPUCompiler", "KernelAbstractions", "LLVM", "LazyArtifacts", "Libdl", "LinearAlgebra", "Logging", "NVTX", "Preferences", "PrettyTables", "Printf", "Random", "Random123", "RandomNumbers", "Reexport", "Requires", "SparseArrays", "Statistics", "UnsafeAtomicsLLVM"] git-tree-sha1 = "f062a48c26ae027f70c44f48f244862aec47bf99" uuid = "052768ef-5323-5732-b1bb-66c8b64840ba" version = "5.0.0" [deps.CUDA.extensions] SpecialFunctionsExt = "SpecialFunctions" [deps.CUDA.weakdeps] SpecialFunctions = "276daf66-3868-5448-9aa4-cd146d93841b" [[deps.CUDA_Driver_jll]] deps = ["Artifacts", "JLLWrappers", "LazyArtifacts", "Libdl", "Pkg"] git-tree-sha1 = "35a37bb72b35964f2895c12c687ae263b4ac170c" uuid = "4ee394cb-3365-5eb0-8335-949819d2adfc" version = "0.6.0+3" [[deps.CUDA_Runtime_Discovery]] deps = ["Libdl"] git-tree-sha1 = "bcc4a23cbbd99c8535a5318455dcf0f2546ec536" uuid = "1af6417a-86b4-443c-805f-a4643ffb695f" version = "0.2.2" [[deps.CUDA_Runtime_jll]] deps = ["Artifacts", "CUDA_Driver_jll", "JLLWrappers", "LazyArtifacts", "Libdl", "TOML"] git-tree-sha1 = "bfe5a693a11522d58392f742243f2b50dc27afd6" uuid = "76a88914-d11a-5bdc-97e0-2f5a05c973a2" version = "0.9.2+0" [[deps.ColorTypes]] deps = ["FixedPointNumbers", "Random"] git-tree-sha1 = "eb7f0f8307f71fac7c606984ea5fb2817275d6e4" uuid = "3da002f7-5984-5a60-b8a6-cbb66c0b333f" version = "0.11.4" [[deps.Colors]] deps = ["ColorTypes", "FixedPointNumbers", "Reexport"] git-tree-sha1 = "fc08e5930ee9a4e03f84bfb5211cb54e7769758a" uuid = "5ae59095-9a9b-59fe-a467-6f913c188581" version = "0.12.10" [[deps.Compat]] deps = ["UUIDs"] git-tree-sha1 = "e460f044ca8b99be31d35fe54fc33a5c33dd8ed7" uuid = "34da2185-b29b-5c13-b0c7-acf172513d20" version = "4.9.0" weakdeps = ["Dates", "LinearAlgebra"] [deps.Compat.extensions] CompatLinearAlgebraExt = "LinearAlgebra" [[deps.CompilerSupportLibraries_jll]] deps = ["Artifacts", "Libdl"] uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae" version = "1.0.5+0" [[deps.Crayons]] git-tree-sha1 = "249fe38abf76d48563e2f4556bebd215aa317e15" uuid = "a8cc5b0e-0ffa-5ad4-8c14-923d3ee1735f" version = "4.1.1" [[deps.DataAPI]] git-tree-sha1 = "8da84edb865b0b5b0100c0666a9bc9a0b71c553c" uuid = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a" version = "1.15.0" [[deps.DataFrames]] deps = ["Compat", "DataAPI", "DataStructures", "Future", "InlineStrings", "InvertedIndices", "IteratorInterfaceExtensions", "LinearAlgebra", "Markdown", "Missings", "PooledArrays", "PrecompileTools", "PrettyTables", "Printf", "REPL", "Random", "Reexport", "SentinelArrays", "SortingAlgorithms", "Statistics", "TableTraits", "Tables", "Unicode"] git-tree-sha1 = "04c738083f29f86e62c8afc341f0967d8717bdb8" uuid = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0" version = "1.6.1" [[deps.DataStructures]] deps = ["Compat", "InteractiveUtils", "OrderedCollections"] git-tree-sha1 = "3dbd312d370723b6bb43ba9d02fc36abade4518d" uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8" version = "0.18.15" [[deps.DataValueInterfaces]] git-tree-sha1 = "bfc1187b79289637fa0ef6d4436ebdfe6905cbd6" uuid = "e2d170a0-9d28-54be-80f0-106bbe20a464" version = "1.0.0" [[deps.Dates]] deps = ["Printf"] uuid = "ade2ca70-3891-5945-98fb-dc099432e06a" [[deps.Downloads]] deps = ["ArgTools", "FileWatching", "LibCURL", "NetworkOptions"] uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6" version = "1.6.0" [[deps.ExprTools]] git-tree-sha1 = "27415f162e6028e81c72b82ef756bf321213b6ec" uuid = "e2ba6199-217a-4e67-a87a-7c52f15ade04" version = "0.1.10" [[deps.FileWatching]] uuid = "7b1f6079-737a-58dc-b8bc-7a2ca5c1b5ee" [[deps.FixedPointNumbers]] deps = ["Statistics"] git-tree-sha1 = "335bfdceacc84c5cdf16aadc768aa5ddfc5383cc" uuid = "53c48c17-4a7d-5ca2-90c5-79b7896eea93" version = "0.8.4" [[deps.Future]] deps = ["Random"] uuid = "9fa8497b-333b-5362-9e8d-4d0656e87820" [[deps.GPUArrays]] deps = ["Adapt", "GPUArraysCore", "LLVM", "LinearAlgebra", "Printf", "Random", "Reexport", "Serialization", "Statistics"] git-tree-sha1 = "8ad8f375ae365aa1eb2f42e2565a40b55a4b69a8" uuid = "0c68f7d7-f131-5f86-a1c3-88cf8149b2d7" version = "9.0.0" [[deps.GPUArraysCore]] deps = ["Adapt"] git-tree-sha1 = "2d6ca471a6c7b536127afccfa7564b5b39227fe0" uuid = "46192b85-c4d5-4398-a991-12ede77f4527" version = "0.1.5" [[deps.GPUCompiler]] deps = ["ExprTools", "InteractiveUtils", "LLVM", "Libdl", "Logging", "Scratch", "TimerOutputs", "UUIDs"] git-tree-sha1 = "fa7dcf83344770adde87b256784155b194e05c9a" uuid = "61eb1bfa-7361-4325-ad38-22787b887f55" version = "0.24.4" [[deps.InlineStrings]] deps = ["Parsers"] git-tree-sha1 = "9cc2baf75c6d09f9da536ddf58eb2f29dedaf461" uuid = "842dd82b-1e85-43dc-bf29-5d0ee9dffc48" version = "1.4.0" [[deps.InteractiveUtils]] deps = ["Markdown"] uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240" [[deps.InvertedIndices]] git-tree-sha1 = "0dc7b50b8d436461be01300fd8cd45aa0274b038" uuid = "41ab1584-1d38-5bbf-9106-f11c6c58b48f" version = "1.3.0" [[deps.IteratorInterfaceExtensions]] git-tree-sha1 = "a3f24677c21f5bbe9d2a714f95dcd58337fb2856" uuid = "82899510-4779-5014-852e-03e436cf321d" version = "1.0.0" [[deps.JLLWrappers]] deps = ["Artifacts", "Preferences"] git-tree-sha1 = "7e5d6779a1e09a36db2a7b6cff50942a0a7d0fca" uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210" version = "1.5.0" [[deps.JuliaNVTXCallbacks_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] git-tree-sha1 = "af433a10f3942e882d3c671aacb203e006a5808f" uuid = "9c1d0b0a-7046-5b2e-a33f-ea22f176ac7e" version = "0.2.1+0" [[deps.KernelAbstractions]] deps = ["Adapt", "Atomix", "InteractiveUtils", "LinearAlgebra", "MacroTools", "PrecompileTools", "Requires", "SparseArrays", "StaticArrays", "UUIDs", "UnsafeAtomics", "UnsafeAtomicsLLVM"] git-tree-sha1 = "4c5875e4c228247e1c2b087669846941fb6e0118" uuid = "63c18a36-062a-441e-b654-da1e3ab1ce7c" version = "0.9.8" [deps.KernelAbstractions.extensions] EnzymeExt = "EnzymeCore" [deps.KernelAbstractions.weakdeps] EnzymeCore = "f151be2c-9106-41f4-ab19-57ee4f262869" [[deps.LLVM]] deps = ["CEnum", "LLVMExtra_jll", "Libdl", "Printf", "Unicode"] git-tree-sha1 = "a9d2ce1d5007b1e8f6c5b89c5a31ff8bd146db5c" uuid = "929cbde3-209d-540e-8aea-75f648917ca0" version = "6.2.1" [[deps.LLVMExtra_jll]] deps = ["Artifacts", "JLLWrappers", "LazyArtifacts", "Libdl", "TOML"] git-tree-sha1 = "7ca6850ae880cc99b59b88517545f91a52020afa" uuid = "dad2f222-ce93-54a1-a47d-0025e8a3acab" version = "0.0.25+0" [[deps.LaTeXStrings]] git-tree-sha1 = "f2355693d6778a178ade15952b7ac47a4ff97996" uuid = "b964fa9f-0449-5b57-a5c2-d3ea65f4040f" version = "1.3.0" [[deps.LazyArtifacts]] deps = ["Artifacts", "Pkg"] uuid = "4af54fe1-eca0-43a8-85a7-787d91b784e3" [[deps.LibCURL]] deps = ["LibCURL_jll", "MozillaCACerts_jll"] uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21" version = "0.6.3" [[deps.LibCURL_jll]] deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"] uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0" version = "7.84.0+0" [[deps.LibGit2]] deps = ["Base64", "NetworkOptions", "Printf", "SHA"] uuid = "76f85450-5226-5b5a-8eaa-529ad045b433" [[deps.LibSSH2_jll]] deps = ["Artifacts", "Libdl", "MbedTLS_jll"] uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8" version = "1.10.2+0" [[deps.Libdl]] uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb" [[deps.LinearAlgebra]] deps = ["Libdl", "OpenBLAS_jll", "libblastrampoline_jll"] uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" [[deps.Logging]] uuid = "56ddb016-857b-54e1-b83d-db4d58db5568" [[deps.MacroTools]] deps = ["Markdown", "Random"] git-tree-sha1 = "9ee1618cbf5240e6d4e0371d6f24065083f60c48" uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09" version = "0.5.11" [[deps.Markdown]] deps = ["Base64"] uuid = "d6f4376e-aef5-505a-96c1-9c027394607a" [[deps.MbedTLS_jll]] deps = ["Artifacts", "Libdl"] uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1" version = "2.28.2+0" [[deps.Missings]] deps = ["DataAPI"] git-tree-sha1 = "f66bdc5de519e8f8ae43bdc598782d35a25b1272" uuid = "e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28" version = "1.1.0" [[deps.MozillaCACerts_jll]] uuid = "14a3606d-f60d-562e-9121-12d972cd8159" version = "2022.10.11" [[deps.NVTX]] deps = ["Colors", "JuliaNVTXCallbacks_jll", "Libdl", "NVTX_jll"] git-tree-sha1 = "8bc9ce4233be3c63f8dcd78ccaf1b63a9c0baa34" uuid = "5da4648a-3479-48b8-97b9-01cb529c0a1f" version = "0.3.3" [[deps.NVTX_jll]] deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] git-tree-sha1 = "ce3269ed42816bf18d500c9f63418d4b0d9f5a3b" uuid = "e98f9f5b-d649-5603-91fd-7774390e6439" version = "3.1.0+2" [[deps.NetworkOptions]] uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908" version = "1.2.0" [[deps.OpenBLAS_jll]] deps = ["Artifacts", "CompilerSupportLibraries_jll", "Libdl"] uuid = "4536629a-c528-5b80-bd46-f80d51c5b363" version = "0.3.21+4" [[deps.OrderedCollections]] git-tree-sha1 = "2e73fe17cac3c62ad1aebe70d44c963c3cfdc3e3" uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d" version = "1.6.2" [[deps.Parsers]] deps = ["Dates", "PrecompileTools", "UUIDs"] git-tree-sha1 = "716e24b21538abc91f6205fd1d8363f39b442851" uuid = "69de0a69-1ddd-5017-9359-2bf0b02dc9f0" version = "2.7.2" [[deps.Pkg]] deps = ["Artifacts", "Dates", "Downloads", "FileWatching", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"] uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f" version = "1.9.2" [[deps.PooledArrays]] deps = ["DataAPI", "Future"] git-tree-sha1 = "36d8b4b899628fb92c2749eb488d884a926614d3" uuid = "2dfb63ee-cc39-5dd5-95bd-886bf059d720" version = "1.4.3" [[deps.PrecompileTools]] deps = ["Preferences"] git-tree-sha1 = "03b4c25b43cb84cee5c90aa9b5ea0a78fd848d2f" uuid = "aea7be01-6a6a-4083-8856-8a6e6704d82a" version = "1.2.0" [[deps.Preferences]] deps = ["TOML"] git-tree-sha1 = "7eb1686b4f04b82f96ed7a4ea5890a4f0c7a09f1" uuid = "21216c6a-2e73-6563-6e65-726566657250" version = "1.4.0" [[deps.PrettyTables]] deps = ["Crayons", "LaTeXStrings", "Markdown", "Printf", "Reexport", "StringManipulation", "Tables"] git-tree-sha1 = "ee094908d720185ddbdc58dbe0c1cbe35453ec7a" uuid = "08abe8d2-0d0c-5749-adfa-8a2ac140af0d" version = "2.2.7" [[deps.Printf]] deps = ["Unicode"] uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7" [[deps.REPL]] deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"] uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb" [[deps.Random]] deps = ["SHA", "Serialization"] uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c" [[deps.Random123]] deps = ["Random", "RandomNumbers"] git-tree-sha1 = "552f30e847641591ba3f39fd1bed559b9deb0ef3" uuid = "74087812-796a-5b5d-8853-05524746bad3" version = "1.6.1" [[deps.RandomNumbers]] deps = ["Random", "Requires"] git-tree-sha1 = "043da614cc7e95c703498a491e2c21f58a2b8111" uuid = "e6cf234a-135c-5ec9-84dd-332b85af5143" version = "1.5.3" [[deps.Reexport]] git-tree-sha1 = "45e428421666073eab6f2da5c9d310d99bb12f9b" uuid = "189a3867-3050-52da-a836-e630ba90ab69" version = "1.2.2" [[deps.Requires]] deps = ["UUIDs"] git-tree-sha1 = "838a3a4188e2ded87a4f9f184b4b0d78a1e91cb7" uuid = "ae029012-a4dd-5104-9daa-d747884805df" version = "1.3.0" [[deps.SHA]] uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce" version = "0.7.0" [[deps.Scratch]] deps = ["Dates"] git-tree-sha1 = "30449ee12237627992a99d5e30ae63e4d78cd24a" uuid = "6c6a2e73-6563-6170-7368-637461726353" version = "1.2.0" [[deps.SentinelArrays]] deps = ["Dates", "Random"] git-tree-sha1 = "04bdff0b09c65ff3e06a05e3eb7b120223da3d39" uuid = "91c51154-3ec4-41a3-a24f-3f23e20d615c" version = "1.4.0" [[deps.Serialization]] uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b" [[deps.Sockets]] uuid = "6462fe0b-24de-5631-8697-dd941f90decc" [[deps.SortingAlgorithms]] deps = ["DataStructures"] git-tree-sha1 = "c60ec5c62180f27efea3ba2908480f8055e17cee" uuid = "a2af1166-a08f-5f64-846c-94a0d3cef48c" version = "1.1.1" [[deps.SparseArrays]] deps = ["Libdl", "LinearAlgebra", "Random", "Serialization", "SuiteSparse_jll"] uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf" [[deps.StaticArrays]] deps = ["LinearAlgebra", "Random", "StaticArraysCore"] git-tree-sha1 = "51621cca8651d9e334a659443a74ce50a3b6dfab" uuid = "90137ffa-7385-5640-81b9-e52037218182" version = "1.6.3" weakdeps = ["Statistics"] [deps.StaticArrays.extensions] StaticArraysStatisticsExt = "Statistics" [[deps.StaticArraysCore]] git-tree-sha1 = "36b3d696ce6366023a0ea192b4cd442268995a0d" uuid = "1e83bf80-4336-4d27-bf5d-d5a4f845583c" version = "1.4.2" [[deps.Statistics]] deps = ["LinearAlgebra", "SparseArrays"] uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" version = "1.9.0" [[deps.StringManipulation]] deps = ["PrecompileTools"] git-tree-sha1 = "a04cabe79c5f01f4d723cc6704070ada0b9d46d5" uuid = "892a3eda-7b42-436c-8928-eab12a02cf0e" version = "0.3.4" [[deps.SuiteSparse_jll]] deps = ["Artifacts", "Libdl", "Pkg", "libblastrampoline_jll"] uuid = "bea87d4a-7f5b-5778-9afe-8cc45184846c" version = "5.10.1+6" [[deps.TOML]] deps = ["Dates"] uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76" version = "1.0.3" [[deps.TableTraits]] deps = ["IteratorInterfaceExtensions"] git-tree-sha1 = "c06b2f539df1c6efa794486abfb6ed2022561a39" uuid = "3783bdb8-4a98-5b6b-af9a-565f29a5fe9c" version = "1.0.1" [[deps.Tables]] deps = ["DataAPI", "DataValueInterfaces", "IteratorInterfaceExtensions", "LinearAlgebra", "OrderedCollections", "TableTraits"] git-tree-sha1 = "a1f34829d5ac0ef499f6d84428bd6b4c71f02ead" uuid = "bd369af6-aec1-5ad0-b16a-f7cc5008161c" version = "1.11.0" [[deps.Tar]] deps = ["ArgTools", "SHA"] uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e" version = "1.10.0" [[deps.Test]] deps = ["InteractiveUtils", "Logging", "Random", "Serialization"] uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40" [[deps.TimerOutputs]] deps = ["ExprTools", "Printf"] git-tree-sha1 = "f548a9e9c490030e545f72074a41edfd0e5bcdd7" uuid = "a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f" version = "0.5.23" [[deps.UUIDs]] deps = ["Random", "SHA"] uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4" [[deps.Unicode]] uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5" [[deps.UnsafeAtomics]] git-tree-sha1 = "6331ac3440856ea1988316b46045303bef658278" uuid = "013be700-e6cd-48c3-b4a1-df204f14c38f" version = "0.2.1" [[deps.UnsafeAtomicsLLVM]] deps = ["LLVM", "UnsafeAtomics"] git-tree-sha1 = "323e3d0acf5e78a56dfae7bd8928c989b4f3083e" uuid = "d80eeb9a-aca5-4d75-85e5-170c8b632249" version = "0.1.3" [[deps.Zlib_jll]] deps = ["Libdl"] uuid = "83775a58-1f1d-513f-b197-d71354ab007a" version = "1.2.13+0" [[deps.libblastrampoline_jll]] deps = ["Artifacts", "Libdl"] uuid = "8e850b90-86db-534c-a0d3-1478176c7d93" version = "5.8.0+0" [[deps.nghttp2_jll]] deps = ["Artifacts", "Libdl"] uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d" version = "1.48.0+0" [[deps.p7zip_jll]] deps = ["Artifacts", "Libdl"] uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0" version = "17.4.0+0" ```
Version info
Details on Julia
``` julia> versioninfo() Julia Version 1.9.3 Commit bed2cd540a1 (2023-08-24 14:43 UTC) Build Info: Official https://julialang.org/ release Platform Info: OS: Linux (x86_64-linux-gnu) CPU: 16 × AMD Ryzen 7 1700 Eight-Core Processor WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-14.0.6 (ORCJIT, znver1) Threads: 1 on 16 virtual cores Environment: JULIA_DEBUG = CUDA_Runtime_jll,CUDA,CUDA_Driver_jll JULIA_NUM_THREADS = 4 ```
Details on CUDA:
``` julia> CUDA.versioninfo() CUDA runtime 12.2, artifact installation CUDA driver 12.2 NVIDIA driver 535.104.5 CUDA libraries: - CUBLAS: 12┌ Debug: cuBLAS (v12.2) function cublasStatus_t cublasGetProperty(libraryPropertyType, int*) called: │ type: type=SOME TYPE; val=0 │ value: type=int; val=POINTER (IN HEX:0x0x7f4b437e97b0) │ Time: 2023-09-20T12:48:33 elapsed from start 0.933333 minutes or 56.000000 seconds │ Process=91174; Thread=139961379438848; GPU=0; Handle=POINTER (IN HEX:0x(nil)) │ COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3) └ @ CUDA.CUBLAS ~/.julia/packages/CUDA/nbRJk/lib/cublas/CUBLAS.jl:224 .┌ Debug: cuBLAS (v12.2) function cublasStatus_t cublasGetProperty(libraryPropertyType, int*) called: │ type: type=SOME TYPE; val=1 │ value: type=int; val=POINTER (IN HEX:0x0x7f4b437e9800) │ Time: 2023-09-20T12:48:33 elapsed from start 0.933333 minutes or 56.000000 seconds │ Process=91174; Thread=139961379438848; GPU=0; Handle=POINTER (IN HEX:0x(nil)) │ COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3) └ @ CUDA.CUBLAS ~/.julia/packages/CUDA/nbRJk/lib/cublas/CUBLAS.jl:224 2┌ Debug: cuBLAS (v12.2) function cublasStatus_t cublasGetProperty(libraryPropertyType, int*) called: │ type: type=SOME TYPE; val=2 │ value: type=int; val=POINTER (IN HEX:0x0x7f4b437e9850) │ Time: 2023-09-20T12:48:33 elapsed from start 0.933333 minutes or 56.000000 seconds │ Process=91174; Thread=139961379438848; GPU=0; Handle=POINTER (IN HEX:0x(nil)) │ COMPILED WITH: GNU GCC/G++ / 6.3.1 20170216 (Red Hat 6.3.1-3) │ └ @ CUDA.CUBLAS ~/.julia/packages/CUDA/nbRJk/lib/cublas/CUBLAS.jl:224 .5 - CURAND: 10.3.3 - CUFFT: 11.0.8 - CUSOLVER: 11.5.2 - CUSPARSE: 12.1.2 - CUPTI: 20.0.0 - NVML: 12.0.0+535.104.5 Julia packages: - CUDA: 5.0.0 - CUDA_Driver_jll: 0.6.0+3 - CUDA_Runtime_jll: 0.9.2+0 Toolchain: - Julia: 1.9.3 - LLVM: 14.0.6 - PTX ISA support: 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3, 6.4, 6.5, 7.0, 7.1, 7.2, 7.3, 7.4, 7.5 - Device capability support: sm_37, sm_50, sm_52, sm_53, sm_60, sm_61, sm_62, sm_70, sm_72, sm_75, sm_80, sm_86 1 device: 0: NVIDIA GeForce GTX 980 (sm_52, 3.288 GiB / 4.000 GiB available) ```
Additional context
I'm using
Base.DL_LOAD_PATHto provide the path for libcuda, since I'm on NixOS, where system library paths are nonstandard:FWIW, a simple addition on CuArrays fails due to another error, which I'm sure is due to NixOS and which I'll try and hunt down on that front.
Failing CuArray addition
``` julia> x = CUDA.fill(1.0f0, 10); y = CUDA.fill(2.0f0, 10) 10-element CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}: 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 julia> y .+= x ERROR: IOError: could not spawn setenv(`/home/boogie/.julia/artifacts/e22703892311db2869af2d2c41ecbbe8e72596c3/bin/ptxas --generate-line-info --verbose --gpu-name sm_52 --output-file /tmp/jl_S7MPMlM5Av.cubin /tmp/jl_QhwZwr83mz.ptx`,[ ---- loads of envars here ---- ]): no such file or directory (ENOENT) Stacktrace: [1] _spawn_primitive(file::String, cmd::Cmd, stdio::Vector{Union{RawFD, IO}}) @ Base ./process.jl:128 [2] #760 @ ./process.jl:139 [inlined] [3] setup_stdios(f::Base.var"#760#761"{Cmd}, stdios::Vector{Union{RawFD, IO}}) @ Base ./process.jl:223 [4] _spawn @ ./process.jl:138 [inlined] [5] _spawn(::Base.CmdRedirect, ::Vector{Union{RawFD, IO}}) (repeats 2 times) @ Base ./process.jl:166 [6] run(::Base.CmdRedirect; wait::Bool) @ Base ./process.jl:483 [7] run @ ./process.jl:477 [inlined] [8] run_and_collect(cmd::Cmd) @ CUDA ~/.julia/packages/CUDA/nbRJk/src/compiler/compilation.jl:313 [9] compile(job::GPUCompiler.CompilerJob) @ CUDA ~/.julia/packages/CUDA/nbRJk/src/compiler/compilation.jl:239 [10] actual_compilation(cache::Dict{Any, CuFunction}, src::Core.MethodInstance, world::UInt64, cfg::GPUCompiler.CompilerConfig{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}, compiler::typeof(CUDA.compile), linker::typeof(CUDA.link)) @ GPUCompiler ~/.julia/packages/GPUCompiler/Cp7sE/src/execution.jl:125 [11] cached_compilation(cache::Dict{Any, CuFunction}, src::Core.MethodInstance, cfg::GPUCompiler.CompilerConfig{GPUCompiler.PTXCompilerTarget, CUDA.CUDACompilerParams}, compiler::Function, linker::Function) @ GPUCompiler ~/.julia/packages/GPUCompiler/Cp7sE/src/execution.jl:103 [12] macro expansion @ ~/.julia/packages/CUDA/nbRJk/src/compiler/execution.jl:323 [inlined] [13] macro expansion @ ./lock.jl:267 [inlined] [14] cufunction(f::GPUArrays.var"#broadcast_kernel#32", tt::Type{Tuple{CUDA.CuKernelContext, CuDeviceVector{Float32, 1}, Base.Broadcast.Broadcasted{CUDA.CuArrayStyle{1}, Tuple{Base.OneTo{Int64}}, typeof(+), Tuple{Base.Broadcast.Extruded{CuDeviceVector{Float32, 1}, Tuple{Bool}, Tuple{Int64}}, Base.Broadcast.Extruded{CuDeviceVector{Float32, 1}, Tuple{Bool}, Tuple{Int64}}}}, Int64}}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ CUDA ~/.julia/packages/CUDA/nbRJk/src/compiler/execution.jl:318 [15] cufunction @ ~/.julia/packages/CUDA/nbRJk/src/compiler/execution.jl:315 [inlined] [16] macro expansion @ ~/.julia/packages/CUDA/nbRJk/src/compiler/execution.jl:104 [inlined] [17] #launch_heuristic#1087 @ ~/.julia/packages/CUDA/nbRJk/src/gpuarrays.jl:17 [inlined] [18] launch_heuristic @ ~/.julia/packages/CUDA/nbRJk/src/gpuarrays.jl:15 [inlined] [19] _copyto! @ ~/.julia/packages/GPUArrays/EZkix/src/host/broadcast.jl:70 [inlined] [20] materialize! @ ~/.julia/packages/GPUArrays/EZkix/src/host/broadcast.jl:46 [inlined] [21] materialize!(dest::CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, bc::Base.Broadcast.Broadcasted{CUDA.CuArrayStyle{1}, Nothing, typeof(+), Tuple{CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}, CuArray{Float32, 1, CUDA.Mem.DeviceBuffer}}}) @ Base.Broadcast ./broadcast.jl:881 [22] top-level scope @ REPL[7]:1 [23] top-level scope @ ~/.julia/packages/CUDA/nbRJk/src/initialization.jl:205 ```