search

JuliaGPU / CUDA.jl

CUDA programming in Julia.
https://juliagpu.org/cuda/
Other
1.21k stars 221 forks source link

Segmentation Fault on Loading CUDA #2453

Closed loganpknudsen closed 3 months ago

loganpknudsen commented 3 months ago

I am trying to load CUDA in my Julia script as follows: using CUDA and when I do so I get the following error message relating to a segmentation fault:


[60718] signal (11.1): Segmentation fault
in expression starting at /glade/u/home/knudsenl/.julia/packages/CUDA_Runtime_jll/YgJCI/.pkg/platform_augmentation.jl:210
Allocations: 2907 (Pool: 2898; Big: 9); GC: 0
ERROR: LoadError: Failed to precompile CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2] to "/glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA_Runtime_jll/jl_XyISs8".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:2468
  [3] compilecache
    @ ./loading.jl:2340 [inlined]
  [4] (::Base.var"#968#969"{Base.PkgId})()
    @ Base ./loading.jl:1974
  [5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
  [6] #mkpidlock#6
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
  [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
  [8] #invokelatest#2
    @ ./essentials.jl:894 [inlined]
  [9] invokelatest
    @ ./essentials.jl:889 [inlined]
 [10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base ./loading.jl:2983
 [11] maybe_cachefile_lock
    @ ./loading.jl:2980 [inlined]
 [12] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1970
 [13] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1812
 [14] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [15] invoke_in_world
    @ ./essentials.jl:923 [inlined]
[16] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1803
 [17] macro expansion
    @ ./loading.jl:1790 [inlined]
 [18] macro expansion
    @ ./lock.jl:267 [inlined]
 [19] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
 [20] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [21] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [22] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
 [23] include
    @ ./Base.jl:495 [inlined]
 [24] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
    @ Base ./loading.jl:2222
[25] top-level scope
    @ stdin:3
in expression starting at /glade/u/home/knudsenl/.julia/packages/CUDA/Tl08O/src/CUDA.jl:1
in expression starting at stdin:3
ERROR: LoadError: Failed to precompile CUDA [052768ef-5323-5732-b1bb-66c8b64840ba] to "/glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA/jl_meOBVB".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:2468
  [3] compilecache
    @ ./loading.jl:2340 [inlined]
  [4] (::Base.var"#968#969"{Base.PkgId})()
    @ Base ./loading.jl:1974
  [5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
  [6] #mkpidlock#6
  @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
  [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
  [8] #invokelatest#2
    @ ./essentials.jl:894 [inlined]
  [9] invokelatest
    @ ./essentials.jl:889 [inlined]
 [10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base ./loading.jl:2983
 [11] maybe_cachefile_lock
    @ ./loading.jl:2980 [inlined]
 [12] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1970
 [13] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1812
 [14] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [15] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [16] _require_prelocked(uuidkey::Base.PkgId, env::String)
 @ Base ./loading.jl:1803
 [17] macro expansion
    @ ./loading.jl:1790 [inlined]
 [18] macro expansion
    @ ./lock.jl:267 [inlined]
 [19] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
 [20] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [21] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [22] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
in expression starting at /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/testcode.jl:1

my versioninfo() for what I am running is:

Julia Version 1.10.2
Commit bd47eca2c8 (2024-03-01 10:14 UTC)
Build Info:

    Note: This is an unofficial build, please report bugs to the project
    responsible for this build and not to the Julia project unless you can
    reproduce the issue using official builds available at https://julialang.org/downloads

Platform Info:
  OS: Linux (x86_64-suse-linux)
  CPU: 72 × Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, cascadelake)
Threads: 1 default, 0 interactive, 1 GC (on 72 virtual cores)
Environment:
  LD_LIBRARY_PATH = /glade/u/apps/casper/23.10/spack/opt/spack/openmpi/4.1.6/oneapi/2023.2.1/dgcv/lib:/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/lib64:/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/nvvm/lib64:/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/extras/CUPTI/lib64:/glade/u/apps/common/23.08/spack/opt/spack/cuda/12.2.1/extras/Debugger/lib64:/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/lib:/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/lib/x64:/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/lib/oclfpga/host/linux64/lib:/glade/u/apps/common/23.08/spack/opt/spack/intel-oneapi-compilers/2023.2.1/compiler/2023.2.1/linux/compiler/lib/intel64_lin:/glade/u/apps/casper/23.10/spack/opt/spack/hdf5/1.12.2/oneapi/2023.2.1/6vf2/lib

and the status check is:

Status `~/.julia/environments/v1.10/Project.toml`
  [c7e460c6] ArgParse v1.2.0
⌃ [052768ef] CUDA v5.4.2
  [9e8cae18] Oceananigans v0.91.4
  [de0858da] Printf
  [9a3f8284] Random
Info Packages marked with ⌃ have new versions available and may be upgradable.

Any help would be appreciated! I have been working with @tomchor to fix this issue.

loganpknudsen commented 3 months ago

A quick update, following this issue I have run 

git clone https://github.com/JuliaBinaryWrappers/CUDA_Runtime_jll.jl
cd CUDA_Runtime_jll.jl
git checkout CUDA_Runtime-v0.14.1+0
JULIA_DEBUG=all julia --project -e 'using CUDA_Runtime_jll'
JULIA_DEBUG=all julia --project -e 'using Pkg; Pkg.instantiate'
JULIA_DEBUG=all julia --project -e 'using CUDA_Runtime_jll; @show Base.BinaryPlatforms.triplet(CUDA_Runtime_jll.host_platform)'

I get the following output:

┌ Debug: Ignoring cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/Preferences/pWSk8_BvCUb.ji for Preferences [21216c6a-2e73-6563-6e65-726566657250] (fafbfcfd-cc3e-9703-001f-6626af1aa549) since it is does not provide desired build_id (fafbfcfd-2d97-b1c6-0017-56458bd4c481)
└ @ Base loading.jl:3058
┌ Debug: Ignoring cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/JLLWrappers/7Zgw7_BvCUb.ji for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210] (fafbfcfd-eaa4-3a3e-001f-6626df4f7b2b) since it is does not provide desired build_id (fafbfcfd-0a46-ca48-0017-564611ab88b0)
└ @ Base loading.jl:3058
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/LazyArtifacts/MRP8l_veGKK.so for LazyArtifacts [4af54fe1-eca0-43a8-85a7-787d91b784e3]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/Preferences/pWSk8_veGKK.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/JLLWrappers/7Zgw7_veGKK.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA_Driver_jll/QJyk7_veGKK.so for CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
└ @ Base loading.jl:1057
┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 12.3.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:130
┌ Debug: Forward-compatible CUDA driver found at /glade/u/home/knudsenl/.julia/artifacts/a86b67fd924e2a8c72d376d301a34b2364281978/lib/libcuda.so; known to be version 12.5.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:184
┌ Debug: Could not use forward compatibility package (error 100)
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:192
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA_Runtime_jll/Hs50y_5HSzE.so for CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2]
└ @ Base loading.jl:1057
Base.BinaryPlatforms.triplet(CUDA_Runtime_jll.host_platform) = "x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx30-cuda+12.5-julia_version+1.10.2-cuda_local+false"

Does this help with the issue? How do I fix the forward compatability?

maleadt commented 3 months ago

What happens if you run with JULIA_CUDA_USE_COMPAT=false?

loganpknudsen commented 3 months ago

So I ran the following:

JULIA_DEBUG=all JULIA_CUDA_USE_COMPAT=false julia --project -e 'using CUDA_Runtime_jll; @show Base.BinaryPlatforms.triplet(CUDA_Runtime_jll.host_platform)'

and the output was:

┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/LazyArtifacts/MRP8l_veGKK.so for LazyArtifacts [4af54fe1-eca0-43a8-85a7-787d91b784e3]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/Preferences/pWSk8_veGKK.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/JLLWrappers/7Zgw7_veGKK.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA_Driver_jll/QJyk7_veGKK.so for CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
└ @ Base loading.jl:1057
┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 12.3.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:130
┌ Debug: CUDA compat preference: false
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:142
┌ Debug: User disallows using forward-compatible driver.
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:144
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA_Runtime_jll/Hs50y_5HSzE.so for CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2]
└ @ Base loading.jl:1057
Base.BinaryPlatforms.triplet(CUDA_Runtime_jll.host_platform) = "x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx30-cuda+12.5-julia_version+1.10.2-cuda_local+false"
maleadt commented 3 months ago

I meant to try and precompile CUDA.jl with that env var set.

loganpknudsen commented 3 months ago

My bad, I got the following after running JULIA_DEBUG=all julia --project="." -e 'using CUDA_Runtime_jll; @show Base.BinaryPlatforms.triplet(CUDA_Runtime_jll.host_platform)':

┌ Debug: Skipping mtime check for file /glade/derecho/scratch/csgteam/temp/spack/casper/23.10/builds/spack-stage-julia-1.10.2-apod2ebbzv2cxtnlmfltk45mbfdgyeq3/spack-src/usr/share/julia/stdlib/v1.10/LazyArtifacts/src/LazyArtifacts.jl used by /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/compiled/v1.10/LazyArtifacts/MRP8l_VacWF.ji, since it is a stdlib
└ @ Base loading.jl:3129
┌ Debug: Loading object cache file /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/compiled/v1.10/LazyArtifacts/MRP8l_VacWF.so for LazyArtifacts [4af54fe1-eca0-43a8-85a7-787d91b784e3]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/Preferences/pWSk8_veGKK.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/JLLWrappers/7Zgw7_veGKK.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA_Driver_jll/QJyk7_veGKK.so for CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
└ @ Base loading.jl:1057
┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 12.3.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:130
┌ Debug: Forward-compatible CUDA driver found at /glade/u/home/knudsenl/.julia/artifacts/a86b67fd924e2a8c72d376d301a34b2364281978/lib/libcuda.so; known to be version 12.5.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:184
┌ Debug: Could not use forward compatibility package (error 100)
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:192
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA_Runtime_jll/Hs50y_5HSzE.so for CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2]
└ @ Base loading.jl:1057
Base.BinaryPlatforms.triplet(CUDA_Runtime_jll.host_platform) = "x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx30-cuda+12.5-julia_version+1.10.2-cuda_local+false"

I believe this is what you meant correct?

loganpknudsen commented 3 months ago

I don't know if this helps. but aomething else I just tried was deleting the Project.toml and Manifest.toml file as well as ~/.julia, and after running the following:

using Pkg
Pkg.activate(".")
Pkg.add("CUDA")
Pkg.instantiate()
using CUDA

I got:

[57577] signal (11.1): Segmentation fault
in expression starting at /glade/u/home/knudsenl/.julia/packages/CUDA_Runtime_jll/YgJCI/.pkg/platform_augmentation.jl:210
Allocations: 2907 (Pool: 2898; Big: 9); GC: 0
  Activating new project at `/glade/derecho/scratch/knudsenl/BottomBoundaryLayer`
  Installing known registries into `~/.julia`
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
   Installed GPUArraysCore ─────────────── v0.1.6
   Installed Crayons ───────────────────── v4.1.1
   Installed TableTraits ───────────────── v1.0.1
   Installed ColorTypes ────────────────── v0.11.5
   Installed Adapt ─────────────────────── v4.0.4
   Installed Scratch ───────────────────── v1.2.1
   Installed Preferences ───────────────── v1.4.3
   Installed PrettyTables ──────────────── v2.3.2
   Installed CUDA_Driver_jll ───────────── v0.9.1+1
   Installed SentinelArrays ────────────── v1.4.5
   Installed DataAPI ───────────────────── v1.16.0
   Installed PooledArrays ──────────────── v1.4.3
   Installed FixedPointNumbers ─────────── v0.8.5
   Installed Tables ────────────────────── v1.12.0
   Installed LLVMLoopInfo ──────────────── v1.0.0
   Installed GPUCompiler ───────────────── v0.26.7
   Installed StaticArraysCore ──────────── v1.4.3
   Installed TimerOutputs ──────────────── v0.5.24
   Installed JLLWrappers ───────────────── v1.5.0
   Installed InlineStrings ─────────────── v1.4.2
   Installed AbstractFFTs ──────────────── v1.5.0
   Installed IteratorInterfaceExtensions ─ v1.0.0
   Installed StaticArrays ──────────────── v1.9.7
   Installed PrecompileTools ───────────── v1.2.1
   Installed DataValueInterfaces ───────── v1.0.0
   Installed CUDA_Runtime_Discovery ────── v0.3.4
   Installed NVTX_jll ──────────────────── v3.1.0+2
   Installed LLVMExtra_jll ─────────────── v0.0.30+0
   Installed LaTeXStrings ──────────────── v1.3.1
   Installed OrderedCollections ────────── v1.6.3
   Installed NVTX ──────────────────────── v0.3.4
   Installed UnsafeAtomicsLLVM ─────────── v0.1.5
   Installed CEnum ─────────────────────── v0.5.0
   Installed InvertedIndices ───────────── v1.3.0
   Installed Reexport ──────────────────── v1.2.2
   Installed JuliaNVTXCallbacks_jll ────── v0.2.1+0
   Installed BFloat16s ─────────────────── v0.5.0
   Installed MacroTools ────────────────── v0.5.13
   Installed DataStructures ────────────── v0.18.20
   Installed Colors ────────────────────── v0.12.11
   Installed KernelAbstractions ────────── v0.9.22
   Installed RandomNumbers ─────────────── v1.5.3
   Installed Missings ──────────────────── v1.2.0
   Installed Compat ────────────────────── v4.15.0
   Installed StringManipulation ────────── v0.3.4
   Installed SortingAlgorithms ─────────── v1.2.1
   Installed UnsafeAtomics ─────────────── v0.2.1
   Installed Atomix ────────────────────── v0.1.0
   Installed LLVM ──────────────────────── v8.0.0
   Installed CUDA ──────────────────────── v5.4.3
    Updating `/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/Project.toml`
  [052768ef] + CUDA v5.4.3
    Updating `/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/Manifest.toml`
ERROR: LoadError: failed process: Process(`/glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/bin/julia -C native -J/glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/lib/julia/sys.so -g1 -O0 --color=no --history-file=no --startup-file=no --project=/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/Project.toml --eval 'append!(empty!(Base.DEPOT_PATH), ["/glade/u/home/knudsenl/.julia", "/glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/local/share/julia", "/glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia"])
append!(empty!(Base.DL_LOAD_PATH), String[])

cd("/glade/u/home/knudsenl/.julia/packages/CUDA_Runtime_jll/YgJCI/.pkg")
include("/glade/u/home/knudsenl/.julia/packages/CUDA_Runtime_jll/YgJCI/.pkg/select_artifacts.jl")
' -t1 --startup-file=no x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx30-julia_version+1.10.2`, ProcessSignaled(11)) [0]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] read(cmd::Cmd)
    @ Base ./process.jl:449
  [3] collect_artifacts(pkg_root::String; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:720
  [4] collect_artifacts
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:706 [inlined]
  [5] check_artifacts_downloaded(pkg_root::String; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:764
  [6] check_artifacts_downloaded
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:763 [inlined]
  [7] is_package_downloaded(manifest_file::String, pkg::Pkg.Types.PackageSpec; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2195
  [8] is_package_downloaded
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2190 [inlined]
  [9] print_status(env::Pkg.Types.EnvCache, old_env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, header::Symbol, uuids::Vector{Base.UUID}, names::Vector{String}; manifest::Bool, diff::Bool, ignore_indent::Bool, outdated::Bool, extensions::Bool, io::IOStream, mode::Pkg.Types.PackageMode, hidden_upgrades_info::Bool, show_usagetips::Bool)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2308
 [10] print_status
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2240 [inlined]
 [11] status(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}; header::Symbol, mode::Pkg.Types.PackageMode, git_diff::Bool, env_diff::Pkg.Types.EnvCache, ignore_indent::Bool, io::IOStream, outdated::Bool, extensions::Bool, hidden_upgrades_info::Bool, show_usagetips::Bool)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2474
 [12] status (repeats 2 times)
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2442 [inlined]
 [13] #show_update#167
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2438 [inlined]
 [14] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:1397
 [15] add
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:1376 [inlined]
 [16] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::@Kwargs{io::IOStream})
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:278
 [17] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::IOStream, kwargs::@Kwargs{})
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:159
 [18] add(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:148
 [19] add
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:147 [inlined]
 [20] add(pkg::String)
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:146
 [21] top-level scope
    @ /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/testcode.jl:3
in expression starting at /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/testcode.jl:3
maleadt commented 3 months ago

You're posting a lot of things here, please take a step back. If I understand your issue correctly:

loganpknudsen commented 3 months ago

It appears you have a correct understanding of the issue. I went back and tried rerunning the previous steps on a GPU to see if that would prompt the segmentation fault, and after running(note that I did this on a new environment):

git clone https://github.com/JuliaBinaryWrappers/CUDA_Runtime_jll.jl
cd CUDA_Runtime_jll.jl
git checkout CUDA_Runtime-v0.14.1+0
JULIA_DEBUG=all julia --project -e 'using Pkg; Pkg.activate(".")'
JULIA_DEBUG=all julia --project -e 'using Pkg; Pkg.add("CUDA")'

I got the following output from trying to add CUDA, and it appears that I am having an issue(both on GPU and CPU) of CUDA_Runtime_jll failing to precompile causing CUDA to undergo a segmentation fault:

  Installing known registries into `~/.julia`
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
┌ Debug: tiered_resolve: trying PRESERVE_ALL
└ @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:1331
   Installed GPUArraysCore ─────────────── v0.1.6
   Installed Crayons ───────────────────── v4.1.1
   Installed Adapt ─────────────────────── v4.0.4
   Installed TableTraits ───────────────── v1.0.1
   Installed Scratch ───────────────────── v1.2.1
   Installed ColorTypes ────────────────── v0.11.5
   Installed Preferences ───────────────── v1.4.3
   Installed PrettyTables ──────────────── v2.3.2
   Installed CUDA_Driver_jll ───────────── v0.9.1+1
   Installed DataAPI ───────────────────── v1.16.0
   Installed SentinelArrays ────────────── v1.4.5
   Installed PooledArrays ──────────────── v1.4.3
   Installed FixedPointNumbers ─────────── v0.8.5
   Installed GPUCompiler ───────────────── v0.26.7
   Installed LLVMLoopInfo ──────────────── v1.0.0
   Installed Tables ────────────────────── v1.12.0
   Installed InlineStrings ─────────────── v1.4.2
   Installed TimerOutputs ──────────────── v0.5.24
   Installed StaticArraysCore ──────────── v1.4.3
   Installed IteratorInterfaceExtensions ─ v1.0.0
   Installed JLLWrappers ───────────────── v1.5.0
   Installed AbstractFFTs ──────────────── v1.5.0
   Installed StaticArrays ──────────────── v1.9.7
   Installed PrecompileTools ───────────── v1.2.1
   Installed NVTX_jll ──────────────────── v3.1.0+2
   Installed DataValueInterfaces ───────── v1.0.0
   Installed CUDA_Runtime_Discovery ────── v0.3.4
   Installed LaTeXStrings ──────────────── v1.3.1
   Installed LLVMExtra_jll ─────────────── v0.0.30+0
   Installed UnsafeAtomicsLLVM ─────────── v0.1.5
   Installed OrderedCollections ────────── v1.6.3
   Installed NVTX ──────────────────────── v0.3.4
   Installed JuliaNVTXCallbacks_jll ────── v0.2.1+0
   Installed InvertedIndices ───────────── v1.3.0
   Installed CEnum ─────────────────────── v0.5.0
   Installed BFloat16s ─────────────────── v0.5.0
   Installed Reexport ──────────────────── v1.2.2
   Installed Random123 ─────────────────── v1.7.0
   Installed GPUArrays ─────────────────── v10.3.0
   Installed DataFrames ────────────────── v1.6.1
   Installed Requires ──────────────────── v1.3.0
   Installed ExprTools ─────────────────── v0.1.10
   Installed Colors ────────────────────── v0.12.11
   Installed MacroTools ────────────────── v0.5.13
   Installed DataStructures ────────────── v0.18.20
   Installed RandomNumbers ─────────────── v1.5.3
   Installed KernelAbstractions ────────── v0.9.22
   Installed CUDA ──────────────────────── v5.4.3
   Installed StringManipulation ────────── v0.3.4
   Installed Compat ────────────────────── v4.15.0
   Installed Missings ──────────────────── v1.2.0
   Installed UnsafeAtomics ─────────────── v0.2.1
   Installed SortingAlgorithms ─────────── v1.2.1
   Installed Atomix ────────────────────── v0.1.0
   Installed LLVM ──────────────────────── v8.0.0
┌ Debug: Downloading artifact from Pkg server
│   name = "CUDA_Driver"
│   artifacts_toml = "/glade/u/home/knudsenl/.julia/packages/CUDA_Driver_jll/PZjqb/Artifacts.toml"
│   platform = Linux x86_64 {cxxstring_abi=cxx11, julia_version=1.10.2, libc=glibc, libgfortran_version=5.0.0, libstdcxx_version=3.4.30}
│   url = "https://pkg.julialang.org/artifact/a86b67fd924e2a8c72d376d301a34b2364281978"
└ @ Pkg.Artifacts /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Artifacts.jl:421
  Downloaded artifact: CUDA_Driver
┌ Debug: Downloading artifact from Pkg server
│   name = "LLVMExtra"
│   artifacts_toml = "/glade/u/home/knudsenl/.julia/packages/LLVMExtra_jll/gZzRL/Artifacts.toml"
│   platform = Linux x86_64 {cxxstring_abi=cxx11, julia_version=1.10.2, libc=glibc, libgfortran_version=5.0.0, libstdcxx_version=3.4.30}
│   url = "https://pkg.julialang.org/artifact/9ba4abb9d5e99d0e3e3b0eed78625b5c90957bfc"
└ @ Pkg.Artifacts /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Artifacts.jl:421
  Downloaded artifact: LLVMExtra
┌ Debug: Downloading artifact from Pkg server
│   name = "NVTX"
│   artifacts_toml = "/glade/u/home/knudsenl/.julia/packages/NVTX_jll/W9h82/Artifacts.toml"
│   platform = Linux x86_64 {cxxstring_abi=cxx11, julia_version=1.10.2, libc=glibc, libgfortran_version=5.0.0, libstdcxx_version=3.4.30}
│   url = "https://pkg.julialang.org/artifact/0b6483311b60bc85d320f64bb66fd2e310916fb1"
└ @ Pkg.Artifacts /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Artifacts.jl:421
  Downloaded artifact: NVTX
┌ Debug: Downloading artifact from Pkg server
│   name = "JuliaNVTXCallbacks"
│   artifacts_toml = "/glade/u/home/knudsenl/.julia/packages/JuliaNVTXCallbacks_jll/AIP88/Artifacts.toml"
│   platform = Linux x86_64 {cxxstring_abi=cxx11, julia_version=1.10.2, libc=glibc, libgfortran_version=5.0.0, libstdcxx_version=3.4.30}
│   url = "https://pkg.julialang.org/artifact/ff17ab283d1f7456123711f21d8b1d7e852f8024"
└ @ Pkg.Artifacts /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Artifacts.jl:421
  Downloaded artifact: JuliaNVTXCallbacks
┌ Debug: Adding include dependency on /usr/lib64/libcuda.so.1
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:78
┌ Debug: CUDA_Driver_jll not available; not selecting an artifact
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:147
┌ Debug: Failed to query CUDA driver version
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:245
┌ Debug: Adding include dependency on /usr/lib64/libcuda.so.1
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:78
┌ Debug: CUDA_Driver_jll not available; not selecting an artifact
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:147
┌ Debug: Failed to query CUDA driver version
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:245
    Updating `/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/Project.toml`
  [052768ef] + CUDA v5.4.3
  [692b3bcd] + JLLWrappers v1.5.0
  [4ee394cb] + CUDA_Driver_jll v0.9.1+1
  [fa267f1f] ~ TOML ⇒ v1.0.3
    Updating `/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/Manifest.toml`
  [621f4979] + AbstractFFTs v1.5.0
  [79e6a3ab] + Adapt v4.0.4
  [a9b6321e] + Atomix v0.1.0
  [ab4f0b2a] + BFloat16s v0.5.0
  [fa961155] + CEnum v0.5.0
  [052768ef] + CUDA v5.4.3
  [1af6417a] + CUDA_Runtime_Discovery v0.3.4
  [3da002f7] + ColorTypes v0.11.5
  [5ae59095] + Colors v0.12.11
  [34da2185] + Compat v4.15.0
  [a8cc5b0e] + Crayons v4.1.1
  [9a962f9c] + DataAPI v1.16.0
  [a93c6f00] + DataFrames v1.6.1
  [864edb3b] + DataStructures v0.18.20
  [e2d170a0] + DataValueInterfaces v1.0.0
  [e2ba6199] + ExprTools v0.1.10
  [53c48c17] + FixedPointNumbers v0.8.5
  [0c68f7d7] + GPUArrays v10.3.0
  [46192b85] + GPUArraysCore v0.1.6
  [61eb1bfa] + GPUCompiler v0.26.7
  [842dd82b] + InlineStrings v1.4.2
  [41ab1584] + InvertedIndices v1.3.0
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [692b3bcd] + JLLWrappers v1.5.0
  [63c18a36] + KernelAbstractions v0.9.22
  [929cbde3] + LLVM v8.0.0
  [8b046642] + LLVMLoopInfo v1.0.0
  [b964fa9f] + LaTeXStrings v1.3.1
  [1914dd2f] + MacroTools v0.5.13
  [e1d29d7a] + Missings v1.2.0
  [5da4648a] + NVTX v0.3.4
  [bac558e1] + OrderedCollections v1.6.3
  [2dfb63ee] + PooledArrays v1.4.3
  [aea7be01] + PrecompileTools v1.2.1
  [21216c6a] + Preferences v1.4.3
  [08abe8d2] + PrettyTables v2.3.2
  [74087812] + Random123 v1.7.0
  [e6cf234a] + RandomNumbers v1.5.3
  [189a3867] + Reexport v1.2.2
  [ae029012] + Requires v1.3.0
  [6c6a2e73] + Scratch v1.2.1
  [91c51154] + SentinelArrays v1.4.5
  [a2af1166] + SortingAlgorithms v1.2.1
  [90137ffa] + StaticArrays v1.9.7
  [1e83bf80] + StaticArraysCore v1.4.3
  [892a3eda] + StringManipulation v0.3.4
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.12.0
  [a759f4b9] + TimerOutputs v0.5.24
  [013be700] + UnsafeAtomics v0.2.1
  [d80eeb9a] + UnsafeAtomicsLLVM v0.1.5
  [4ee394cb] + CUDA_Driver_jll v0.9.1+1
  [9c1d0b0a] + JuliaNVTXCallbacks_jll v0.2.1+0
  [dad2f222] + LLVMExtra_jll v0.0.30+0
  [e98f9f5b] + NVTX_jll v3.1.0+2
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [9fa8497b] + Future
  [b77e0a4c] + InteractiveUtils
  [4af54fe1] + LazyArtifacts
  [b27032c2] + LibCURL v0.6.4
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.10.0
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays v1.10.0
  [10745b16] + Statistics v1.10.0
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll v1.1.0+0
  [deac9b47] + LibCURL_jll v8.4.0+0
  [e37daf67] + LibGit2_jll v1.6.4+0
  [29816b5a] + LibSSH2_jll v1.11.0+1
  [c8ffd9c3] + MbedTLS_jll v2.28.2+1
  [14a3606d] + MozillaCACerts_jll v2023.1.10
  [4536629a] + OpenBLAS_jll v0.3.23+4
  [bea87d4a] + SuiteSparse_jll v7.2.1+1
  [83775a58] + Zlib_jll v1.2.13+1
  [8e850b90] + libblastrampoline_jll v5.8.0+1
  [8e850ede] + nghttp2_jll v1.52.0+1
  [3f19e933] + p7zip_jll v17.4.0+2
┌ Debug: Adding include dependency on /usr/lib64/libcuda.so.1
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:78
┌ Debug: Precompiling CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
└ @ Base loading.jl:2353
┌ Debug: Precompiling JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:2353
┌ Debug: Precompiling Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:2353
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/Preferences/pWSk8_a3zg0.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/Preferences/pWSk8_a3zg0.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/JLLWrappers/7Zgw7_a3zg0.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/Preferences/pWSk8_a3zg0.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/JLLWrappers/7Zgw7_a3zg0.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:1057
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.10/CUDA_Driver_jll/QJyk7_a3zg0.so for CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
└ @ Base loading.jl:1057
┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 12.3.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:130
┌ Debug: Forward-compatible CUDA driver found at /glade/u/home/knudsenl/.julia/artifacts/a86b67fd924e2a8c72d376d301a34b2364281978/lib/libcuda.so; known to be version 12.5.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:184
┌ Debug: Could not use forward compatibility package (error 803)
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:192

[7090] signal (11.1): Segmentation fault
in expression starting at /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:144
Allocations: 605840 (Pool: 605079; Big: 761); GC: 1
ERROR: failed process: Process(`/glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/bin/julia -C native -J/glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/lib/julia/sys.so -g1 -O0 --color=no --history-file=no --startup-file=no --project=/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/Project.toml --eval 'append!(empty!(Base.DEPOT_PATH), ["/glade/u/home/knudsenl/.julia", "/glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/local/share/julia", "/glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia"])
append!(empty!(Base.DL_LOAD_PATH), String[])

cd("/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg")
include("/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/select_artifacts.jl")
' -t1 --startup-file=no x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx30-julia_version+1.10.2`, ProcessSignaled(11)) [0]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] read(cmd::Cmd)
    @ Base ./process.jl:449
  [3] collect_artifacts(pkg_root::String; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:720
  [4] collect_artifacts
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:706 [inlined]
  [5] check_artifacts_downloaded(pkg_root::String; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:764
  [6] check_artifacts_downloaded
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:763 [inlined]
  [7] is_package_downloaded(manifest_file::String, pkg::Pkg.Types.PackageSpec; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2195
  [8] is_package_downloaded
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2190 [inlined]
  [9] #10
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:132 [inlined]
 [10] _all
    @ ./reduce.jl:1288 [inlined]
 [11] all
    @ ./reducedim.jl:1023 [inlined]
 [12] is_instantiated(env::Pkg.Types.EnvCache; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:132
 [13] is_instantiated
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:116 [inlined]
 [14] instantiate(ctx::Pkg.Types.Context; manifest::Nothing, update_registry::Bool, verbose::Bool, platform::Base.BinaryPlatforms.Platform, allow_build::Bool, allow_autoprecomp::Bool, kwargs::@Kwargs{})
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:1820
 [15] precompile(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; internal_call::Bool, strict::Bool, warn_loaded::Bool, already_instantiated::Bool, timing::Bool, _from_loading::Bool, kwargs::@Kwargs{})
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:1082
 [16] precompile
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:1078 [inlined]
 [17] #_auto_precompile#6
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Pkg.jl:805 [inlined]
 [18] _auto_precompile (repeats 2 times)
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/Pkg.jl:803 [inlined]
 [19] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::@Kwargs{})
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:160
 [20] add(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:148
 [21] add
    @ /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:147 [inlined]
 [22] add(pkg::String)
    @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.10.2/gcc/7.5.0/apod/share/julia/stdlib/v1.10/Pkg/src/API.jl:146
 [23] top-level scope
    @ none:1

Does this help make the issue clearer?

maleadt commented 3 months ago

That confirms my understanding, however, can you try that again now with JULIA_CUDA_USE_COMPAT=false set in your environment? If that works, you can keep it around, or set it permanently as a preference: https://github.com/JuliaGPU/CUDA.jl/blob/beccab19247738a1be3ec173ad4495e579e317d2/LocalPreferences.toml#L19-L23

loganpknudsen commented 3 months ago

Sorry for the late response, I just ran that and the output was the following(it seemed to work!):

  Activating project at `/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl`
knudsenl@casper-login2:/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl> JULIA_DEBUG=all JULIA_CUDA_USE_COMPAT=false julia --project -e 'using Pkg; Pkg.add("CUDA")'
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
┌ Debug: tiered_resolve: trying PRESERVE_ALL
└ @ Pkg.Operations /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1328
┌ Debug: Adding include dependency on /usr/lib64/libcuda.so.1
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:78
┌ Debug: CUDA_Driver_jll not available; not selecting an artifact
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:147
┌ Debug: Failed to query CUDA driver version
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:245
┌ Debug: Adding include dependency on /usr/lib64/libcuda.so.1
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:78
┌ Debug: CUDA_Driver_jll not available; not selecting an artifact
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:147
┌ Debug: Failed to query CUDA driver version
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:245
    Updating `/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/Project.toml`
  [052768ef] + CUDA v5.4.3
  [692b3bcd] + JLLWrappers v1.5.0
  [4ee394cb] + CUDA_Driver_jll v0.9.1+1
  [fa267f1f] ~ TOML ⇒ v1.0.3
    Updating `/glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/Manifest.toml`
  [621f4979] + AbstractFFTs v1.5.0
  [79e6a3ab] + Adapt v4.0.4
  [a9b6321e] + Atomix v0.1.0
  [ab4f0b2a] + BFloat16s v0.5.0
  [fa961155] + CEnum v0.5.0
  [052768ef] + CUDA v5.4.3
  [1af6417a] + CUDA_Runtime_Discovery v0.3.4
  [3da002f7] + ColorTypes v0.11.5
  [5ae59095] + Colors v0.12.11
  [34da2185] + Compat v4.15.0
  [a8cc5b0e] + Crayons v4.1.1
  [9a962f9c] + DataAPI v1.16.0
  [a93c6f00] + DataFrames v1.6.1
  [864edb3b] + DataStructures v0.18.20
  [e2d170a0] + DataValueInterfaces v1.0.0
  [e2ba6199] + ExprTools v0.1.10
  [53c48c17] + FixedPointNumbers v0.8.5
⌅ [0c68f7d7] + GPUArrays v10.2.3
  [46192b85] + GPUArraysCore v0.1.6
  [61eb1bfa] + GPUCompiler v0.26.7
  [842dd82b] + InlineStrings v1.4.2
  [41ab1584] + InvertedIndices v1.3.0
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [692b3bcd] + JLLWrappers v1.5.0
  [63c18a36] + KernelAbstractions v0.9.22
  [929cbde3] + LLVM v8.0.0
  [8b046642] + LLVMLoopInfo v1.0.0
  [b964fa9f] + LaTeXStrings v1.3.1
  [1914dd2f] + MacroTools v0.5.13
  [e1d29d7a] + Missings v1.2.0
  [5da4648a] + NVTX v0.3.4
  [bac558e1] + OrderedCollections v1.6.3
  [2dfb63ee] + PooledArrays v1.4.3
  [aea7be01] + PrecompileTools v1.2.1
  [21216c6a] + Preferences v1.4.3
  [08abe8d2] + PrettyTables v2.3.2
  [74087812] + Random123 v1.7.0
  [e6cf234a] + RandomNumbers v1.5.3
  [189a3867] + Reexport v1.2.2
  [ae029012] + Requires v1.3.0
  [6c6a2e73] + Scratch v1.2.1
  [91c51154] + SentinelArrays v1.4.5
  [a2af1166] + SortingAlgorithms v1.2.1
  [90137ffa] + StaticArrays v1.9.7
  [1e83bf80] + StaticArraysCore v1.4.3
  [892a3eda] + StringManipulation v0.3.4
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.12.0
  [a759f4b9] + TimerOutputs v0.5.24
  [013be700] + UnsafeAtomics v0.2.1
  [d80eeb9a] + UnsafeAtomicsLLVM v0.1.5
  [4ee394cb] + CUDA_Driver_jll v0.9.1+1
  [9c1d0b0a] + JuliaNVTXCallbacks_jll v0.2.1+0
  [dad2f222] + LLVMExtra_jll v0.0.30+0
  [e98f9f5b] + NVTX_jll v3.1.0+2
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [9fa8497b] + Future
  [b77e0a4c] + InteractiveUtils
  [4af54fe1] + LazyArtifacts
  [b27032c2] + LibCURL v0.6.3
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.9.2
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics v1.9.0
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll v1.0.5+0
  [deac9b47] + LibCURL_jll v7.84.0+0
  [29816b5a] + LibSSH2_jll v1.10.2+0
  [c8ffd9c3] + MbedTLS_jll v2.28.2+0
  [14a3606d] + MozillaCACerts_jll v2022.10.11
  [4536629a] + OpenBLAS_jll v0.3.21+4
  [bea87d4a] + SuiteSparse_jll v5.10.1+6
  [83775a58] + Zlib_jll v1.2.13+0
  [8e850b90] + libblastrampoline_jll v5.8.0+0
  [8e850ede] + nghttp2_jll v1.48.0+0
  [3f19e933] + p7zip_jll v17.4.0+0
        Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m`
┌ Debug: Adding include dependency on /usr/lib64/libcuda.so.1
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:78
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/Preferences/pWSk8_VphDT.so for Preferences [21216c6a-2e73-6563-6e65-726566657250]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/JLLWrappers/7Zgw7_VphDT.so for JLLWrappers [692b3bcd-3c85-4b1f-b108-f13ce0eb3210]
└ @ Base loading.jl:1051
┌ Debug: Loading object cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/CUDA_Driver_jll/QJyk7_VphDT.so for CUDA_Driver_jll [4ee394cb-3365-5eb0-8335-949819d2adfc]
└ @ Base loading.jl:1051
┌ Debug: System CUDA driver found at libcuda.so.1, detected as version 12.3.0
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:130
┌ Debug: CUDA compat preference: false
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:142
┌ Debug: User disallows using forward-compatible driver.
└ @ CUDA_Driver_jll ~/.julia/packages/CUDA_Driver_jll/PZjqb/src/wrappers/x86_64-linux-gnu.jl:144
┌ Debug: Using CUDA_Driver_jll for driver discovery
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:152
┌ Debug: Found CUDA driver at 'libcuda.so.1'
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:179
┌ Debug: CUDA driver version: 12.3.0
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:248
┌ Debug: Selected CUDA toolkit: 12.5.0
└ @ Main /glade/derecho/scratch/knudsenl/BottomBoundaryLayer/CUDA_Runtime_jll.jl/.pkg/platform_augmentation.jl:279
┌ Warning: Circular dependency detected. Precompilation will be skipped for:
│   CUDA_Runtime_jll [76a88914-d11a-5bdc-97e0-2f5a05c973a2]
│   CUDA [052768ef-5323-5732-b1bb-66c8b64840ba]
└ @ Pkg.API /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/stdlib/v1.9/Pkg/src/API.jl:1234
┌ Debug: Rejecting cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/MbedTLS_jll/u5NEn_X8D0f.ji for  [top-level] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 0
└ @ Base loading.jl:2730
┌ Debug: Rejecting cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/Zlib_jll/xjq3Q_X8D0f.ji for  [top-level] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 0
└ @ Base loading.jl:2730
┌ Debug: Rejecting cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/InlineStrings/prmY3_X8D0f.ji for  [top-level] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 0
└ @ Base loading.jl:2730
┌ Debug: Rejecting cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/Statistics/ERcPL_X8D0f.ji for  [top-level] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 0
└ @ Base loading.jl:2730
┌ Debug: Rejecting cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/SuiteSparse_jll/ME9At_X8D0f.ji for  [top-level] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 0
└ @ Base loading.jl:2730
┌ Debug: Skipping mtime check for file /glade/derecho/scratch/csgteam/temp/spack/casper/23.10/builds/spack-stage-julia-1.9.2-mjeadxih745lj3s24lbol2ou7lpwqtse/spack-src/usr/share/julia/stdlib/v1.9/MbedTLS_jll/src/MbedTLS_jll.jl used by /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/compiled/v1.9/MbedTLS_jll/u5NEn_1a2YF.ji, since it is a stdlib
└ @ Base loading.jl:2835
┌ Debug: Skipping mtime check for file /glade/derecho/scratch/csgteam/temp/spack/casper/23.10/builds/spack-stage-julia-1.9.2-mjeadxih745lj3s24lbol2ou7lpwqtse/spack-src/usr/share/julia/stdlib/v1.9/Zlib_jll/src/Zlib_jll.jl used by /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/compiled/v1.9/Zlib_jll/xjq3Q_1a2YF.ji, since it is a stdlib
└ @ Base loading.jl:2835
┌ Debug: Skipping mtime check for file /glade/derecho/scratch/csgteam/temp/spack/casper/23.10/builds/spack-stage-julia-1.9.2-mjeadxih745lj3s24lbol2ou7lpwqtse/spack-src/usr/share/julia/stdlib/v1.9/Statistics/src/Statistics.jl used by /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/compiled/v1.9/Statistics/ERcPL_1a2YF.ji, since it is a stdlib
└ @ Base loading.jl:2835
┌ Debug: Skipping mtime check for file /glade/derecho/scratch/csgteam/temp/spack/casper/23.10/builds/spack-stage-julia-1.9.2-mjeadxih745lj3s24lbol2ou7lpwqtse/spack-src/usr/share/julia/stdlib/v1.9/SuiteSparse_jll/src/SuiteSparse_jll.jl used by /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/compiled/v1.9/SuiteSparse_jll/ME9At_1a2YF.ji, since it is a stdlib
└ @ Base loading.jl:2835
┌ Debug: Rejecting cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/Statistics/ERcPL_X8D0f.ji for Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 0
└ @ Base loading.jl:2730
┌ Debug: Rejecting cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/LibSSH2_jll/K6mup_X8D0f.ji for  [top-level] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 0
└ @ Base loading.jl:2730
┌ Debug: Rejecting cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/Statistics/ERcPL_X8D0f.ji for Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 0
└ @ Base loading.jl:2730
┌ Debug: Rejecting cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/Statistics/ERcPL_X8D0f.ji for Statistics [10745b16-79ce-11e8-11f9-7d13ad32a3b2] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 0
└ @ Base loading.jl:2730
┌ Debug: Skipping mtime check for file /glade/derecho/scratch/csgteam/temp/spack/casper/23.10/builds/spack-stage-julia-1.9.2-mjeadxih745lj3s24lbol2ou7lpwqtse/spack-src/usr/share/julia/stdlib/v1.9/Statistics/src/Statistics.jl used by /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/compiled/v1.9/Statistics/ERcPL_1a2YF.ji, since it is a stdlib
└ @ Base loading.jl:2835
┌ Debug: Skipping mtime check for file /glade/derecho/scratch/csgteam/temp/spack/casper/23.10/builds/spack-stage-julia-1.9.2-mjeadxih745lj3s24lbol2ou7lpwqtse/spack-src/usr/share/julia/stdlib/v1.9/LibSSH2_jll/src/LibSSH2_jll.jl used by /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/compiled/v1.9/LibSSH2_jll/K6mup_1a2YF.ji, since it is a stdlib
└ @ Base loading.jl:2835
┌ Debug: Skipping mtime check for file /glade/derecho/scratch/csgteam/temp/spack/casper/23.10/builds/spack-stage-julia-1.9.2-mjeadxih745lj3s24lbol2ou7lpwqtse/spack-src/usr/share/julia/stdlib/v1.9/Statistics/src/Statistics.jl used by /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/compiled/v1.9/Statistics/ERcPL_1a2YF.ji, since it is a stdlib
└ @ Base loading.jl:2835
┌ Debug: Skipping mtime check for file /glade/derecho/scratch/csgteam/temp/spack/casper/23.10/builds/spack-stage-julia-1.9.2-mjeadxih745lj3s24lbol2ou7lpwqtse/spack-src/usr/share/julia/stdlib/v1.9/Statistics/src/Statistics.jl used by /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/compiled/v1.9/Statistics/ERcPL_1a2YF.ji, since it is a stdlib
└ @ Base loading.jl:2835
┌ Debug: Rejecting cache file /glade/u/home/knudsenl/.julia/compiled/v1.9/MbedTLS_jll/u5NEn_X8D0f.ji for MbedTLS_jll [c8ffd9c3-330d-5841-b78e-0817d7145fa1] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
│   cache file:      use_pkgimages = false, debug_level = 1, check_bounds = 0, inline = true, opt_level = 0
└ @ Base loading.jl:2730
┌ Debug: Skipping mtime check for file /glade/derecho/scratch/csgteam/temp/spack/casper/23.10/builds/spack-stage-julia-1.9.2-mjeadxih745lj3s24lbol2ou7lpwqtse/spack-src/usr/share/julia/stdlib/v1.9/MbedTLS_jll/src/MbedTLS_jll.jl used by /glade/u/apps/casper/23.10/spack/opt/spack/julia/1.9.2/gcc/7.5.0/mjea/share/julia/compiled/v1.9/MbedTLS_jll/u5NEn_1a2YF.ji, since it is a stdlib
└ @ Base loading.jl:2835
loganpknudsen commented 3 months ago

Thank you to everyone who took the time to respond to this issue! I am still unclear on what exactly was driving the error I was getting(I am open to continuing to work on this issue but feel free to close it), but I was able to fix it by creating a new environment in a clean directory and running the following modules on Casper for the UCAR HPC resources:

module --force purge
module --ignore-cache load ncarenv/23.10 gcc ncarcompilers netcdf
module --ignore-cache load cuda
module --ignore-cache load julia/1.10.2

and then running my script. Thanks again for all the help!

maleadt commented 3 months ago

Great, thanks for following up. I think we can close this in favor of https://github.com/JuliaGPU/CUDA.jl/issues/2446 then, which should resolve the issue by more safely handling the forwards-compatible library.