Error when running example code

[Wenzhe-Ma]

I am new to GPU computing. When I try to run the following example code: using DiffEqGPU, OrdinaryDiffEq, StaticArrays, CUDA

function lorenz2(u, p, t) σ = p[1] ρ = p[2] β = p[3] du1 = σ (u[2] - u[1]) du2 = u[1] (ρ - u[3]) - u[2] du3 = u[1] u[2] - β u[3] return SVector{3}(du1, du2, du3) end

u0 = @SVector [1.0f0; 0.0f0; 0.0f0] tspan = (0.0f0, 10.0f0) p = @SVector [10.0f0, 28.0f0, 8 / 3.0f0] prob = ODEProblem{false}(lorenz2, u0, tspan, p) prob_func = (prob, i, repeat) -> remake(prob, p = (@SVector rand(Float32, 3)) .* p) monteprob = EnsembleProblem(prob, prob_func = prob_func, safetycopy = false) sol = solve(monteprob, GPUTsit5(), EnsembleGPUKernel(CUDA.CUDABackend()), trajectories = 10_000, saveat = 1.0f0)

I get the following error: CUDA error: an illegal memory access was encountered (code 700, ERROR_ILLEGAL_ADDRESS)

Stacktrace: [1] throw_api_error(res::CUDA.cudaError_enum) @ CUDA ~/.julia/packages/CUDA/p5OVK/lib/cudadrv/libcuda.jl:27 [2] macro expansion @ ~/.julia/packages/CUDA/p5OVK/lib/cudadrv/libcuda.jl:35 [inlined] [3] cuMemAllocAsync @ ~/.julia/packages/CUDA/p5OVK/lib/utils/call.jl:26 [inlined] [4] alloc(::Type{CUDA.Mem.DeviceBuffer}, bytesize::Int64; async::Bool, stream::CuStream, pool::Nothing) @ CUDA.Mem ~/.julia/packages/CUDA/p5OVK/lib/cudadrv/memory.jl:83 [5] alloc @ ~/.julia/packages/CUDA/p5OVK/lib/cudadrv/memory.jl:71 [inlined] [6] actual_alloc(bytes::Int64; async::Bool, stream::CuStream) @ CUDA ~/.julia/packages/CUDA/p5OVK/src/pool.jl:56 [7] actual_alloc @ ~/.julia/packages/CUDA/p5OVK/src/pool.jl:50 [inlined] [8] macro expansion @ ~/.julia/packages/CUDA/p5OVK/src/pool.jl:327 [inlined] [9] macro expansion @ ./timing.jl:393 [inlined] [10] #_alloc#164 @ ~/.julia/packages/CUDA/p5OVK/src/pool.jl:409 [inlined] [11] _alloc @ ~/.julia/packages/CUDA/p5OVK/src/pool.jl:404 [inlined] [12] #alloc#163 @ ~/.julia/packages/CUDA/p5OVK/src/pool.jl:394 [inlined] [13] alloc @ ~/.julia/packages/CUDA/p5OVK/src/pool.jl:388 [inlined] [14] CuArray{ODEProblem{SVector{3, Float32}, Tuple{Float32, Float32}, false, SVector{3, Float32}, ODEFunction{false, SciMLBase.AutoSpecialize, typeof(lorenz2), LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing, Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, 1, CUDA.Mem.DeviceBuffer}(#unused#::UndefInitializer, dims::Tuple{Int64}) @ CUDA ~/.julia/packages/CUDA/p5OVK/src/array.jl:93 [15] CuArray @ ~/.julia/packages/CUDA/p5OVK/src/array.jl:342 [inlined] [16] CuArray @ ~/.julia/packages/CUDA/p5OVK/src/array.jl:347 [inlined] [17] CuArray @ ~/.julia/packages/CUDA/p5OVK/src/array.jl:356 [inlined] [18] convert @ ~/.julia/packages/GPUArrays/t0LfC/src/host/construction.jl:4 [inlined] [19] adapt_storage @ ~/.julia/packages/CUDA/p5OVK/src/array.jl:575 [inlined] [20] adapt_structure @ ~/.julia/packages/Adapt/UtItS/src/Adapt.jl:57 [inlined] [21] adapt @ ~/.julia/packages/Adapt/UtItS/src/Adapt.jl:40 [inlined] [22] adapt_storage @ ~/.julia/packages/CUDA/p5OVK/src/CUDAKernels.jl:29 [inlined] [23] adapt_structure @ ~/.julia/packages/Adapt/UtItS/src/Adapt.jl:57 [inlined] [24] adapt @ ~/.julia/packages/Adapt/UtItS/src/Adapt.jl:40 [inlined] [25] batch_solve_up_kernel(ensembleprob::EnsembleProblem{ODEProblem{SVector{3, Float32}, Tuple{Float32, Float32}, false, SVector{3, Float32}, ODEFunction{false, SciMLBase.AutoSpecialize, typeof(lorenz2), LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing, Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, var"#47#48", typeof(SciMLBase.DEFAULT_OUTPUT_FUNC), typeof(SciMLBase.DEFAULT_REDUCTION), Nothing}, probs::Vector{ODEProblem{SVector{3, Float32}, Tuple{Float32, Float32}, false, SVector{3, Float32}, ODEFunction{false, SciMLBase.AutoSpecialize, typeof(lorenz2), LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing, Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}}, alg::GPUTsit5, ensemblealg::EnsembleGPUKernel{CUDABackend}, I::UnitRange{Int64}, adaptive::Bool; kwargs::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:saveat, :unstable_check), Tuple{Float32, DiffEqGPU.var"#24#30"}}}) @ DiffEqGPU ~/.julia/packages/DiffEqGPU/dzUW8/src/DiffEqGPU.jl:785 [26] batch_solve(ensembleprob::EnsembleProblem{ODEProblem{SVector{3, Float32}, Tuple{Float32, Float32}, false, SVector{3, Float32}, ODEFunction{false, SciMLBase.AutoSpecialize, typeof(lorenz2), LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing, Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, var"#47#48", typeof(SciMLBase.DEFAULT_OUTPUT_FUNC), typeof(SciMLBase.DEFAULT_REDUCTION), Nothing}, alg::GPUTsit5, ensemblealg::EnsembleGPUKernel{CUDABackend}, I::UnitRange{Int64}, adaptive::Bool; kwargs::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:unstable_check, :saveat), Tuple{DiffEqGPU.var"#24#30", Float32}}}) @ DiffEqGPU ~/.julia/packages/DiffEqGPU/dzUW8/src/DiffEqGPU.jl:729 [27] macro expansion @ ./timing.jl:393 [inlined] [28] solve(ensembleprob::EnsembleProblem{ODEProblem{SVector{3, Float32}, Tuple{Float32, Float32}, false, SVector{3, Float32}, ODEFunction{false, SciMLBase.AutoSpecialize, typeof(lorenz2), LinearAlgebra.UniformScaling{Bool}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED), Nothing, Nothing}, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}, SciMLBase.StandardODEProblem}, var"#47#48", typeof(SciMLBase.DEFAULT_OUTPUT_FUNC), typeof(SciMLBase.DEFAULT_REDUCTION), Nothing}, alg::GPUTsit5, ensemblealg::EnsembleGPUKernel{CUDABackend}; trajectories::Int64, batch_size::Int64, unstable_check::Function, adaptive::Bool, kwargs::Base.Pairs{Symbol, Float32, Tuple{Symbol}, NamedTuple{(:saveat,), Tuple{Float32}}}) @ DiffEqGPU ~/.julia/packages/DiffEqGPU/dzUW8/src/DiffEqGPU.jl:593 [29] solve @ ~/.julia/packages/DiffEqGPU/dzUW8/src/DiffEqGPU.jl:541 [inlined] [30] #solve#32 @ ~/.julia/packages/DiffEqBase/G15op/src/solve.jl:996 [inlined] [31] top-level scope @ In[93]:19

I appreciate if you could help me to understand what could be the error here.

Thanks, Wenzhe

[ChrisRackauckas]

That looks like CUDA driver issues. Do you have an old CUDA driver installed?

[Wenzhe-Ma]

Thanks for you quick response Chris! My current driver versions are: NVIDIA-SMI 530.30.02 Driver Version: 530.30.02 CUDA Version: 12.1 Is it OK for DiffEqGPU package?

[Wenzhe-Ma]

The problems is gone. I was using 1.9.0 and since I switched to 1.9.1 the error is gone.

[ChrisRackauckas]

Okay interesting. Yeah it looked more like a CUDA bug than a DiffEqGPU bug, so its not unreasonable that it was compiler bug handled in the patch release. Cool, cheers!