enstrophy and cuda problem

[liasiegelman]

When creating a diagnostic for enstrophy I get the following error (no problem for KE and PE)

julia> PE = Diagnostic(potential_energy, prob; nsteps=nsteps)
Diagnostic
  ├─── calc: potential_energy
  ├─── prob: FourierFlows.Problem{DataType, CuArray{ComplexF64, 2}, Float64, CuArray{ComplexF64, 2}}
  ├─── data: 1000001-element Vector{Int64}
  ├────── t: 1000001-element Vector{Float64}
  ├── steps: 1000001-element Vector{Int64}
  ├─── freq: 1
  └────── i: 1

julia> Z = Diagnostic(enstrophy, prob; nsteps=nsteps)
ERROR: GPU compilation of kernel broadcast_kernel(CUDA.CuKernelContext, CuDeviceMatrix{ComplexF64, 1}, Base.Broadcast.Broadcasted{CUDA.CuArrayStyle{2}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}, typeof(+), Tuple{Base.Broadcast.Extruded{CuDeviceMatrix{ComplexF64, 1}, Tuple{Bool, Bool}, Tuple{Int64, Int64}}, Base.Broadcast.Extruded{Matrix{ComplexF64}, Tuple{Bool, Bool}, Tuple{Int64, Int64}}}}, Int64) failed
KernelError: passing and using non-bitstype argument

Argument 4 to your kernel function is of type Base.Broadcast.Broadcasted{CUDA.CuArrayStyle{2}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}, typeof(+), Tuple{Base.Broadcast.Extruded{CuDeviceMatrix{ComplexF64, 1}, Tuple{Bool, Bool}, Tuple{Int64, Int64}}, Base.Broadcast.Extruded{Matrix{ComplexF64}, Tuple{Bool, Bool}, Tuple{Int64, Int64}}}}, which is not isbits:
  .args is of type Tuple{Base.Broadcast.Extruded{CuDeviceMatrix{ComplexF64, 1}, Tuple{Bool, Bool}, Tuple{Int64, Int64}}, Base.Broadcast.Extruded{Matrix{ComplexF64}, Tuple{Bool, Bool}, Tuple{Int64, Int64}}} which is not isbits.
    .2 is of type Base.Broadcast.Extruded{Matrix{ComplexF64}, Tuple{Bool, Bool}, Tuple{Int64, Int64}} which is not isbits.
      .x is of type Matrix{ComplexF64} which is not isbits.

Stacktrace:
  [1] check_invocation(job::GPUCompiler.CompilerJob)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/fG3xK/src/validation.jl:66
  [2] macro expansion
    @ ~/.julia/packages/GPUCompiler/fG3xK/src/driver.jl:318 [inlined]
  [3] macro expansion
    @ ~/.julia/packages/TimerOutputs/jgSVI/src/TimerOutput.jl:252 [inlined]
  [4] macro expansion
    @ ~/.julia/packages/GPUCompiler/fG3xK/src/driver.jl:317 [inlined]
  [5] emit_asm(job::GPUCompiler.CompilerJob, ir::LLVM.Module; strip::Bool, validate::Bool, format::LLVM.API.LLVMCodeGenFileType)
    @ GPUCompiler ~/.julia/packages/GPUCompiler/fG3xK/src/utils.jl:62
  [6] cufunction_compile(job::GPUCompiler.CompilerJob)
    @ CUDA ~/.julia/packages/CUDA/DL5Zo/src/compiler/execution.jl:317
  [7] cached_compilation(cache::Dict{UInt64, Any}, job::GPUCompiler.CompilerJob, compiler::typeof(CUDA.cufunction_compile), linker::typeof(CUDA.cufunction_link))
    @ GPUCompiler ~/.julia/packages/GPUCompiler/fG3xK/src/cache.jl:89
  [8] cufunction(f::GPUArrays.var"#broadcast_kernel#17", tt::Type{Tuple{CUDA.CuKernelContext, CuDeviceMatrix{ComplexF64, 1}, Base.Broadcast.Broadcasted{CUDA.CuArrayStyle{2}, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}, typeof(+), Tuple{Base.Broadcast.Extruded{CuDeviceMatrix{ComplexF64, 1}, Tuple{Bool, Bool}, Tuple{Int64, Int64}}, Base.Broadcast.Extruded{Matrix{ComplexF64}, Tuple{Bool, Bool}, Tuple{Int64, Int64}}}}, Int64}}; name::Nothing, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ CUDA ~/.julia/packages/CUDA/DL5Zo/src/compiler/execution.jl:288
  [9] cufunction
    @ ~/.julia/packages/CUDA/DL5Zo/src/compiler/execution.jl:282 [inlined]
 [10] macro expansion
    @ ~/.julia/packages/CUDA/DL5Zo/src/compiler/execution.jl:102 [inlined]
 [11] #launch_heuristic#241
    @ ~/.julia/packages/CUDA/DL5Zo/src/gpuarrays.jl:17 [inlined]
 [12] _copyto!
    @ ~/.julia/packages/GPUArrays/gok9K/src/host/broadcast.jl:73 [inlined]
 [13] copyto!
    @ ~/.julia/packages/GPUArrays/gok9K/src/host/broadcast.jl:56 [inlined]
 [14] copy
    @ ~/.julia/packages/GPUArrays/gok9K/src/host/broadcast.jl:47 [inlined]
 [15] materialize
    @ ./broadcast.jl:883 [inlined]
 [16] broadcast_preserving_zero_d
    @ ./broadcast.jl:872 [inlined]
 [17] +(A::CuArray{ComplexF64, 2}, B::Matrix{ComplexF64})
    @ Base ./arraymath.jl:39
 [18] enstrophy(sol::CuArray{ComplexF64, 2}, vars::GeophysicalFlows.SingleLayerQG.Vars{CuArray{Float64, 2}, CuArray{ComplexF64, 2}, Nothing, Nothing}, params::GeophysicalFlows.SingleLayerQG.Params{Float64, Matrix{Float64}, Matrix{ComplexF64}, Nothing}, grid::TwoDGrid{Float64, CuArray{Float64, 2}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}, CUDA.CUFFT.cCuFFTPlan{ComplexF64, -1, false, 2}, CUDA.CUFFT.rCuFFTPlan{Float64, -1, false, 2}, UnitRange{Int64}})
    @ GeophysicalFlows.SingleLayerQG ~/.julia/packages/GeophysicalFlows/Ig4RB/src/singlelayerqg.jl:493
 [19] enstrophy
    @ ~/.julia/packages/GeophysicalFlows/Ig4RB/src/singlelayerqg.jl:496 [inlined]
 [20] Diagnostic(calc::typeof(enstrophy), prob::FourierFlows.Problem{DataType, CuArray{ComplexF64, 2}, Float64, CuArray{ComplexF64, 2}}; freq::Int64, nsteps::Int64, ndata::Int64)
    @ FourierFlows ~/.julia/packages/FourierFlows/IWexK/src/diagnostics.jl:37
 [21] top-level scope
    @ REPL[83]:1
 [22] top-level scope
    @ ~/.julia/packages/CUDA/DL5Zo/src/initialization.jl:52

[navidcy]

Can you post your whole script?

[navidcy]

The params seem to include a matrix. Probably that’s the issue.

[navidcy]

Can you print typeof(prob.params.eta) and typeof(prib.params.etah)?

[liasiegelman]

You are correct Navid, thanks ! I failed to create a CUDA array somehow. Will fix this after a surf ! Thanks :)

[navidcy]

I’m opening it to attempt to make this user friendlier.

[liasiegelman]

Sure maybe there's a way to implement an error message such that when dev=GPU(), and eta is not a cuda array it returns an error message when constructing the problem (prob = SingleLayerQG.Problem(dev; nx=n, Lx=L, eta=η, deformation_radius=deformation_radius, dt=dt, stepper=stepper))

[navidcy]

Was thinking the constructor would convert it to a CuArray if dev is GPU.