search

SciML / DiffEqFlux.jl

Pre-built implicit layer architectures with O(1) backprop, GPUs, and stiff+non-stiff DE solvers, demonstrating scientific machine learning (SciML) and physics-informed machine learning methods
https://docs.sciml.ai/DiffEqFlux/stable
MIT License
871 stars 157 forks source link

Neural ODE example with GPU does not work #243

Closed matutenun closed 4 years ago

matutenun commented 4 years ago

I have been playing with your awesome tools, but cant make this one to work with GPU The example from https://diffeqflux.sciml.ai/dev/GPUs/#Neural-ODE-Example-1

using DiffEqFlux, OrdinaryDiffEq, Flux, Optim, Plots, CuArrays, DiffEqSensitivity
CuArrays.allowscalar(false) # Makes sure no slow operations are occuring

# Generate Data
u0 = Float32[2.0; 0.0]
datasize = 30
tspan = (0.0f0, 1.5f0)
tsteps = range(tspan[1], tspan[2], length = datasize)
function trueODEfunc(du, u, p, t)
    true_A = [-0.1 2.0; -2.0 -0.1]
    du .= ((u.^3)'true_A)'
end
prob_trueode = ODEProblem(trueODEfunc, u0, tspan)
# Make the data into a GPU-based array if the user has a GPU
ode_data = gpu(solve(prob_trueode, Tsit5(), saveat = tsteps))

dudt2 = FastChain((x, p) -> x.^3,
                  FastDense(2, 50, tanh),
                  FastDense(50, 2))
u0 = Float32[2.0; 0.0] |> gpu
p = initial_params(dudt2) |> gpu
prob_neuralode = NeuralODE(dudt2, tspan, Tsit5(), saveat = tsteps)

function predict_neuralode(p)
  gpu(prob_neuralode(u0,p))
end
function loss_neuralode(p)
    pred = predict_neuralode(p)
    loss = sum(abs2, ode_data .- pred)
    return loss, pred
end
# Callback function to observe training
list_plots = []
iter = 0
callback = function (p, l, pred; doplot = false)
  global list_plots, iter
  if iter == 0
    list_plots = []
  end
  iter += 1
  display(l)
  # plot current prediction against data
  plt = scatter(tsteps, Array(ode_data[1,:]), label = "data")
  scatter!(plt, tsteps, Array(pred[1,:]), label = "prediction")
  push!(list_plots, plt)
  if doplot
    display(plot(plt))
  end
  return false
end
result_neuralode = DiffEqFlux.sciml_train(loss_neuralode, p,
                                          ADAM(0.05), cb = callback,
                                          maxiters = 300)

when running this i get 


Training: 100%|█████████████████████████████████████████| Time: 0:00:03
scalar getindex is disallowed

Stacktrace:
 [1] error(::String) at .\error.jl:33
 [2] assertscalar(::String) at C:\Users\matias\.julia\packages\GPUArrays\WZupy\src\host\indexing.jl:41
 [3] getindex(::CuArray{Float32,1,Nothing}, ::Int64) at C:\Users\matias\.julia\packages\GPUArrays\WZupy\src\host\indexing.jl:96
 [4] getindex at C:\Users\matias\.julia\packages\ReverseDiff\uy0uk\src\tracked.jl:268 [inlined]
 [5] _broadcast_getindex at .\broadcast.jl:597 [inlined]
 [6] _getindex at .\broadcast.jl:627 [inlined] (repeats 2 times)
 [7] _broadcast_getindex at .\broadcast.jl:603 [inlined]
 [8] getindex at .\broadcast.jl:564 [inlined]
 [9] macro expansion at .\broadcast.jl:910 [inlined]
 [10] macro expansion at .\simdloop.jl:77 [inlined]
 [11] copyto! at .\broadcast.jl:909 [inlined]
 [12] copyto! at .\broadcast.jl:864 [inlined]
 [13] copy at .\broadcast.jl:840 [inlined]
 [14] materialize at .\broadcast.jl:820 [inlined]
 [15] (::var"#18#19")(::ReverseDiff.TrackedArray{Float32,Float32,1,CuArray{Float32,1,Nothing},CuArray{Float32,1,Nothing}}, ::ReverseDiff.TrackedArray{Float32,Float32,1,CuArray{Float32,1,Nothing},CuArray{Float32,1,Nothing}}) at .\In[9]:18
 [16] applychain at C:\Users\matias\.julia\packages\DiffEqFlux\YmiIM\src\fast_layers.jl:20 [inlined]
 [17] FastChain at C:\Users\matias\.julia\packages\DiffEqFlux\YmiIM\src\fast_layers.jl:21 [inlined]
 [18] dudt_ at C:\Users\matias\.julia\packages\DiffEqFlux\YmiIM\src\neural_de.jl:60 [inlined]
 [19] ODEFunction at C:\Users\matias\.julia\packages\DiffEqBase\8A5uo\src\diffeqfunction.jl:248 [inlined]
 [20] (::DiffEqSensitivity.var"#48#51"{ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#18#19",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}})(::ReverseDiff.TrackedArray{Float32,Float32,1,CuArray{Float32,1,Nothing},CuArray{Float32,1,Nothing}}, ::ReverseDiff.TrackedArray{Float32,Float32,1,CuArray{Float32,1,Nothing},CuArray{Float32,1,Nothing}}, ::ReverseDiff.TrackedArray{Float32,Float32,1,Array{Float32,1},Array{Float32,1}}) at C:\Users\matias\.julia\packages\DiffEqSensitivity\ykcWV\src\local_sensitivity\adjoint_common.jl:89
 [21] ReverseDiff.GradientTape(::Function, ::Tuple{CuArray{Float32,1,Nothing},CuArray{Float32,1,Nothing},Array{Float32,1}}, ::ReverseDiff.GradientConfig{Tuple{ReverseDiff.TrackedArray{Float32,Float32,1,CuArray{Float32,1,Nothing},CuArray{Float32,1,Nothing}},ReverseDiff.TrackedArray{Float32,Float32,1,CuArray{Float32,1,Nothing},CuArray{Float32,1,Nothing}},ReverseDiff.TrackedArray{Float32,Float32,1,Array{Float32,1},Array{Float32,1}}}}) at C:\Users\matias\.julia\packages\ReverseDiff\uy0uk\src\api\tape.jl:207
 [22] GradientTape at C:\Users\matias\.julia\packages\ReverseDiff\uy0uk\src\api\tape.jl:204 [inlined]
 [23] adjointdiffcache(::Function, ::InterpolatingAdjoint{0,true,Val{:central},ReverseDiffVJP{true}}, ::Bool, ::ODESolution{Float32,2,Array{CuArray{Float32,1,Nothing},1},Nothing,Nothing,Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},ODEProblem{CuArray{Float32,1,Nothing},Tuple{Float32,Float32},false,CuArray{Float32,1,Nothing},ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#18#19",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#18#19",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Array{CuArray{Float32,1,Nothing},1},Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}},DiffEqBase.DEStats}, ::Nothing; quad::Bool) at C:\Users\matias\.julia\packages\DiffEqSensitivity\ykcWV\src\local_sensitivity\adjoint_common.jl:88
 [24] adjointdiffcache at C:\Users\matias\.julia\packages\DiffEqSensitivity\ykcWV\src\local_sensitivity\adjoint_common.jl:25 [inlined]
 [25] DiffEqSensitivity.ODEInterpolatingAdjointSensitivityFunction(::Function, ::InterpolatingAdjoint{0,true,Val{:central},ReverseDiffVJP{true}}, ::Bool, ::ODESolution{Float32,2,Array{CuArray{Float32,1,Nothing},1},Nothing,Nothing,Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},ODEProblem{CuArray{Float32,1,Nothing},Tuple{Float32,Float32},false,CuArray{Float32,1,Nothing},ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#18#19",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#18#19",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Array{CuArray{Float32,1,Nothing},1},Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}},DiffEqBase.DEStats}, ::Nothing, ::Array{Float32,1}, ::Nothing, ::NamedTuple{(:reltol, :abstol),Tuple{Float64,Float64}}) at C:\Users\matias\.julia\packages\DiffEqSensitivity\ykcWV\src\local_sensitivity\interpolating_adjoint.jl:36
 [26] ODEAdjointProblem(::ODESolution{Float32,2,Array{CuArray{Float32,1,Nothing},1},Nothing,Nothing,Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},ODEProblem{CuArray{Float32,1,Nothing},Tuple{Float32,Float32},false,CuArray{Float32,1,Nothing},ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#18#19",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#18#19",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Array{CuArray{Float32,1,Nothing},1},Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}},DiffEqBase.DEStats}, ::InterpolatingAdjoint{0,true,Val{:central},ReverseDiffVJP{true}}, ::DiffEqSensitivity.var"#df#78"{CuArray{Float32,2,Nothing},CuArray{Float32,1,Nothing}}, ::StepRangeLen{Float32,Float64,Float64}, ::Nothing; checkpoints::Array{Float32,1}, callback::CallbackSet{Tuple{},Tuple{}}, reltol::Float64, abstol::Float64, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at C:\Users\matias\.julia\packages\DiffEqSensitivity\ykcWV\src\local_sensitivity\interpolating_adjoint.jl:104
 [27] _adjoint_sensitivities(::ODESolution{Float32,2,Array{CuArray{Float32,1,Nothing},1},Nothing,Nothing,Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},ODEProblem{CuArray{Float32,1,Nothing},Tuple{Float32,Float32},false,CuArray{Float32,1,Nothing},ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#18#19",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#18#19",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Array{CuArray{Float32,1,Nothing},1},Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}},DiffEqBase.DEStats}, ::InterpolatingAdjoint{0,true,Val{:central},ReverseDiffVJP{true}}, ::Tsit5, ::DiffEqSensitivity.var"#df#78"{CuArray{Float32,2,Nothing},CuArray{Float32,1,Nothing}}, ::StepRangeLen{Float32,Float64,Float64}, ::Nothing; abstol::Float64, reltol::Float64, checkpoints::Array{Float32,1}, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at C:\Users\matias\.julia\packages\DiffEqSensitivity\ykcWV\src\local_sensitivity\sensitivity_interface.jl:13
 [28] _adjoint_sensitivities(::ODESolution{Float32,2,Array{CuArray{Float32,1,Nothing},1},Nothing,Nothing,Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},ODEProblem{CuArray{Float32,1,Nothing},Tuple{Float32,Float32},false,CuArray{Float32,1,Nothing},ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#18#19",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#18#19",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Array{CuArray{Float32,1,Nothing},1},Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}},DiffEqBase.DEStats}, ::InterpolatingAdjoint{0,true,Val{:central},ReverseDiffVJP{true}}, ::Tsit5, ::Function, ::StepRangeLen{Float32,Float64,Float64}, ::Nothing) at C:\Users\matias\.julia\packages\DiffEqSensitivity\ykcWV\src\local_sensitivity\sensitivity_interface.jl:13 (repeats 2 times)
 [29] adjoint_sensitivities(::ODESolution{Float32,2,Array{CuArray{Float32,1,Nothing},1},Nothing,Nothing,Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},ODEProblem{CuArray{Float32,1,Nothing},Tuple{Float32,Float32},false,CuArray{Float32,1,Nothing},ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#18#19",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#18#19",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Array{CuArray{Float32,1,Nothing},1},Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}},DiffEqBase.DEStats}, ::Tsit5, ::Vararg{Any,N} where N; sensealg::InterpolatingAdjoint{0,true,Val{:central},ReverseDiffVJP{true}}, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at C:\Users\matias\.julia\packages\DiffEqSensitivity\ykcWV\src\local_sensitivity\sensitivity_interface.jl:6
 [30] (::DiffEqSensitivity.var"#adjoint_sensitivity_backpass#77"{Tsit5,InterpolatingAdjoint{0,true,Val{:central},ReverseDiffVJP{true}},CuArray{Float32,1,Nothing},CuArray{Float32,1,Nothing},Tuple{}})(::CuArray{Float32,2,Nothing}) at C:\Users\matias\.julia\packages\DiffEqSensitivity\ykcWV\src\local_sensitivity\concrete_solve.jl:72
 [31] #565#back at C:\Users\matias\.julia\packages\ZygoteRules\6nssF\src\adjoint.jl:55 [inlined]
 [32] #173 at C:\Users\matias\.julia\packages\Zygote\1aQlT\src\lib\lib.jl:182 [inlined]
 [33] #344#back at C:\Users\matias\.julia\packages\ZygoteRules\6nssF\src\adjoint.jl:49 [inlined]
 [34] NeuralODE at C:\Users\matias\.julia\packages\DiffEqFlux\YmiIM\src\neural_de.jl:62 [inlined]
 [35] (::typeof(∂(λ)))(::CuArray{Float32,2,Nothing}) at C:\Users\matias\.julia\packages\Zygote\1aQlT\src\compiler\interface2.jl:0
 [36] predict_neuralode at .\In[9]:26 [inlined]
 [37] (::typeof(∂(predict_neuralode)))(::CuArray{Float32,2,Nothing}) at C:\Users\matias\.julia\packages\Zygote\1aQlT\src\compiler\interface2.jl:0
 [38] loss_neuralode at .\In[9]:29 [inlined]
 [39] (::typeof(∂(loss_neuralode)))(::Tuple{Float32,Nothing}) at C:\Users\matias\.julia\packages\Zygote\1aQlT\src\compiler\interface2.jl:0
 [40] #173 at C:\Users\matias\.julia\packages\Zygote\1aQlT\src\lib\lib.jl:182 [inlined]
 [41] (::Zygote.var"#344#back#175"{Zygote.var"#173#174"{typeof(∂(loss_neuralode)),Tuple{Tuple{Nothing},Tuple{}}}})(::Tuple{Float32,Nothing}) at C:\Users\matias\.julia\packages\ZygoteRules\6nssF\src\adjoint.jl:49
 [42] #24 at C:\Users\matias\.julia\packages\DiffEqFlux\YmiIM\src\train.jl:99 [inlined]
 [43] (::typeof(∂(λ)))(::Float32) at C:\Users\matias\.julia\packages\Zygote\1aQlT\src\compiler\interface2.jl:0
 [44] (::Zygote.var"#48#49"{Zygote.Params,Zygote.Context,typeof(∂(λ))})(::Float32) at C:\Users\matias\.julia\packages\Zygote\1aQlT\src\compiler\interface.jl:167
 [45] gradient(::Function, ::Zygote.Params) at C:\Users\matias\.julia\packages\Zygote\1aQlT\src\compiler\interface.jl:45
 [46] macro expansion at C:\Users\matias\.julia\packages\DiffEqFlux\YmiIM\src\train.jl:98 [inlined]
 [47] macro expansion at C:\Users\matias\.julia\packages\ProgressLogging\g8xnW\src\ProgressLogging.jl:328 [inlined]
 [48] (::DiffEqFlux.var"#23#28"{DiffEqFlux.var"#27#32",Int64,Bool,Bool,typeof(loss_neuralode),CuArray{Float32,1,Nothing},Zygote.Params})() at C:\Users\matias\.julia\packages\DiffEqFlux\YmiIM\src\train.jl:43
 [49] with_logstate(::DiffEqFlux.var"#23#28"{DiffEqFlux.var"#27#32",Int64,Bool,Bool,typeof(loss_neuralode),CuArray{Float32,1,Nothing},Zygote.Params}, ::Base.CoreLogging.LogState) at .\logging.jl:398
 [50] with_logger at .\logging.jl:505 [inlined]
 [51] maybe_with_logger(::DiffEqFlux.var"#23#28"{DiffEqFlux.var"#27#32",Int64,Bool,Bool,typeof(loss_neuralode),CuArray{Float32,1,Nothing},Zygote.Params}, ::LoggingExtras.TeeLogger{Tuple{LoggingExtras.EarlyFilteredLogger{ConsoleProgressMonitor.ProgressLogger,DiffEqBase.var"#10#12"},LoggingExtras.EarlyFilteredLogger{Base.CoreLogging.SimpleLogger,DiffEqBase.var"#11#13"}}}) at C:\Users\matias\.julia\packages\DiffEqBase\8A5uo\src\utils.jl:259
 [52] sciml_train(::Function, ::CuArray{Float32,1,Nothing}, ::ADAM, ::Base.Iterators.Cycle{Tuple{DiffEqFlux.NullData}}; cb::Function, maxiters::Int64, progress::Bool, save_best::Bool) at C:\Users\matias\.julia\packages\DiffEqFlux\YmiIM\src\train.jl:42
​
if I change the second line : 

CuArrays.allowscalar(false) 
to 
CuArrays.allowscalar(false)

the error  i get is 

┌ Warning: Performing scalar operations on GPU arrays: This is very slow, consider disallowing these operations with `allowscalar(false)`
└ @ GPUArrays C:\Users\matias\.julia\packages\GPUArrays\WZupy\src\host\indexing.jl:43
Training: 100%|█████████████████████████████████████████| Time: 0:00:03
ArgumentError: cannot take the CPU address of a CuArray{Float32,2,CuArray{Float32,1,Nothing}}

Stacktrace:
 [1] unsafe_convert(::Type{Ptr{Float32}}, ::CuArray{Float32,2,CuArray{Float32,1,Nothing}}) at C:\Users\matias\.julia\packages\CuArrays\4Q1BY\src\array.jl:226
 [2] gemv!(::Char, ::Float32, ::CuArray{Float32,2,CuArray{Float32,1,Nothing}}, ::Array{Float32,1}, ::Float32, ::Array{Float32,1}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\LinearAlgebra\src\blas.jl:587
 [3] gemv!(::Array{Float32,1}, ::Char, ::CuArray{Float32,2,CuArray{Float32,1,Nothing}}, ::Array{Float32,1}, ::Bool, ::Bool) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\LinearAlgebra\src\matmul.jl:470
 [4] mul! at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\LinearAlgebra\src\matmul.jl:66 [inlined]
 [5] mul! at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\LinearAlgebra\src\matmul.jl:208 [inlined]
 [6] * at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\LinearAlgebra\src\matmul.jl:47 [inlined]
 [7] record_mul at C:\Users\matias\.julia\packages\ReverseDiff\uy0uk\src\derivatives\linalg\arithmetic.jl:173 [inlined]
 [8] *(::ReverseDiff.TrackedArray{Float32,Float32,2,CuArray{Float32,2,CuArray{Float32,1,Nothing}},CuArray{Float32,2,CuArray{Float32,1,Nothing}}}, ::Array{ReverseDiff.TrackedReal{Float32,Float32,Nothing},1}) at C:\Users\matias\.julia\packages\ReverseDiff\uy0uk\src\derivatives\linalg\arithmetic.jl:217
 [9] (::FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}})(::Array{ReverseDiff.TrackedReal{Float32,Float32,Nothing},1}, ::ReverseDiff.TrackedArray{Float32,Float32,1,CuArray{Float32,1,Nothing},CuArray{Float32,1,Nothing}}) at C:\Users\matias\.julia\packages\DiffEqFlux\YmiIM\src\fast_layers.jl:37
 [10] applychain(::Tuple{FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}, ::Array{ReverseDiff.TrackedReal{Float32,Float32,Nothing},1}, ::ReverseDiff.TrackedArray{Float32,Float32,1,CuArray{Float32,1,Nothing},CuArray{Float32,1,Nothing}}) at C:\Users\matias\.julia\packages\DiffEqFlux\YmiIM\src\fast_layers.jl:20 (repeats 2 times)
 [11] FastChain at C:\Users\matias\.julia\packages\DiffEqFlux\YmiIM\src\fast_layers.jl:21 [inlined]
 [12] dudt_ at C:\Users\matias\.julia\packages\DiffEqFlux\YmiIM\src\neural_de.jl:60 [inlined]
 [13] ODEFunction at C:\Users\matias\.julia\packages\DiffEqBase\8A5uo\src\diffeqfunction.jl:248 [inlined]
 [14] (::DiffEqSensitivity.var"#48#51"{ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#20#21",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing}})(::ReverseDiff.TrackedArray{Float32,Float32,1,CuArray{Float32,1,Nothing},CuArray{Float32,1,Nothing}}, ::ReverseDiff.TrackedArray{Float32,Float32,1,CuArray{Float32,1,Nothing},CuArray{Float32,1,Nothing}}, ::ReverseDiff.TrackedArray{Float32,Float32,1,Array{Float32,1},Array{Float32,1}}) at C:\Users\matias\.julia\packages\DiffEqSensitivity\ykcWV\src\local_sensitivity\adjoint_common.jl:89
 [15] ReverseDiff.GradientTape(::Function, ::Tuple{CuArray{Float32,1,Nothing},CuArray{Float32,1,Nothing},Array{Float32,1}}, ::ReverseDiff.GradientConfig{Tuple{ReverseDiff.TrackedArray{Float32,Float32,1,CuArray{Float32,1,Nothing},CuArray{Float32,1,Nothing}},ReverseDiff.TrackedArray{Float32,Float32,1,CuArray{Float32,1,Nothing},CuArray{Float32,1,Nothing}},ReverseDiff.TrackedArray{Float32,Float32,1,Array{Float32,1},Array{Float32,1}}}}) at C:\Users\matias\.julia\packages\ReverseDiff\uy0uk\src\api\tape.jl:207
 [16] GradientTape at C:\Users\matias\.julia\packages\ReverseDiff\uy0uk\src\api\tape.jl:204 [inlined]
 [17] adjointdiffcache(::Function, ::InterpolatingAdjoint{0,true,Val{:central},ReverseDiffVJP{true}}, ::Bool, ::ODESolution{Float32,2,Array{CuArray{Float32,1,Nothing},1},Nothing,Nothing,Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},ODEProblem{CuArray{Float32,1,Nothing},Tuple{Float32,Float32},false,CuArray{Float32,1,Nothing},ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#20#21",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#20#21",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Array{CuArray{Float32,1,Nothing},1},Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}},DiffEqBase.DEStats}, ::Nothing; quad::Bool) at C:\Users\matias\.julia\packages\DiffEqSensitivity\ykcWV\src\local_sensitivity\adjoint_common.jl:88
 [18] adjointdiffcache at C:\Users\matias\.julia\packages\DiffEqSensitivity\ykcWV\src\local_sensitivity\adjoint_common.jl:25 [inlined]
 [19] DiffEqSensitivity.ODEInterpolatingAdjointSensitivityFunction(::Function, ::InterpolatingAdjoint{0,true,Val{:central},ReverseDiffVJP{true}}, ::Bool, ::ODESolution{Float32,2,Array{CuArray{Float32,1,Nothing},1},Nothing,Nothing,Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},ODEProblem{CuArray{Float32,1,Nothing},Tuple{Float32,Float32},false,CuArray{Float32,1,Nothing},ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#20#21",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#20#21",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Array{CuArray{Float32,1,Nothing},1},Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}},DiffEqBase.DEStats}, ::Nothing, ::Array{Float32,1}, ::Nothing, ::NamedTuple{(:reltol, :abstol),Tuple{Float64,Float64}}) at C:\Users\matias\.julia\packages\DiffEqSensitivity\ykcWV\src\local_sensitivity\interpolating_adjoint.jl:36
 [20] ODEAdjointProblem(::ODESolution{Float32,2,Array{CuArray{Float32,1,Nothing},1},Nothing,Nothing,Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},ODEProblem{CuArray{Float32,1,Nothing},Tuple{Float32,Float32},false,CuArray{Float32,1,Nothing},ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#20#21",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#20#21",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Array{CuArray{Float32,1,Nothing},1},Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}},DiffEqBase.DEStats}, ::InterpolatingAdjoint{0,true,Val{:central},ReverseDiffVJP{true}}, ::DiffEqSensitivity.var"#df#78"{CuArray{Float32,2,Nothing},CuArray{Float32,1,Nothing}}, ::StepRangeLen{Float32,Float64,Float64}, ::Nothing; checkpoints::Array{Float32,1}, callback::CallbackSet{Tuple{},Tuple{}}, reltol::Float64, abstol::Float64, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at C:\Users\matias\.julia\packages\DiffEqSensitivity\ykcWV\src\local_sensitivity\interpolating_adjoint.jl:104
 [21] _adjoint_sensitivities(::ODESolution{Float32,2,Array{CuArray{Float32,1,Nothing},1},Nothing,Nothing,Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},ODEProblem{CuArray{Float32,1,Nothing},Tuple{Float32,Float32},false,CuArray{Float32,1,Nothing},ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#20#21",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#20#21",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Array{CuArray{Float32,1,Nothing},1},Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}},DiffEqBase.DEStats}, ::InterpolatingAdjoint{0,true,Val{:central},ReverseDiffVJP{true}}, ::Tsit5, ::DiffEqSensitivity.var"#df#78"{CuArray{Float32,2,Nothing},CuArray{Float32,1,Nothing}}, ::StepRangeLen{Float32,Float64,Float64}, ::Nothing; abstol::Float64, reltol::Float64, checkpoints::Array{Float32,1}, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at C:\Users\matias\.julia\packages\DiffEqSensitivity\ykcWV\src\local_sensitivity\sensitivity_interface.jl:13
 [22] _adjoint_sensitivities(::ODESolution{Float32,2,Array{CuArray{Float32,1,Nothing},1},Nothing,Nothing,Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},ODEProblem{CuArray{Float32,1,Nothing},Tuple{Float32,Float32},false,CuArray{Float32,1,Nothing},ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#20#21",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#20#21",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Array{CuArray{Float32,1,Nothing},1},Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}},DiffEqBase.DEStats}, ::InterpolatingAdjoint{0,true,Val{:central},ReverseDiffVJP{true}}, ::Tsit5, ::Function, ::StepRangeLen{Float32,Float64,Float64}, ::Nothing) at C:\Users\matias\.julia\packages\DiffEqSensitivity\ykcWV\src\local_sensitivity\sensitivity_interface.jl:13 (repeats 2 times)
 [23] adjoint_sensitivities(::ODESolution{Float32,2,Array{CuArray{Float32,1,Nothing},1},Nothing,Nothing,Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},ODEProblem{CuArray{Float32,1,Nothing},Tuple{Float32,Float32},false,CuArray{Float32,1,Nothing},ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#20#21",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},DiffEqBase.StandardODEProblem},Tsit5,OrdinaryDiffEq.InterpolationData{ODEFunction{false,DiffEqFlux.var"#dudt_#92"{NeuralODE{FastChain{Tuple{var"#20#21",FastDense{typeof(tanh),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}},FastDense{typeof(identity),DiffEqFlux.var"#initial_params#75"{typeof(Flux.glorot_uniform),typeof(Flux.zeros),Int64,Int64}}}},Array{Float32,1},Nothing,Tuple{Float32,Float32},Tsit5,Tuple{},Base.Iterators.Pairs{Symbol,StepRangeLen{Float32,Float64,Float64},Tuple{Symbol},NamedTuple{(:saveat,),Tuple{StepRangeLen{Float32,Float64,Float64}}}}}},LinearAlgebra.UniformScaling{Bool},Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing,Nothing},Array{CuArray{Float32,1,Nothing},1},Array{Float32,1},Array{Array{CuArray{Float32,1,Nothing},1},1},OrdinaryDiffEq.Tsit5ConstantCache{Float32,Float32}},DiffEqBase.DEStats}, ::Tsit5, ::Vararg{Any,N} where N; sensealg::InterpolatingAdjoint{0,true,Val{:central},ReverseDiffVJP{true}}, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at C:\Users\matias\.julia\packages\DiffEqSensitivity\ykcWV\src\local_sensitivity\sensitivity_interface.jl:6
 [24] (::DiffEqSensitivity.var"#adjoint_sensitivity_backpass#77"{Tsit5,InterpolatingAdjoint{0,true,Val{:central},ReverseDiffVJP{true}},CuArray{Float32,1,Nothing},CuArray{Float32,1,Nothing},Tuple{}})(::CuArray{Float32,2,Nothing}) at C:\Users\matias\.julia\packages\DiffEqSensitivity\ykcWV\src\local_sensitivity\concrete_solve.jl:72
 [25] #565#back at C:\Users\matias\.julia\packages\ZygoteRules\6nssF\src\adjoint.jl:55 [inlined]
 [26] #173 at C:\Users\matias\.julia\packages\Zygote\1aQlT\src\lib\lib.jl:182 [inlined]
 [27] #344#back at C:\Users\matias\.julia\packages\ZygoteRules\6nssF\src\adjoint.jl:49 [inlined]
 [28] NeuralODE at C:\Users\matias\.julia\packages\DiffEqFlux\YmiIM\src\neural_de.jl:62 [inlined]
 [29] (::typeof(∂(λ)))(::CuArray{Float32,2,Nothing}) at C:\Users\matias\.julia\packages\Zygote\1aQlT\src\compiler\interface2.jl:0
 [30] predict_neuralode at .\In[11]:27 [inlined]
 [31] (::typeof(∂(predict_neuralode)))(::CuArray{Float32,2,Nothing}) at C:\Users\matias\.julia\packages\Zygote\1aQlT\src\compiler\interface2.jl:0
 [32] loss_neuralode at .\In[11]:30 [inlined]
 [33] (::typeof(∂(loss_neuralode)))(::Tuple{Float32,Nothing}) at C:\Users\matias\.julia\packages\Zygote\1aQlT\src\compiler\interface2.jl:0
 [34] #173 at C:\Users\matias\.julia\packages\Zygote\1aQlT\src\lib\lib.jl:182 [inlined]
 [35] (::Zygote.var"#344#back#175"{Zygote.var"#173#174"{typeof(∂(loss_neuralode)),Tuple{Tuple{Nothing},Tuple{}}}})(::Tuple{Float32,Nothing}) at C:\Users\matias\.julia\packages\ZygoteRules\6nssF\src\adjoint.jl:49
 [36] #24 at C:\Users\matias\.julia\packages\DiffEqFlux\YmiIM\src\train.jl:99 [inlined]
 [37] (::typeof(∂(λ)))(::Float32) at C:\Users\matias\.julia\packages\Zygote\1aQlT\src\compiler\interface2.jl:0
 [38] (::Zygote.var"#48#49"{Zygote.Params,Zygote.Context,typeof(∂(λ))})(::Float32) at C:\Users\matias\.julia\packages\Zygote\1aQlT\src\compiler\interface.jl:167
 [39] gradient(::Function, ::Zygote.Params) at C:\Users\matias\.julia\packages\Zygote\1aQlT\src\compiler\interface.jl:45
 [40] macro expansion at C:\Users\matias\.julia\packages\DiffEqFlux\YmiIM\src\train.jl:98 [inlined]
 [41] macro expansion at C:\Users\matias\.julia\packages\ProgressLogging\g8xnW\src\ProgressLogging.jl:328 [inlined]
 [42] (::DiffEqFlux.var"#23#28"{var"#15#17",Int64,Bool,Bool,typeof(loss_neuralode),CuArray{Float32,1,Nothing},Zygote.Params})() at C:\Users\matias\.julia\packages\DiffEqFlux\YmiIM\src\train.jl:43
 [43] with_logstate(::DiffEqFlux.var"#23#28"{var"#15#17",Int64,Bool,Bool,typeof(loss_neuralode),CuArray{Float32,1,Nothing},Zygote.Params}, ::Base.CoreLogging.LogState) at .\logging.jl:398
 [44] with_logger at .\logging.jl:505 [inlined]
 [45] maybe_with_logger(::DiffEqFlux.var"#23#28"{var"#15#17",Int64,Bool,Bool,typeof(loss_neuralode),CuArray{Float32,1,Nothing},Zygote.Params}, ::LoggingExtras.TeeLogger{Tuple{LoggingExtras.EarlyFilteredLogger{ConsoleProgressMonitor.ProgressLogger,DiffEqBase.var"#10#12"},LoggingExtras.EarlyFilteredLogger{Base.CoreLogging.SimpleLogger,DiffEqBase.var"#11#13"}}}) at C:\Users\matias\.julia\packages\DiffEqBase\8A5uo\src\utils.jl:259
 [46] sciml_train(::Function, ::CuArray{Float32,1,Nothing}, ::ADAM, ::Base.Iterators.Cycle{Tuple{DiffEqFlux.NullData}}; cb::Function, maxiters::Int64, progress::Bool, save_best::Bool) at C:\Users\matias\.julia\packages\DiffEqFlux\YmiIM\src\train.jl:42
 [47] top-level scope at In[12]:1
​```

any ideas what could be the problem? 
thanks a lot

Matias
ChrisRackauckas commented 4 years ago

Seems like you just opened a few issues, so let's pin this down. I just copy pasted and ran the code just fine, so at least it's an installation issue. Are you on DiffEqFlux v1.10.1? Are you on Julia v1.4?

matutenun commented 4 years ago

yes, thanks! I was using a previous version of DiffEqFlux. I updated everything, now the example does run. Note: It seems a minor issue, but I get this message/error below after finishing "scalar getindex is disallowed"

... loss: 0.223: 100%|█████████████████████████████████████████| Time: 0:04:52 0.22293958f0

scalar getindex is disallowed

Stacktrace: [1] error(::String) at .\error.jl:33 [2] assertscalar(::String) at C:\Users\matias.julia\packages\GPUArrays\OXvxB\src\host\indexing.jl:41 [3] getindex(::CuArray{Float32,1,Nothing}, ::Int64) at C:\Users\matias.julia\packages\GPUArrays\OXvxB\src\host\indexing.jl:96 [4] iterate at .\abstractarray.jl:913 [inlined] [5] iterate at .\abstractarray.jl:911 [inlined] [6] iterate at .\iterators.jl:641 [inlined] [7] iterate at .\iterators.jl:639 [inlined] [8] iterate at .\generator.jl:44 [inlined] [9] collect(::Base.Generator{Base.Iterators.Take{CuArray{Float32,1,Nothing}},Optim.var"#2#4"}) at .\array.jl:665 [10] show(::IOContext{Base.GenericIOBuffer{Array{UInt8,1}}}, ::Optim.MultivariateOptimizationResults{ADAM,Float64,CuArray{Float32,1,Nothing},Float64,Float32,Nothing,Bool}) at C:\Users\matias.julia\packages\Optim\UkDyx\src\types.jl:242 [11] show at .\multimedia.jl:47 [inlined] [12] limitstringmime(::MIME{Symbol("text/plain")}, ::Optim.MultivariateOptimizationResults{ADAM,Float64,CuArray{Float32,1,Nothing},Float64,Float32,Nothing,Bool}) at C:\Users\matias.julia\packages\IJulia\DrVMH\src\inline.jl:43 [13] display_mimestring(::MIME{Symbol("text/plain")}, ::Optim.MultivariateOptimizationResults{ADAM,Float64,CuArray{Float32,1,Nothing},Float64,Float32,Nothing,Bool}) at C:\Users\matias.julia\packages\IJulia\DrVMH\src\display.jl:67 [14] display_dict(::Optim.MultivariateOptimizationResults{ADAM,Float64,CuArray{Float32,1,Nothing},Float64,Float32,Nothing,Bool}) at C:\Users\matias.julia\packages\IJulia\DrVMH\src\display.jl:96 [15] #invokelatest#1 at .\essentials.jl:712 [inlined] [16] invokelatest at .\essentials.jl:711 [inlined] [17] execute_request(::ZMQ.Socket, ::IJulia.Msg) at C:\Users\matias.julia\packages\IJulia\DrVMH\src\execute_request.jl:112 [18] #invokelatest#1 at .\essentials.jl:712 [inlined] [19] invokelatest at .\essentials.jl:711 [inlined] [20] eventloop(::ZMQ.Socket) at C:\Users\matias.julia\packages\IJulia\DrVMH\src\eventloop.jl:8 [21] (::IJulia.var"#15#18")() at .\task.jl:358

matutenun commented 4 years ago

which is the same error as before , but it does finish the convergence, as I can use the optimized parameters in result_neuralode.minimizer

ChrisRackauckas commented 4 years ago

Is that in a new REPL session? Can you share your status? i.e. ]st? I'm curious what versions of the GPU packages you're on since I know on the latest ones it works (just tested)

matutenun commented 4 years ago

it was from jupyterlab typing ]st

Status C:\Users\matias\.julia\environments\v1.4\Project.toml [fbb218c0] BSON v0.2.6 [6e4b80f9] BenchmarkTools v0.5.0 [336ed68f] CSV v0.6.2 [3895d2a7] CUDAapi v4.0.0 [c5f51814] CUDAdrv v6.3.0 [be33ccc6] CUDAnative v3.1.0 [5ae59095] Colors v0.12.0 [3a865a2d] CuArrays v2.2.0 [2445eb08] DataDrivenDiffEq v0.3.1 [a93c6f00] DataFrames v0.21.0 [aae7a2af] DiffEqFlux v1.10.1 [41bf760c] DiffEqSensitivity v6.14.2 [0c46a032] DifferentialEquations v6.14.0 [b4f34e82] Distances v0.8.2 [31c24e10] Distributions v0.23.2 [ced4e74d] DistributionsAD v0.5.2 [5789e2e9] FileIO v1.3.0 [587475ba] Flux v0.10.4 [f6369f11] ForwardDiff v0.10.10 [0c68f7d7] GPUArrays v3.3.0 [7073ff75] IJulia v1.21.2 [6218d12a] ImageMagick v1.1.5 [4e3cecfd] ImageShow v0.2.3 [916415d5] Images v0.22.2 [c601a237] Interact v0.10.3 [18b7da76] JuliaAcademyData v0.1.0 #master (https://github.com/JuliaComputing/JuliaAcademyData.jl) [e5e0dc1b] Juno v0.8.1 [961ee093] ModelingToolkit v3.3.0 [429524aa] Optim v0.20.6 [1dea7af3] OrdinaryDiffEq v5.38.1 [65888b18] ParameterizedFunctions v5.3.0 [58dd65bb] Plotly v0.3.0 [91a5bcdd] Plots v1.2.5 [d330b81b] PyPlot v2.9.0 [1fd47b50] QuadGK v2.3.1 [90137ffa] StaticArrays v0.12.3 [6fc51010] Surrogates v1.1.2 [6aa5eb33] TaylorSeries v0.10.3 [0f1e0344] WebIO v0.8.14 [e88e6eb3] Zygote v0.4.20

ChrisRackauckas commented 4 years ago

Could you completely restart and try again? I'm curious if Jupyter Lab is doing some kind of caching that blocks the update until you're in a completely new session.

matutenun commented 4 years ago

from the repl I get something similar, after finishing the convergence,

julia> \\ ERROR: syntax: "\" is not a unary operator Stacktrace:

ChrisRackauckas commented 4 years ago

Okay cool, I expect Optim will have issue showing the result right now (though you can index it). We should make it not count maximum iterations as a failure though. That means it worked if it's an ML optimizer like from Flux.

matutenun commented 4 years ago

indeed if i use BFGS it does not work, the error :

scalar getindex is disallowed

Stacktrace: [1] error(::String) at .\error.jl:33 [2] assertscalar(::String) at C:\Users\matias.julia\packages\GPUArrays\OXvxB\src\host\indexing.jl:41 [3] getindex(::CuArray{Float32,1,Nothing}, ::Int64) at C:\Users\matias.julia\packages\GPUArrays\OXvxB\src\host\indexing.jl:96 [4] iterate at .\abstractarray.jl:913 [inlined] [5] iterate at .\abstractarray.jl:911 [inlined] [6] generic_normInf at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\LinearAlgebra\src\generic.jl:445 [inlined] [7] normInf at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\LinearAlgebra\src\generic.jl:536 [inlined] [8] norm(::CuArray{Float32,1,Nothing}, ::Float64) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\LinearAlgebra\src\generic.jl:611 [9] initial_state(::BFGS{LineSearches.InitialStatic{Float64},LineSearches.HagerZhang{Float64,Base.RefValue{Bool}},Nothing,Float64,Flat}, ::Optim.Options{Float64,DiffEqFlux.var"#_cb#46"{var"#15#17",Base.Iterators.Cycle{Tuple{DiffEqFlux.NullData}}}}, ::TwiceDifferentiable{Float32,CuArray{Float32,1,Nothing},Array{Float32,2},CuArray{Float32,1,Nothing}}, ::CuArray{Float32,1,Nothing}) at C:\Users\matias.julia\packages\Optim\UkDyx\src\multivariate\solvers\first_order\bfgs.jl:74 [10] optimize(::TwiceDifferentiable{Float32,CuArray{Float32,1,Nothing},Array{Float32,2},CuArray{Float32,1,Nothing}}, ::CuArray{Float32,1,Nothing}, ::BFGS{LineSearches.InitialStatic{Float64},LineSearches.HagerZhang{Float64,Base.RefValue{Bool}},Nothing,Float64,Flat}, ::Optim.Options{Float64,DiffEqFlux.var"#_cb#46"{var"#15#17",Base.Iterators.Cycle{Tuple{DiffEqFlux.NullData}}}}) at C:\Users\matias.julia\packages\Optim\UkDyx\src\multivariate\optimize\optimize.jl:33 [11] sciml_train(::Function, ::CuArray{Float32,1,Nothing}, ::BFGS{LineSearches.InitialStatic{Float64},LineSearches.HagerZhang{Float64,Base.RefValue{Bool}},Nothing,Float64,Flat}, ::Base.Iterators.Cycle{Tuple{DiffEqFlux.NullData}}; cb::Function, maxiters::Int64, diffmode::DiffEqFlux.ZygoteDiffMode, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at C:\Users\matias.julia\packages\DiffEqFlux\9CzLb\src\train.jl:269

ChrisRackauckas commented 4 years ago

Yes that's a known upstream issue: https://github.com/SciML/DiffEqFlux.jl/issues/238

matutenun commented 4 years ago

what i did noticed is that if i benchmark the case of using GPU vs CPU (same example) is two orders of magnitud faster with CPU

with GPU

BenchmarkTools.Trial: memory estimate: 13.26 GiB allocs estimate: 416496234

minimum time: 210.416 s (2.08% GC) median time: 210.416 s (2.08% GC) mean time: 210.416 s (2.08% GC) maximum time: 210.416 s (2.08% GC)

samples: 1 evals/sample: 1

with CPU

BenchmarkTools.Trial: memory estimate: 1.76 GiB allocs estimate: 64750578

minimum time: 5.393 s (6.27% GC) median time: 5.393 s (6.27% GC) mean time: 5.393 s (6.27% GC) maximum time: 5.393 s (6.27% GC)

samples: 1 evals/sample: 1

ChrisRackauckas commented 4 years ago

Yes, size 10/20 matrices are not good for GPUs. It's a demonstration but not really a tutorial that's great for GPUs. You want larger NNs for that.

ChrisRackauckas commented 4 years ago

In fact, you can probably get another order of magnitude faster by hyper optimizing for the CPU here with some StaticDense stuff.

matutenun commented 4 years ago

cool, thanks for the an swers. aweome set of tools, I will keep playing with them and learn to use them.

ChrisRackauckas commented 4 years ago

For more questions, and just general chatter on the topics, feel free to join the Julia Slack: https://slackinvite.julialang.org/ . We're in the #diffeq-bridged and the #machine-learning channels.