When running with failmode flag the predict = model(data) breaks:
(line 73 is reshape before Dense)
ERROR: LoadError: MethodError: no method matching (::BSON.__deserialized_types__.var"#3#5")(::Array{Float32,4})
The applicable method may be too new: running in world age 26854, while current world is 26855.
Closest candidates are:
#3(::Any) at /scratch/jchen975/train/case300/train.jl:73 (method too new to be called from this world context.)
Stacktrace:
[1] applychain(::Tuple{BSON.__deserialized_types__.var"#3#5",Dense{typeof(identity),Array{Float32,2},Array{Float32,1}}}, ::Arr ay{Float32,4}) at /home/jchen975/.julia/packages/Flux/oX9Pi/src/layers/basic.jl:30 (repeats 4 times)
[2] (::Chain{Tuple{Conv{2,2,typeof(elu),Array{Float32,4},Array{Float32,1}},Conv{2,2,typeof(elu),Array{Float32,4},Array{Float32 ,1}},Conv{2,2,typeof(elu),Array{Float32,4},Array{Float32,1}},BSON.__deserialized_types__.var"#3#5",Dense{typeof(identity),Array {Float32,2},Array{Float32,1}}}})(::Array{Float32,4}) at /home/jchen975/.julia/packages/Flux/oX9Pi/src/layers/basic.jl:32
[3] forward(::Array{Float32,3}, ::String, ::String, ::Float64, ::Int64, ::Bool, ::Bool) at /scratch/jchen975/train/case300/tra in2.jl:378
[4] main(::Array{String,1}) at /scratch/jchen975/train/case300/train.jl:472
[5] top-level scope at /scratch/jchen975/train/case300/train.jl:591
[6] include at ./boot.jl:328 [inlined]
[7] include_relative(::Module, ::String) at ./loading.jl:1105
[8] include(::Module, ::String) at ./Base.jl:31
[9] exec_options(::Base.JLOptions) at ./client.jl:287
[10] _start() at ./client.jl:460
in expression starting at /scratch/jchen975/train/case300/train.jl:591
But it works fine in REPL (login node doesn't have GPU so |> gpu was no-op). Must. Investigate.
When running with
failmodeflag thepredict = model(data)breaks: (line 73 isreshapebeforeDense)But it works fine in REPL (login node doesn't have GPU so
|> gpuwas no-op). Must. Investigate.