Error when running Flux.train! under VS Code debugger

[karimn]

The below code runs fine but when I run it under the VS Code debugger I get an ErrorException: Method is @generated; try 'code_lowered' instead.

using Flux

input = rand(28)
label = rand(10)

model = Chain(
  Dense(28 => 10)
)

model(input)

opt_state = Flux.setup(Adam(), model)

for epoch in 1:100
  Flux.train!(model, [(input, label)], opt_state) do m, x, y
    Flux.mse(m(x), y)
  end
end

Stacktrace:
 [1] train!(loss::var"#1#2", model::Chain{Tuple{Dense{typeof(identity), Matrix{Float32}, Vector{Float32}}}}, data::Vector{Tuple{Vector{Float64}, Vector{Float64}}}, opt::NamedTuple{(:layers,), Tuple{Tuple{NamedTuple{(:weight, :bias, :σ), Tuple{Optimisers.Leaf{Optimisers.Adam{Float64}, Tuple{Matrix{Float32}, Matrix{Float32}, Tuple{Float64, Float64}}}, Optimisers.Leaf{Optimisers.Adam{Float64}, Tuple{Vector{Float32}, Vector{Float32}, Tuple{Float64, Float64}}}, Tuple{}}}}}}; cb::Nothing)
   @ Flux.Train ~/.julia/packages/ProgressLogging/6KXlp/src/ProgressLogging.jl:385
 [2] train!(loss::var"#1#2", model::Chain{Tuple{Dense{typeof(identity), Matrix{Float32}, Vector{Float32}}}}, data::Vector{Tuple{Vector{Float64}, Vector{Float64}}}, opt::NamedTuple{(:layers,), Tuple{Tuple{NamedTuple{(:weight, :bias, :σ), Tuple{Optimisers.Leaf{Optimisers.Adam{Float64}, Tuple{Matrix{Float32}, Matrix{Float32}, Tuple{Float64, Float64}}}, Optimisers.Leaf{Optimisers.Adam{Float64}, Tuple{Vector{Float32}, Vector{Float32}, Tuple{Float64, Float64}}}, Tuple{}}}}}})
   @ Flux.Train ~/.julia/packages/Flux/FWgS0/src/train.jl:102
 [3] top-level scope
   @ ~/Code/grokking-drl/test_flux.jl:15

  [052768ef] CUDA v4.1.4
  [587475ba] Flux v0.13.15
  [b98c9c47] Pipe v1.3.0
  [438e738f] PyCall v1.95.1

I'm using Julia v1.8.5.

[ToucheSir]

That's because the AD engine Flux uses by default, Zygote, uses generated functions extensively. This warning is new to me though, because I've successfully used the same debugger on Zygote code before. You may want to investigate this further with the Julia VS Code extension folks.

[karimn]

Thank you for the quick response. Ok, I'll reach out the VS Code extension folks. Is there an alternative debugger that you use for debugging Flux code?

[ToucheSir]

Kind of, but my workflow is neither typical nor particularly straightforward :sweat_smile:.

For model-level debugging I typically stay away from line-by-line debuggers like the VSCode one because they can deoptimize tight loops and make getting to your breakpoint take too long. There are ways around this such as enabling compiled mode for certain modules, but it's a little finicky to do when developing these packages. https://github.com/JuliaDebug/Infiltrator.jl is a great tool if you can make do without line-by-line stepping, maybe have a look at that.

For debugging code which uses Zygote, I occasionally use https://github.com/JuliaDebug/Debugger.jl, which is also what the VSCode extension uses under the hood. The reason I don't use VSCode directly is because I need to enable a setting (IR view) to step into generated functions. I would not recommend trying this unless you're comfortable reading Julia AST and IR.