Open DrChainsaw opened 1 year ago
One thing we could try is adding @nospecialize
to some update!
methods? Or even to a whole block of its code.
Yeah, I guess that is the simplest option. Won't it hurt the SciML μs-hunting cases?
maybe @ChrisRackauckas has some advice here
Make a flamegraph of the compile times on a standard case to see what actually matters. See for example https://github.com/SciML/DifferentialEquations.jl/issues/786#issuecomment-991334836 . That will tell you what is taking all of the compile time and should be focused on.
Removing specialization doesn't necessarily make compilation faster. Sometimes, the fact that it can remove inference at a level can actually make it worse, so check and see if it's inference time or llvm time.
Won't it hurt the SciML μs-hunting cases?
That's fine. If we need something separate we can do SimpleOptimization.jl. We already have SimpleDiffEq.jl and SimpleNonlienarSolve.jl which are specailized implementation for things like GPU kernels and ultra-small problems on the same interface. If we need a SimpleOptimization.jl for a SimpleGradientDecent and SimpleAdam then I wouldn't be too surprised. So if you go and make Optimisers the best thing for neural networks we won't mind and we can just make sure Optimization.jl has multiple options with the right trade-offs.
Is there some known remedy for explosive compile times when updating deeply nested models?
I did this to prevent spurious stalls with NaiveGAflux, but maybe there is a better way.