unhelpful `ReadOnlyMemoryError` message

EnzymeAD / Enzyme.jl

Julia bindings for the Enzyme automatic differentiator

https://enzyme.mit.edu

MIT License

458 stars 66 forks source link

Open gaelforget opened 1 month ago

gaelforget commented 1 month ago

I am trying to differentiate this function (main branch) and use the resulting adjoint.

using ECCO, Enzyme
x=[0.004]
glacier_ad(x)=autodiff(Reverse, ECCO.glacier_model.integrate, Active(x[1]))
glacier_ad(x)

The autodiff step does return a function,glacier_ad, but this function errors out, and I have no idea what to do to resolve the issue.

julia> glacier_ad(x)
ERROR: ReadOnlyMemoryError()
Stacktrace:
 [1] forward_problem
   @ ECCO.jl/src/mountain_glacier.jl:19

I looked through the docs of Enzyme.jl and searched for ReadOnlyMemoryError via github but did not immediately find any help.

How should I proceed to debug this? Where is the error message actually coming from? Am I missing something obvious?

Please advise.

wsmoses commented 1 month ago

That error is being thrown by Julia itself, not Enzyme.

Do you have a stacktrace or error log?

gaelforget commented 1 month ago

Thank you very much for the quick feedback.

The above is all the whole error message / stack trace.

Maybe a hint as to what happens : setting the number of time step is set to 6*5000 works, but at 8*5000 I get the above error message.

Does this make sense to you?

gaelforget commented 1 month ago

I just pushed (this commit) a version that works by setting nt=6*5000 here
To reproduce the error it suffice to edit the file and set nt=8*5000, e.g. with Revise.jl

gaelforget commented 1 month ago

Should have probably mentioned that results reported above were done with Version 1.10.5.
with Version 1.11.0 , the nt=6*5000 test crashes Julia as seen in this CI run
but it seems that Enzyme + 1.11 has broader issues based on that CI run

wsmoses commented 1 month ago

Will need to investigate this further, but perhaps it's useful to apply Checkpointing.jl on your timestep loop?

Enzyme hasn't yet adapted to 1.11 changes so the 1.11 error is expected (and you should use 1.10 for now)

cc @swilliamson7 @heimbach @michel2323