search

mabarnes / moment_kinetics

Other
2 stars 4 forks source link

If a NaN is found, stop the simulation #132

Closed johnomotani closed 1 year ago

johnomotani commented 1 year ago

When writing output, check for NaNs and Allreduce/Bcast the result so that the simulation can stop cleanly. This probably won't cost much, and prevents time/energy being wasted running simulations that are silently producing junk.

mrhardman commented 1 year ago

I am discovering that my 2D1V simulations go unstable (unclear yet if numerical or physical) and end up printing NaNs. Is this branch ready to merge?

johnomotani commented 1 year ago

I'd based on the previous PRs, but that was just because it was the last thing I'd added in the stuff I'd been working on. Can rebase onto master easily, then should be merge-able (I hope the test failures were caused by a different PR, so should go away then). Will do that now.

johnomotani commented 1 year ago

I think the CI tests just got confused when I changed the base branch. The 'push' tests pass fine, but the 'pull' tests are getting a bug from an old version of save-time-for-run. Going to merge and hope it's OK.