Layered userobjects lagged by one timestep when using ExternalProblem

[aprilnovak]

Bug Description

Layered userobjects that operate on variables created by ExternalProblem::addExternalVariables and written in ExternalProblem::syncSolutions are lagged by one time step relative to the same setup performed with FEProblemBase. This gives an incorrect rendering of that userobject (via SpatialUserObject) for the first time step of an external problem's solve.

This issue is quite similar conceptually to #17534, where a dummy AuxVariable/AuxKernel must be present in external problem input files in order to prevent postprocessors operating on externally-added/written variables from also being lagged. But for this particular case with layered userobjects, I couldn't find a similar way around this bug (such as adding a dummy userobject).

I'm not sure if this particular bug is strictly related to LayeredIntegral userobjects and its derived classes - it might be more generally applicable to all userobjects. I just didn't test that.

Steps to Reproduce

I have attached a minimum working example with a very slimmed down BasicExternalProblem that creates a CONSTANT MONOMIAL variable heat_source and writes a value of 12345.0 to it within syncSolutions. Even though a ElementExtremeValue postprocessor (as well as the Exodus output file) show that heat_source is written correctly at the end of time step 1, a LayeredIntegral evaluates to zero for time step 1. If you add some time dependence to that written variable, you can confirm that the userobject is indeed lagged (and not just not computed for the first time step).

The same input using FEProblemBase gives the expected behavior.

Impact

Prevents work from being done - the coupling of 3-D OpenMC/NekRS models in Cardinal to lower-dimensional MOOSE applications cannot be performed without correct behavior of the layered userobjects (because the coupling data on the first time step is wrong). Also prevents correct rendering of various convenient postprocessing operations, such as the OpenMC fission distribution integrated over each z-plane.

[aprilnovak]

example.zip

[GiudGiud]

where do we stand on this? Seems like we just need an additional execute_on flag maybe?

[aprilnovak]

I don't think it's an additional execute_on flag -- there is different behavior solely depending on whether an auxvariable is created by ExternalProblem::addExternalVariables vs. MOOSE's regular processes. I don't think there should be any difference in that behavior.

[GiudGiud]

@DanielYankura this is a good one to look at too. We don't fully understand what's going on and it's pretty consequential. You ll want to have logs of Debug/show_execution_order and compare to get more info

[aprilnovak]

Thanks for taking a look at this!