Need a capability to restart a transient calculation from a steady-state eigenvalue calculation

YaqiWang commented 2 years ago

Reason

This feature is frequently required by neutronics calculations but we will get errors if we attempt to do so currently. In neutronics, we typically establish a steady-state solution with an eigenvalue calculation, then we use the eigenvalue solution as the initial condition for a transient calculation. The eigenvalue is typically loaded as well and used to scale the kernels corresponding to the eigen kernels in the eigenvalue calculation. The eigenvalue calculation and the later transient calculation may have multiapps with lots of subapps for multiphysics calculations. We used to use the exodus file restart and our custom binary restart, but they both have limitations especially with lots of subapps. It is highly desired we can perform the restart with the checkpoint files.

Design

I created a test in a branch of my forked repository, which I will push shortly. The test is the simplest case for pursuing further more complicated restart calculations. We also desire a verbose control parameter for some screen printouts so that we can check what quantities are actually restarted by MOOSE to double check if things work as expected. Tag @jortensi.

Design points:

[ ] Perhaps a getRestartableData API that when paired with declareRestartableData is somewhat analogous to the getMaterialProperty/declareProperty API pair where both APIs return references, just one of them is writable and the other is not. Then when the declarer does not exist in the restarted simulation, an object that gets will still trigger the load of the written restartable data.
[ ] Handle the difference in vector set between the two different simulation stages

Impact

Make the neutronics multiphysics transient simulations more user friendly.

YaqiWang commented 2 years ago

Once this basic functionality is supported, I can add more sophisticated tests with mutiapps. A less used feature is to restart an eigenvalue calculation from another eigenvalue calculation.

YaqiWang commented 2 years ago

We had a quick discussion on this today. Few things suggested by @friedmud :

let MOOSE handle the system checkpoint file. Probably we still use libMesh mesh xdr files, but for the equation system, we may do it at MOOSE level. This design thinking is aligned with the design for the system where more flexibility on controlling solution vectors is desired.
a dedicated input block for restart. Parameters like remapping variables from the variables in the file to the variables on the target system can be put underneath. A parameter controlling the verbose level of restarting is desired. Whenever a variable is restarted and users set a IC for the variable, we possibly want to issue a warning in the code to let the users be aware.
my own suggestion. When this flexible restart capability is in place, we can deprecate the restart capability from Exodus files. Exodus file does not support high order elemental variables.

If you have more comments, please feel free to add. @roystgnr could be interested in this.

roystgnr commented 2 years ago

IIRC several years ago we put basic remapping into libMesh::System I/O: variable data would be read into variables of the same name regardless of whether the variable ordering/numbering had changed, and any variables that existed in the input file but not in the new system could be ignored without throwing an error, and any variables that existed in the system but not in the file would be zero-initialized. That was all we needed for our purposes at the time (using low-number-of-species hypersonics solves as initial guesses to speed up more expensive models with more chemical species, or vice-versa to look at model error) but there might be room here to go more flexible still, if you have variable name changes or variables with non-zero "defaults" or something else more complicated to consider.

On the other hand, we'll probably need better test coverage of what's already available before we start adding more flexibility. That hypersonics code isn't public and hasn't been part of libMesh CI for a long time, and I don't recall if we have any tests of the feature elsewhere, and if we haven't been testing it for years it could easily have regressed.

lindsayad commented 2 years ago

For my own cross-referencing, we're attempting to combat the opposite problem (stored data has more vectors than the simulation into which we are trying to load) in #17445 with the corresponding issue being #17444

lindsayad commented 2 years ago

Other restart issue: #17465

YaqiWang commented 2 years ago

This is the place in our app where we add the kernels for both transient or eigenvalue through an action:

      InputParameters params = _factory.getValidParams("DiffusionFission");
      params.set<NonlinearVariableName>("variable") = scalarFluxName(g);

      if (_equation == "eigenvalue" && _problem->vectorTagExists("eigen"))
        params.set<std::vector<TagName>>("extra_vector_tags").push_back("eigen");
      else
        params.set<PostprocessorName>("inverse_scaling") = eigenPostprocessor();
      ...
      // added the kernel with this params.

so this eigenPostprocessor() should be pointing to the eigenvalue to be loaded from the checkpoint file. Currently, we made this postprocessor a Receiver and loaded the value manually. If just with MOOSE, we will possibly have something like

[Postprocessors]
  [eigenvalue]
    type = Reciever
    restart_from_file = true
  []
[]

We had two ways of restarting our application, one is a bool parameter in our action restart_transport_system, that controls if we want to load variables, eigenvalue in the exodus file in the mesh block, another is to have a separate input block using a user object to load variables/eigenvalue in a previous written custom binary file. Ideally, with is capability in place, we want something like

[TransportSystems]
  restart_file_base = steady_out_cp/LATEST
  ...
[]

to restart variables, eigenvalue added by the action.

Currently in MOOSE moose/test/tests/restart/restart_transient_from_steady/restart_from_steady.i, we have

[Problem]
  restart_file_base = steady_out_cp/LATEST
  skip_additional_restart_data = true
[]

it will be good to have something like

[Restart]
  restart_file_base = steady_out_cp/LATEST
  # load mesh, variables/restartable data/sub-applications existing on both the file and the current system
  default_restart = 'mesh equation_system restartable_data sub-application'
[]

or

[Restart]
  restart_file_base = steady_out_cp/LATEST
  default_restart = 'restartable_data'
  variables_to_restart = 'var1 var2'
  sub_applications_to_restart = 'multiapp1 multiapp2'
[]

With our application actions, we probably do not want the default equation_system restart.

We currently have this syntax

[Variables]
  [u]
    initial_from_file_var = u
  []
[]

and we can still support it even though it requires the mesh file to be a checkpoint file and it can be replaced with

[Restart]
  ...
  variables_to_restart = 'u'
[]

or we can remove the Variables/*/initial_from_file_var syntax.

friedmud commented 2 years ago

Assigning myself here - I'm going to transition the restart system over to using the MOOSE internal backup capability for the main app (like we do for sub-apps). That will give us more control about what vectors we restart and how.

YaqiWang commented 1 year ago

If the transient system has stateful material properties which should be restarted from material properties from the steady-state solution. But the material properties on steady-state are not stateful. I guess we will need to handle this. This just came across my mind yesterday while I was working on a restart fix in Griffin.

lindsayad commented 1 year ago

I think we may seeing the impacts of ^ in https://github.com/idaholab/moose/discussions/22721

GiudGiud commented 1 year ago

@friedmud do you have an update on this?

YaqiWang commented 1 year ago

@loganharbour I updated my branch to include one more test that restarts a transient from eigenvalue calculation with a subapp. I modified a little in the previous two commits (one for restarting transient from eigen another for restarting eigen from eigen), by hardcoded the eigenvalue in the input to see the null transient and immediate convergence of eigenvalue calculation. Because I hardcoded the value in the restarting eigenvalue input file, the eigenvalue of the restarted calculation is one. I am not able to restart the transient from eigen with subapp even I hardcoded the eigenvalue. I possibly missing something. The point for that test is to maintain null transient with subapp.

I made a small cleanup in a new commit (the first one), with which we can remove an existing input file in the test folder although I did not proceed to remove that file.

loganharbour commented 1 year ago

This is pending testing by the Griffin team. Stefano and Javi plan on using it shortly. We will likely need some way to get the eigenvalue behind the scenes, but that task should be simple.

GiudGiud commented 3 weeks ago

@loganharbour I think you have addressed this?

loganharbour commented 2 weeks ago

Yes, addressed in https://github.com/idaholab/moose/pull/24510 and https://github.com/idaholab/moose/pull/25415.

@YaqiWang please confirm. I believe the only remaining issues are convenience items (the ones listed in the meeting two weeks ago).

YaqiWang commented 2 weeks ago

Yes, I think this should have been closed.

GiudGiud commented 2 weeks ago

Closing as addressed in https://github.com/idaholab/moose/pull/24510 and https://github.com/idaholab/moose/pull/25415.

idaholab / moose