Contribute to Multiple Named Matrices/Vectors via Tagging

[friedmud]

Description of the enhancement or error report

A new capability needs to be added to MOOSE to allow for the creation/filling of multiple Matrices/Vectors. This will be accomplished by declaring new matrices/vectors in a System using "tags"... and then objects that contribute to matrices/vectors (Kernels, BCs, etc.) would then be assigned tags that they would contribute to.

We are going to create BundleVector and BundleMatrix that manage multiple vectors and matrices. The BundleVector and BundleMatrix should work as if they were regular vector and regular matrix, except that they accept tags. Each tag corresponds to a matrix/vector, and when we do operations on the BundleVector/matrix, all the matrices/vectors with which the passed in tags or internal tags are associated should accept the operations.

Rationale for the enhancement or information for reproducing the error

Many use-cases exist for a capability that would allow MOOSE to create multiple residual/jacobians and have objects selectively contribute to them. A few examples are generalized eigenvalue solvers, explicit implementations that would like to use lumped mass matrices, producing a mass matrix for inspection and better "save in" capability (more efficient).

Identified impact

The impact will be significant. Most of the work will be done in Assembly with some modifications to SystemBase, etc.

[friedmud]

@fdkong and myself did a bunch of design work this morning to figure out what the tagging should do. The above picture are the 4 use-cases that we identified... and we think we have a design that will satisfy them. Here it is:

1. There is a 1 to 1 relationship between a tag and a vector/matrix. This is critical to make things easy.
   1. Vector tags and Matrix tags are separate (they always have been)
   2. Tags are added independently of adding vectors/matrices
   3. Vectors/matrices are then associated with tags through a separate associateVectorToTag() and associateMatirxToTag() calls.
   4. Nonlinear problems will have three vector tags by default: residual, time and nontime
   5. residual is the final combined residual. The residual vector that comes in from the PETSc callback will be associated with that tag at the top of computeResidual()
   6. time is the residual for TimeKernel derived Kernels
   7. nontime is the residual for all other Kernels
   8. time and nontime are only to be used before the callback to TimeIntegrator::postResidual() in NonlinearSystem::computeResidualInternal(). After that - all objects should contribute to residual directly.
   9. The job of TimeIntegrator::postResidual() is to combine the time and nontime vectors into the residual vector
   10. Kernels default to vector_tags of either time or nontime depending on what they are
   11. Kernels default to matrix_tags of system which will be the tag for the main matrix in the nonlinear system.
   12. All other "Kernels" default to the same things as Kernels (InterfaceKernels, DGKernels, etc.)
   13. IntegratedBCs/NeumannBCs default to the same things as Kernels
   14. NodalBCs/DirichletBCs default to vector_tags of residual and matrix_tags of system
   15. All objects will have input_parameters of vector_tags and matrix_tags to allow the specification of where stuff goes from the input file
   16. All objects will also have extra_vector_tags and extra_matrix_tags so as not to override the defaults (this is what the new save_in will be).
   17. computeResidual() will take a std::set of TagIDs... that will restrict what objects are run during that residual computation.
   18. The "normal" compute residual operation will pass in the residual from the callback and the residual, time and nontime tags.

[YaqiWang]

Thanks for sharing! I have a few questions:

Item 6, you did not make kernels default to time+residual or nontime+residual because of reasons for efficiency?

Item 11, not like save_in, which requires an auxiliary variable, tagging needs a solution vector. How the user create a solution vector and visualize it?

Item 4, you implemented the block Jacobian, can the tag be used to create part of the solution vector or matrix?

[friedmud]

Item 6, you did not make kernels default to time+residual or nontime+residual because of reasons for efficiency?

In 6 Kernel contributions are separate and then combined in 6v by the TimeIntegrator... it has to be this way because of the special ways they get combined by higher-order time integrators. Look at postResidual() here for instance: https://github.com/idaholab/moose/blob/devel/framework/src/timeintegrators/AStableDirk4.C#L131

Item 11, not like save_in, which requires an auxiliary variable, tagging needs a solution vector. How the user create a solution vector and visualize it?

For 11 the extra vectors/matrices will get created automatically if you add extra tags. If you want those to come out as auxiliary variables (so you can look at them) we'll have a separate mechanism for doing that (I haven't completely decided on that just yet - we can make it more manual or more automatic depending).

Item 4, you implemented the block Jacobian, can the tag be used to create part of the solution vector or matrix?

I don't know for sure yet. Let's revisit this once we get this initial thing in.

[friedmud]

More on the "part of residual or matrix": You will definitely be able to pass in a subset of tags to computeResidual() and only get the objects with those tags contributions.

[friedmud]

More on save_in: realize that part of the reason we're going this direction is so that we can make save_in scalable and efficient. It's currently TERRIBLY inefficient... so even contributing these to separate residual vectors and then copying them to auxvariables (if you want to see them) will be WAY more efficient than what we're currently doing.

Not only that - but many uses of save_in are NOT for output/visualization. Those won't need to create auxiliary vectors at all now. They will be able to just couple directly to these new residual vectors (we'll have a new mechanism for doing exactly that). ALSO: it will allow us to save residual contributions exactly as they are computed for non-linear-lagrange shape functions.

[YaqiWang]

Make sense. So are you saying we can have something like addCoupledTaggedVariable(variable_list, tag_name) and later couple them with our Coupleable interface? That sounds good to me. Also be warned that Rattlesnake currently uses save-in to avoid redoing complicated calculations in kernels within postprocessors. For example (grad v1, D grad u), instead of doing it all in postprocessors, what we are doing is to have a kernel (grad test, D grad u) to save in the residual and then do a simple inner product of the save-in and v1 with nodal postprocessor. When save-in is removed, you will have to have a plan on using tagging to do this.

[friedmud]

coupledTaggedResidual(var, tag)? Something like that. Who knows...