Enabling ExplicitSystem in FEProblem

[snschune]

For SN-DFEM with TransportUpdateExecutioner the Jacobian is not required. Without the Jacobian, I will be able to run much larger problems than now.

FEProblem either uses a NonlinearSystem or a EigenSystem. I want to add a third option for using an ExplicitSystem that does not maintain the Jacobian. So rename use_nonlinear to system_type and then use an enum to switch between the three options.

[andrsd]

IIRC, you can easily deallocate the jacobian matrix of the TransientNonlinearImplicitSystem. Go look at PhysicsBasedPreconditioner.C in the c-tor how to do it. That might be an easier way to do what you need...

[YaqiWang]

I think that will still create the memory usage spike, which will be the limitation of large simulations and something we want to avoid.

[andrsd]

preconditioner is added before init, so libMesh will allocate something for the sparse matrix object, but it will be empty (i.e. with no entries). Maybe @roystgnr or @jwpeterson could confirm this?

[jwpeterson]

I looked at the code this morning with @snschune and from what I can tell EquationSystems::init() calls System::init() and this allocates memory for the matrix based on the DofMap. So even if you would deallocate it later, the memory usage would initially spike.

[andrsd]

Then, we are getting the same memory spike for PBP...

[snschune]

@andrsd We have observed a memory spike before, but I am not sure if it was in particular for PBP. Peak memory consumption is a limiting factor for us (in contrast to other applications?). If you think it'd be helpful I can produce the memory footprint for an application with diagonal preconditioner and PBP.

I'll let this issue sit here until @permcody and @friedmud had time to chime in.

[friedmud]

(Note: I'm going to use Moose:: for stuff that's in MOOSE even though there really is no namespace because it's tricky to disambiguate between libMesh and MOOSE for this Systems stuff)

I'm following - but it's complicated.

It looks like you want to be able to compute a residual for an ExplicitSystem. It's tricky because everything is so tied up with the NonlinearSystem in MOOSE. It would be a pretty major undertaking to allow us to swap out an ExplicitSystem for the NonlinearSystem. It's not impossible... just REALLY freaking difficult.

The problem is the templated inheritance of SystemTempl. It permanently ties Moose::NonlinearSystem to libMesh::TransientNonlinearImplicitSystem because it's specified in the templated inheritance.

Here's my best shot at how to fix this:

• Remove SystemTempl. It doesn't do much anyway. Take all of the stuff that doesn't need a particular type of libmesh::System and move it down into the base Moose::SystemBase class. Take everything else and move it up.

  The only real thing we lose is the templated sys() function to return the particular type of libmesh::System. We should replace it with a SystemBase::sys() that returns a generic libMesh::System. I don't know yet what repercussions that will have...

What this gives us is the ability to switch what type of libMesh::System we create inside Moose::NonlinearSystem. So, at run time, we could now create a libMesh::TransientExpclicitSystem to be the underlying system for Moose::NonlinearSystem.

However, this has consequences: namely, MANY things in Moose::NonlinearSystem are tied to the idea of the underlying system being a libMesh::TransientNonlinearImplicitSystem. We have two ways to deal with this:

1. Make Moose::NonlinearSystem an intermediate base class with two children: Moose::ImplicitNonlinearSystem and Moose::ExplicitNonlinearSystem. That will keep all the logic nice and separate... but might also be a drag for keeping both of them up to date. I don't know.
2. Use something like _is_implicit in Moose::NonlinearSystem to shortcircuit all methods that are doing something that requires an underlying libMesh::TransientNonlinearImplicitSystem.

I personally like (1) better... but it will definitely be a lot of work. I think that in the long run it will be less error prone though.

[friedmud]

A quick note: you can't just change out what type of system is being built in FEProblem because FEProblem depends on the interface in NonlinearSystem.

What EigenProblem does to get around this is that it inherits from NonlinearSystem. However, that won't fix your problem because that still means the underlying libMesh system is libMesh::TransientNonlinearImplicitSystem with a matrix....

[YaqiWang]

This will impact Moose::FEProblem and Moose::Executioner quite significantly. Maybe a much simpler way is to ask libMesh::TransientNonlinearImplicitSystem for more flexibility of construction and initialization?

[friedmud]

I thought about that... but it's NOT their fault. It's OUR design screwup. In libMesh, "ImplicitSystem" means there is a matrix... period. We're the ones that hardcoded that into OUR code...

[snschune]

I am all for @friedmud suggestion (1) even though I hoped for an easier fix of this problem. I'll do a memory scaling study (with angular directions & mesh refinement) to see how urgent this is: can I get above several millions of cells and >100 angular directions without using all of falcon?

[YaqiWang]

We may still want the nonlinear solver. But in libmesh, there is no NonlinearExplicitSystem. Even we limit ourselves to linear system, ExplicitSystem does not provide a solver. So we have to build the linear solver in MOOSE. But other than this, I guess over 90 percent of the existing codes can go into NonlinearSystem (now a base class). Problem is that the name is now misleading because the derived class maybe not a nonlinear system. It could be better to create a base class named with PrimalSystem and derive two classes, NonlinearSystem and LinearSystem.

Or we could create NonlinearExplicitSystem in libMesh and use it. Then the names of these three classes will be @friedmud suggested.

@snschune I am not against doing this now because this sounds a doable plan.

[YaqiWang]

While I was on a libMesh issue https://github.com/libMesh/libmesh/pull/1378, I verified that we do not have to change the system type in MOOSE. libMesh systems will allow us to construct vectors, matrices optionally. This issue can thus be closed. Tag @permcody

[permcody]

Closed based on a reasonable workaround described above.