MueLu: possibly use AMG V-cycle as coarse solver in region MG problems

mayrmt commented 5 years ago

Enhancement

@trilinos/muelu

Enable the region MG algorithm to use another (unstructured) AMG hierarchy with just one sweep through the V-cycle instead of applying a direct solver on the coarsest region MG level.

A particular use case is the following: region MG is applied to partially structured meshes. Regional coarsening has reduced the coarse size of each region to a single (or a few) "element", but there are many of these very small coarse regions now. From a global perspective, this looks like an unstructured mesh. Hence, AMG is the perfect candidate to further coarsen such a problem.

Definition of Done

[x] Add command line arguments to en-/disable this feature
[x] Compute nullspace of composite coarse level operator
[x] Pass coordinates along to enable rebalancing
[x] Setup the additional AMG hierarchy, starting from the composite coarse level operator
[x] On the region MG coarse level: replace direct solver by one sweep through an AMG V-cycle
[x] Add tests

Possible Solution

The region MG hierarchies store should be able to provide regional nullspaces on the coarse level. They can be combined to form a composite coarse level nullspace to be used in the setup of the additional AMG hierarchy.

Interested Parties

@lucbv @rstumin @pohm01

mayrmt commented 5 years ago

@lucbv @rstumin I have some preliminary results for using AMG as a coarse solver. In essence, if the regionMG hierarchy has still large coarse level operator (e.g. as it would be the case in a true HHG scenario), then replacing the direct solver by an AMG hierarchy reduces setup time. Numbers of iterations change, but solver time is constant.

Here are some numbers: 9 regions (1 per proc), nx = ny = 1000x1000.

regionMG coarse size = 254016 rows
- 2 levels region MG + direct solver: setup = 16 s, solve = 40.8 s
- 2 levels regionMG + 3 levels AMG: 1.18 s, solve = 40.5
regionMG coarse size = 65025 rows
- 3 levels region MG + direct solver: setup = 2.1 s, solve = 62.6 s
- 3 levels regionMG + 2 levels AMG: 1.0 s, solve = 62.8 s

I wanted to look into rebalancing, since the AMG hierarchy is rather small compared to the regionMG hierarchy. Therefore, we need access to the coordinates of the coarse level composite operator. It's not clear to me, how I can extract them from the regionMG hierarchy's coarse level. To call level->Get(), the generating factory needs to be specified. Though, what do I have to put there?

mayrmt commented 5 years ago

@trilinos/muelu How can I tell the hierarchy via the expert xml-file to keep certain quantities around, in particular the nullspace and the coordinates vectors?

To complete this issue, I need the region MG hierarchy to keep the nullspace and coordinates vectors, such that I can extract them on the coarse level of the region hierarchy and pass them to the AMG hierarchy, that will be built on of it.

lucbv commented 5 years ago

@mayrmt I talked with Chris yesterday and he suggested that an initial hack is to set the coordinates and nullspace to be exported to file, this should guarantee that they are kept in the hierarchy. Try running: grep -rin "exportdata" packages/muelu/src/ to see how MueLu handles the export of various outputs.

tawiesn commented 5 years ago

@mayrmt @lucbv I think for the expert-style xml file you need to look for "DataToWrite" and it only supports the "Matrices" (i.e. A), prolongators and restrictors. You would need to extend it for Coordinates and Nullspace. So, you have to touch the code anyway. I think the most elegant way is to add a general class "KeepDataFactory" which accepts a variable name and the generating factory and stores the data on the same level under the same (or a similar) name with the NoFactory placeholder as generating factory. In general, though, one should think about whether you actually want to keep the data (or not just write a factory which actually uses the data?

tawiesn commented 5 years ago

@mayrmt One can also add a general factory "WriteDataToFileFactory" which accepts all kinds of data types and writes them to a file on the disk. Could be useful for postprocessing or some statistics.

mayrmt commented 5 years ago

@lucbv FYI: There has been an earlier discussion on the keep mechanism in #2268.

lucbv commented 4 years ago

@mayrmt PR #6515 adds the final piece needed for coarse AMG solve in SetupRegionHierarchy. I pretty much re-implemented the construction of the nullspace as it would be done in Galeri, this I way I do not have to fiddle with the keep mechanism for the nullspace which eventually would have to be removed later...

mayrmt commented 4 years ago

We have a capability to use AMG V-cycles as coarse solver. Hence, this issue can be closed. If we need to change existing capabilities in the future, feel free to re-open or file a new issue.

trilinos / Trilinos