Switch to blocked Thyra linear algebra types

[ibaned]

Lays the foundation for block preconditioning.

[ibaned]

• If the model evaluator returns blocked objects, there is a casting issue unless we actually use Teko
• The Discretization is right now responsible for the DOF maps, consider separating DOF maps from the mesh backends
• Consider using some or all of Panzer

[ibaned]

wiki page

[bartgol]

I want to revive this issue. In a few weeks (maybe ~2 months?) I may have some time to spend on Albany (not sure yet). I wanted to seriously think about moving toward a blocked handling of system of PDEs. As we discussed in the past, this would require a Thyra refactor. At the page that @ibaned linked, there are some tentative items I wrote a (long) while ago, on what needs to happen (imo) to move toward thyra. They are obviously up for discussion, but I wanted people to think about what they think about thyra layers, and maybe share they thoughts of what they expect from a block handling of PDE systems.

For instance: should we have a single big discretization or one discretization per PDE block? Should blocks be hierarchical? By that I mean that a block of, say, 12 PDE could have 1 block of 6 PDEs and 1 block that has 2 subblocks of 3 PDEs. Should we still assemble on a 'global' basis, or on a 'per block' basis? Or maybe we should assemble on a discretization basis? The latter seems appealing, since different discretizations could have different basis and number of nodes, so different layouts...

There are many things to think about for a block vision. However, what I wanted to focus first was the thyra layer which would replace all references to Tpetra/Epetra throughout the majority of the code; this means leaving *petra stuff only where you can't avoid it, which is anywhere you need to actually read/write the vectors/matrices (gather/scatter and write/read to/from mesh).

I know @mperego did some work in Trilinos to unform Epetra/Tpetra support in NOX and Piro (I think). In particular, there were things that were implemented for Epetra but not for Thyra builds, like sensitivities in Piro. More precisely, some of the PiroEpetra classes had things that the corresponding Piro_ classes did not have. Notice that the Piro_* classes are not only for Tpetra (I think). They are general, relying on Thyra (so one can use those for both Epetra and Tpetra).

[bartgol]

WIP PR #334 is currently addressing this.