Using matrix to store and manipulate distributed data

[zachmprince]

Reason

In the stochastic tools, we are attempting transfer solutions from many sub applications back to the main app and then perform operations across solutions. We think that the most straightforward way to do this is to transfer these solutions to a distributed dense matrix and perform linear algebra operations on it.

Design

The main design we are thinking is to build a dense matrix with a custom parallel distribution, which will either have each processor own a certain amount of columns (column-parallel) or a certain amount of rows (row-parallel). Is this possible in petsc or libmesh? If so, what kind of operations can we perform on this kind of matrix.

Impact

More straightforward data storage and manipulation in stochastic tools.

@aeslaughter @grmnptr @fdkong @roystgnr

[roystgnr]

PETSc has MATMPIDENSE. Not sure if we've ever used it before or not. I'm pretty sure it won't scale well, though. My first thought would be to build a full SparsityPattern in each sub application, add the ability to permute SparsityPattern data (or at least offset it, depending on how complicated your sub->main permutation operator is) and to take the union of multiple Sparsity patterns, so you can use the result of that to initialize a new SparseMatrix and then in the end you still have a collective sparse matrix to work with.