search

NexGenAnalytics / Trilinos

Primary repository for the Trilinos Project
https://trilinos.org/
Other
0 stars 2 forks source link

Integrate Quotient Graph into `miniEM` #6

Closed JacobDomagala closed 1 year ago

egboman commented 1 year ago

Quotient graph is already an option in MueLu, so only need to change the xml file in miniEM. Note the rebalancing method is specified two or three times (for different subproblems).

JacobDomagala commented 1 year ago

Changes needed to switch from MJ to Quotient: https://github.com/NexGenAnalytics/Trilinos/commit/e21cf850044c37bd23f9ec706cefe1529054ea54

JacobDomagala commented 1 year ago

Also we'll probably want to update miniEM to read HIP config file (this will need some changes to source code)

JacobDomagala commented 1 year ago

I've created new branch with changes needed: https://github.com/NexGenAnalytics/Trilinos/tree/zoltan2-test-quotient-with-miniem

JacobDomagala commented 1 year ago

OK, we probably will want to enable ParMetis aswell (since QuotientAlg uses ParmetisAlg). The issue is that I don't see parmetis module, hopefully we won't have to build it from source UPDATE:

  1. I was able to update miniEM to read HIP config files (changes needed are on my branch)
  2. When using default values in maxwell-large.xml the Zoltan2 algorithm is not used, I had to change the values to 80 (for each dimension), only then partitioning is actually used (and it fails, see more info below)

Command I used: srun -t 00:10:00 -A ${PROJECT_NAME} -N 16 PanzerMiniEM_BlockPrec.exe --stacked-timer --solver=MueLu-RefMaxwell --numTimeSteps=3 --linAlgebra=Tpetra --inputFile=maxwell-large.xml

Ok so the failing part. As I mentioned in the comment above, we will actually need to enable ParMetis in our build script, right now it's missing and the exception is thrown when QuotientAlg is trying to build internal ParMetis alg.

egboman commented 1 year ago

Yes, you will need ParMetis. Sorry forgot about that. I believe it's in a module on Crusher?

JacobDomagala commented 1 year ago

I don't see it (running module spider parmetis doesn't show any results). I've seen module load parmetis/4.0.3 in old build scripts, but this module is no longer present.

egboman commented 1 year ago

A couple thoughts on how to proceed: 1) I will ask olcf-help what happened to the parmetis module. I also can't find it, but I think superlu-dist needs it, so must be somewhere? 2) I'll check if I can remove the dependency on ParMetis in the Quotient algorithm. In principle, we could use Zoltan/PHG instead but this requires changing the Zoltan2 code and might not be straightforward in practice.

egboman commented 1 year ago

The OLCF folks promptly installed Parmetis for us:

module load parmetis/4.0.3

We will need to update the Trilinos build script to use $OLCF_PARMETIS_ROOT

JacobDomagala commented 1 year ago

OK, the parmetis package is there, but I think we also need metis

EDIT. While metis is not a viable module, the library is present on the machine (${OLCF_PARMETIS_ROOT}/../metis-5.1.0-ialj45dt3hh66bnl3vslxlduihz7i5dy)