search

LLNL / libROM

Model reduction library with an emphasis on large scale parallelism and linear subspace methods
https://www.librom.net
Other
198 stars 36 forks source link

Parallel snapshot generation with a custom MPI communicator #197

Open dreamer2368 opened 1 year ago

dreamer2368 commented 1 year ago

Currently BasisReader, BasisWriter, and BasisGenerator are initialized with only MPI_COMM_WORLD. This has been a limitation, particularly for generating samples in parallel. This happens when multiple samples can be run in parallel, but no communication is needed between each sample, so each sample can have a different communicator or even simply run with one process.

Did anybody come across with this situation, and how did you work around this? Or should we change the initializations so that each class can be initialized with a custom MPI Communicator?

chldkdtn commented 1 year ago

This is a good point. I think your domain decomposition ROM will be the first application where parallel sampling can get benefits. I think your suggestion makes sense, but it has to be done so that all previous usage cases do not break. @dylan-copeland , any thought?

dylan-copeland commented 1 year ago

@dreamer2368 you are right that the code assumes sampling of a variable in the global domain distributed with MPI_COMM_WORLD, and I believe you are the first person to need independent sampling and basis generation on different processes. This will require generalizations of the classes you mentioned, to allow for constructing them with different communicators. It would also be possible to have a serial version without any MPI, which assumes each variable is independent of other processes. This would be simpler to implement than allowing for different communicators, but it would not work in the most general case of variables parallelized within subdomains.