adamantine-sim / adamantine

Software to simulate heat transfer for additive manufacturing
https://adamantine-sim.github.io/adamantine/
Other
37 stars 10 forks source link

Use MPI groups to parallelize data assimilation #252

Closed Rombur closed 8 months ago

Rombur commented 9 months ago

With this PR, we can finally run data assimilation in parallel. We can use up to as many processors as there are ensemble members. The way this is done is by splitting the global MPI communicator (we use MPI_COMM_WORLD) in multiple local communicators and then using these local communicators where needed. A lot of the PR is just about choosing the right communicator. The data assimilation itself is still done in serial on rank 0. This means that all the vectors need to be move to the rank 0 and out of it after they have been updated. There are two difficulties with this operation:

  1. there is no facility in deal.II to move an entire vector to a single processor and so we have to ship the data and rebuild the vector by hand
  2. the algorithm is more complicated than it should be because I wrote it to support more processors than there are ensemble members but it is not working right now. I think we should keep that extra code because it's still a step in the right direction and we can come back to the issue later.