Could we use MPI Cartesian Communicators?

xcompact3d / x3d2

https://xcompact3d.github.io/x3d2

BSD 3-Clause "New" or "Revised" License

3 stars 4 forks source link

Could we use MPI Cartesian Communicators? #90

Open pbartholomew08 opened 4 months ago

pbartholomew08 commented 4 months ago

tdsops_t contains information about where in the parallel decomposition it is. This is very similar to the MPI_CartComm and it may be beneficial to make use of this.

semi-h commented 4 months ago

We may want to use nccl in CUDA backend and rccl for AMD and I guess they don't have a CartComm like support, I cound't find anything online. Also, with the iterative solver I think we'll be able to run cases that doesn't really a good fit for CartComm. For example a backward facing step, where there is no mesh and no ranks in that gap part. A custom communicator is still an option here and I think at least nccl supports it, but not sure if its any better than current approach.

Nanoseb commented 2 months ago

side benefit(/drawback) of #113, cartesian comm from 2decomp will be used for the OMP backend when using FFTs