devitocodes / devito

DSL and compiler framework for automated finite-differences and stencil computation
http://www.devitoproject.org
MIT License
557 stars 226 forks source link

SparseFunctions + MPI + coordinates #928

Open FabioLuporini opened 5 years ago

FabioLuporini commented 5 years ago

Assume rank0, out of 4 MPI ranks, has 8 items

a b c d e f g h

Now assume that, upon Operator application, the sparse points get distributed via _dist_scatter to the 4 ranks and the result is

a c  --- b d g --- e f --- h

Now, upon returning from Operator application, via _dist_gather, the sparse points get stored either as

a c b d g e f h

OR

a b e h c d f g

(to be double checked)

So the layout is different than it was upon data allocation/initialization.

Instead of making sure the layout is deterministically the same, currently the coordinates are rather reshuffled the same way the sparse points are, so that the mapping position -> value is correct

This is horrible. Possible approaches to improve this:

mloubout commented 4 months ago

Isn't this fixed?

FabioLuporini commented 4 months ago

I think so, but I haven't touched that code in ages, so... 😬