jipolanco
commented
2 years ago Hi, right now I can't look at this in detail, but the issue seems to come from a broadcasting operation involving arrays with different memory layouts (as implied by the different Permutation{...} and NoPermutation in the error message). Such permutations are implicitly applied when performing transforms using PencilFFTs.jl (but they can be explicitly disabled when creating a plan).
My guess is that you're broadcasting a Fourier-transformed 3D PencilArray with a regular 3D Array. In that case, the issue is related to #42, see https://github.com/jipolanco/PencilArrays.jl/issues/42#issuecomment-1029766858 for details. Unfortunately this case is currently not well-supported in PencilArrays.
To fix this, the broadcasting implementation should be generalised to be able to combine arrays with different memory layouts. Currently, broadcasting assumes that all arrays have the same layout, and thus broadcasting is performed contiguously in memory. A possible fix would allow a slower path in which not all arrays are iterated contiguously (but this would require scalar indexing, not ideal on GPUs).
In any case, I don't think this is an issue with constraints on the number of processes or grid points (there are no such constraints that I'm aware of, other than obvious ones such as grid points >= ranks). I would guess that, in the cases where your code doesn't fail, it's just a coincidence (e.g. the local number of grid points is the same along all directions), but I would still expect the results to be wrong, since the memory layouts are different.
tomchor
I've been trying to run this code, which uses
PencilArraysin the background to do the decomposition.Depending on the value I put for the grid sizes
Nx,Ny,Nzand the number of ranks in each directionRx,Ry,RzI get an error. For example I noticed that whenNx*Rx == Ny*Ry == Nz*Rzthe code runs successfully. But if that condition isn't satisfied (for example if I setNx=Ny=Nz=8,Rx=Rz=1andRy=2) I get an error like this:... the error continues.
So, my question is: is that a basic constraint when setting up the pencils?
A broader point is that I wasn't able to find anything in the docs stating constraints when setting up the pencils. Is that information there? If not, maybe it's worth including a small section discussing it?
Thanks!