It is possible to avoid packing when sending halos with unstructured grids. In order to do this, the ordering of the mesh vertices has to be such that first indices address the internal nodes, then come all the halo nodes for first neighbor, then - for second neighbor, and so on. To illustrate, if you have a local domain of 10 internal nodes and 12 halo nodes, with 4 neighbors, then the vector should be stored in memory such
where 0 means local domain interior, and 1-4 denote neighbor ids. This way when you submit the send you only have to take the pointer to the first entry required by a given neighbor and submit that to the transport backend. No packing is needed, since the sent values are already contiguous in memory. Of course, the receiver has to perform normal unpacking, as he then extracts the contiguous buffer into non-contiguous locations in the local domain part.
To achieve this, wither the user has to provide a domain description, in which the vertex IDs have already been reordered to fulfill the above, or we could perform such renumbering internally in GHEX (at users' request) and return the new vertex number permutation to the user. GHEX's init function would then behave as a 'local' (per-subdomain) node reordering routine.
It is possible to avoid packing when sending halos with unstructured grids. In order to do this, the ordering of the mesh vertices has to be such that first indices address the internal nodes, then come all the halo nodes for first neighbor, then - for second neighbor, and so on. To illustrate, if you have a local domain of 10 internal nodes and 12 halo nodes, with 4 neighbors, then the vector should be stored in memory such
where
0means local domain interior, and1-4denote neighbor ids. This way when you submit thesendyou only have to take the pointer to the first entry required by a given neighbor and submit that to the transport backend. No packing is needed, since the sent values are already contiguous in memory. Of course, the receiver has to perform normal unpacking, as he then extracts the contiguous buffer into non-contiguous locations in the local domain part.To achieve this, wither the user has to provide a domain description, in which the vertex IDs have already been reordered to fulfill the above, or we could perform such renumbering internally in GHEX (at users' request) and return the new vertex number permutation to the user. GHEX's init function would then behave as a 'local' (per-subdomain) node reordering routine.