support returning array of pids per element

[cwsmith]

To support some of the operations described in #108, we should provide a function that returns a CSR (on device) of pids per element.

[Angelyr]

I can take care of this.

[cwsmith]

@zhangchonglin In the array of particle ids that is returned, does the order of PIDs within each element matter? In the current version of the function, within each element, they will not match the order they are stored in the GPU particle structure.

[zhangchonglin]

The objective is to operate on particles within each mesh element. After the PID of a particle is obtained from this function, we then need to access other particle information corresponding to this PID, for example particle velocity. If the PID does not match the order they are stored in the GPU particle structure, can we still access the velocity information correctly, for example the access of particle information in XGCm: https://github.com/SCOREC/xgcm/blob/7ba88539f2cc32bb16bc2a46d3d68bfb6bbb7ac0/src/xgcm_ion_push.cpp#L45-L46. I am just using above XGCm code as an example, but the idea will be similar.

[cwsmith]

Yes, that should be fine. We can add a sanity check/test for that type of usage.

[zhangchonglin]

I misunderstood your previous comment. If it's just the order is different, then I think we should be fine. I assume in your returned PID array, if I want to know the position of a particle with index i (your index), then I can do, for example:

• pid = PID(i)
• Vector2d actual_pts = {x_c(pid, 0), x_c(pid, 1)};

The particle parallel_for is accessing a different set of pid index consistent with GPU particle structure memory layout. The mesh based operation and particle based parallel_for operation are separate and not interfering with each other.