use non-pickled API (Isends/Recv in place of isend/recv), which should allow for more particles per send/receive transaction. This refactor required:
Adding two additional types to type_, which are MPI representations of type_.particle_record and of a new "turnstile event", which is used to coordinate halting. This is necessary, since the non-pickled API can only handle custom types when given MPI representations of those types.
Adding a global variable outside of object mode, called requests, which contains a list of request/buffer pairs. This is necessary because objmode frees the memory associated with all local variables, regardless of outstanding references, and asynchronous sends need their corresponding request object and message buffer to persist until they are complete. Since objmode also disallows direct global variable access, requests and buffers are passed to a full Python (non-object mode) function, which then handles storage of outstanding requests and cleanup of resolved requests.
coordinate/detect halting conditions without the use of collective operations, which should allow ranks to exchange work without requiring synchronization with all other ranks. This is achieved through the addition of two new message types (in addition to the messages that simply transfer particles - which now has tag=1).
dd_turnstile_event messages (tag=2), which communicate a busy_delta and send_delta, which respectively represent changes to the number of ranks that are occupied with work and the number of particles that have been sent but not received. This message type is only sent to rank zero, which acts as custodian to busy_total and send_total values, representing the counts affected by these deltas. Once both of these counts reach zero as part of the same transaction, rank zero notifies all other ranks that the system has reached a halting condition. Rank zero does not send this message type to itself, instead directly updating the count when necessary.
halt messages (tag=3), which communicate that a halting condition has been reached, and which is sent out by rank zero to all other ranks once the aforementioned halting condition has been reached.
use an all-purpose dd_distribute_bank function, which de-duplicates the bank-sending logic performed along each cardinal direction.
use dd_check_in and dd_check_out functions, that set up and close out the global state required to coordinate halting and clean up outstanding request objects.
store numba-mode-compatible domain-decomposition-related information in a newly created domain_decomp field of the type_.global_ type.
Revised domain decomposition functionality to:
Isends/Recvin place ofisend/recv), which should allow for more particles per send/receive transaction. This refactor required:type_, which are MPI representations oftype_.particle_recordand of a new "turnstile event", which is used to coordinate halting. This is necessary, since the non-pickled API can only handle custom types when given MPI representations of those types.requests, which contains a list of request/buffer pairs. This is necessary becauseobjmodefrees the memory associated with all local variables, regardless of outstanding references, and asynchronous sends need their corresponding request object and message buffer to persist until they are complete. Sinceobjmodealso disallows direct global variable access, requests and buffers are passed to a full Python (non-object mode) function, which then handles storage of outstanding requests and cleanup of resolved requests.tag=1).dd_turnstile_eventmessages (tag=2), which communicate abusy_deltaandsend_delta, which respectively represent changes to the number of ranks that are occupied with work and the number of particles that have been sent but not received. This message type is only sent to rank zero, which acts as custodian tobusy_totalandsend_totalvalues, representing the counts affected by these deltas. Once both of these counts reach zero as part of the same transaction, rank zero notifies all other ranks that the system has reached a halting condition. Rank zero does not send this message type to itself, instead directly updating the count when necessary.haltmessages (tag=3), which communicate that a halting condition has been reached, and which is sent out by rank zero to all other ranks once the aforementioned halting condition has been reached.dd_distribute_bankfunction, which de-duplicates the bank-sending logic performed along each cardinal direction.dd_check_inanddd_check_outfunctions, that set up and close out the global state required to coordinate halting and clean up outstanding request objects.domain_decompfield of thetype_.global_type.