[IDAG] Communication & Receive Arbitration

[fknorr]

This is the second PR in the Instruction Graph series, and the first one to touch the execution side of the new runtime. It adds the necessary infrastructure to perform in-place sends and receives from (host) buffers and to match pilots to their corresponding payloads. See #249 for context.

This does not really interfere with #249 nor is it based on that PR, the only overlap being pilot.h.

communicator

class communicator is an abstraction of MPI_Comm that provides all operations to transmit and receive pilots and perform asynchronous sends / receives on strided buffer-allocation data with known message ids. It does not spawn its own thread but relies on polling from the executor thread (which currently is the main thread in tests).

Edit: clang-tidy gives me a heap of MPI check warnings about re-using requests and memory leaks, but I'm convinced all of these are false positives (see the second github-actions review below).

receive_arbiter

Instead of linearizing and packing transfer data into frames, the IDAG runtime transfers metadata ahead-of-time through pilot_messages and performs in-place MPI_Isend/Irecv operations (via the communicator abstraction). Since receive_instructions and pilots, and MPI_Recv'd data can appear in any order, a state machine must be maintained for every transfer at execution time to drive every receive operation to completion.

I call this method receive arbitration. It requires some added complexity to deal with the fact that incoming send-boxes do not need to match received regions in the IDAG exactly. In this regard, it is similar to the legacy buffer_transfer_manager. Like communicators, the receive arbiter is also polled from the executor thread and does not add its own concurrency.

The receive-arbiter tests mock the communicator to generate all possible sequences of events observable at execution time for each test case, to account for the non-deterministic behavior in the cluster.

async_event

Type-erased events are needed for many different situations in IDAG execution. Since this PR is the first to require them, it adds async_event as a generic construct.

[github-actions[bot]]

Check-perf-impact results: (19e9c0e63e2eff8d602cc7a81b622c19)

:question: No new benchmark data submitted. :question:
Please re-run the microbenchmarks and include the results if your commit could potentially affect performance.

[coveralls]

Pull Request Test Coverage Report for Build 8340717687

Details

• 299 of 301 (99.34%) changed or added relevant lines in 6 files are covered.
• 1 unchanged line in 1 file lost coverage.
• Overall coverage increased (+0.09%) to 94.724%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/mpi_communicator.cc	138	139	99.28%
src/receive_arbiter.cc	129	130	99.23%
<!--	Total:	299	301	99.34%	-->

Files with Coverage Reduction	New Missed Lines	%
include/grid.h	1	96.81%
<!--	Total:	1		-->

Totals
Change from base Build 8329571092:	0.09%
Covered Lines:	6872
Relevant Lines:	7071

---

💛 - Coveralls

[github-actions[bot]]

Check-perf-impact results: (7b849de16ff11660b98988ab0b032db7)

:question: No new benchmark data submitted. :question:
Please re-run the microbenchmarks and include the results if your commit could potentially affect performance.

[fknorr]

CI is unreliable at the moment (either a Github or slurmactiond issue or both). Everything passes locally for me and formatting is also correct, so I'm doing a blind merge here.