Using MMAP to optimize I/O access

flux-framework / dyad

DYAD: DYnamic and Asynchronous Data Streamliner

GNU Lesser General Public License v3.0

7 stars 5 forks source link

Using MMAP to optimize I/O access #36

Open hariharan-devarajan opened 1 year ago

hariharan-devarajan commented 1 year ago

Here are some places where we could optimize data access using MMap.

Producer writes
Broker read_all
consumer reading.

Additionally, we can switch the consumer file to a dyad-specific cache file on SSD so that external apps do not access it by mistake. This is because the consistency of that cache file is only known by DYAD.

ilumsden commented 2 months ago

Now that we are using UCX buffers, we need to make sure of the following:

UCX-allocated buffers are page aligned
UCX-allocated buffers comply with requirements for POSIX mmap
UCX supports transferring data to/from specialized memory allocators (e.g., hipMalloc on El Cap)