xerial / larray

Large off-heap arrays and mmap files for Scala and Java
Apache License 2.0
400 stars 43 forks source link

Shared memory without a backing file #78

Open ctrueden opened 1 year ago

ctrueden commented 1 year ago

It is possible to create a named buffer of shared memory that is not backed by a file on the filesystem. In Python since v3.8, the multiprocessing.shared_memory module enables this in a cross-platform way via its SharedMemory constructor. You can then pass that name to other processes to create a SharedMemory there pointing to the same shared memory buffer. Here is the Python code, which uses shm_open on POSIX, or CreateFileMapping with an INVALID_HANDLE_VALUE plus non-null name string on Windows (and OpenFileMapping for already-existing named buffers).

I was hoping to use LArray to do something similar in Java, avoiding the platform-specific issues of shared memory buffer creation, but it looks like LArrayNative's mmap only supports memory mapped files? At least, this is the only place I see CreateFileMapping used, and it does not use INVALID_HANDLE_VALUE, nor populate any name string.

Is there something I'm missing? Would it make sense to add a new public static native long mmap(String name, boolean create, long size); method to LArrayNative.java that works in an analogous way to Python's SharedMemory?