Support memory-mapped files

[dead-claudia]

Is your feature request related to a problem? Please describe. I'd like to be able to read and modify large file-based data sets without either 1. paginating them into memory explicitly (mildly inefficient and eats RAM up rapidly) or 2. doing frequent random file access (extremely inefficient for magnetic hard drives).

Describe the solution you'd like Integration of mmap/munmap/msync for Linux, and equivalent APIs in Windows in a new fs API. This API would return a memory map handle with a few pieces of functionality:

• A Buffer whose underlying array buffer encapsulates a pointer over the memory-mapped region.
• A method to synchronize the memory with the underlying file (via msync on *nix).
• A method to release the reference to the file (via munmap on *nix), detaching the buffer's underlying array buffer. This would also be done implicitly when the handle itself gets collected, to mitigate leaks.

I'm omitting support for Linux's mremap as there's no direct equivalent in Windows (and the closest equivalent it has is in a part of it that's filled with various caveats).

Describe alternatives you've considered See the "without either..." part of the problem description.

[Trott]

@nodejs/fs

[dead-claudia]

Of course, there is a risk of SIGBUS-based crashes, but that can be avoided through careful management of the file system. (This would necessarily have to be warned about in the docs.)

[martinheidegger]

Prior work in userland.

• mmap.js
• mmap-mio
• mmap
• mmap-utils

[bnoordhuis]

As the author of the original mmap module: commit bnoordhuis/node-mmap@0f81e51c48 (10 years ago!) mentions why mmap is a terrible idea in an async runtime - it performs blocking I/O.

1. paginating them into memory explicitly (mildly inefficient and eats RAM up rapidly) or 2. doing frequent random file access (extremely inefficient for magnetic hard drives).

That's no different with memory-mapped files. They're not some magic bullet.

[martinheidegger]

Convince me otherwise. :-)

I came to this topic by looking at n2 and faiss which both use mmap to scale for data that should be in RAM but cant because of it's size.

[github-actions[bot]]

There has been no activity on this feature request for 5 months and it is unlikely to be implemented. It will be closed 6 months after the last non-automated comment.

For more information on how the project manages feature requests, please consult the feature request management document.

[dead-claudia]

bump to not be stale

[bnoordhuis]

I'll go ahead and close this to avoid further content-free bumps. No maintainers spoke out in favor and I personally don't see the need for this to live in core, rather the opposite.