Why mmap?

[JensRantil]

Just curious, why mmap the files? It seems like simply writing serially and reading serially also should work, no? Would also give higher control over flushing and buffering.

[mangalaman93]

A couple of reasons -

• It avoids the copy of data from page cache (disk page cache that kernel maintains) to application memory. mmap directly maps the files into application virtual memory.
• We plan to build more data structures in future, such as BigArray, and plan to reuse some of the components that we have built for bigqueue. mmap will work better in that case.

[JensRantil]

I see. Some quick thoughts:

It avoids the copy of data from page cache (disk page cache that kernel maintains) to application memory. mmap directly maps the files into application virtual memory.

Looks like you can do that with a regular file, also: https://askubuntu.com/a/159012

We plan to build more data structures in future, such as BigArray, and plan to reuse some of the components that we have built for bigqueue. mmap will work better in that case.

👍