Implement shm_open() and shm_unlink()

[linkmauve]

Those two functions are defined in POSIX.1-2001, and are used to get a shared memory file descriptor, usually mapped to a file in /dev/shm, and in emscripten’s case would map well to memfs.

[nyuichi]

:+1:

[stale[bot]]

This issue has been automatically marked as stale because there has been no activity in the past 2 years. It will be closed automatically if no further activity occurs in the next 7 days. Feel free to re-open at any time if this issue is still relevant.

[vadimkantorov]

@sbc100 Could you please reopen this issue? Or is shm_open/shm_unlink now supported on emscripten? (could be supported if /dev/shm is checked and is made available via MEMFS (probably easiest) or via a native impl of MEMFS)

[sbc100]

While we could emulate these system calls, its probably best to recommend that folks compile their project assuming that these don't exist, since we don't actually have any memory mapping capabilities in emscripten. Just like mmap, for example, its better to recommend folks simply use malloc.

[vadimkantorov]

@sbc100 At the end of the day I am looking to solve two problems:

• how to get a fd for some blob of bytes embedded in the binary which I execute (e.g. to access some ZIP archive bytes appended to the binary)
• how to get a fd for some byte buffer in memory (e.g. after on-the-fly decompressing a ZIP embedded into the binary into a buffer we need to return a fd to it)

[sbc100]

I did notice in #18062 that emscripten does support fmemopen.. but that returns a FILE* and no a fd (and fileno() doesn't work on that file handle). Would that be enough for your use case?

For embedding files as bytes in your wasm binary we have --embed-file we does exactly that. Is that useful to you? The embedded file are then available in the emscripten VFS.

[sbc100]

You can also use the JS FS_createDataFile API and pass a heap subarray along with canOwn=true. See https://github.com/emscripten-core/emscripten/blob/b3c25673f53974ff1f30df43701dc63af7cbbdc4/src/library.js#L2843-L2844 for an example of how to do that.

[vadimkantorov]

Would that be enough for your use case?

Only some of them.

Essentially I am trying implement a (read-only) analogue of memFS / virtual files - but in a portable way, so that I can hopefully use the same C code (so hopefully without relying on JS filesystem infra for compiling to emscripten, and with regular desktop musl libc, and with cosmo libc. So one way is to have memfd_create/shm_open emulation shim in emscripten (asserting that /dev/shm is available), another way is maybe overriding open/seek/read to redirect to fopen/fseek/fread working with fmemopen-produced FILE* (which should be retrieved from some array by fd of special range - e.g. fds greater than 1_000_000_000 could mean these in-memory files)

So I maybe wonder if it's in general a viable direction to move the memfs / virtual fs code to C? Then it could be used/tested even in general C programs (e.g. linking this memfs to a musl-built busybox can be useful in practice). Cosmo libc supports something like this in their fork of musl libc - known as zipos - allowing transparent read-only access to contents of zip-file appended to the binary. This could also allow building a somewhat decoupled support of data packages which can then be consumed outside of JS context and even in general C-based apps.

[sbc100]

If you are looking for VFS solution in C code then you should probably look at WASMFS if you haven't already. Its the new emscripten VFS layer that is designed to replace the JS one eventually, and its written in C/C++.

[vadimkantorov]

Ah, very interesting, I'll take a look at it. I'll also keep you posted on my proof-of-concepts of non-emscripten-coupled read-only/overlayfs-like/transparent zip decompression virtual FS :) For this limited usecase it can be useful to have this written in a portable way (or at least maybe having portable helpers for implementing this), embedding some files into the binary, and allowing portable data packages (maybe zips) to be overlayed. Maybe even existing formats like iso/zip or initramfs/squashfs could be reused for this. Then we can use existing tools for working with these formats.