File locking

[sunfishcode]

Sqlite and the Emscripten cache file are two cases where file locking has come up in real-world use cases.

The POSIX way to do file locking, with fcntl and F_SETLK/F_GETLK-style locks has the unfortunate property of being per-process, which is unfortunate both for being contrary to what applications actually want, and because we'd prefer to avoid exposing details like which host processes things are running in to wasm programs. (POSIX is considering fixing this, but until they do and systems update to the new mechanism, we need to work with what current systems provide.)

Here are some notes on a WASI file locking API might want to consider:

• The API should support byte-range locks, but permit implementations to lock more than requested, potentially up to the whole file. For example, FreeBSD has fcntl locks with the unfortunate POSIX behavior, and flock with the desirable per-file-description behavior, however flock only supports locking whole files at a time.
• Discretionary file locking only is probably enough for now. Mandatory locking isn't reliably available in many host environments.
• A lock can be shared (multiple locks can be held at once) or exclusive (only one lock).
• Only one lock can be held on a given byte range of a given file description at a time.
• Locks can be upgraded from shared to exclusive.
• For now, acquiring a lock blocks until the lock is available.
• It may be useful to permit implementations to report that locking is unsupported on the host filesystem. Some old NFS implementations, and possibly other networked filesystems, don't support locking.

[Mohsen7s]

Any update on this? Is it going to be implemented ?

[sunfishcode]

The next step here is for someone to propose a specific feature set, so that we can evaluate whether it's sufficient for meaningful use cases, and whether it's implementable on popular platforms.

As an aside, looking at the FreeBSD flock man page, it's not clear that it really is per-file-descriptor. It says it's per-file, which suggests that it's process-wide like fcntl locking is, which would similarly introduce a dependency on the host process boundary.

[sunfishcode]

FreeBSD's flock does appear to be per-file-description, rather than per-process.

I'm going to propose an API which is essentially flock, with shared (read-only) and exclusive (read-write) locking. It applies only to whole files, and not byte ranges. It is advisory, meaning there's no guarantee other processes aren't accessing the file. We can implement it on Windows with LockFileEx. It's on Linux, macOS, and most Unix variants as flock.

[andrewrk]

Thanks @sunfishcode - looking forward to this.

https://github.com/ziglang/zig/issues/13589

https://github.com/ziglang/zig/blob/2823fcabd1974550b889a56e8ada0eb52f3d2080/src/Cache.zig#L859-L886

Please share with me an early draft if you can and I will test-pilot the API.

[bjorn3]

This is implemented by https://github.com/WebAssembly/wasi-filesystem/pull/69, right?

[andrewrk]

Thanks for pointing that out - I hadn't noticed.

[sunfishcode]

69 is now merged, and on track to be in preview2. This doesn't yet cover byte ranges or mandatory locking, but those can be considered separately.

[cberner]

Was this released as part of preview2? I'm not able to find it in the wasi crate

[sunfishcode]

Unfortunately, file locking was removed around the time of preview2's release in order to keep the scope down, as there were a lot of other things to do to get preview2 released. Now that preview2 is released and the release cadence is established, we could re-introduce it. So I'll re-open this issue to track that.