The sdk needs a simple and reliabe way to write to an existing file on disk when operating in non-browser environments like the cli and Sherlock.
Node fs.writeFile is not atomic. This exposes us to partial writes e.g. if a process exits or if there are concurrent writes to the same file, which can leave the file in a corrupted state.
We currently use a global lock to avoid concurrent writes by cooperating sdk tasks, but this is heavy and doesn't scale to larger numbers of files. It also doesn't block 3rd party writers.
As we build more sophisticated multi-file storage for v2 persistence, having a simpler atomic write capability will also allow us to explore more options e.g. for persisted secondary indexes, without resorting to heavy global locks for everything.
Proposal
extend NodeishFs with fs.rename
This would allow the sdk to write store files in 2 phases, similar to npm/write-file-atomic
write temp file
rename temp file -> store file
alternatives:
bypass lix when not in browser and only use NodeishFs for browser apps (nope)
detect corruption and add a recovery mechanism (complex)
ignore the issue and use git recovery (looks bad!)
pray for no file corruption because of partial writes 😇
Context
MESDK-126 (https://github.com/opral/inlang-message-sdk/issues/86)
The sdk needs a simple and reliabe way to write to an existing file on disk when operating in non-browser environments like the cli and Sherlock.
Node fs.writeFile is not atomic. This exposes us to partial writes e.g. if a process exits or if there are concurrent writes to the same file, which can leave the file in a corrupted state.
We currently use a global lock to avoid concurrent writes by cooperating sdk tasks, but this is heavy and doesn't scale to larger numbers of files. It also doesn't block 3rd party writers.
As we build more sophisticated multi-file storage for v2 persistence, having a simpler atomic write capability will also allow us to explore more options e.g. for persisted secondary indexes, without resorting to heavy global locks for everything.
Proposal
extend NodeishFs with fs.rename
This would allow the sdk to write store files in 2 phases, similar to npm/write-file-atomic
alternatives: