search

ncruces / go-sqlite3

Go bindings to SQLite using wazero
https://pkg.go.dev/github.com/ncruces/go-sqlite3
MIT License
499 stars 16 forks source link

Implement shared memory WAL. #71

Closed ncruces closed 6 months ago

ncruces commented 7 months ago

This intends to implement shared memory WAL for both 64-bit Linux and macOS (amd64/arm64).

This depends on https://github.com/tetratelabs/wazero/pull/2157 (or some potentially better solution) to ensure Wasm memory is mmaped.

Current solution already passes mptest but still needs stability, portability and documentation work.

ncruces commented 7 months ago

Portability is done: only Linux and macOS (on amd64 and arm64) are supported.

illumos (and other 64-bit Linux archs) could potentially be supported, but they'd be hard to test; BSDs don't work due to locking; Windows is out due to memory remapping; 32-bit doesn't have enough address space.

ncruces commented 7 months ago

Mostly done. Still missing read-only mode, https://github.com/tetratelabs/wazero/pull/2157 improvements, and documentation.

ncruces commented 7 months ago

Will need an improved design over https://github.com/tetratelabs/wazero/pull/2157.

ncruces commented 7 months ago

This is pretty much ready for testing, @infogulch.

This should also make these bindings fully compatible with Litestream, @Koeng101. Suggestions welcome for a Litestream test setup.

It also opens the way for SQLite memory mapped IO, and WAL-Mode Blocking Locks.

Koeng101 commented 7 months ago

I'll test out later! Very exciting!

ncruces commented 6 months ago

Windows may be possible after all. For reference: https://devblogs.microsoft.com/oldnewthing/20240201-00/?p=109346

Shared memory primitives should work fine on any Unix OS, as implemented. In practice, they're useless without OFD locks, so are unavailable on BSD. illumos could work though, but is disabled for being impossible to test.

ncruces commented 6 months ago

https://github.com/tetratelabs/wazero/pull/2177 was merged, and will be in the next release.

ncruces commented 6 months ago

It also opens the way for SQLite memory mapped IO, and WAL-Mode Blocking Locks.

I briefly looked into memory mapped IO, and I won't be pursuing it at this time. It has numerous caveats, especially if not implemented exactly like SQLite does it (mapping the entire file, hoping the size doesn't change often). Trying to implement the API straightforwardly (map/unmap this page), leads to a bunch of syscalls, memory fragmentation, wasted space. Also not a fan of "crashes on IO errors."