search

opral / lix-sdk

1 stars 0 forks source link

Discussion: do we add nodish fs support for SQLocal #25

Closed martin-lysk closed 2 weeks ago

martin-lysk commented 3 months ago

Context

Lix's uses SQLocal as adapter to Kysely. With our own fork of Sqlocal we support OPFS and Memory as storage base for the database opend by Sqlocal.

This works to open and close inlangs underlying Sql database in communication with lix but eventually Lix needs to be persisted also in envs other than OPFS.

One thing we considered: supporting the native sqlite https://www.npmjs.com/package/better-sqlite3 in case lix runs in node.

I would like to take a closer look to the use cases for such a fs based storage.

Consumer of a lix document:

Tests (read+write)

-> We want to run tests based on existing lix files (reading and writing binary array should be enought)

Paraglide (read only) - node based apps with access to nodishfs(?)

-> can work for lix in memory -> loads the whole lix project via inlangs loadProjectFromDirectory (it uses fs only on creation and via the importers)

-> Even if we load the file from fs it could load the byte array

Sherlock (read only*) - vs-code extension - with access to nodish fs when running in native client

-> need to be able to read a byte array - we don't store the lix file (we only store the exported json for now)

-> if we have multiple open instances #miniapps - we would need a different mechanism to synchronize an open lix

Parrot (read/write) - Figma plugin - No OPFS but also no nodishfs. With the clientStorage api (https://www.figma.com/plugin-docs/api/figma-clientStorage/) provided by figma we will run into storage problems quickly

"Each plugin gets a total of 1MB of storage. You can approximate the quota used by a key/value pair by adding the size of the key to the size of the value converted to JSON. The main exception is that Uint8Array values counts as the size of the Uint8Array, rather than the array converted to JSON."

This might give us some new lix requirements like store only the history chain and the head data and load the rest on demand. But nodishFs would help us here at all.

Proposal

skip nodish fs support for lix for now

@jan.johannes @samuel.stroschein

samuelstroschein commented 3 months ago

Agree. Skip node fs entirely by having

OPFS
in memory (sherlock needs to persist the blob itself)

EDIT: If sqlocal runs in memory on node