Absurd-sql

[bokolob]

Hello. Don’t you have any ideas about implementing persistense on block level? Like absurd-sql does? It uses idb as a filesystem for sql.js

[jepiqueau]

@bokolob Thanks for your proposal. I looked quickly to absurd-sql, interesting but this package does not seem to be maintained (no version published since 2 years) and required a special version of sql-js. I will not go that way.

[bokolob]

I know, however the code there is really simple and it's possible to embrace it (move in you repo :). Because now there is problems with writing offline-first apps.

There is a strong community around capacitor and jeepsqlite, I think they will be able to help.

BTW, there are forks - https://github.com/kikko-land/better-absurd-sql

[jepiqueau]

@bokolob what are the problems you encounter in writing offline-first apps

[bokolob]

1) I have quite big amount of data (rows in table), possible about several Gbs. It's fast to have all of them in memory, but I'm not sure that it's ok for browser 2) Saving whole data at once into idb consumes too much time, saving after each commit is unacceptable for me, it would be better to save them partially. 3) Now I call saveToStore by timer, and some date could be lost if tab/browser is closed before save

So I'm looking for another approaches, I don't want to migrate to something like dexie.js, because it will led to have two versions of repositories (SQL and no-sql).

[jepiqueau]

@bokolob Where this data are stored somewhere in a remote server did you try getFromHTTPRequest giving the url? if the database was originally stored locally to the disk will this have a value if i try to read it from disk load it to memory so you can use it and after save it to local disk if there are some changes

[bokolob]

Data is formed by multiple ajax requests to server. Could you please explain your point deeply?

[jepiqueau]

@bokolob i am looking into the file-system-access which implement the File System Access API which seems to allow read and write access through file/directory pickers as an alternative to IndexedDB for persistent storage.

[bokolob]

Would db be written at once? I have tested absurd - it works :)

[bokolob]

And I forgot about another problem. What about different tabs access to same database? If a few of them call saveToStore there will be a race. I'm not 100% sure that absurd-sql solves it, but it needs some WebWorker, which could be a Shared one.

[jepiqueau]

@bokolob i look quickly at absurd-sql, looks interesting but it seems that it is required to work with WebWorkers which means that the sql code must be in a separate worker file. This is really specific to a Web app and not on an app running native and web. jeep-sqlite has been developed to be used by the web plugin part of the @capacitor-community/sqlite which allows developing apps with the same code for native and web. So i do not not want to make it web specific. Did you try to use absurd-sql without the use of web workers?. it is also requiring a specific @jlongster/sql.js package which is not inline with the latest sql.js package. sql.js works with the database in memory, so whatever you do to store your database to a disk using file-system-accessapi or to IndexedDB , it will required the whole transfer of the memory db buffer. which is fine for "small" size database but not large one. absurd.sql use IndexedDB as a file system which is a pretty nice idea so if my understand is correct no database buffer in memory which allows large databases.

[bokolob]

Hello. Your point about web workers isn’t clear for me. Why not to put part of jeep-sqlite into worker? It will affect only web, native doesn’t need to be changed.

I would go further and try sharedWorker ( or serviceWorker)