Persistent storage on limited HTML5 environment

[kapilaramji]

Hi.

In reference to this: https://github.com/kripken/sql.js/wiki/Persisting-a-Modified-Database

While working in an environment (android html5) which only allows localStorage in the format of a utf-8 string... the following method of converting to an 8intArray does not work.

How do I save in such an environment?

-Kapila

[ltearno]

Once i was :

• converting to a uint array,
• compressing this with zlib compiled by emscripten,
• encoding that into a string
• storing that in the local storage.

I don't remembrer how much i saved, but it worked! Le mar. 12 janv. 2016 05:10, Kapila Ramji notifications@github.com a écrit :

Hi.

In reference to this: https://github.com/kripken/sql.js/wiki/Persisting-a-Modified-Database

While working in an environment (android html5) which only allows localStorage in the format of a utf-8 string... the following method of converting to an 8intArray does not work.

How do I save in such an environment?

-Kapila

— Reply to this email directly or view it on GitHub https://github.com/kripken/sql.js/issues/132.

[lgrignon]

Hello everyone, There is IMHO too few information about localStorage quota on the internet, and since it is the default recommended way of persisting sql.js database, I want to expose the results of my small research here.

According to those threads (thread 1, thread 2), it is not currently possible to increase localStorage quota. This therefore seems a very poor solution for a persistent database storage.

My use case was using sql.js as a TypeORM driver, which provides an option to make sql.js database persistent. This option's implementation uses, behind the scene, the sql.js recommended pattern here. Since it uses localStorage by default, I quickly fell into the quota trap.

Nevertheless, I discovered that this driver offers an interesting alternative way of persisting the sql.js database using localforage (which transparently uses indexedDB instead of localStorage). Please, see their implementation here.

Using localforage, it is pretty easy to replace the recommended pattern from sql.js to use localforage instead of localStorage

// save
await window.localforage.setItem("mydata", toBinString(db.export()));
// restore
var db = new SQL.Database(toBinArray(await window.localforage.getItem("mydata")));

I hope this will help other people :)

[kaizhu256]

localforage/indexeddb has a hard limitation in chrome as well of ~240mb per sql.js database. its probably enough for most, but don't expect to do gigabyte-scale, persistent data-science in browsers anytime soon.

[lovasoa]

It may be worth adding a small section about persistence in the README...

[lgrignon]

@kaizhu256 I can remember that I was able to store pretty huge database in IndexedDB. According to this article, the storage limitation of IndexedDB would rather be calculated based on available disk space. However, there might be a limitation on the size of a single DB record because sql.js persistence uses a unique entry (for both localStorage and localForage). I will have to test soon so I will post my result for a big database here.