Discussion: Local Forage, moving beyond LS

[Ramblurr]

Hey @fsmanuel

We've been using this addon in production for about a year now and we're hitting the limits of LocalStorage. Some of our users are hitting the 5MB size limit, and nearly all of them are affected by the slow performance of LS when there are many little reads and writes.

Our deployment target are devices as a hybrid app, and we have to support a range of devices that have varying abysmal support for WebSQL and IndexedDB. So I've been keen on migrating to an abstraction library such as Mozilla's LocalForage. https://mozilla.github.io/localForage/

LocalForage is an abstraction layer on top of LS, WebSQL, IndexedDB, and even Cordova SQLite. It provides a sync and async getItem/setItem API.

There's an existing ember-localforage-adapter project, however its implementation is rudimentary and not particularly performant. see: https://github.com/genkgo/ember-localforage-adapter/issues/43

I would really like to migrate ember-local-storage to use the localforage API to get the benefit of increased storage options. The question is how to implement it, should it go directly into the project or become a fork?

Do you have any thoughts on this?

Edit: Hit save before I was finished.

[fsmanuel]

Hey @Ramblurr

Great to hear that the addon is useful. Your inquiry is 100% valid and I always wondered when the first complains about the storage limitations pop up. I haven't had a look into localForage yet but I'll try to look at the API this week.

I think we can make it a feature to use localForage as the backend abstraction.

Do you mind to set up a kind of roadmap so we can discuss the details of such an implementation?

[fsmanuel]

@Ramblurr do you need it for the adapter, the storageFor or both?

[fsmanuel]

@Ramblurr and one more question. Are you on the current release with your production app?

[fsmanuel]

@Ramblurr I run through the code and marked all the places where we should add changes to make it localForage compatible. It doesn't seem to be too much work. I created a branch feature/local-forage-support you can open PR's against it if you have time to resolve some of the TODOs.

Here is a kind of roadmap:

• [ ] Add an initializer (blueprint) to generate the localForage config
• [ ] Add a config option to activate localForage backends (add bower or npm build, app.import(), etc)
  • [ ] Add sessionStorage driver
• [ ] Resolve all TODO: localForage see #155 and changes for more details
• [ ] Update README
  • [ ] How to run on localForage
  • [ ] How to add more drivers, etc

[Ramblurr]

@fsmanuel No, I'm on the 2nd latest version. I'll be upgrading as part of working on localforage however.

Thanks for the roadmap! I'll send some PRs your way.

[Ramblurr]

Just saw your first question: We're only using the Ember Data adapter, and that's all we'd need it for.

Jumping into this this week. Looking at your TODOs it seems you plan on replacing the original functionality with LF, and then having the original / default local storage behavior routed through LF. Is that correct?

[fsmanuel]

@Ramblurr I used the dot notation for interacting with localStorage. The first step would be to convert the code to the localStorage API (using setItem and getItem, etc.) If it is done we can test it (the tests should run without any problem I guess) and then implement specific LF functionality/compatibility.

If you like we can skype or talk in the slack channel.

[Ramblurr]

@fsmanuel Just wanted to let you know that we've decided to stop using ember-l-s and indeed any key/value store altogether in our app.

With the amount of data we're dealing with, and in particular the way we need to query the data on certain indices, it just doesn't make sense to continue to use a K/V based store.

Ember-local-storage served us great for a long time, but we're moving to an Indexeddb based solution for now. Thanks for all the great help!

[fsmanuel]

@Ramblurr Sorry for the late reply. Thanks for all the feedback and help! Much obliged.

[timbaas]

I've also took a look at making this possible, mainly because iOS can clear the localStorage if a device is full (this could be because [] is used instead of getItem and setItem: https://forum.ionicframework.com/t/ios-localstorage-persistence/20004).

Another problem I came across is the "storage" event, which isn't triggered/supported by localForage.

[fsmanuel]

@timbaas The plane to build on localForage is still current. Right now I have no free cycles for it. I guess it will be better in 1-2 weeks. If you have time to help you can take over #155

[akshaisarma]

@fsmanuel Any updates on this feature? I would love to have this!

Edit: I found https://github.com/genkgo/ember-localforage-adapter and it seems to work fine. I'll test it more

[fsmanuel]

@akshaisarma yeah ember-localforage-adapter is a good choice but the performance isn't that great if you have a lot of records. I'm still very interested in bringing localForage into ember-local-storage and will work on it in February.

[akshaisarma]

@fsmanuel I didn't realize that ember-localforage-adapter rewrote the whole storage for each CRUD operation. That does not seem to be very sustainable for medium-large data stores. But I really need a larger solution than what local storage provides. I'll be waiting eagerly for ember-local-storage to support this.

I am new to Ember and don't have many free cycles at the moment to learn what adding localforage means to this addon (I only use the adapter) but if I can help in anyway, let me know and I can try and give it a shot.

[akshaisarma]

@fsmanuel I was wondering if you got a chance to look at this? I may have some free time later this month and can try and give it a shot if you'd like.

[Mogball]

@fsmanuel I am looking into localforage support. Thing is, localforage has an asynchronous API which would require restructuring some code.

locaforage has a driver for localStorage (and one for sessionStorage that requires an additional package, localforage-sessionStorage) so it'd be possible to just go with that and make everything use the async API.

Any thoughts?