Data seem to be lost... :-P

[brolund]

Me and @codification seems to have lost data.

Thinkable reason is that storage server returned 404 for the statuses, the code then assumes that the store is empty and start writing anew, hence overwriting existing data.

[brolund]

Possible solutions for this:

• Start writing a local backup.
• Start writing data as events (key: 1,2,3,...) with latest key stored in store.
• Alert user when the first write is made just to make sure

[brolund]

I'm going for an alert for the user when writing to a seemingly empty store.

[jancborchardt]

Hm, an alert just letting users know that there is a problem is not a really good solution. Rather we should try to solve it in the backend as much as possible. When the storage returns 404, how can it start overwriting existing stuff? Or is that a problem which needs to be solved in remoteStorage.js?

Also, cc @skddc @galfert was there a known problem with 5apps remoteStorage?

[brolund]

Yup, this is a temporary fix because I wanted to do a release before a presentation I'm doing today at our office. :-)

We should start storing data at the client (I guess that is what you mean by backend ;-)) to avoid asking the user as much as possible.

I have filed a question at 5apps about this.

[raucao]

The new remoteStorage.js follows a client-first approach anyway. You only talk to localStorage and it manages remote data asynchronously.

Regarding the 404: if your code is written in a way that you store lots of data in a single key, you have a problem with syncing and using the same app on many devices anyway. In order for your app to work properly in a distributed setting, you should separate data into as many small units as possible and try not to sync whole collections at once.

That said, we don't know of any lost data, and it's actually kind of impossible with our kind of redundancy to just lose something on the server side. It's indeed possible that something would return a single 404 when there's a problem (we'll have to look into it, as it should really only happen when there's nothing stored at that key), but definitely not constantly over a longer time span.

[brolund]

Thanks @skddc!

Looking forward to the new remoteStorage! Will we be able to list keys (pref even public keys for any user)? That would solve a lot of problems and allow for pure event sourcing. :-)

Due to the problem of finding data without listing of keys (afaiko), we do have all data in a few keys atm.

It could have been a 404 or that the server responded with 200 and null/undefined, in other cases nothing should be (over)written. We do read-update-check (3 requests) when we write data to minimize the risk when using several devices, but, as stated, listing keys would get us away from that.

[jancborchardt]

As far as I know, listing keys will be possible in the next version of remoteStorage.js – @nilclass probably knows more.

[brolund]

Oh, and I changed the title since it didn't make sense. :-)

[nilclass]

Yes, you can list keys in remoteStorage 0.7. The baseClient has getListing, which most modules wrap with a more useful listing method (such as contacts.list()). For public stuff, there is the public module (part of the root module), which gives you remoteStorage.public.getPublicItems(). That actually stores a separate index under /public/publishedItems, use remoteStorage.root.addToPublicItems() to add to that.

[brolund]

Excellent! I foresee an upgrade in the near future. :-)

[brolund]

Do you have a ballpark estimate for when the 0.7 will be out in the public storages?

[nilclass]

There are a few issues for 0.7 left, it will probably be released once 5apps and owncloud are compatible. Currently I know the state of implementation of neither of them :) But the client side code is rather stable, so development can (and should) switch to 0.7 as soon as possible. I was about to write here, that to build a working app sync is not strictly required - but since you're building a social app I agree it doesn't really make sense to have localStorage only...

[raucao]

For 5apps, we'll talk it over next weekend at the Unhosted Unconference, and probably release the new storage backend one or two weeks after that. It will be a soft migration, where you can use both the old and the new API at the same time for a while.

[brolund]

I'll close this, we don't seem to have this problem anymore.