search

remotestorage / remotestorage.js

⬡ JavaScript client library for integrating remoteStorage in apps
https://remotestoragejs.readthedocs.io
MIT License
2.31k stars 141 forks source link

Schema versioning and migrations? #858

Open lewisl9029 opened 9 years ago

lewisl9029 commented 9 years ago

I'm trying to figure out how to handle data schema versioning and migrations with remotestorage.

Is this something that remotestorage has implemented at the moment?

If so, where can I find some documentation or examples on how to create/perform migrations?

If not, is this somewhere on the roadmap? Or is it something developers are expected to implement on top of remotestorage on their own?

Thanks!

raucao commented 9 years ago

As of yet, this is an unsolved problem. The idea for a generic migration library/tool came up before, of course. I think ideally it should be an add-on for remoteStorage.js, which app developers can use within their apps somehow.

Afaik nobody currently plans to work on this, but a few people would likely be more than happy to assist someone who would start throwing around ideas and/or working on it.

lewisl9029 commented 9 years ago

Thanks for the follow up.

The main concern I have about implementing this on my own is how best to handle synchronization between clients of conflicting versions. I think it might be best to simply disable synchronization when versions mismatch, but I'm unsure how to best implement this behavior on top of the library (maybe a separate module to store just version information and sync'ing and checking that before any other module?).

I'd love to hear from people with more working experience with remoteStorage or other data synchronization platforms on how to approach versioning and migrations. I feel like this is a fairly crucial piece of functionality to get right before launching anything into production.

michielbdejong commented 9 years ago

This question is about module development, right? OK, I came up with a system:

So say you have the pictures module and want to start allowing pictures that appear in two albums. The approach for this is to allow JSON-LD documents to appear in albums, and they contain a reference to the actual photo. This would indeed break clients that use the current module, where the assumption is safe that any file that is in an album, is an image file.

I would suggest the following procedure:

In general: we can standardize that "highest data module version used" indicator now already, without knowning when and why we might at some point make breaking changes to any of the modules. We can now already make each module do a check for e.g. /.highestMajorVersion, and if it's anything higher than 1, display a warning to the user saying "This app may not display all your data correctly"

raucao commented 9 years ago

I think the idea of keeping a meta data file for this makes sense. And why not have one with a more general filename with all the information in it from the start?

Ragnis commented 9 years ago

Wouldn't it be cleaner to use a versioned JSON-LD ID instead of keeping the version info in a separate file?

On Tue, Apr 14, 2015 at 5:10 PM, Sebastian Kippe notifications@github.com wrote:

I think the idea of keeping a meta data file for this makes sense. And why not have one with a more general filename with all the information in it from the start?

— Reply to this email directly or view it on GitHub https://github.com/remotestorage/remotestorage.js/issues/858#issuecomment-92866700 .

raucao commented 9 years ago

Yes, but that only helps you with JSON objects and doesn't solve the problem of changing directory structures for example.

Edit: Just to be clear, I still think that's a very good idea for JSON objects/schemas and other files don't need to be versioned anyway. Right?

raucao commented 9 years ago

Turns out we (at @5apps) need to migrate some data in commercial RS-enabled apps of one of our clients soon, so we need to solve this on the way. Timeframe is about 1-2 months from now.