Migrate orbitdb data for current node to the new shared data layer.

[zachferland]

We should be able to implement a migration function that takes the data from our backup and writes it to our new store. We could also get the same effect a level up by just simulating users/opening stores on both our pinning node and the new one. Both are probably similar amounts of work, but the first we can get a bit better guarantees. This store is the first attempt at do this, will still need more validation and testing stories in the future before actually moving over.

[zachferland]

This will not be too difficult, we can create a module to run a process on our pinning node that iterates through all key/vals stored on our pinning server and copies them to our new store. Leveldb is use by orbit, this module will create a leveldb instance (ref same paths/config), there you can create an iterator which is bound over all keys, each key val can be iterated over and written to a redis instance we plan to use as the shard orbitdb-cache data layer.

[oed]

@zachferland Could the approach you're describing cause some data loss if a store is currently being used with the old node while the migration is taking place?

[zachferland]

sure, because its a separate process that doesnt know anything about the node, I think we will just need to plan a down time window. Its not worth the additional work to keep 100% uptime through it. Also if did happen by chance, would likely be able to sync data again in future.

[oed]

Sounds good.

[zachferland]

Moving back to backlog for next sprint, was started, but actual implementation was not started.

[zachferland]

As reference here - https://github.com/3box/orbit-db-cache-redis/issues/1, a little bit more work, or manual, as each store/db actually has its own store. So need to iterate folders, then open dbs in each and open an orbit-db-redis cache instance in each to copy over.

[zachferland]

Created the following utillity CL tool to transfer data from a local orbitdb path to a redis instance. Then the redis instance can be used with the redis orbit db cache.

https://github.com/3box/orbit-cache-to-redis

At the moment, this can only copy data from stores which are not open, otherwise those caches will have locks. We will have to consider how we want to run this when we switch over. We might want to close all stores, before running. Or run multiple times in a best effort. Also running during time of very low traffic would help.

[oed]

Let's plan some downtime to make sure it's done correctly.