Migrate backend from single ardb to grid of 0-stor.

[rkhamis]

Issue migrated from [https://api.github.com/repos/zero-os/0-hub/issues/29](), opened by @zaibon

The backend architecture of the backend of the 0-hub need to be improve. We need to migrate from a single ardb instance to a multiple instance of 0-stors.

• [ ] All data flist data as well as flist itself needs to be on 0-stor
• [ ] Decent python client for creating & uploading of info & flists
• [ ] metadata in etcd of zero-stor client. So the mapping between the name of the flist and the key of the flist in 0-stor.

[rkhamis]

*commented by @despiegk remark: redundancy is in client?

lets create architecture doc*

[rkhamis]

*commented by @maxux

Summary of the changes needed

First of all, we will discard any backward compatibility with the new code. We can keep the old infrastructure up (as long as it can survive), but new flist and backend will not serve the old as-it.

New flist creation

The hub needs to be able to upload files on the new 0-stor backend, which requires a python client.

The current flist implementation only stores a list of chunks with their encryption key. The new 0-stor backend will different kind of information:

• We could re-use the current implementation and only store one chunks which could be the filekey in the backend, but this need a lookup on the etcd to get the metadata back
• We can save the metadata returned by the client and save them on the flist, so we can skip the etcd lookup which will dramatically improve performance and scalabilty.

To save the metadata needed by the 0-stor, we need to change the chunk list to map the chunk metadata structure: https://github.com/zero-os/0-stor/blob/master/client/README.md#metadata

Using flist on 0-fs

The 0-fs program should support both old and new version of the flist. The configuration url already uses protocol prefix on the URI, we can easily support new things and keep the old support. If the flist contains all the metadata, it can directly request files without asking metadata from etcd.

Roadmap overview

• [ ] 0-stor: deploy a 0-stor cluster usable by the hub
• [ ] jumpscale: adapt the flist structure and implementation to support new metadata from 0-stor
• [ ] 0-hub: change the implementation to support the new flist format and upload files to 0-stor
• [ ] 0-fs: add 0-stor support URI and client*

[rkhamis]

commented by @GlenDC In order to support (and thus implement this feature), issue https://github.com/zero-os/0-stor/issues/352 will have to be resolved. But for that some collective brainstorming and discussion will be required, as there is currently not yet a proposed solution for that issue. As long as that issue isn't resolved you won't be able to store data that is available (in read-only mode) for public.

[rkhamis]

*commented by @GlenDC

remark: redundancy is in client?

The 0-stor client provides erasure coding or replication as storage strategies to provide redundancy to data. When data gets corrupted (but is still readable) it can also be repaired back into an optimal state.*

[maxux]

Since we are using 0-db as backend, there is nothing to do regarding the hub itself. It's database side to choose what to do.

We should move to https://github.com/threefoldtech/0-stor_v2 and 0-db offload feature.