Archives for backing up and restoring nodes

[vyzo]

We are reaching the stage where it makes sense to start backing up long running nodes. We can do this by using archives, as provisioned by the RFC4 design, which would allow datasets to be easily exported, distributed, and imported by any node in the network.

Operationally, we should start with the simple goal of having a portable backup/restore mechanism through an API.

There are two basic api endpots required:

• /export: given a query, export an archive containing all relevant statements and objects.
• /import: which takes an archive and merges its contents to the local store

The API is sufficient to provide the mechanism to export and import datasets and solve the immediate issue of node backup and restore.

The obvious next step is to distribute through IPFS, and publish archive statements that link to the IPFS objects.

[parkan]

how are you thinking of serializing these?

[vyzo]

Not sure yet. The most portable representation is a tarball, with each statement and each object in their own file (with some directory structure perhaps).

[vyzo]

On the other extreme, we can do something like having sql dumps for statement db and raw sst files for data; not sure if it's a reasonable choice for a portable data format though.

[parkan]

Ok, so we should have some kind of archive manifest with format version etc

[vyzo]

We have a practical Implementation with tarballs containing statements and objects:

135

https://github.com/mediachain/aleph/pull/186 https://github.com/mediachain/aleph/pull/185

[parkan]

:+1: I think this completes the task