Ian: ocap-merkledag MVP

[zenhack]

https://github.com/zenhack/ocap-merkledag is the foundation of my plans for Sandstorm's backup system, but in the shorter term it even a less integrated version as an app would be useful as a data-dumptruck for my own backups. Soonish I hope to get an MVP of this out. This will consist of a Sandstorm app with basically no UI other than an offer iframe, and a command-line tool that you can use to shove stuff into it. From there I'll build a UI and such, but this issue is for the MVP.

[ocdtrekkie]

Since it was a popular ask in #14, can I ask roughly where this sits today?

[zenhack]

The current state is, I have a cli based server & client that can store & retrieve a directory tree, but:

1. It's unacceptably slow, for reasons whose roots are mostly inside haskell-capnp, which I'm working on fixing currently. Prob a few days to a week of actually working on it to get these issues fixed-enough to get back to working ocap-md proper.
2. While I have a plan for enforcing ACID properties, I haven't implemented some bits of this yet, so it's likely to eat your data in the event of a crash.
3. There is of course no sandstorm packaging & no web server in front of it yet; it's just doing tcp on localhost.

[ocdtrekkie]

Okay, so probably still a bit too early to be one of my short-term deliverables. A backup product that eats your data is probably problematic.

I am thinking maybe the next time around when assembling projects, we may be able to set a goal to release this as an app for general backup use, and then the next milestone after that would be the integration for Sandstorm use.

[zenhack]

Re: backups, the way I've been thinking of it the Sandstorm integration is possibly separable; I've been thinking of a workflow like this:

• Go to grain settings page.
• Hit setup backups
• Sandstorm triggers a powerbox request for some well-known backup interface, which can be satisfied by whatever apps.
• You set a schedule.
• Sandstorm periodically feeds the .zip to the backup capability that was chosen.

...you could also do default policies for grains you haven't set up explicitly. You'd likely want to add some checks to make sure you're not backing a grain up into itself.

So ocap-md could satisfy this interface, but it would also allow us to just add a trivial thing to davros or such that just stores the zip file in a directory.

[ocdtrekkie]

That all sounds excellent. I think we probably would also want to be able to set that default at a server-wide level, in addition to a user-wide level, and probably put it into the setup flow eventually.

Presumably if our goal is to create an easy to use cloud platform that makes good decisions, it should probably set up at least minimally sufficient backups by default.

[zenhack]

Some progress on this one! I ended up re-writing the little bit that I had in Go, rather than continue to fight the perf issues I was seeing in Haskell. I kinda knew using Haskell for this might have been a mistake going in.

Anyway, relative to the stated functionality from before:

• It's actually fast enough to use.
• In principle it shouldn't eat your data. That said, I would want to do more aggressive stress testing before trusting it with anything super important. The storage format is very simple though, so I'd give very good odds to me having actually gotten it right.
• There is actually sandstorm stuff; if you've got a Go toolchain installed you should be able to clone the repo onto a sandstorm box and do make dev (it uses raw spk). The UI just shows the text "Hello, World", but you can generate a webkey and interact with the grain using the cli.

This is very nearly at the MVP I described, but before anyone else tries to use this:

• I may want muck with the storage format a bit before I commit myself to having to care about backwards compatibility.
• Probably should be treated as experimental for a while.