pietgeursen commented 5 years ago

:tada: World's first rusty replication :tada:

About

In short, this PR should let you try out:

(from rust) publish a new message.
(from rust ) connect to a (modded) js ssb-server that requests your feed and replicate it to js.
(from js) publish a reply to the rusty message
(from rust) connect to the js ssb-server and replicate the js server's feed.
(from rust) call createHistoryStream for the js feed and see the js reply in the console.

Setup

We want to force the js server to always request the feed of the rust bot.

Clone and npm install ssb-server

In the node_modules/ssb-replicate folder, edit legacy.js, line 351 to look like:

function fallback () {
  //if we are not configured to use EBT, then fallback to createHistoryStream
  if(replicate_self) return
  replicate_self = true
  replicate({id: "<KEY_OF_RUST_ID>", sequence: 0})
  replicate({id: ssbServer.id, sequence: toSend[ssbServer.id] || 0})
}

We want to switch off ebt as well.

Edit bin.js in the top level of ssb-server and comment out line 61 eg.
```
//.use(require('ssb-ebt'))
```

Make a secret file for the rust code to use that is different from the js-bot.

One other important caveat

The path to the offset file and sqlite file are hard coded to be /tmp/ssb-db.offset and /tmp/ssb-db.sqlite3 in the handleReplicationRequests command. That means you must use those same paths for the other commands.

I need to hard code them otherwise we get borrow checker errors because of the async lambda function around line 365.

Let's do it!

Publish our message:

./target/release/rustle --secret-file <path-to-secret> publish --dbpath /tmp/ssb-db.sqlite3 --offsetpath /tmp/ssb-db.offset --content '{"type":"post", "text": "hello from RUST!"}'

and you should see something like:

published a new message!
{
  "key": "%sdWbyCRzELsK70UTgKdWBDeVx6LHt680AzrI39tE5Ns=.sha256",
  "value": {
    "previous": "%iT6lKuNaTgcixW4LrG6u0L32i3pnSfDw2CYmcd2JVN4=.sha256",
    "author": "@SpDxZ/N2pG0NemSXhPbzbEy4KUH3028+z83FtbqyTEY=.ed25519",
    "sequence": 2,
    "timestamp": 0,
    "hash": "sha256",
    "content": {
      "text": "hello from RUST!",
      "type": "post"
    },
    "signature": "S5062Za77eF11HzAp1lW14aQdVIO6T+xdvS/df+d7MFP8UPficXi+wSQ29rM4kVY6f0OcfwDtv5us1ejGSaUDw==.sig.ed25519"
  }
}

Get the js ssb-server to replicate us:

With the modded ssb-server running.

./target/release/rustle --secret-file <path-to-secret> handleReplicationRequests --key <key-of-js-ssb-server-without-@-or-.ed25519> --dbpath /tmp/ssb-db.sqlite3 --offsetpath /tmp/ssb-db.offset

You should see:

client connected
In: 1 strm:true end:false Json {"name":["createHistoryStream"],"args":[{"id":"@SpDxZ/N2pG0NemSXhPbzbEy4KUH3028+z83FtbqyTEY=.ed25519","seq":1,"live":true,"keys":false}],"type":"source"}
got a chs request.
retrieved 2 entries
sent all entries

Now on the js side, go createHistoryStream for the rust feed id. You should see your rusty message :1st_place_medal:

Post a reply to it:

ssb-server publish --type post --root <rust-post-key-id> --text "OMG hello rust!"

Get the js feed into rust:

./target/release/rustle --secret-file ~/.ssb/secret replicatefeed --feed <js-server-id> --key <key-of-js-ssb-server-without-@-or-.ed25519> --dbpath /tmp/ssb-db.sqlite3 --offsetpath /tmp/ssb-db.offset

You should see:

Latest sequence for feed is 7
client connected
got 1 new messages from server
validated messages
verified messages
appended 1 new messages to db

And now to see the reply from js:

./target/release/rustle --secret-file ~/.ssb/secret createHistoryStream --dbpath /tmp/ssb-db.sqlite3 --offsetpath /tmp/ssb-db.offset --feed <js-server-id>

And you should see:

{"key":"%yLOqT6+w6gSzYZ9SqICc0pKD7PzAnjOQNZ0Ub7OJJik=.sha256","value":{"previous":"%RDSVWmArd1pTTE/uIog0lrDYqqE73SDG9NinNPOxDA8=.sha256","sequence":7,"author":"@z7oRp9UDnnWdCprvjBRs+PIKKaNhSdL6zLSI99aujTY=.ed25519","timestamp":1574679772837,"hash":"sha256","content":{"type":"post","root":"%iT6lKuNaTgcixW4LrG6u0L32i3pnSfDw2CYmcd2JVN4=.sha256","text":"OMG hello rust"},"signature":"cuCiPzzhvqaap2wWLqM0Qt8973D+uGHSQnVk3DGFqn3XVILIuEXFBGFRvx38Ot+nvlk2Kam6gOThILO1ekUsCg==.sig.ed25519"},"timestamp":1574679772838}

Fast replication

Also for fun, try replicating a big feed. The validation / verification / appending is all batched to it's super fast!

Noisy diff

OMG, the diff is super noisy, sorry, I always run cargo fmt before I push.

cc @ahdinosaur @AljoschaMeyer @mmckegg

sbillig commented 5 years ago

Awesome work, @pietgeursen! Tested it out, and it works.