[question]SSB Spec? - Githubissues

jjbarr commented 8 years ago

I am interested in using SSB in a platform/language-agnostic environment. Is there any sort of specification of the SSB protocol, for implementation in other environmentd?

ahdinosaur commented 8 years ago

hey @qwertyuiop924, you might be interested in https://github.com/ssbc/docs, published on https://ssbc.github.io/docs/ and https://scuttlebot.io. :smiley:

do you want to expand further on what you have in mind? myself and probably others would be interested to hear.

jjbarr commented 8 years ago

Well, I was wondering if the actual Secure Scuttlebutt protocol was documented, such that it could be implemented in another programming language without digging through the source. You explain some of it, like identities, but the majority of the protocol is entirely unspecified, meaning that you have to dig through the source to get that information.

To contrast, your peers (dat, IPFS, Matrix, etc.) and the majority of widely adopted protocols today (HTTP, SMTP, LDAP, SSH, Telnet, IMAP, POP, etc.) have rigourous specifications of the protocol (on their website for the former, as RFCs for the latter), detailing all, or at least most, of what you need to know to create a working implementation of the protocol. Most of these protocols (all save dat, AFAICT) have multiple implementations, which function well together, so you can choose whichever one you want: this isn't a coincedence.

Right now, if you want to use SSB, you have to use Node. I don't hate Node or anything, but plenty of people do, and there are uses to which it is ill-suited. I would rather that it would be possible to use SSB in other languages as well, which would have the addtional benefit of ensuring that the success or failure of the protocol isn't tied to the success or failure of any one language or runtime. With a formally defined spec, it becomes dramatically easier for anyone who wishes to implement the SSB protocol to do so.

Possible difficulties for you in doing this is the fact that I gather you use many external libraries in the protocol itself: IIRC, an external lib is used for LAN discovery in Scuttlebot. If you wanted to specify your protocol, you'd have to specify what these libraries did as part of the spec, as they are part of your protocol on the wire. And you'd probably want to specify Scuttlebot, as it is downstack from most actual uses of SSB.

Hoped this helped explain.

ahdinosaur commented 8 years ago

hey @qwertyuiop924, you're totally right.

i too wish there was a formal specification to explain the protocol in great detail. the problem is the protocol has iterated quickly based on rapid prototyping of our working software and writing good specifications is really difficult.

so, do you want to help document what you find lacking? if you have any specific questions about the protocol, feel free to ask here or on the SSB network (using patchwork or patchbay).

Right now, if you want to use SSB, you have to use Node.

even without a formal spec, here's a proof-of-concept go implementation thanks to @cryptix. :smile:

jjbarr commented 8 years ago

Neat.

Really, I just wish the protocol was better defined. Like, for instance, what's muxrpc got to do with SSB? How are SSB messages formatted? All that stuff. If I dig up the answers, I might even put in a PR to document it (noting that it's unstable, and subject to change, naturally).

cryptix commented 8 years ago

secret-handshake and muxrpc don't have much to do with ssb per-se but it transports the gossip communication.

jjbarr commented 8 years ago

Ah. So they're downstack from ssb? That explains a lot.

dominictarr commented 8 years ago

@qwertyuiop924 yeah, maintaining multiple protocols at this current point in development would slow us down a lot. Each change would have to be synchronized over multiple implementations, but I completely agree that this is important to eventually have, but not yet.

what language do you prefer?

which is direction is downstack? muxrpc and secret-handshake are dependencies of ssb.

I'm happy to answer any questions you have if you would like to organize those answers into documentation

cryptix commented 8 years ago

@qwertyuiop924 re downstack: shs[muxrpc[ssb]] is roughly like ip[tcp[http]] from a capsulation standpoint. You could do all kinds of distributed systems with shs and muxrpc.

pfrazee commented 8 years ago

@cryptix just to fill out your comparison a little more clearly, ip[tcp[shs[muxrpc[ssb]]]] is the full communication stack, and it's comparable to ip[tcp[tls[http[app-msgs]]]]

jjbarr commented 8 years ago

@dominictarr

I get it, and am willing to wait.

Downstack means below the component in the tech stack, so yes, dependancies are downstack. As is node, the OS, the BIOS, and the hardware itself.

Favorite language? Scheme, closely followed by CL, Lua, JS, and C

BrianOtto commented 7 years ago

I agree with @qwertyuiop924, SSB really needs an official specification (even if that is still a moving target). The current documentation is too high level and mostly focused on Scuttlebot. It does not cover lower level things a developer, in another language, would need to know.

Things like ...

How do I negotiate the secret handshake? e.g.
- The SHS protocol should be explained like I'm five
- i.e. What is happening in the "challenge" and "accept" steps?
- i.e. What is the process for exchanging keys and encrypting data?
How do I communicate with the server? e.g.
- A detailed description of the packet stream codec and what the different flags do
- i.e. How are the packets split up and encrypted during transit?
- i.e. What is the JSON object format being used?
How do I format the different message types in my feed? e.g.
- The JSON format and an explanation of the different types (about, contact / friends, vote, pub)
- i.e. How are private messages formatted differently?
- i.e. How do I request / search for blobs?

I have been trying to build a mobile client for SSB, and it has been very difficult to understand how everything works together. However, having said all that, I was able to eventually write a barebones client that answers some of these questions.

It gets rid of most of the dependencies and simplifies the logic so that the protocol is clearer. I am posting it here in case anyone else is looking to write a client outside of the Node ecosphere.

https://github.com/GrokBB/Nebula/blob/master/doc/reference/ssb-client.js

This process was much more difficult than it should have been, especially with the amount of dependencies I had to go through to understand everything. There are some things I still don't understand, and some parts I will have to come back to later (like the different message types). So an official specification would be greatly appreciated :)

jamen commented 7 years ago

Dang, that is really cool @BrianOtto. I'll just chime in as well: I would really like to see a spec eventually so I can try and implement something for Erlang or Elixir.

staltz commented 7 years ago

I agree we need a spec, I just think we need a volunteer for that (instead of pre-appointing someone).

aschrijver commented 7 years ago

BrianOtto commented 7 years ago

I have started to write up what I know here ...

https://github.com/ssbc/secure-scuttlebutt/wiki/Getting-Started https://github.com/ssbc/secure-scuttlebutt/wiki/Packet-Stream-Codec

I will add more as time permits. Feel free to review and correct any errors, or add more details!

andrewtj commented 6 years ago

Is the protocol still moving fast?

cryptix commented 6 years ago

Since it hasn't been linked here yet: Scuttlebutt Protocol Guide

Is the protocol still moving fast?

i'm not aware of any breaking change to the network layer yet.

staltz commented 6 years ago

This issue might be never closed unless we consider the protocol guide (link by cryptix above) done as version 1. More actionable issues can be opened, regarding that guide. In particular, the issues should be opened in that repo: https://github.com/ssbc/scuttlebutt-protocol-guide

(Hope people are fine with closing this issue)

dominictarr commented 6 years ago

@andrewtj the protocol is being developed, but we try to do so without breaking things. we do not intend to support legacy protocols for ever, but we do intend to design for graceful upgrades. There is some emerging stuff that is not in that guide, but that covers everything currently "mainstream" ssb

CrowdHailer commented 6 years ago

I'm interested in writing a erlang pub. Would also be interested in a full spec, and helping improve it as possible, even if it is flux

dominictarr commented 6 years ago

@CrowdHailer yup, see the protocol guide: https://github.com/ssbc/scuttlebutt-protocol-guide

dominictarr commented 6 years ago

@CrowdHailer also if you are doing erlang stuff, also talk to @jiangplus who tells me he's working on exlir port

CrowdHailer commented 6 years ago

@dominictarr thanks for the link. I have actually found the protocol guide and opened an issue asking for the improvement of adding concrete examples for all the steps.

@jiangplus Would be interested in seeing what you have done if it is on github. I am actually an elixir developer and am only using erlang as an experiment. I would be quite keen to contribute to an elixir version

AljoschaMeyer commented 6 years ago

Moid (@Sur8RwcDh6kBjub8pLZpHNWDfuuRpYVyCHrVo+TdA/4=.ed25519 on ssb) also did some work with ssb in erlang: https://viewer.scuttlebot.io/%25d5gv%2FqcOJiI2ow7VWpZ20LJASksEOtP4asjW28nMyvw%3D.sha256

ssbc / ssb-db

[question]SSB Spec? #154