DEP: Ephemeral message extension

[pfrazee]

Another DEP proposal

This DEP defines the non-standard em extension message used in the Dat replication protocol. This message provides a way to send arbitrary application data to a peer through an existing connection.

This is not meant to supersede https://github.com/datprotocol/DEPs/pull/27 and will probably be used in combination with it.

[pfrazee]

Here's a lab API I'm considering for Beaker to leverage both this and #27 https://gist.github.com/pfrazee/7259f9201d44417c777803984715e1c4

[RangerMauve]

This is awesome! The proposed Beaker API is really elegant, too.

Only questions:

• Why 1000 bytes?
• What happens when there's a parsing error for the payload?
• Are the peer IDs going to be random per dat site?

[pfrazee]

Why 1000 bytes?

I originally had it at 256 bytes to fit within a single packet (with some clearance) but I decided that's just too much of a pain for users.

I figure we should cap it at some size to discourage DoS attacks (ie "hey parse this 100mb json message!"). I'm open to discussing the exact size, but 1kb seemed reasonable.

What happens when there's a parsing error for the payload?

I figure you just drop the message. There's no acknowledgement of receipt and so an error won't come back either.

Are the peer IDs going to be random per dat site?

They're random per connection, at the moment.

[RangerMauve]

Also, question: In the future would it be weird if I did the "approve" review action when reviewing stuff even though I'm not part of the WG? I almost did it by reflex but wasn't sure if it would be weird to do so.

[pfrazee]

Straight into the dungeon with you

[pfrazee]

Changes made. I removed any max payload length. Do we want to readd that?

[bnewbold]

Could go either way on payload length... if we do limit it i'd leave the door open to increasing the size in the future, but never decreasing the message size.

[pfrazee]

Is this 👍 to publish as a draft?

[bnewbold]

It would be nice to at least mention that payload size should be kept "reasonable", but I don't have specific language to recommend. I wouldn't block on that.

Skimming back over this, I don't see a privacy/security section. Even as an "Informative" and "Draft", I think there should be a section, and at least a "user/developer beware" statement. This in addition to the statement under drawbacks. Off the top of my head:

• messages are encrypted on the wire and have the same global observer privacy properties as regular hypercore messages in that regard (eg, "what can your gateway router or ISP inspect/filter/block").
• the discovery key semantics mean that you know you are connecting to a peer that knows the discovery key. however, within that set of users (which could be large/public/everybody if the discovery key is a public app), there is no man-in-the-middle protection; any protocol trying to implement authentication or secure message using ephemeral messages needs to treat them like any other datagram over the open internet; hypercore does not provide a trustworthy end-to-end or peer-to-peer messaging channel. as with hypercore, the current status of the network is that a gateway can "trivially" control discovery of peers (unless DNS-over-TLS or DNSSEC or similar), and as well anybody can join the swarm, so man-in-the-middling is a real thing for all users, not an advanced-persistent-threat scenario; this isn't as much of a concern with regular hypercore because you're verifying the data you receive by signature, but ephemeral messages aren't signed with the feed key (unless the application developer does so themselves)

That last point kind of ran on, sorry about that.

It would be nice to link to the implementation of extension messages, but we don't have the wire protocol DEP published yet. For reference, the message type is 15 (0x0F). Also the semantics: if ephemeral messages are being used, no other extension messages can be sent unambiguously, and the "scope" of extension messages is to the connection, not to any particular feed or channel (though I guess it's implicitly tied to the first/primary feed that was used to initiate the connection).

Of the above, the only thing i'd consider blocking is an additional security/privacy section. Apologies for not noticing and requesting earlier; I can draft something if you want.

[pfrazee]

@bnewbold I'm happy to write it, and happy for you to! I'd probably steal what you wrote their either way. Just LMK

[bnewbold]

If you're up for writing it please do, and feel free to take anything from the above.

[pfrazee]

@bnewbold glad you suggested that section, it was needed. Ready for review.

[bnewbold]

We should discuss in the WG meeting, but after thinking about it more I think the above security issue is serious enough that it feels like we're handing developers a foot-gun by publishing this as a DEP, even a Draft. This doesn't feel "safe by default": there is a large burden on application developers and users to understand the security/privacy semantics.

[pfrazee]

We decided to withdraw this proposal. The WG was uncomfortable giving this spec approval given its security & privacy characteristics. Applications are (as always) free to use similar designs in their own extension messages.

[RangerMauve]

Is the session data proposal still good to go?

[pfrazee]

@bnewbold do you have any feelings about that? It's not fundamentally different in its security & privacy properties.

[RangerMauve]

Also, if this is being revoked, does that mean the datPeers API in beaker is going away? Would it be possible to keep something like it for sessionData if it's just ephemeral messages that are going away?

[pfrazee]

No the datPeer API is a go as planned. We decided not to make it an official DEP

[aral]

Sorry to revive a closed PR but regarding the privacy concerns, wouldn’t they be addressed by having the ephemeral messages encrypted by the feed key as part of the ephemeral extension protocol itself (instead of leaving it to userland?)

It feels like this would be preferable to every userland application creating its own method of handling ephemeral messaging.

[pfrazee]

@aral AFAIK that is the case -- all messages over the hypercore-protocol are encrypted using the feed key that arranged the connection. The concern is that the feed key is not secret enough for general use. For instance, if the communication were occurring over a popular app's dat.

There's a lot of work planned & under way with the discovery and connection layer. Discovery is shifting to hyperswarm and connection-layer encryption is (last I heard) moving to a NOISE implementation. The reasons for this include:

• Enabling authenticated dat connections, so that you can specify a network ACL (eg "only sync with aral").
• Hyperswarm moves discovery onto a DHT, which improves resilience. Hopefully that work will improve overall connection-reliability as well.
• We have plans to make the discovery+connection layer a separate reusable stack which Beaker will expose as the Peer Socket API. This will make it possible to arrange connections without piggybacking on the replication streams and that should make the connections fit their intended uses a little better in terms of controlling who participates in them.