discv5: new packet format proposal

[fjl]

This issue is a proposal for changing the discv5 packet format. I have included a lot of background information so everyone will be able to participate in the discussion.

There are a couple of requirements that drive the packet format design:

• Mutual authentication of the communication partners. This is why we have the handshake. The handshake also doubles as a traffic amplification countermeasure.
• We want active protocol participants to be able to distinguish discovery traffic from other traffic on the same UDP port for multiplexing and upgrade purposes.
• Deep packet inspection resistance, i.e. it should be difficult for a passive observer to determine whether a network packet belongs to the discovery protocol purely by looking at the packet content.

Why enable protocol identification for active protocol participants?

Protocol identification means that the application receiving the packet can distinguish discv5 packets from other UDP packets arriving at the same UDP port. This is different from DPI because the application understands the protocol and may have additional information which the firewall doesn't have.

Not all protocols include explicit identification. In fact, most Internet protocols don't because the port number already identifies the application sufficiently.

For peer-to-peer communication, protocol identification is useful because getting the traffic through NAT is hard enough to do for one port. Best if you can get the most use out of that port. Being able to confirm the protocol also enables deployment of new protocol versions alongside the existing version. For example, since discv4 packets can be matched by checking for a valid packet hash, we can run discv4 and discv5 on the same UDP port. Any traffic that isn't explicitly identified as discv4 can be passed on to the discv5 packet handler.

Why attempt DPI resistance?

I think it's important to keep DPI in mind when creating peer-to-peer protocols. Commercial-grade networking equipment provides built-in features for blocking peer-to-peer traffic (see Cisco, Meraki, Juniper docs), specifically file sharing protocols, because organizations don't want to expose themselves to legal risk. Most peer-to-peer networking activity is not illegal, but if it's easy to block peer-to-peer traffic for networking admins, they'll probably just do it to feel safer.

There are several ways in which a DPI-based firewall can identify traffic:

• Simple packet content signatures, i.e. the firewall runs a stateless script to check if data at certain offsets in the packet matches the signature. This is very common because it's easy and cheap to do.
• Stateful processing: for certain protocols, the firewall can follow along with the data flow to determine the session ID contained in the traffic. It can then block other traffic using the session ID.
• Inter-packet timing/size/direction analysis. This can be surprisingly effective even for encrypted protocols and is usually employed by dedicated commercial firewall solutions.

The main worry with DPI isn't that someone would block discv5 explicitly, it's rather that the protocol might eventually end up in a DPI vendor signature list. The discovery network can be a shared resource for multiple protocols. If a single application is deemed malicious and is blocked using DPI for that reason, all other non-malicious protocols will also be affected.

Designing the protocol wire format to avoid static signature matching is the simplest and most effective thing to evade most DPI. Working against timing/size analysis directly in the wire protocol is more complicated, but implementations can make themselves harder to identify by adding randomized delays or packet padding.

Note that DPI evasion measures do not make the protocol impossible to block in general, it just means DPI alone isn't enough to block it. It's still possible to block (or whitelist) the traffic based on port numbers, for example. Truly determined firewall operators could also just participate in the protocol and learn about node endpoints this way.

Protocol identification in the current discv5 wire protocol

The current discv5 wire protocol does not permit protocol identification explicitly. Worse yet, there is also no way to identify if an otherwise valid discovery packet is truly intended for the node which is receiving it. This causes the type confusion issue described in issue #131:

If node A sends a packet to an endpoint, assuming it belongs to node B, it uses a tag of A xor sha256(B). But if the node behind the endpoint is actually node C, the protocol fails. Node C receives the message and derives src-id = (A xor sha256(B)) xor sha256(C) which is bogus. It then sends WHOAREYOU back to the derived src-id, which node A doesn't recognize because the tag on WHOAREYOU depends on the destination ID.

DPI resistance of the current discv5 wire protocol

While DPI resistance was a design goal initially, we sort-of abandoned it later when the handshake was added. I just re-checked it and turns out static matching is possible because the protocol contains plain text RLP.

WHOAREYOU packets with empty sequence number can be matched like this:

def isWhoareyou(p):
    return len(p) == 80 and p[32:34] == b'\xEF\x8C' and p[46] == 0xA0 and p[79] == 0x80

Another possible static signature is the authentication response header:

def isAuthResp(p):
    return len(p) > 87 and p[35] == 0x8C and p[48] == 0xA0 and p[81:87] == b"\x83gcm\xB8\x40"

If either of these functions were used for blocking, discovery is effectively disabled. The selectors are not perfect and could be improved to reduce the false positive rate, but as an example, they show that static identification is possible.

Proposal Overview

For reference, the current outermost packet encoding is:

packet = tag || auth-header || message

where tag is a multi-purpose 32-byte value, auth-header is plain text RLP of varying size depending on the handshake state, and message is an encrypted container for the actual protocol message. This encoding is very compact, in some cases even optimally compact, but has a number of real-world shortcomings:

• Nobody really likes the tag construction and it doesn't guard against misidentification of nodes or the WHOAREYOU packet.
• The auth-header is multi-purpose and tries to hide the node record from passive observers in a clumsy way. During the handshake, some parts of auth-header are strongly encrypted with a one-time use key. This leads to a lot of complexity in the spec.
• The protocol can still be statically identified.

The current encoding honestly just feels a little cobbled together. Given all those issues, I have decided to redo the outer packet encoding one last time, in a more principled way, before releasing the first stable protocol spec.

There are two proposals here. The first proposal defines a packet encoding using mostly fixed-size fields. The new format permits protocol identification explicitly (through the checksum) and encodes handshake state explicitly (in flag). The second proposal builds on the first and adds 'DPI blinding', effectively removing all plaintext for passive observers.

Proposal 1

The discv5 protocol deals with three distinct kinds of packets:

• Ordinary message packets, which carry an encrypted/authenticated message.
• WHOAREYOU packets, which are sent when the recipient of an ordinary message packet cannot decrypt/authenticate the packet's message.
• Handshake message packets, which are sent following WHOAREYOU. These packets establish a new session and carry handshake-related data in addition to the encrypted/authenticated message.

In the following definitions, we assume that the sender of a packet has knowledge of its own 256bit node ID (src-id) and the node ID of the packet destination (dest-id). When sending any packet except WHOAREYOU, the sender also generates a unique 96-bit nonce value.

All packets start with a fixed-size header, followed by a variable-length authdata section, followed by the encrypted/authenticated message.

packet        = header || authdata || message
header        = checksum || src-id || flag || authdata-size
message       = aesgcm_encrypt(initiator-key, nonce, message-plaintext, header || authdata)
checksum      = crc64("discv5" || dest-id)
authdata-size = uint16    -- byte length of authdata
flag          = uint8     -- packet type identifier

The checksum field should be recomputed by the recipient based on its own node ID. The recipient may then verify whether the packet is truly a discv5 packet sent to the correct node. If the checksum doesn't match, the recipient should simply ignore the packet.

The flag field identifies the kind of packet and determines authdata content.

Ordinary Message Packet (flag = 0)

For message packets, the authdata section is just the 96-bit AES/GCM nonce:

authdata      = nonce
authdata-size = 12

WHOAREYOU Packet (flag = 1)

In WHOAREYOU packets, the authdata section contains information for the verification procedure.

authdata      = request-nonce || id-nonce || enr-seq
authdata-size = 52
request-nonce = uint96    -- nonce of request packet that couldn't be decrypted
id-nonce      = uint256   -- random bytes
enr-seq       = uint64    -- ENR sequence number of the requesting node

Handshake Message Packet (flag = 2)

For handshake message packets, the authdata section has variable size since public key and signature sizes depend on the ENR identity scheme. For the "v4" identity scheme, we assume 64-byte signature size and 33 bytes of (compressed) public key size.

authdata starts with a fixed-size authdata-head component, followed by the ID signature, ephemeral public key and optional node record. The record field may be omitted if the enr-seq of WHOAREYOU is recent enough, i.e. when it matches the current sequence number of the sending node.

authdata      = authdata-head || id-signature || eph-pubkey || record
authdata-size = 15 + sig-size + eph-key-size + len(record)
authdata-head = version || nonce || sig-size || eph-key-size
version       = uint8     -- value: 1
sig-size      = uint8     -- value: 64 for ID scheme "v4"
eph-key-size  = uint8     -- value: 33 for ID scheme "v4"

Proposal 2 - DPI blinding

The encoding defined in the first proposal transmits all header information as plain text. This is fine, and does not affect the authentication property of the protocol in any way.

However, it may permit passive observers of discovery traffic to identify the protocol and uncover the node IDs which are communicating. Since the sender of a packet knows the destination node ID, and every node is aware of its own node ID, we can use the destination node ID as a symmetric encryption key for protocol metadata.

There are downsides to this obfuscation step: protocol debugging with standard tools (i.e. tcpdump) becomes impossible, and the additional iv element adds 16 bytes of packet size. We should carefully consider whether DPI resistance is worth enough to warrant the additional complexity and packet size.

packet        = iv || masked-header || message
masked-header = aesctr_encrypt(masking-key, iv, plain-header)
plain-header  = header || authdata
masking-key   = dest-id[:16]
iv            = uint128   -- random data unique to packet

The image below shows the masked-header with a thick black border. Note that message is not part of masked-header since it is already encrypted using AES/GCM.

Decrypting the masked header data works as follows: The recipient first reads the iv and constructs an AES/CTR stream cipher using its own node ID as the key and iv as the initialization vector. It can then decrypt the header part, verify the checksum, read authdata-size, and finally read the remaining authdata.

[decanus]

@fjl, finally managed to read. Proposal 1 seems relatively sound imo, however I am not yet certain if Proposal 2 is necessary. I need to think about this a little further, don't quote me on this but potentially the same guarantees could be achieved if the transport of packets was done over quic. Quic it already ensures that middleboxes can't inspect packets, additionally this may be more sound as it would be using a transport google is already using for most of its webservices making it potentially even harder for middleboxes to block. I may be wrong though, I am no expert on the matter.

[fjl]

Quic it already ensures that middleboxes can't inspect packets

What QUIC is trying to prevent there is something else though. With TCP, all control information is transmitted in the clear. There are certain networking appliances which can optimize TCP traffic or apply advanced QoS rules by changing the parameters of a TCP connection passing through. QUIC makes this type of optimization impossible by design. It's not concerned with blocking.

additionally this may be more sound as it would be using a transport google is already using for most of its webservices making it potentially even harder for middleboxes to block

I can see the potential benefit of masquerading as a web protocol, but also think it's going to be quite hard to make the protocol truly look like it's a legit QUIC connection.

[fjl]

Here are some benchmark results for proposal 1 vs. proposal 2. These are from this go code on a 2,5 GHz Intel Core i7.

DecodeHandshakePingSecp256k1 is the handshake worst-case including ENR signature verification. DecodePing is decryption & decoding of a ping packet with session keys.

name                               old time/op  new time/op  delta
V5_DecodeHandshakePingSecp256k1-8   526µs ± 0%   528µs ± 0%   ~     (p=1.000 n=1+1)
V5_DecodePing-8                    2.04µs ± 0%  3.51µs ± 0%   ~     (p=1.000 n=1+1)

I think it's fair to say the additional masking crypto makes no difference at all.

[zilm13]

@fjl Did header mask add any extra size to the packet or we have only IV packet size increase (16 bytes)?

[fjl]

The size increase is the added 16 bytes IV.

[fjl]

Just did another round of benchmarks on my phone, which has a Snapdragon 660 processor and sits roughly in the middle when it comes to smartphone processor performance.

V5_DecodeHandshakePingSecp256k1-6  1.29ms ± 1%  1.29ms ± 1%   ~     (p=1.000 n=3+3)
V5_DecodePing-6                    12.2µs ± 0%  19.6µs ± 4%   ~     (p=0.100 n=3+3)

[mkalinin]

There is a possibility to get rid of crc64 computation that becomes redundant if both proposals are applied. Since dest-node-id is used to derive a secret key, zero string can be used instead of crc64 as a key checksum value. This type of checksum should be pretty secure to use in this particular case as there is no risk to compromise the key.

Probably, we can also reduce the size of checksum from 64 to 32 bits. Assuming IV is randomly generated (so does the secret key derived from randomly generated dest-node-id) we can think of 2**32 randomly distributed checksums which is pretty enough to cover all nodes in all networks ever. Even though there is a collision, its cost should be negligible.

UPD:

• however, with either 64 or 32 bit checksum, masked part of the message fits into 4 AES blocks (440 vs 408 bits); therefore there is probably no reason in reducing checksum value
• in CTR mode receiver can initialise a cipher stream with IV, read checksum (first 32 or 64 bits) check if they are equal to zero and either discard the packet or continue processing it

[fjl]

Thanks for the feedback. In the spec update (#157) the checksum is already removed.