Networking: NOISE handshake and transport encryption

[Frando]

For a working Dat2 / hypercore 8 implementation in Rust, we'll have to implement the NOISE handshaking and transport encryption.

I'll document what I found out while looking into this.

I don't think there's any "official" documentation about the NOISE handshake and transport encryption in hypercore yet. Looking into the code reveals:

• The handshake pattern used is Noise_XX_25519_XChaChaPoly_BLAKE2 (see noise-protocol/handshake-state and simple-hypercore-protocol/handshake). The handshake uses a state machine module called simple-handshake, which then uses noise-protocol, a Javascript implementation of some parts of the NOISE spec that uses sodium-native for the actual crypto.
• The handhshaking messages are sent over the binary stream with a varint length prefix (see simple-hypercore-protocol/lib/handshake.
• During the handshake, the payload being transmitted from each side is a protocol buffers encoded message with a 24 byte nonce (random bytes), created in simple-hypercore-protocol/index.js
• After finishing the handshake, the transport encryption is not done by noise-protocol (in other NOISE impls, there's usually a function to switch from the handshake phase directly into transport phase, where the cipher type is the same in transport mode as in the handshake).
• Instead, the transport is encrypted with XSalsa20 directly in simple-hypercore-protocol with sodium.crypto_stream_xor (see simple-hypercore-protocol/lib/xor.js and the sodium docs), with instances for receiving and transmitting, where for each the keys are coming from the split result of the NOISE handshake and the nonces are the payloads that were transmitted during the handshake. sodium.crypto_stream_xor resolves to crypto_stream_xsalsa20_xor which, in sodium-native, uses crypto_stream_xsalsa20
• The messages in transport mode are not length-delimited. Each buffer is decrypted as it floats in.

I tried to connect to a nodejs hypercore-protocol stream from Rust, however I hit a few roadblocks. I started with snow as it seems to be the most complete NOISE impl in rust. Following are, I think, what's missing to make connecting to a hypercore-protocol stream:

• [ ] snow doesn't support the XChaChaPoly cipher. I created an issue for that. There's a rust impl of XChaChaPoly in chacha20poly1305.
• [ ] In a quick test where I added XChaChaPoly to snow I still couldn't handshake with a hypercore-protocol stream (decryption error on either side), this needs to be investigated more
• [ ] snow doesn't allow to switch cipher types between the handshake phase and the transport phase, nor does it currently allow to access the split results directly

[Frando]

Some more investigations into the transport encryption after the handshake is complete:

The NOISE spec states with regard to transport messages:

A Noise transport message is simply an AEAD ciphertext that is less than or equal to 65535 bytes in length, and that consists of an encrypted payload plus 16 bytes of authentication data. The details depend on the AEAD cipher function, e.g. AES256-GCM, or ChaCha20-Poly1305, but typically the authentication data is either a 16-byte authentication tag appended to the ciphertext, or a 16-byte synthetic IV prepended to the ciphertext. (source)

and also

Applications must handle any framing or additional length fields for Noise messages (source)

(which in my tests was very true - the read_message method of the transport state in snow can only decrypt messages that are exactly the same messages as created by the write_message call on the other side, thus needing length-prefixes during transport)

---

hypercore-protocol in transport uses a streaming XSalsa20 cipher from libsodium, where the docs state:

The ciphertext is the message combined with the output of the stream cipher using the XOR operation, and doesn't include any authentication tag. (source)

So hypercore-protocol follows NOISE for the handshake, but does not use NOISE for the transport encryption, instead it uses XSalsa20 for streaming encryption with no authentication tags and no need for length-delimited messages for the decryption. This means that by-the-spec NOISE frameworks cannot be used for the transport phase of a hypercore-protocol stream.

[Frando]

@mafintosh / @emilbayes: Could you check if these findings about hypercore-protocol are correct, and clarify if there's a reason why hypercore-protocol does not stick to the NOISE spec for the transport phase?

[Frando]

I pushed a client/server example of how far I got with the handshaking, see here for details.

[emilbayes]

There are some things I did in noise-protocol that are "non-standard" but that I want to fix:

• Use chacha20-poly1305 instead of xchacha20-poly1305. Not a requirement to be standard, but barely any other implementations have xchacha20. I also send a nonce over after the initial handshake which is also "non-standard". It's hard to talk about standards with noise tho, since it's a framework for writing protocol handshakes and not a strict standard
• I use the sodium kx APIs for Diffie-Hellman. I thought this was safer before I really learnt how the protocol works as this hashes the symmetric key derived from the DH function. Since everything in noise is "hash and mix" this doesn't matter, and just makes it harder to write an implementation in other languages.
• I never really considered transport encryption part of noise, but when I read the spec now I see that there's extensive instructions on what transport encryption is under the noise framework. I should include this in noise-protocol

The above points I am going to remedy soon, but these are also the only parts that are noise here. The rest is how hypercore has decided to use noise under various constraints.

I know that hypercore uses "dummy keys" and I think it uses "channel binding" as part of the new capabilities system. The application responsibilities also makes it more tricky to get different implementations to talk to each other: http://www.noiseprotocol.org/noise.html#application-responsibilities

Maybe it's worth it to consider using a pregenerated noise implementation from https://noiseexplorer.com and modify that to how hypercore uses noise?

[Frando]

@emilbayes cool, thanks for this info.

With your standard-dh branch I got one step forward: Now the handshake between node and rust semi-completes - the initiator finishes correctly, the not-inititiator dies with a decrypt error.

I set up a repo to better test this: https://github.com/Frando/rust-node-noise-handshake

Feel free to chime in there or help to debug :-D could also be a good base for proper integration tests, it has a one-command runner that starts both server and client.

[Frando]

Now some more progress here, yeeha:

https://github.com/Frando/hypercore-protocol-rs

With the standard-dh branch of noise-protocol in Node.js, and two patches to snow ((1), (2)), I now have a mostly working (yet very basic) implementation of hypercore-protocol 7 (Hypercore 8 / Dat 2) in Rust: Pass the handshake, set up the transport encryption, open channels, verify capabilities and send and receive messages. Next would be to integrate with hypercore!

Also if someone would want to review the API and implementation of the repo linked above, I'd be very interested :smile:

[Frando]

This is all fixed and can be closed :)

See hypercore-protocol-rs. My PRs to snow got merged, and the new Hypercore 9 release switched the handshake algorithm. It also switched the handshake cipher from XChaChaPoly to ChaChaPoly, so one of the PRs to snow wasn't even needed in the end.

Anyway - the master branch of hypercore-protocol-rs can handshake, verify cabalities and exchange all messages with a nodejs hypercore now. Once snow gets a new release, I'll publish a first preview release.