Add support for Kernel TLS

[alexw91]

Problem: Some users may see performance/latency issues due to syscall and mem copy overhead to copy data from one file descriptor to another. We can reduce this overhead by integrating with the Kernel TLS API's if we're on a Linux Kernel that supports it.

Links:

• https://netdevconf.org/1.2/papers/ktls.pdf
• https://blog.filippo.io/playing-with-kernel-tls-in-linux-4-13-and-go/
• https://lwn.net/Articles/666509/

Proposed Solution: Add support for Kernel TLS, and add API's to enable/disable it.

[wcs1only]

Has there been any proposals for what the s2n API for KTLS would be?

For me two possibilities come to mind:

Expose kTLS Socket

We could add a s2n_handshake_ktls() API that performs a handshake as normal and returns an fd to a kTLS socket. The userspace application then can use the fd as they would any other socket, and would be responsible for managing its lifecycle. S2n could free up its corresponding connection structure and otherwise forget about the connection from this point forward.

Pros:

• Probably the easiest to implement
• s2n wouldn't need to manage differences in platform specific socket APIs. (FreeBSD supports non-blocking sendfile(), Linux does not, etc.).
• We might be able to reclaim all of the memory overhead to s2n connections. (~48KB/connection today)

Cons:

• We might loose the ability to manage blinding on shutdown, or any other post handshake state.
• Requires a different pattern of use that may put a lot of integration work on the userspace application that is possibly already integrated with s2n.

Create new zerocopy s2n APIs

We could add a kTLS configuration parameter that would let s2n maintain ownership of the connection. The userspace application could still interact via s2n_send(), s2n_shutdown(), etc. In order to get the actual performance benefits we would need to add s2n_sendfile() and s2n_splice(), etc.

Pros:

• Existing s2n applications might be able to migrate rather easily. Same overall flow, just switch out s2n_sendfile/splice when writing the body of a message.
• Allows s2n to still manage TLS control messages. (Not sure what we're doing here today).

Cons:

• Will need platform agnostic zero-copy syscall APIs.

Would love some maintainer feedback on this, or some discussion/proposal documentation. I have a few weeks downtime and I may try and write up a PoC for 1, just because it seems like I might be able to fit it into the time I have.

[dougch]

Thanks for the comments, working to get some feedback.

[dougch]

After a team discussion, we'd like to investigate one other possible performance improvement approach before committing to kTLS.

[armanbilge]

The userspace application then can use the fd as they would any other socket

There's a significant pro to this approach (or con to the "zerocopy s2n APIs" proposal). When the userspace application can directly interact with the fd, it would be able to use io_uring for reading/writing on it efficiently. Since s2n doesn't use io_uring this option isn't available with the zerocopy approach.

Anyway, I think unstable support for kTLS landed in https://github.com/aws/s2n-tls/pull/4217, not sure what approach was taken in the end :)