Implement SecureTransport option for SSL/TLS

[gdamore]

On macOS and iOS platforms, it would be nice to make use of the native OSX SecureTransport for TLS support. This would prevent needing to depend on mbedTLS. It's even callback based, so this should be a good fit.

[gdamore]

Would love to have expert help from a mac developer here.

[joemoulton]

This seems to align with what I'm working on, given that I am developing a RethinkDB client driver for which I have chosen to make use of NNG + mbedtls for use in a project with iOS and macOS client applications. The release builds of the mbedtls libs for ARMv7/ARM64 on iOS take up worth ~1.6 MB of space that I don't want in my client app distribution so I will offer my services.

[joemoulton]

I've hacked together a Secure Transport implementation that allows me to trivially use SSLRead and SSLWrite. Given the poor documentation and scattered example snippets across SO, this was really nontrivial compared to OpenSSL or MBEDTLS with only a root CA. Key points to consider:

1. SSLSetCertificate can only be set with a SecIdentityRef before the SSLHandshake routine is started. SecIdentityRefs only get created when a valid keypair + certificate with matching hash get added to the device keychain store. The only way to successfully add these keypair + certificate combinations to result in creating a SecIdentityRef is with a PKCS12 store file containing enough information to create the identity in tandem with the SecPKCS12Import method or generating the certificates + keypairs on the device, over the network on-the-fly using SecKeyGeneratePair/SecKeyRandomPair and adding your cert in DER format to the keychain.

2. Since you can't get around item 1 if you are only working with a ca root cert that has only a public key, you have to customize the SSL handshake to do the authentication against the peer connection yourself using the SecTrustAPI with your ca root cert. Again, format of cert file must be DER to create a valid SecCertificateRef.

Now that I have SecureTransport working though I am sure that I'd rather use it in favor of a 3rd party MBEDTLS dependency, even though mbed is way more user friendly. I am still on the fence about whether to use nanomsg in my Darwin apps and RDB client driver. I would really like to. I'd rather not manage all that stuff myself. I am almost ready to acquiesce that posix aio was the right choice, I just want to make absolutely sure that zero copy sockets are not actually possible first. I'm not sure it would be possible in the write case, but have you tested and explicitly ruled out using sendfile and reading from the file's file descriptor with mmap? Even without sendfile, this kind of architecture is the only thing that makes lowest latency client media streaming possible for services like Pandora/Spotify and Netflix/Hulu and they are definitely using sendfile on the server side. Either way I think I can probably live with aio performance if it is the next best thing. The real kicker for me is having mutexes done away with in the name of custom locking mechanisms like I have mentioned elsewhere.

[gdamore]

When your content is static, or is already in a file descriptor, sendfile can be interesting as it allows the kernel to effect the file transfer without crossing into user space. So for streaming where the content is a media file, and the destination a TCP socket, it works great. (Not so much with HTTPS I'll note though.)

For a system like NNG it's hard to see how we'd make meaningful use of this.

On a client system, its hard to imagine that the latency difference would be an issue. We're usually talking at most a few 10s of MBs per second, and latency tolerations are up in the dozens of msec. The system call latency increase is a few tends of microseconds at most usually. (Can be single digit, really.)

For running on a heavily loaded server, the situation is vastly different. But again, to make much use of this presupposes you have static content, and a nice handy file descriptor to pipe through. HTTPS or any encryption breaks that (unless you use an inline proxy), as does any other kind of dynamic content generation.

If load is that big a problem, I would recommend either selecting better scale out designs (NNG can help there), or more exotic approaches (like porting NNG to run in kernel space along with your application -- NNG was designed with such approaches in mind btw.)

I'm a little concerned about the API changes to use SecureTransport -- meaning it sounds like applications will need to be modified to support SecureTransport -- it can't just be dropped into place. That's actually kind of surprising.

[joemoulton]

These are great answers.

However, I'm talking about using sendfile for socket receive on the client side (though I don't see yet why it wouldn't also be potentially viable in the write direction) regardless of whether the network data is encrypted or not, essentially using the file on disk as a message queue. Where does sendfile say the file needs to be static? It just says that the file descriptor needs to be able to be mmapped, right? Sendfile would write the data read from socket directly to file on disk using an mmapped file descriptor. You read from the front of the file with an mmapped file descriptor when bytes are available. Yes being slaved to a 3rd party socket based SSL api will inhibit decrypting when reading, but you get the idea.

I would really like to try scaling with some of your protocols, but again, I can't seem to get past the unkown 'Protocol error' when using a tls+tcp transport. I have only been successful with byte streams. Eventually I will want to try porting NNG to kernel space on desktop platforms but I am focused on iOS atm.

Re SecureTransport misgivings, you are referring to having to make the SSLRead/SSLWrite calls on a Secure Transport SSLContext which triggers the callbacks which have access to the socket, correct?

[gdamore]

mmap only makes sense for a file. you can't mmap a socket.

The idea behind sendfile is to basically get the kernel to copy the data for you between two different file descriptors. We might in theory have a file descriptor at one end (the network pipe), but we definitely don't at the other end (the application).

It turns out even for the network socket, its a lot more complex because we have message framing that we do in the protocol layer, which can't be done via sendfile. We could orchestrate that, but it would be a lot more complex, with little value.

It's also the case that we don't necessarily have just a single network file descriptor at the other end. For example, inproc doesn't go to the kernel at all, TLS will require extra transforms, and so might websocket depending on whether we are on the client or the server. Then there is the case of fanout and retry, where we need a copy of the data anyway. Ultimately sendfile is just an impedance mismatch, and any attempt to use it with NNG is almost certainly a case of premature optimization.

Your protocol error issues are concerning... both ends need to be speaking an SP implementation (nanomsg, NNG, and mangos are the main implementations, although there are a few others that are less mature) and have appropriate peer protocols (e.g. PUB matched with SUB etc.) Which SP protocol are using?

If you're using the byte streams directly, then you probably aren't speaking an SP protocol.

[joemoulton]

I am aware that you can't mmap a socket. I am talking about reading directly from the buffer mmapped to the file on disk. While the man pages for sendfile seem to indicate it only works in one direction (file->socket), this SO post seems to indicate that flipping the file descriptor and socket descriptors will work for a read from socket to file. Possible or not? Also note that FreeBSD sendfile is different from linux sendfile.

https://stackoverflow.com/questions/20235843/how-to-receive-a-file-using-sendfile

[joemoulton]

I am definitely not speaking to another SP implementation. Again, I am using nng for a rethinkdb client driver in C, that interfaces with the RethinkDB service over TCP wrapped in SSL by another service. Thanks for the clarification.

[gdamore]

And now, with macOS 10.14 and iOS 12, Apple is deprecating Secure Transport. Making matters worse, it's so darned hard to separate out the Objective C stuff in their new "network objects" API from the C.

I'm probably going to just close this for now -- I don't have the energy to keep up with Apple's rate of change, and so far none of the folks who've contributed economically to the project have requested support for any of the apple platforms beyond what we already have.

At least for macOS mbed (and going forward we will support others) Just Works.