protocol stability

[cgwalters]

The protocol isn't documented today as far as I can tell. But I think this crate is implicitly committing to its stability - because any time one is doing IPC, the possibility of version drift between the binaries (and hence versions of this crate) comes into play. I'm sure the protocol wouldn't change across Rust semvers of this crate - but do you see it changing ever?

In one use case I have that's similar to this crate, I have the client send the (device, inode) pair of /proc/self/exe over to the server, and the server verifies they match. (This also of course covers the higher-level protocol that applications would use on top of this crate; a related but distinct issue)

One reason I ask this is I have a use case for doing IPC between Rust and Go, and I'm today doing JSON over SOCK_SEQPACKET but...while the great thing about SEQPACKET is the kernel takes care of the framing, it has less support in the userspace ecosystem, and a simple "[header with length][payload]" protocol has well-understood tooling because it's how a vast array of TCP protocols work.

TL;DR: Would you say we could document the "raw" protocol used here and commit to its stability approximately ~forever?

[mitsuhiko]

I think the answer is a) yes but also b) before we do that, what should that protocol be. Is what is there today sufficient? I would say the answer is no.

[cgwalters]

Did you have anything particular in mind that might be missing from the protocol?

The main thing I can think of here would be supporting an explicit initialization phase. This would mesh with https://github.com/mitsuhiko/tokio-unix-ipc/issues/4

I glanced at https://dbus.freedesktop.org/doc/dbus-specification.html#message-protocol (particularly the negotiate/auth phases) but I don't think we want any of that.

Probably simplest to have something like an initial packet that has a magic string as a sanity check and peer credentials (I think we could make it mandatory on Unix)?

[mitsuhiko]

At the very least I would want to put a version into the header so that additional fields can be added later. Also a magic never hurts.

[cgwalters]

At the very least I would want to put a version into the header so that additional fields can be added later.

Hmm...anything like that seems like a new protocol basically because both the client and server would need to understand it for it to be useful.

WDYT about the below? It's basically:

• Add a hello message with a magic string and space for version negotiation
• hello message is also the new place for SCM_CREDENTIALS
• ~header uses varints~ (edit: oops no that can't work efficiently)

Protocol

Client hello message

The initial connection from client to server consists of the string value v=iUt0 in ASCII encoding, terminated by a newline. (The characters chosen here are an acronym for "ipc-unix-tokio", the project name in reverse because "TUI" is a much more common acronym).

In this initial message, if Unix domain credentials passing (SCM_CREDENTIALS) is expected by the server, the client should provide it at this point.

The server SHOULD ignore any data between v=iUt0 and the trailing newline; this enables space for future protocol negotiation (for example, the client in a potential new version could send v=iUt0,1) to signal compatibility with both version 0 and 1).

Message framing

All further messages take the following form:

[HEADER][MESSAGE]

The header is composed of two unsigned 32 bit integers.

struct MsgHeader {
    payload_len: u32,
    fd_count: u32,
}

The payload_len describes how many bytes to read for the rest of the message. The fd_count is how many file descriptors have been passed via SCM_RIGHTS.

These integers are written in host-native endianness; there is no provision or support for using this protocol across network boundaries. There are plenty of other protocols for that.

Further messages

After the initial "hello message" described above, the ordering of all further messages is under the control of the application. For example, it is likely that most applications would want to have the client also send an additional message at the time of connection, but this is not required. It's also valid to send and queue multiple messages, etc.

[cgwalters]

So this isn't urgent, but I have kind of narrowed in on this crate as what I'd like to use to de-duplicate a few ad-hoc IPC reimplementations I have in a few places.

Any opinions on giving me co-maintainership? How important is this crate to you? I could fork of course, but I was definitely hoping to share maintenance. (Though my SOP nowadays for crates is to transfer ownership to the coreos Github organization to be team-maintained).

[mitsuhiko]

I’m happy to add you to the crate. My only ask is to ensure the mac version continues working. Could easily be ensured if macOS is added to github actions.

[cgwalters]

Thanks, seems reasonable. I don't personally own a Mac but yeah CI should cover it. At least it's a Unix system. I think I said this earlier but I also in the past investigated https://github.com/servo/ipc-channel as an option but while that code is overall good too, it's heavily weighted by trying to also support Windows, and it predates the creation of nix/rustix for safe abstractions - and most important of all it also predates async rust. async is just nicer than channels.

For code I work on I tend to prefer having someone else review it. On my team @lucab is a go-to person for Rust reviews, I may just manually tag him for now.

[mitsuhiko]

Sounds good. Generally I'm also super happy to do reviews. I would try to ignore windows here. At a later point one can always do an abstraction on top of this and a hypothetical tokio-windows-ipc.

[cgwalters]

I've still been investigating things and recently came across the fact that Cap'n Proto now supports fd passing though not in the Rust bindings yet - but I may actually move in that direction instead.

(Cap'n Proto also has its own whole level of complexity, but it may eventually be warranted for some of the use cases I'm looking at)

[benwis]

I'd love to see and would be happy to help support some form of RPC over unix sockets! I'm not sure if Cap'n Proto or Protobuf better fits this use case, but I'd prefer protobuf just because it makes it easier to send over a web socket.