setsockopt(2) and reuse

[steveklabnik]

Issue by AngusL Sunday Jul 20, 2014 at 17:56 GMT

For earlier discussion, see https://github.com/rust-lang/rust/issues/15835

This issue was labelled with: A-libs in the Rust repository

---

I see there's been discussion of SO_REUSEADDR in the past. Mooted was that Windows allows later sockets to set the option and bind to endpoints which already have sockets without the option set, permitting a partial denial of service. (Later mitigated somewhat by SO_EXCLUSIVEADDRUSE.) TcpListener::bind currently uses this option to ignore the TIME_WAIT state, allowing faster re-binding. This introduces more-or-less the same problem in exchange for a benefit that's probably more useful in development than production.

There are other platform variations which make it difficult to find a unifying interface, particularly for TCP. In lieu of such an interface, I'd suggest exposing setsockopt(2) - even from an obviously platform-specific namespace. My motivation is a multicast discovery protocol, where I'd like multiple agents on one machine to be able to see each other. As far as I can tell, I need to either handle this separately or reinvent much of the UdpSocket wheel - which would be a shame.

[tomwhoiscontrary]

@AngusL opens:

I see there's been discussion of SO_REUSEADDR in the past.

I don't know what this refers to. If anyone remembers, could we have a link to it?

[tomwhoiscontrary]

I'd never heard of that Windows option, so here's Microsoft's advice on using SO_REUSEADDR and SO_EXCLUSIVEADDRUSE. Edited highlights:

The SO_REUSEADDR option has very few uses in normal applications aside from multicast sockets where data is delivered to all of the sockets bound on the same port. Otherwise, any application that sets this socket option should be redesigned to remove the dependency since it is eminently vulnerable to "socket hijacking". [...] All server applications must set SO_EXCLUSIVEADDRUSE for a strong level of socket security. Not only does it prevent malicious software from hijacking the port, but it also indicates whether or not another application is bound to the requested port.

So, AIUI, on Windows, multicast sockets where you want to share the port should have SO_REUSEADDR set, and all other sockets should have SO_EXCLUSIVEADDRUSE set.

[tomwhoiscontrary]

Personally, at the moment, i'm on UNIX, and i'd like to set SO_REUSEADDR on my multicast sockets. With the standard library as it stands, there's no way to do this, because sys_common::net::UdpSocket::bind goes straight from creating the socket to binding it:

    let sock = Socket::new(addr, c::SOCK_DGRAM)?;
    let (addrp, len) = addr.into_inner();
    cvt(unsafe { c::bind(*sock.as_inner(), addrp, len) })?;

I would love to have a way to set SO_REUSEADDR here, either specifically, or via a more general way to set options before binding.

Meta-question: what is the bar to making changes to this code? If i come up with an idea that sounds okay and submit a PR, is there a chance it will be merged? Or does this need to go through some kind of process?

Off the top of my head, i would suggest that the existing bind method becomes a method specifically for binding non-reusable sockets, and that both std::net::UdpSocket and sys_common::net::UdpSocket grow a new method called something like bind_reusable, which creates a reusable socket. The non-reusable bind should probably set SO_EXCLUSIVEADDRUSE on Windows; the reusable bind should set SO_REUSEADDR, and perhaps also SO_REUSEPORT on Linux.

Alternatively, we could have the two bind methods on std::net::UdpSocket, but change the existing bind method on sys_common::net::UdpSocket to take a boolean flag or enum specifying reusability. I don't know to what extent you consider sys_common an implementation detail as opposed to an API.

Thoughts?

[drrlvn]

Another use-case: since connect() can return EADDRNOTAVAIL when no ephemeral ports are available I would like to set SO_REUSEADDR on the socket before connect()ing, but there's currently no way to do that.

[phhusson]

I think UDP sockets should have the same behaviour as TCP and simply reuse port everytime.

I agree it's a little weird TCP ignores TIME_WAIT, but anyway, this decision has already been made a long time ago, and it looks like noone had issue with it.

[keepsimple1]

Have there been any follow-ups on this topic? I'm using multicast UDP socket as well (in fact, published mdns-sd crate) and would love to see std::net supports SO_REUSEADDR at least for UDP sockets.

[jnordwick]

I think some people here might be confusing SO_REUSEADDR and SO_REUSEPORT.

SO_REUSEADDR is used mainly for bypassing the TIME_WAIT on the side initiating the close when the other side doesn't respond. The side receiving the close message does not have this problem, and it used for TCP, not UDP since UDP doesn't have close termination messages - or any other control messages. It allows the initiating side to create a connection using the same local address/port pair while even when in TIME_WAIT, except not to the same remote address/port pair.

SO_REUSEPORT is completely different. It is used mostly for UDP sockets to allow multiple programs to list to the same local address/port pair. It is most used for load balancing across threads and also to allow multiple multicast listeners on the same machine as sender. The first socket bound there must have the option supplied and al the others must also supply it as well as have the same effective UID. There is a TCP use for SO_REUSEPORT. It is used to load balance accept calls.

(So it isn't actually this clear in practice since the flags come from different lineages. Under multicast they can be the same, SO_REUSEADDR also has some notes in terms of wait is the same address w.r.t. INADDR_ANY, and of course implementation differences).

@keepsimple1 What were you needing to use SO_REUSEADDR for UDP? You can use SO_REUSEPORT (but still not possible in std::net as you are trying to fix already).

[tomwhoiscontrary]

@jnordwick It may be worth noting that SO_REUSEADDR also allows processes to share ports. Indeed, it is the traditional and, i believe, only POSIX-compliant way to do this. SO_REUSEPORT is Linux-specific, and was added in 3.9, a mere nine years ago. FreeBSD has a SO_REUSEPORT, but i think it does something a bit different. I have no idea what MacOS has.

But yes, anyone on Linux today almost certainly wants SO_REUSEPORT for the specific use cases discussed.

Hence why i thought it was worthwhile creating a slightly higher-level API here, where you call a bind function appropriate to your use case, and it sets the right options for whatever platform you're on. Hey, on FreeBSD maybe that involves PCBGROUP, who knows.

[jnordwick]

SO_REUSEPORT is Linux-specific, and was added in 3.9, a mere nine years ago.

I didn't know it was so recent. I thought I remember using it before then, but might have been languishing in svn or I might not remember how recent that was.

I think on FreeBSD you use SO_REUSEPORT_LB to get similar behavior to Linux. I have done some network programming but not a lot (or very recently) on FreeBSD.

Here is an excellent rundown I found: https://stackoverflow.com/a/14388707/701211

To do a good BSD sockets layer with rust-like semantics and values, I agree you need to lift the API little. I dont know if higher is the right word, but definitely sideways. There are too many platform difference I think once you progress beyond the basics. Ever things as simple as sockopts behave differently or have to be called at different times (eg, before or after an accept() to stick) so code needs to be aware of this unless the runtime can somehow manage that.

Now, I write a lot of high performance, very low latency network code on Linux, and mostly on epoll, but lately also io_uring (set aside until it improves: io_uring + MSG_ZEROCOPY = pain) and PACKET_MMAP w/ eBPF (think io_uring without the rings). Maybe Rust shjould just go all out, support a workable BSD sockets and write new socket api that isnt' driven by BSD sockets and dosn't have to be tied to their cruft. open/close/send/recv isn't all there is out there. You shouldn't even need a read call.

[keepsimple1]

@jnordwick

@keepsimple1 What were you needing to use SO_REUSEADDR for UDP? You can use SO_REUSEPORT (but still not possible in std::net as you are trying to fix already).

Actually I use both in my application as it runs on both Unix-like platforms and Windows.