search

cloudflare / tableflip

Graceful process restarts in Go
BSD 3-Clause "New" or "Revised" License
2.88k stars 147 forks source link

Support binding to the same address with SO_REUSEPORT #45

Closed john-cai closed 4 years ago

john-cai commented 4 years ago

This allows a second listener to bind to the same address. fixes: https://github.com/cloudflare/tableflip/issues/46

lmb commented 4 years ago

Hi John, thanks for your contribution!

I was reading up on SO_REUSEPORT, and found the following:

To prevent "port hijacking", there is one special limitation: All sockets that want to share the same address and port combination must belong to processes that share the same effective user ID! So one user cannot "steal" ports of another user. This is some special magic to somewhat compensate for the missing SO_EXCLBIND/SO_EXCLUSIVEADDRUSE flags.

This means that enabling SO_REUSEPORT unconditionally can lead to two unrelated applications sharing the same address. Another quirk is that multiple calls to Upgrader.Listen with the same arguments will succeed, which might confuse the library. Finally, there is a race condition that means you will lose connections, see https://github.blog/2016-12-01-glb-part-2-haproxy-zero-downtime-zero-delay-reloads-with-multibinder/#haproxy-almost-safe-reloads for details.

The correct thing would be to use SO_REUSEADDR, if only Linux had BSD semantics. As it stands it doesn't work:

$ socat - tcp-l:1234,reuseaddr &
$ socat - tcp-l:1234,reuseaddr,bind=127.0.0.1
2020/03/26 11:39:56 socat[12138] E bind(5, {AF=2 127.0.0.1:1234}, 16): Address already in use

So as far as I can see, it's not possible to do a seamless transition from specific IP to INADDR_ANY :(

If you still want to use SO_REUSEPORT despite the drawbacks I mentioned, we could add a way for you to specify a custom control function or maybe even a custom dialer.

lmb commented 4 years ago

@john-cai ping. Any thoughts?

lmb commented 4 years ago

Ping @john-cai

john-cai commented 4 years ago

thanks @lmb ! Yeah that's a good find. Given the security concern, I think we'll try to work around this issue on our end.

john-cai commented 4 years ago

we could add a way for you to specify a custom control function or maybe even a custom dialer.

@lmb actually, after trying to work around this issue on our end, we feel that maybe it'd be easier if we could use a custom dialer. Is that something you're still open to?

lmb commented 4 years ago

Sure, a custom dialer sounds good to me. Can you do that in a new PR though?