search

rust-lang / socket2

Advanced configuration options for sockets.
https://docs.rs/socket2
Apache License 2.0
682 stars 222 forks source link

Support for sending multicast packets on multiple interfaces with a single socket #86

Open bltavares opened 4 years ago

bltavares commented 4 years ago

Hi there @alexcrichton,

Thanks a lot for the project. It has helped me to get started with a multicast-based project, and it was amazing. I was able to send messages and find peers, until I couldn't anymore. I've dig deep into the depths of the network stack, to find, with amusement, that network is chaos.

One of my computers could find the other announcements, but the other computer would not find the first one. After some analyses, I realized this is related to having multiple interfaces, which is very common on laptops (wifi and cable) and Windows machines with HyperV (and WSL2). To my surprise, I could listen to multicast packets on any of the interfaces, and it would only send on a specific one instead of a broadcast to all.

socket2-rs already supports most of the calls to have a "Single socket, multi interface listeners". If we bind to 0.0.0.0, we can use Socket::join_multicast_v4 for each interface ip (enumerating it is outside the package scope).

As we bind to 0.0.0.0, selecting the interface to use for sending the packets is delegated to the OS, and it might end-up selecting the interface that does not talk to the other computers, leading to partial discovery and something that I find very often with my dual network computer.

I found this StackOverflow answer that shows that it would be possible to use the already available Socket::set_multicast_if_v4 call to determine which interface to use before sending the packet, so we can hop around to respond on the same interface that we received the packet.

As you may already know, the default socket API does not return the origin interface on the packet, which made me despaired to find alternatives and question my own sanity of why I'm shaving this yak. I was surprised to find some incantations using IP_PKTINFO and/or/~I'm not sure anymore~ IP_RECVIF to load the information from a "magic" place of the network stack, using CMSG_FIRSTHDR, CMSG_NXTHDR, CMSG_DATA, and WSARecvMsg on Windows.

It looks like this is what Avahi does the interface hopping already, in order to use a single socket to broadcast messages on all interfaces and locate peers happily ~until the next network topology breaks it again~.

My question here; TL; DR; Would it make sense for the socket2-rs project to expose this extra information from CMSG_DATA and WSARecvMsg?

I do think I could make something as a separate crate, but it would also benefit a lot from the already available Windows/Unix infrastructure to build this project on the sys.rs file. If you do think it fits this project, I could try to get something running, but I would also be super ok to ask to have this on another crate.

The alternative seems to be create a socket per interface and put a MultiSocket struct grouping them, which would avoid all this side-quest, but it is not resource efficient, and I'm afraid I would displease Ferris by opening too many sockets.

If you have any other suggestions from your experience, I would love to hear them as well! This is way out of my depth hehehe

Thank you a lot for your attention and time to read this up :)

(PS: I've tried to make this issue funny to read, and I'm sorry ahead of time if it ended up too verbose, or came up weird. This hobby project is leading to so many rabbit holes but I'm learning a lot!)

alexcrichton commented 4 years ago

Thanks for the explanation and the thorough issue report! I suspect that the bindings here are pretty simple so I think they'd work well in this crate. In general this crate is "mostly single-line wrappers around system functionality", so as long as it fits that bill it should be good to add.

bltavares commented 4 years ago

Hi @alexcrichton, thanks for the comment. I've started exploring the design space and doing something usable across platforms required many lines of unsafe code.

I've started to do that on a fork of socket2, but as it kept growing I felt like it would be a bit too big for the project. So I've moved those out to a new crate using some of the OS-specific traits to get access to the socket handle, which was great!

I've put the functionality on multicast-socket, and maybe there are a few things that could be added back here, such as socket.set_pktinfo(true), or exposing to_s_addr, which would help me reduce some copy-and-paste.

The actual sendmsg and recvmsg seems to be a bit more complex to be packaged as cross-patform generic code while trying to be a "single-line wrapper" and I wonder if it would be worth extracting out.

Thanks a lot for providing the AsRawFd and AsRawSocket crate, which allowed to access the underlying handler to wrap things on that nice little crate :) I've tested it on Android, Windows, Linux, Mac and even MIPS and I can do mdns discovery now!

Feel free to either close the issue or suggest any of the code to be contributed back that fits project. Thanks for the attention!

Thomasdezeeuw commented 3 years ago

@bltavares what concretely has to be done to solve this issue? In other words what is socket2 missing (on the master branch)?

bltavares commented 3 years ago

I think we could close the issue, as it is already possible to use the AsRawFd to access things from a socket2::Socket, but there are a few things that could be ported over to socket2, but I'm not sure if that is the intention of the crate.

This is not required to work, more of a list of ideas when using socket2 on this specific use-case. Given the crate can be used through the AsRawFd trait I would be ok to close this issue and only pick some of the ideas that match the crate goal :)

Thomasdezeeuw commented 3 years ago

I think we could close the issue, as it is already possible to use the AsRawFd to access things from a socket2::Socket, but there are a few things that could be ported over to socket2, but I'm not sure if that is the intention of the crate.

* Expose [to_s_addr](https://github.com/bltavares/multicast-socket/blob/d62ec76551b050364bacf4f9540492279644077c/src/win.rs#L443-L449) to avoid copy-and-paste

Is SockAddr::{family, len, as_ptr} sufficient for this? https://github.com/rust-lang/socket2/blob/9c4a404d5e44df1ecc56d62d35ee22109a886b5f/src/sockaddr.rs#L65-L78

* Add `socket.set_pkt_info(true)` that wraps [this piece](https://github.com/bltavares/multicast-socket/blob/d62ec76551b050364bacf4f9540492279644077c/src/win.rs#L443-L449) similarly to how `socket.set_reuse_address(true)` works

A pr for this would definitely be accepted.

* A more generic way to find socket extensions, such as [WSARecvMsgExtension](https://github.com/bltavares/multicast-socket/blob/d62ec76551b050364bacf4f9540492279644077c/src/win.rs#L150) on windows

* A `socket.bind_multicast` which abstracts [the differences between platforms](https://github.com/bltavares/multicast-socket/blob/d62ec76551b050364bacf4f9540492279644077c/src/win.rs#L158-L160)

These would be a bit harder but we can work on it. At the very least we could expose the OS specific functions.

* Expose [recvmsg](https://github.com/bltavares/multicast-socket/blob/d62ec76551b050364bacf4f9540492279644077c/src/unix.rs), and [sendmsg](https://github.com/bltavares/multicast-socket/blob/d62ec76551b050364bacf4f9540492279644077c/src/unix.rs#L184) (and on windows, the equivalents from [an extension method WSA*](https://github.com/bltavares/multicast-socket/blob/d62ec76551b050364bacf4f9540492279644077c/src/win.rs#L306)) to be able to make syscalls with more extensions

We expose Socket::recv_with_flags: https://github.com/rust-lang/socket2/blob/9c4a404d5e44df1ecc56d62d35ee22109a886b5f/src/socket.rs#L318-L328

And Socket::recv_vectored_with_flags: https://github.com/rust-lang/socket2/blob/9c4a404d5e44df1ecc56d62d35ee22109a886b5f/src/socket.rs#L363-L381

Or do you need access to msghdr structure?

JonathanPlasse commented 1 year ago