Multiple Public IPv4s on the server don't work

Describe the bug On a server with 2 ipv4s on eth0, you can only connect to the hysteria server using one of the ips.

To Reproduce Bind an additional ipv4 to your server, use ip addr add to add the ip to et0. Check via both ssh and some running sing-box proxy that both ips work as expected on the server. Now try to connect to the hysteria server. By my experiments, the following happens:

in your ip route settings the default has src specified which then you can only connect to hysteria with that said src
your ip route default has no src specified, which then only the first ip listed in ip addr list under eth0 can connect.
you manually specify the listen address in the hysteria server config to the public ip you want and then that ip works. originally it was empty to allow all ipv4 and ipv6 entries but that did not allow the second ipv4, only the by changing listen to second_ip:443 was I able to connect with the second ip.

IPv6 is not affected, I could connect with ipv6 to the server no matter the case.

Expected behavior When listen ip is not set, any and all ipv4 public addresses should be able to be used to connect to the server.

Logs I checked hysteria logs and no info of the client popped up using the second ip as if I wasn't even connecting to the server. However I have confirmed that the ip pings and I can work with other ports and services on the server using it.

Device and Operating System Ubuntu 24.04 LTS x86_64

Additional context My current workaround is running another config on a different port and specifying listen ip and connecting with the second ip to that config. I also saw a mention of this problem in sing-box github issues however no info was provided there as the issue was pushed onto hysteria. I am using hysteria core {"version": "v2.5.0", "platform": "linux", "arch": "amd64", "channel": "release"}.

Hello,

Unlike TCP, UDP have no accept(2) and no session sockets. Therefore, for a UDP socket, there is no significant difference between replying to a packet and sending a packet to a new destination.

When a UDP socket sends a packet, the source address would be the address it bind(2) to, if it bind(2) to INADDR_ANY (0.0.0.0), the source address would be determined by routine selection.

You can use ip route get to perform a routing selection. For an example, to find the source address to be used when sending a UDP packet to 8.8.8.8, you can use the following command:

ip route get 8.8.8.8

The output would be like

8.8.8.8 via 192.0.2.65 dev eth0 src 192.0.2.1 uid 0
    cache

In this example, the 192.0.2.65 is referred to as the preferred source address, or prefsrc. As for a UDP socket bind(2) on INADDR_ANY, if the interface eth0 has multiple addresses configured, the UDP packet sent to other addresses will respond with the prefsrc.

Due to the limitations of Berkeley sockets (and Linux syscalls), it is impossible to set the source address per packet. In fact, since QUIC take this into account, it is only when there is a symmetric NAT on the client side, or one of the addresses is blocked by the GFW, that a difference between the request address and the response address can cause connection problems.

To solve this problem, some UDP servers (especially DNS servers) use multiple sockets that listen on all known addresses, ensuring that the request address is always the same as the response address. But implementing such a mechanism for quic-go would be inefficient. I'd recommended solving this problem with DNAT.

For example, for the above output of ip route get command and the case of hysteria port is 443. You can use the following iptables rule.

iptables -t nat -A PREROUTING -i eth0 -p udp --dport 443 -j DNAT --to-destination 192.0.2.65:443

Then the conntrack will take care of the source address on the response packet.

Hello,

Unlike TCP, UDP have no accept(2) and no session sockets. Therefore, for a UDP socket, there is no significant difference between replying to a packet and sending a packet to a new destination.

When a UDP socket sends a packet, the source address would be the address it bind(2) to, if it bind(2) to INADDR_ANY (0.0.0.0), the source address would be determined by routine selection.

You can use ip route get to perform a routing selection. For an example, to find the source address to be used when sending a UDP packet to 8.8.8.8, you can use the following command:
ip route get 8.8.8.8
The output would be like
8.8.8.8 via 192.0.2.65 dev eth0 src 192.0.2.1 uid 0
    cache
In this example, the 192.0.2.65 is referred to as the preferred source address, or prefsrc. As for a UDP socket bind(2) on INADDR_ANY, if the interface eth0 has multiple addresses configured, the UDP packet sent to other addresses will respond with the prefsrc.

Due to the limitations of Berkeley sockets (and Linux syscalls), it is impossible to set the source address per packet. In fact, since QUIC take this into account, it is only when there is a symmetric NAT on the client side, or one of the addresses is blocked by the GFW, that a difference between the request address and the response address can cause connection problems.

To solve this problem, some UDP servers (especially DNS servers) use multiple sockets that listen on all known addresses, ensuring that the request address is always the same as the response address. But implementing such a mechanism for quic-go would be inefficient. I'd recommended solving this problem with DNAT.

For example, for the above output of ip route get command and the case of hysteria port is 443. You can use the following iptables rule.
iptables -t nat -i eth0 -p udp --dport 443 -j DNAT --to-destination 192.0.2.65:443
Then the conntrack will take care of the source address on the response packet.

Greetings, sorry for the late reply. If I've understood correctly and by the things I've read the past few days, since udp connections are stateless, source ip can't be set per packet as you've said, and therefore it is theoretically impossible to send a packet to a client specifically using the ip that client connected with since that source address cannot be set for the packet the server is sending per each packet nor would it be efficient. So in the case that one ip is inaccessible to a client and the server tries to send packets via that ip by iptables rules then no connection will be established. This is the gist of what I understood, please correct me if I'm wrong. If so I believe that my current workaround would be the only logical solution as to changing iptables rules would still be the same as only setting one ip as the output source of the server, so it really is necessary to have multiple instances running on different ports for the multiple ips for the result I'm looking for. I did try changing iptables rules so that the source would be the preferred source of device eth0, in which case was 172.31.1.1 but to no avail, the result output source ip was still the IP address in ip addr list with the lesser metric value.

If you could confirm that I have understood correctly and there's no solution to what I'm looking for, I will close this issue.

Thank you.

Sorry, I wrote the above answer on a phone, so I made a mistake in the iptables command.

It should be

iptables -t nat -A PREROUTING -i eth0 -p udp --dport 443 -j DNAT --to-destination 192.0.2.65:443

Don't forget to change the eth0 and 192.0.2.65 to the values from the output of the ip route get command. Referring to a non-existent interface or address will not cause an error, it just won't work.

Sorry, I wrote the above answer on a phone, so I made a mistake in the iptables command.

It should be
iptables -t nat -A PREROUTING -i eth0 -p udp --dport 443 -j DNAT --to-destination 192.0.2.65:443
Don't forget to change the eth0 and 192.0.2.65 to the values from the output of the ip route get command. Referring to a non-existent interface or address will not cause an error, it just won't work.

Indeed that is the command I used, I knew I was supposed to append it to PREROUTING chain However after doing ip route get and finding the preferred source to be 172.31.1.1 and then doing the command as

iptables -t nat -A PREROUTING -i eth0 -p udp --dport 443 -j DNAT --to-destination 172.31.1.1:443

There was no difference in behavior, If I added the secondary ip into the listen parameter of hysteria config, then only the second public ip worked and If I left the listen ip empty only the first public ip worked.

The tests are done on two ISPs, one which blocks the second ip and allows the first one, and one which blocks the first ip and allows the second one.

I'll restate the behavior I wish to happen just in case, as I'm pretty sure this setup you mentioned would work in ordinary circumstances. My problem is that I have clients on ISPs that only one of the two public ipv4s are unblocked, so I need the server to communicate with a client only using the public ip that the client used to connect to the server. However from what I've seen the server chooses the output public ip of it's packets based on ip route rules that can't be influenced per packet due to the nature of udp not being a roundtrip packet that keeps the data of source ip used in communication. As you mentioned on your first reply this only happens with udp as tcp packets of the server on other services work exactly as expected.

Are the two IPs (of two ISPs) on two interfaces (e.g. eth0 & eth1) or two IPs on a single interface?

Are the two IPs (of two ISPs) on two interfaces (e.g. eth0 & eth1) or two IPs on a single interface?

I see that I was too vague in my explanation of the situation so far, I apologize for that. Allow me to explain in detail:

I have a cloud server from hetzner running as the hysteria server. It has two public ipv4s assigned to it, both on eth 0, as I attach the result of ip addr list below;

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 96:00:03:91:85:ca brd ff:ff:ff:ff:ff:ff
    inet 78.46.245.171/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet 188.245.88.209/32 metric 100 scope global dynamic eth0
       valid_lft 82602sec preferred_lft 82602sec
    inet6 2a01:4f8:c012:b45f::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::9400:3ff:fe91:85ca/64 scope link
       valid_lft forever preferred_lft forever

Let's call 78.46.245.171 as the first ip and 188.245.88.209 as the second ip. Now I've confirmed both ips are accessible from the internet. I am running two instances of hysteria with identical configurations on different ports one is listening on port 443 with no listen address specified the other is listening on port 449 with listen address set to 188.245.88.209.

The following is the result of ip route list:

default via 172.31.1.1 dev eth0 proto dhcp
default via 172.31.1.1 dev eth0 proto dhcp src 188.245.88.209 metric 100
172.31.1.1 dev eth0 proto dhcp scope link src 188.245.88.209 metric 100
185.12.64.1 via 172.31.1.1 dev eth0 proto dhcp src 188.245.88.209 metric 100
185.12.64.2 via 172.31.1.1 dev eth0 proto dhcp src 188.245.88.209 metric 100

The first default rule I add manually at the start of the hysteria service as to allow the first ip to act as the primary ip address of the server. Without it, the server instance on port 443 does not communicate with my client via the first ip, which I believe is because that the server tries to respond to my client via the second ip which is blocked by my home internet isp. Therefore I had to change the ip route rule so that it responds to all connections with the first ip for it to work. However this caused me another problem, Now on my mobile network isp which blocks the first ip and allows the second ip I cannot connect to the server instance on port 443. This was not an issue until I added the ip route rule which to me implies that the server was responding to all connections via the second ip. To fix this issue temporarily I created another instance on the server on port 449 specifically listening on the second ip and that made it work on my mobile network implying that the server on port 449 only listens to the second ip and answers with that ip as well.

Now I wish for this to be resolved on just one instance of the server. Without needing to specify listen ip in the config, I want all clients that connect to the server via the first ip to get responds from that same ip and the same behavior for the second ip. However I think due to the nature of udp not being a roundtrip packet, this is impossible as the server has no idea this result that it is sending to some client should exit via which ip, so it just uses the ip route rules which would give it the ip with the lower metric, i.e. the first ip.

I hope this clarified the situation I am facing. Sadly I do not have access to a network that has both IPs unblocked as to check whether the server on port 443 is functioning whenever both IPs are available or not, but that would not solve the issue with my current ISPs to begin with.

I've tested the iptables rules against this use case (two IPs on one interface), and it works on my end.

Maybe you can also try this, which works for both conditions:

iptables -t nat -A PREROUTING -d $secondary_address -p udp --dport 443 -j DNAT --to-destination $primary_address:443

If this still does not work, there may be some other rules preventing it from being effective, please run iptables-save to get a dump of any existing rules and paste them here.

I've tested the iptables rules against this use case (two IPs on one interface), and it works on my end.

Maybe you can also try this, which works for both conditions:
iptables -t nat -A PREROUTING -d $secondary_address -p udp --dport 443 -j DNAT --to-destination $primary_address:443
If this still does not work, there may be some other rules preventing it from being effective, please run iptables-save to get a dump of any existing rules and paste them here.

Thank you so much, this indeed fixed the problem. Inspired by your solution I changed the scripts for port hopping that is run before the server is up and added the primary address at the end of the DNAT rule specified there without the need of the -d parameter and it worked as well. I thank you for the time you spent for this issue and the guidance you gave me.

apernet / hysteria

Multiple Public IPv4s on the server don't work #1169