Allows real VPN tunnel over Hysteria

rgwan commented 8 months ago

Is your feature request related to a problem? Please describe. I want to run VXLAN or GRE over UDP tunnel over Hysteria 2. But these protocols're not connection-oriented protocol.

When I set encap-dport encap-sport option, when tunnel sends packet to the internet, it always send from encap-sport to encap-dport, when tunnel receives packet from the internet, it always receive from encap-dport. I don't know how to properly configure conntrack feature to allows GREoUDP or VXLAN reply to the request port.

Describe the solution you'd like Add an option specify a certain host address (maybe port range) to server side. Like:

persistTun:
  - listen: 4780-4790 # listen port, must to listen when the daemon starts
    remote: 127.0.0.4 # May need ACL or something
    bindIP: 127.0.0.1 # if request IP address/port match the port (listen) range and remote IP address, the UDP session must to bind to the bindIP, and source port = destination (request) port

Add these lines to the client-side config file:

udpForwarding:
  - listen: 127.0.0.3:4789 # On client-side
    remote: 127.0.0.4:4789 # Server-side destination
    timeout: 20s

And apply these configuration both on client side and server-side:

Client side:

Setup a VXLAN/GREoUDP tunnel, which localEP is 127.0.0.1:4789, destination EP is 127.0.0.3:4789

Sever side:

Setup a VXLAN/GREoUDP tunnel, which localEP is 127.0.0.4:4789, destination EP is 127.0.0.1:4789

When client wants to talk to server, client sends from local EP 127.0.0.1:4789 to destEP 127.0.0.3:4789, and Hysteria 2 create a new UDP session to server. Server forwards this initial packet from 127.0.0.1:4789 to 127.0.0.4:4789, reaches the another EP of tunnel.

When server wants reply to client, server sends from 127.0.0.4:4789 to 127.0.0.1:4789, and things going backwards.

Describe alternatives you've considered

Add these lines to the client-side config file:

udpForwarding:
  - listen: 127.0.0.3:4789 # On client-side
    remote: 127.0.0.4:4789 # Server-side destination
    remote-source: 127.0.0.1:4789 # Server-side source IP and address
    timeout: 20s

Additional context

rgwan commented 8 months ago

I made a dirty hack at here , and it works.

My setup is:

remote-side:

modprobe fou
ip fou add port 4789 ipproto 47 local 127.0.0.4
ip link add gre1 type gre remote 127.0.0.1 local 127.0.0.4 encap-dport 4789 encap-sport 4789 encap fou
ip link set gre1 up
ip addr add 10.97.0.1/30 dev gre1

client-side:

modprobe fou
ip fou add port 4789 ipproto 47 local 127.0.0.3
ip link add gre1 type gre local 127.0.0.3 remote 127.0.0.4 encap-dport 4789 encap-sport 4789 encap fou
ip link set gre1 up
ip addr add 10.97.0./30 dev gre1

Need to add these lines to your client-config yaml file:

udpForwarding:
  - listen: 127.0.0.4:4789
    remote: 127.0.0.4:4789
    timeout: 5s

But specify port range, session expiry (seems the timeout value is not work) and ACL needs lots of work. I'm not familiar with Go or Hysteria 2, Anyone interested in this please give me some help. Thank you all!

Session expiry maybe the biggest issue with HY2. Now it may not run without BFD, maybe causes persist occupation of source port. We must to figure a way to close (or re-use) the old UDP session in server when the new UDP session being initiate by client. Btw statically assigned port range is dirty, too. Maybe we should use destination IP address (like 127.0.0.4) to decide whether or not to establish VPN tunnel (Due to the limitation of Linux kernel itself, we have to configure tunnel's source port and dest port to the same port). So it needs some new attributes (vpn endpoint destination address, tunnel port binding address, acl to access VPN ...) in server's config file.

Or we can design a new request for establish a GRE/VXLAN tunnel, by executing some command (from script) after ListenUDP to create tunnel on the server side dynamically, so it wouldn't be limited by fixed port number at all. When client disconnects or reconnects, HY2 server side destroys the previous tunnel (maybe not necessary) and socket handle, and re-create it when user specify reconnect. Personally I think this solution is way more complex than the first one above it, I don't recommend it because real-VPN demand is not so important for most users.

rgwan commented 8 months ago

Btw it seems MTU discovery have some issue. Every packet longer than 1170 byte will be fragment to two packets. Packet capture on outgoing interface clearly shows that:

19:17:20.317224 IP B > A: UDP, length 1219
19:17:20.317260 IP B > A: UDP, length 1219
19:17:20.317278 IP A > B: UDP, length 40
19:17:20.317315 IP A > B: UDP, length 93
19:17:20.317324 IP A > B: UDP, length 93
19:17:20.317437 IP B > A: UDP, length 1219
19:17:20.317470 IP B > A: UDP, length 1219
19:17:20.317496 IP A > B: UDP, length 40
19:17:20.317518 IP B > A: UDP, length 1219

PMTUD in UDP tunnel may not works correctly.

haruue commented 8 months ago

Running a Layer 3 tunnel over Hysteria UDP is a really, really bad idea. Hysteria does nothing to speed up the UDP it forwards, nor does it speed up TCP in your Layer 3 tunnel.

If you want to speed up TCP in your "BGP network", please run Hysteria over your Layer 3 SDN instead.

If you just want to masquerade your layer 3 tunnel with QUIC, please note that

QUIC is stateful, there is always one server and one client, and it is not possible to establish a connection from server to client. In general, it is not a good choice for carrying a stateless protocol like GRE.
In your use case, I think you just need a UDP forwarder on the server side, which will bind to a fixed addrport for the forwarding socket to your tunnel server. I tried this with socat(1) but it did not work, so I just wrote one with go https://gist.github.com/haruue/d480071ce4278e120876784bcdb781f4

rgwan commented 8 months ago

Running a Layer 3 tunnel over Hysteria UDP is a really, really bad idea. Hysteria does nothing to speed up the UDP it forwards, nor does it speed up TCP in your Layer 3 tunnel.

If you want to speed up TCP in your "BGP network", please run Hysteria over your Layer 3 SDN instead.

If you just want to masquerade your layer 3 tunnel with QUIC, please note that
* QUIC is stateful, there is always one server and one client, and it is not possible to establish a connection from server to client. In general, it is not a good choice for carrying a stateless protocol like GRE.

* In your use case, I think you just need a UDP forwarder on the server side, which will bind to a fixed addrport for the forwarding socket to your tunnel server. I tried this with `socat(1)` but it did not work, so I just wrote one with go https://gist.github.com/haruue/d480071ce4278e120876784bcdb781f4

Well I just want to need a proper anti-censor VPN, which allows MPLS or SRv6 payload, and keep the encapsulation cost as low as possible. And I don't mind that issue (server can't initiate a connection to client), like OpenConnect.

Of course a proper VPN tunnel doesn't have the ability to tolerant any packet loss/buffer bloat or other network loss. It's a problem in some use case. On the other hand, WireGuard over Hysteria 2 is much more costly (double encryption for nothing) than GRE over Hysteria 2. So I really don't have much choice.

Use another daemon to forward UDP traffic to the GRE over UDP endpoint is a way to implement this, but it increase the latency, context switch cost, and so on. Implement some persist UDP session in Hysteria 2 is simpler than use another daemon.

Considering some really stingy cloud provider offers 1/2 core VPS, so I perfer to implement this as a built-in function.

But if I add some optimization like FEC for better performance, it might be not easy to integrate it to Hysteria 2. So let us continue the discussion and find out a better way.

rgwan commented 8 months ago

Running a Layer 3 tunnel over Hysteria UDP is a really, really bad idea. Hysteria does nothing to speed up the UDP it forwards, nor does it speed up TCP in your Layer 3 tunnel.

If you want to speed up TCP in your "BGP network", please run Hysteria over your Layer 3 SDN instead.

If you just want to masquerade your layer 3 tunnel with QUIC, please note that
* QUIC is stateful, there is always one server and one client, and it is not possible to establish a connection from server to client. In general, it is not a good choice for carrying a stateless protocol like GRE.

* In your use case, I think you just need a UDP forwarder on the server side, which will bind to a fixed addrport for the forwarding socket to your tunnel server. I tried this with `socat(1)` but it did not work, so I just wrote one with go https://gist.github.com/haruue/d480071ce4278e120876784bcdb781f4

By the way I don't know why Hysteria 2 only generates QUIC packet shorter than 1219 bytes. It might be a bug about PMTUD.

egg1234 commented 8 months ago

可以考虑使用 https://github.com/ginuerzh/gost 项目的gost生成tap/tun接口，使用用hysteria做udp端口转发，以下方案只使用gost 2.11.1版，其他gost版本没有测试过

remote服务器生成tap接口的命令 ./gost -L=tap://:4321?net=172.26.0.2/24 or ./gost -L=tun://:4321?net=172.26.0.2/24

remote服务器配置hysteria服务器端

ip a 命令检查虚拟接口状态

生成tap接口的本地客户端命令 ./gost -L=tap://:0/本机网卡的ip地址:4321?net=172.26.0.1/24 or ./gost -L=tun://:0/本机网卡的ip地址:4321?net=172.26.0.1/24

本地客户端的hysteria配置udp端口转发（因为gost的tap/tun接口转发只能使用udp包）

"relay_udps": [
  {
  "listen": "127.0.0.1:4321",
  "remote": "服务器网卡的ip地址（如果是类似GCP那种网卡是私网段的直接使用这个私网段地址）:4321",
  "timeout": 300
  }
]

ip a 命令检查虚拟接口状态

gost生成的tap/tun接口没有二次加密，作用类似GRE隧道，经过iperf3的测速，tap接口的转发性能比tun接口高几倍，以上方案使用hysteria 1.3.5搭建并使用端口跳跃，hysteria 2.x没有试过

apernet / hysteria

Allows real VPN tunnel over Hysteria #963