Open awptechnologies opened 8 months ago
@awptechnologies
I tried to host hbbs/hbbr and a minimalist admin server in a Kubernetes cluster.
It uses Haproxy 2.7 as the ingress proxy and a modified Haproxy-ingress-controler 1.10 as the Haproxy controler.
First issue is that Haproxy cannot handle udp -except for quic- and it probably never supports it.
Second issue hbbs will see the IP address of Haproxy and shows a log like this one:
2024-03-26T07:41:22.860694378Z [2024-03-26 07:41:22.860521 +00:00] DEBUG [src/rendezvous_server.rs:1097] Tcp connection from [::ffff:172.28.5.218]:58786, ws: false
2024-03-26T07:41:22.861155579Z [2024-03-26 07:41:22.861075 +00:00] DEBUG [src/rendezvous_server.rs:1137] Tcp connection from [::ffff:172.28.5.218]:58786 closed
but this is the generated haproxy config:
frontend https
mode http
bind 0.0.0.0:443 name v4 crt /etc/haproxy-ingress/certs/frontend ssl alpn h2,http/1.1
bind [::]:443 name v6 crt /etc/haproxy-ingress/certs/frontend ssl alpn h2,http/1.1
http-request set-var(txn.base) base
http-request set-var(txn.path) path
http-request set-var(txn.host) req.hdr(Host),field(1,:),lower
http-request set-var(txn.host_match) var(txn.host),map(/etc/haproxy-ingress/maps/host.map)
http-request set-var(txn.host_match) var(txn.host),regsub(^[^.]*,,),map(/etc/haproxy-ingress/maps/host.map,'') if !{ var(txn.host_match) -m found }
http-request set-var(txn.path_match) var(txn.host_match),concat(,txn.path,),map(/etc/haproxy-ingress/maps/path-exact.map)
http-request set-var(txn.path_match) var(txn.host_match),concat(,txn.path,),map_beg(/etc/haproxy-ingress/maps/path-prefix.map) if !{ var(txn.path_match) -m found }
http-request auth realm Protected-Content if { var(txn.path_match) -m dom 706d41ac94352467187ae04f79c5413b } !{ http_auth_group(briac-code-server-hcf-coder-ingress-coder-briac-cf) authenticated-users }
http-request auth realm Protected-Content if { var(txn.path_match) -m dom 1a8a05768edae0436e25b1e49cc824c2 } !{ http_auth_group(longhorn-system-longhorn-ui) authenticated-users }
http-request auth realm Protected-Content if { var(txn.path_match) -m dom 67785cfd7a61cf6cf21ad6ac536622cd } !{ http_auth_group(monitoring-prometheus-kube-prometheus-prometheus) authenticated-users }
http-request set-header X-Forwarded-Proto https
use_backend %[var(txn.path_match),field(1,.)]
default_backend _default-local-service_http
frontend sctgdesk-hbbr-hbbs-gateway-port-21115
mode tcp
bind 0.0.0.0:21115 name v4
bind [::]:21115 name v6
option tcplog
default_backend sctgdesk_port-21115
frontend sctgdesk-hbbr-hbbs-gateway-port-21116
mode tcp
bind 0.0.0.0:21116 name v4
bind [::]:21116 name v6
option tcplog
default_backend sctgdesk_port-21116
frontend sctgdesk-hbbr-hbbs-gateway-port-21117
mode tcp
bind 0.0.0.0:21117 name v4
bind [::]:21117 name v6
option tcplog
default_backend sctgdesk_port-21117
frontend sctgdesk-hbbr-hbbs-gateway-port-21118
mode tcp
bind 0.0.0.0:21118 name v4
bind [::]:21118 name v6
option tcplog
default_backend sctgdesk_port-21118
frontend sctgdesk-hbbr-hbbs-gateway-port-21119
mode tcp
bind 0.0.0.0:21119 name v4
bind [::]:21119 name v6
option tcplog
default_backend sctgdesk_port-21119
backend sctgdesk_port-21116
mode tcp
default-server check
server SRV_1 172.28.2.86:21116 enabled
backend sctgdesk_port-21117
mode tcp
default-server check
server SRV_1 172.28.2.86:21117 enabled
backend sctgdesk_port-21118
mode tcp
default-server check
server SRV_1 172.28.2.86:21118 enabled
backend sctgdesk_port-21119
mode tcp
default-server check
server SRV_1 172.28.2.86:21119 enabled
backend sctgdesk_sctgdesk-api-server-service_port-21114
mode http
balance roundrobin
option forwardfor
no option abortonclose
default-server check
server SRV_1 172.28.5.91:21114 enabled
I also added a iptables rule for forwarding udp:
iptables -t nat -A PREROUTING -i enp0s6 -p udp --dport 21116 -j DNAT --to-destination 172.31.255.221:21116
But yet it doesn't work… I studied the possibility to use Haproxy v2 protocol. It needs a small modification of the code. I made some tests at https://github.com/sctg-development/proxyv2-test/blob/main/src/main.rs#L47-L89 …
In fact it will be better to study the possibility to use the Websockets. I use the RustDesk web it uses websockets on ports 21118/21119. I just modified the code for using secure websockets on 21120/21121 . It is easy to proxy websockets and to proxy secure websockets to websockets. hbbs/hbbr use tokio_tungstenite for getting the inner stream via websocket, we'll need to add that support on RustDesk
I made a fork publically available with my test https://github.com/sctg-development/sctgdesk-server I don't have yet enough time to test but docker image are built at https://hub.docker.com/search?q=eltorio%2Frustdesk-server and the built package are at https://github.com/sctg-development/sctgdesk-server/releases But as I wrote before Haproxy cannot handle UDP, you need something else.
Another idea may use QUIC which can be enabled in HAProxy, I saw that the hbb_common code contains a quic feature. Currently it does not compile.
Reverse proxy is supported from what i can tell in the research i have done. i am having troubles setting up a relay server that is accessible from outside my network through the reverse proxy.