Quilkin not proxying on IPv4

thijsvanloef commented 6 months ago

What happened: I want to use a simple quilkin proxy to proxy request from a VPS to a self hosted server in my home network. The application is running inside a docker container that exposes port 8211 over IPv4(udp).

While trying to proxy the request the request seems to never "hit" quilkin. tcpdump udp port 8211 on the host with quilkin shows packages arriving. But quilkin never proxies them to the configured host.

What you expected to happen:

Quilkin proxying the traffic to the service host

How to reproduce it (as minimally and precisely as possible):

Start Quilkin: quilkin -p 8211 -t other_host:8211
Try to connect to the server via Quilkin: nc -u quilkin_host 8211
Quilkin does not proxy the traffix

Anything else we need to know?:

Environment:

Quilkin version: 0.7.0
Execution environment (binary, container, etc): Binairy
Operating system: Ubuntu 22.04.3 on Intel based CPU and Self compiled Ubuntu 22.04.3 on ARM64 host
Custom filters? (Yes/No - if so, what do they do?): No
Log(s):

```log ./x86_64-unknown-linux-gnu/release/quilkin proxy -p 8211 -t REDACTED:8211 2024-03-11T20:34:49.163947Z INFO quilkin::cli: src/cli.rs: Starting Quilkin version="0.7.0" commit="4ad8942a41cbe6cc2823602fade0dedba49f71d2" 2024-03-11T20:34:49.164334Z INFO quilkin::cli::admin: src/cli/admin.rs: Starting admin endpoint address=[::]:8000 2024-03-11T20:34:49.164814Z INFO quilkin::cli::proxy: src/cli/proxy.rs: Starting port=8211 proxy_id="REDACTED" 2024-03-11T20:34:49.165288Z INFO quilkin::cli::proxy: src/cli/proxy.rs: Quilkin is ready ```
Others:

Netstat on Quilkin host ```log REDACTED@REDACTED:~$ netstat -a | grep 8211 udp6 0 0 [::]:8211 [::]:* udp6 0 0 [::]:8211 [::]:* udp6 0 0 [::]:8211 [::]:* udp6 0 0 [::]:8211 [::]:* ``` Netstat on service host ```log REDACTED@REDACTED:~$ netstat -a | grep 8211 udp 0 0 0.0.0.0:8211 0.0.0.0:* ```

Please let me know if this is intended behavior, or if you need more information.

markmandel commented 6 months ago

other_host and quilkin_host - what address are you using for that? A DNS entry, an ipv4 address? Something else?

markmandel commented 6 months ago

Also What happens if you turn on more verbose logging with the proxy? https://googleforgames.github.io/quilkin/v0.7.0/book/deployment/admin.html?highlight=quilkin%3Dtrace#logging

thijsvanloef commented 6 months ago

Thanks for your quick response!

Apologies it seems like a made a mistake with the version, I'm using 0.8.0-dev, but it does not seem to work on 0.7.0 either.

other_host and quilkin_host - what address are you using for that? A DNS entry, an ipv4 address? Something else?

palworld game -> palworld.domain.tld:8211 -> quilkin_host -> ipv4address:8211(other_host)

Also What happens if you turn on more verbose logging with the proxy?

REDACTED@REDACTED:~$ quilkin proxy -p 8211 --to ip_address:8211
2024-03-11T21:10:36.185757Z  INFO quilkin::cli: src/cli.rs: Starting Quilkin version="0.8.0-dev" commit="dddc9b2203b6a66f1bfc4f448fa3912fde78ae8c"
2024-03-11T21:10:36.185803Z DEBUG quilkin::cli: src/cli.rs: config parameters cli=Cli { no_admin: false, config: "quilkin.yaml", admin_address: None, quiet: false, command: Proxy(Proxy { management_server: [], mmdb: None, port: 8211, qcmp_port: 7600, to: [ip_address:8211], idle_request_interval_secs: 30, workers: None }), log_format: Auto }
2024-03-11T21:10:36.185903Z DEBUG quilkin::cli: src/cli.rs: provided path not found path=quilkin.yaml
2024-03-11T21:10:36.185920Z DEBUG quilkin::cli: src/cli.rs: /etc path not found path=quilkin.yaml
2024-03-11T21:10:36.186035Z  INFO quilkin::cli::admin: src/cli/admin.rs: Starting admin endpoint address=[::]:8000
2024-03-11T21:10:36.186229Z TRACE run: quilkin::config::watch: src/config/watch.rs: change detected watchers=0
2024-03-11T21:10:36.186250Z  INFO run: quilkin::cli::proxy: src/cli/proxy.rs: Starting port=8211 proxy_id="saturn"
2024-03-11T21:10:36.186890Z  INFO run: quilkin::cli::proxy: src/cli/proxy.rs: Quilkin is ready
2024-03-11T21:10:36.186987Z TRACE quilkin::config::slot: src/config/slot.rs: storing new value
2024-03-11T21:11:36.188362Z TRACE quilkin::config::slot: src/config/slot.rs: storing new value
2024-03-11T21:12:36.189635Z TRACE quilkin::config::slot: src/config/slot.rs: storing new value
2024-03-11T21:13:36.190140Z TRACE quilkin::config::slot: src/config/slot.rs: storing new value
2024-03-11T21:14:36.191195Z TRACE quilkin::config::slot: src/config/slot.rs: storing new value
2024-03-11T21:15:36.192517Z TRACE quilkin::config::slot: src/config/slot.rs: storing new value

markmandel commented 6 months ago

So you don't see any logging when you send a packet?

thijsvanloef commented 6 months ago

None in Quilkin, but it shows up in tcpdump:

REDACTED@REDACTED:~$ sudo tcpdump udp port 8211
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp0s3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
21:44:01.723544 IP REDACTED.fixed.kpn.net.63073 > instance-20230324-1226.subnet12162055.vcn12162055.oraclevcn.com.8211: UDP, length 5
21:45:09.323443 IP REDACTED.fixed.kpn.net.55189 > instance-20230324-1226.subnet12162055.vcn12162055.oraclevcn.com.8211: UDP, length 53
21:45:10.323743 IP REDACTED.fixed.kpn.net.55189 > instance-20230324-1226.subnet12162055.vcn12162055.oraclevcn.com.8211: UDP, length 47
21:45:11.331857 IP REDACTED.fixed.kpn.net.55189 > instance-20230324-1226.subnet12162055.vcn12162055.oraclevcn.com.8211: UDP, length 48
21:45:12.340263 IP REDACTED.fixed.kpn.net.55189 > instance-20230324-1226.subnet12162055.vcn12162055.oraclevcn.com.8211: UDP, length 48
21:45:13.348558 IP REDACTED.fixed.kpn.net.55189 > instance-20230324-1226.subnet12162055.vcn12162055.oraclevcn.com.8211: UDP, length 52
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel

Using nc -u palworld.domain.tld 8211 and the palworld game

markmandel commented 6 months ago

This says to me that Quilkin isn't receiving any packets unfortunately.

Running through the netcat quickstart with the latest dev, I see the following log lines when data is flowing through.

❯ ./quilkin proxy --to 127.0.0.1:8080
2024-03-11T22:37:55.523788Z  INFO quilkin::cli: src/cli.rs: Starting Quilkin version="0.8.0-dev" commit="dddc9b2203b6a66f1bfc4f448fa3912fde78ae8c"
2024-03-11T22:37:55.523859Z DEBUG quilkin::cli: src/cli.rs: config parameters cli=Cli { no_admin: false, config: "quilkin.yaml", admin_address: None, quiet: false, command: Proxy(Proxy { management_server: [], mmdb: None, port: 7777, qcmp_port: 7600, to: [127.0.0.1:8080], idle_request_interval_secs: 30, workers: None }), log_format: Auto }
2024-03-11T22:37:55.523917Z DEBUG quilkin::cli: src/cli.rs: provided path not found path=quilkin.yaml
2024-03-11T22:37:55.523935Z DEBUG quilkin::cli: src/cli.rs: /etc path not found path=quilkin.yaml
2024-03-11T22:37:55.524113Z  INFO quilkin::cli::admin: src/cli/admin.rs: Starting admin endpoint address=[::]:8000
2024-03-11T22:37:55.524296Z TRACE run: quilkin::config::watch: src/config/watch.rs: change detected watchers=0
2024-03-11T22:37:55.524326Z  INFO run: quilkin::cli::proxy: src/cli/proxy.rs: Starting port=7777 proxy_id="au-dev.australia-southeast2-c.c.agones-mark-dev.internal"
2024-03-11T22:37:55.594972Z TRACE quilkin::config::slot: src/config/slot.rs: storing new value
2024-03-11T22:37:55.595642Z  INFO run: quilkin::cli::proxy: src/cli/proxy.rs: Quilkin is ready
2024-03-11T22:38:10.407619Z TRACE quilkin::net::maxmind_db: src/net/maxmind_db.rs: skipping mmdb telemetry, no maxmind database available
2024-03-11T22:38:10.407656Z TRACE quilkin::cli::proxy: src/cli/proxy.rs: received packet from downstream id=17 size=5 source=127.0.0.1:42664
2024-03-11T22:38:10.407748Z TRACE quilkin::cli::proxy::sessions: src/cli/proxy/sessions.rs: SessionPool::get source=127.0.0.1:42664 dest=127.0.0.1:8080
2024-03-11T22:38:10.407775Z TRACE quilkin::cli::proxy::sessions: src/cli/proxy/sessions.rs: creating new socket for session source=127.0.0.1:42664 dest=127.0.0.1:8080
2024-03-11T22:38:10.408261Z TRACE quilkin::cli::proxy::sessions: src/cli/proxy/sessions.rs: reusing socket for session source=127.0.0.1:42664 dest=127.0.0.1:8080
2024-03-11T22:38:10.408319Z DEBUG quilkin::cli::proxy::sessions: src/cli/proxy/sessions.rs: Session created source=127.0.0.1:42664 dest=127.0.0.1:8080
2024-03-11T22:38:10.408331Z TRACE quilkin::cli::proxy::sessions: src/cli/proxy/sessions.rs: inserting session into map
2024-03-11T22:38:10.408338Z TRACE quilkin::cli::proxy::sessions: src/cli/proxy/sessions.rs: session inserted
2024-03-11T22:38:10.408524Z TRACE quilkin::cli::proxy::sessions: src/cli/proxy/sessions.rs: sending packet upstream send_addr=127.0.0.1:8080 length=5
2024-03-11T22:38:10.410783Z TRACE quilkin::cli::proxy::sessions: src/cli/proxy/sessions.rs: received packet from upstream source=127.0.0.1:8080 dest=127.0.0.1:42664 length=5
2024-03-11T22:38:10.410822Z TRACE quilkin::cli::proxy::sessions: src/cli/proxy/sessions.rs: sending packet downstream source=127.0.0.1:8080 dest=127.0.0.1:42664 length=5
2024-03-11T22:38:55.596483Z TRACE quilkin::config::slot: src/config/slot.rs: storing new value

So maybe something's not quite right on your VPS setup?

thijsvanloef commented 6 months ago

Hi @markmandel Thanks for taking the time to troubleshoot with me. I found the solution and feel incredibly stupid. I am used to UFW on ubuntu, and disabled that on the host but apparently iptables was also installed. Added a rule allowing udp on port 8211 and it started working.

The upside I guess is that I built and tested quilkin on an ARM64 host and it works like a charm.

Sorry for the inconvenience.

googleforgames / quilkin

Quilkin not proxying on IPv4 #904