search

status-im / infra-eth-cluster

Infrastructure for Status-go fleets
https://github.com/status-im/status-go
0 stars 0 forks source link

Statusd seeing external connections from docker bridge IP #35

Closed jakubgs closed 3 years ago

jakubgs commented 4 years ago

While investigating the spammer @cammellos has noticed that some of our logs list the Docker bridge IP as the peer IP:

message:handling peer id=0x1c5e26ae98c9679a292ecaad4f1ce5c79faf592c568c90a69473ddbebf9e145e ip=172.17.0.1 pid:767 logsource:node-05.gc-us-central1-a.eth.prod program:docker/statusd-whisper logsource_ip:104.154.230.58 facility:22 @version:1 type:rsyslog facility_name:local6 host:10.2.1.68 fleet:eth.prod severity:3

See 172.17.0.1. This is weird and if it obscures actual peer IPs it makes it more difficult to diagnose and mitigate such attacks.

jakubgs commented 4 years ago

I general Docker creates a virtual docker0 interface to route through it all traffic to and from containers:

admin@mail-01.do-ams3.eth.test:~ % docker network list        
NETWORK ID          NAME                DRIVER              SCOPE
d52875344fc1        bridge              bridge              local
edd5a41d73cd        host                host                local
811db6e5eb89        none                null                local
admin@mail-01.do-ams3.eth.test:~ % ip addr | grep -A 5 docker0:
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:a6:2c:0f:ee brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:a6ff:fe2c:fee/64 scope link 
       valid_lft forever preferred_lft forever
admin@mail-01.do-ams3.eth.test:~ % ip route | grep docker0
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
jakubgs commented 4 years ago

If you inspect the bridge you can see that the 172.17.0.1 address is the gateway:

admin@mail-01.do-ams3.eth.test:~ % docker inspect bridge
[
    {
        "Name": "bridge",
        "Id": "d52875344fc180fc13eeb64bcb85a24e32df737454cb878ea75c18bc150e3c68",
        "Created": "2020-08-25T21:02:55.170049315Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "56f95582358b759f188b4b170a278e0a507f8c7bb70b5396b17053962ab5b923": {
                "Name": "statusd-mail",
                "EndpointID": "f7eb37ec298267933c842b165dcbeba31e50bedd8e7200b347ae02a7b68b3c44",
                "MacAddress": "02:42:ac:11:00:03",
                "IPv4Address": "172.17.0.3/16",
                "IPv6Address": ""
            },
            "8fd8a70a297ab8dcd1baa67b50c24265a2f4f5b10a3ff80aef25440c9aa2ee77": {
                "Name": "statusd-mail-db",
                "EndpointID": "406230be3348eb69530952b79153f3f21d9345e1537a5f66ec98559907fefa61",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "172.17.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]
cammellos commented 4 years ago

maybe related https://github.com/moby/moby/issues/15086

On Sun, Sep 13, 2020, 18:44 Jakub notifications@github.com wrote:

If you inspect the bridge you can see that the 172.17.0.1 address is the gateway:

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/status-im/infra-eth-cluster/issues/35#issuecomment-691694977, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHYJMFPPLHO7P7HUOLYSMDSFTZFHANCNFSM4RKYFAZA .

jakubgs commented 4 years ago

If I run a container myself with netcat listening on 9000 and then connect from the local machine via my WLAN interface I see:

 > d run --rm -it -p 9000:9000 alpine:latest
/ # ip addr | grep inet
    inet 127.0.0.1/8 scope host lo
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
/ # nc -l -v -p 9000
listening on 0.0.0.0:9000 ...
connect to 172.17.0.2:9000 from 192.168.7.104:58532 (192.168.7.104:58532)
test
^Cpunt!

Which shows the correct IP. And if I check the connection via netstat on the host system it looks correct:

 > sudo netstat -pnt | grep 9000
tcp        0      0 192.168.7.104:58538     192.168.7.104:9000      ESTABLISHED 2785/nc

As it does within the container:

/ # netstat -pnt
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 172.17.0.2:9000         192.168.7.104:58538     ESTABLISHED 7/nc

Except the local listen address is different due to the docker-proxy:

 > ps ax | grep docker-proxy 
 2599 ?        Sl     0:00 .../docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 9000 -container-ip 172.17.0.2 -container-port 9000
jakubgs commented 4 years ago

It appears that the docker-proxy process is called the "Userland proxy" based on the config flags:

  "userland-proxy": false,
  "userland-proxy-path": "/usr/libexec/docker-proxy",

https://docs.docker.com/engine/reference/commandline/dockerd/

And according to the issue Andrea linked and the one related to disabling Userland proxy by default it's enabled.

jakubgs commented 4 years ago

I have attempted the same test as before but on mail-01.do-ams3.eth.test and using the already exposed 30504 port:

admin@mail-01.do-ams3.eth.test:~ % d run --rm -it -p 30504:30504 --entrypoint=/usr/bin/nc alpine -lvnp 30504
listening on [::]:30504 ...
connect to [::ffff:172.17.0.3]:30504 from [::ffff:176.109.132.66]:42446 ([::ffff:176.109.132.66]:42446)
test
^Cpunt!

And it appeared correctly as my current public IP: 176.109.132.66

So there must be something else that causes the IP to be obscured by the userland proxy.

jakubgs commented 4 years ago

Ahh, and there we have it, now it's seen as 172.17.0.1:

admin@mail-01.do-ams3.eth.test:~ % d run --rm -it --name test -p 30504:30504 --entrypoint=/usr/bin/nc alpine -lvnp 30504
listening on [::]:30504 ...
connect to [::ffff:172.17.0.3]:30504 from [::ffff:172.17.0.1]:53306 ([::ffff:172.17.0.1]:53306)
test
^Cpunt!

I did the same thing as before, except I used port 443 instead of the 30504 the service is listening on.

So the issue isn't just docker, it's docker combined with the firewall rule we use to redirect traffic from 443 to 30504.

jakubgs commented 4 years ago

The rule looks like this:

admin@mail-01.do-ams3.eth.test:~ % sudo iptables -L -t nat | grep Redirect
REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:https /* Redirect 443 to 30504 */ redir ports 30504

And the actual iptables command looks like this:

admin@mail-01.do-ams3.eth.test:~ % grep Redirect /etc/iptables/rules.v4 
-A PREROUTING -i eth0 -p tcp -m tcp --dport 443 -m comment --comment "Redirect 443 to 30504" -j REDIRECT --to-ports 30504

And is configured here: https://github.com/status-im/infra-eth-cluster/blob/11f9f58989caa72cb877b826c73340d0f151cb1b/ansible/roles/statusd-mailsrv/tasks/firewall.yml#L17-L30

I added this rule because go-ethereum doesn't accept ports bellow 1000 for DevP2P listening port.

jakubgs commented 4 years ago

At least that's the reason I recall for the change in https://github.com/status-im/status-go/commit/e30cf0bc as a result of https://github.com/status-im/status-react/issues/6351.

jakubgs commented 4 years ago

I tried changing config.json file to use 443 as the default port via ListenAddr and it doesn't seem to have caused any failures:

INFO [09-13|18:39:47.490] UDP listener up                net=enode://123qwe@[::]:443
INFO [09-13|18:39:47.492] Started P2P networking         self="enode://123qwe@127.0.0.1:443?discport=0"

So either that was never the case and I misremember, or one of our go-ethereum updates removed that limitation.

jakubgs commented 4 years ago

So in theory we should be able to invert the current setup and make the 443 the default and 30504 the alternative/fallback via the iptables NAT rule, which I theory should fix the issues with IPs.

jakubgs commented 4 years ago

I have switched around the listen port and alternative port - 30504 and 443 respectively - in 8d9700c738574dcc5acd9bdbb7d2e4c8d750346f end deployed to test.

So far it looks good. If Cammellos is fine with this fix we'll then go to staging and have testers play with it and then prod.

jakubgs commented 4 years ago

@cammellos have you played around with the test fleet? Does it this resolve the issue?

If so maybe we should push the changes to staging and have the test team have a go.

jakubgs commented 4 years ago

According to Andrea the test fleet is fine so I'm trying to deploy to staging but I found out something about the iptables module.

It appears that setting state: absent does not actually result in the rule being added. I tested this on mail-01.do-ams3.eth.staging and now there are for example two conflicting NAT rules:

admin@mail-01.do-ams3.eth.staging:~ % sudo iptables -L PREROUTING -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:30504 /* Redirect 30504 to 443 */ redir ports 443
REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:https /* Redirect 443 to 30504 */ redir ports 30504
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL
jakubgs commented 4 years ago

The simplest way is to just script it with the shell module and using the --line-numbers flag for iptables:

- name: Disable old enode port
  shell: |
    for NUM in $(iptables -L DOCKER-USER --line-numbers \
      | grep "Enable enode {{ mailsrv_cont_name }}" \
      | cut -d" " -f1); do
      iptables -D DOCKER-USER $NUM
    done
  with_items:
    - tcp
    - udp

- name: 'Disable old alternative port: {{ mailsrv_listen_port }}'
  shell: |
    for NUM in $(iptables -L PREROUTING -t nat --line-numbers \
      | grep "Redirect {{ mailsrv_listen_port }} to {{ mailsrv_alt_port }}" \
      | cut -d" " -f1); do
      iptables -D PREROUTING $NUM -t nat
    done

- name: 'Disable alternative enode port: {{ mailsrv_listen_port }}'
  shell: |
    for NUM in $(iptables -L INPUT --line-numbers \
      | grep "Enable alt enode {{ mailsrv_cont_name }}" \
      | cut -d" " -f1); do
      iptables -D INPUT $NUM
    done
jakubgs commented 4 years ago

Okay, I've deployed the changes to all staging hosts and now they use 443 as the main port.

jakubgs commented 4 years ago

In summary:

If this turns out to work fine on staging I'd like to roll this out to prod eventually for easier SPAM mitigation in the future.

@churik @Serhy could you please do some communication testing using the staging fleet? No rush tho.

churik commented 4 years ago

endless "connecting" when I'm switching to staging fleet - > 10% tests are failed due to this Logs: status_logs (2).zip UPDATE: eventually it connects (>3 mins) but time is too long

jakubgs commented 4 years ago

I have tested the fleet with node-canary and it works fine for history nodes:

while read HOST ENODE; do
  echo -ne "$HOST\t";
  ./build/bin/node-canary -log=ERROR -mailserver "$ENODE" \
    && echo SUCCESS;
done < <(curl -s https://fleets.status.im | jq -r '.fleets["eth.staging"].mail | to_entries[] | "\(.key)\t\(.value)"')
mail-01.ac-cn-hongkong-c.eth.staging    SUCCESS
mail-01.do-ams3.eth.staging SUCCESS
mail-01.gc-us-central1-a.eth.staging    SUCCESS

But when I test the regular whisper nodes it fails:

node-01.ac-cn-hongkong-c.eth.staging    ERROR[09-23|16:54:19.123] Error detected while waiting for static peer to be added package=status-go/cmd/node-canary error="wait for peer: timeout"
node-01.do-ams3.eth.staging ERROR[09-23|16:54:24.193] Error detected while waiting for static peer to be added package=status-go/cmd/node-canary error="wait for peer: timeout"
node-01.gc-us-central1-a.eth.staging    ERROR[09-23|16:54:29.271] Error detected while waiting for static peer to be added package=status-go/cmd/node-canary error="wait for peer: timeout"

So there is an issue.

jakubgs commented 4 years ago

It appears that the ports show up as closed from outside:

 > curl -sL fleets.status.im | jq -r '.fleets["eth.staging"].whisper | keys[]' | xargs sudo nmap -Pn -p443  
Starting Nmap 7.80 ( https://nmap.org ) at 2020-09-23 16:56 CEST
Nmap scan report for node-01.ac-cn-hongkong-c.eth.staging (47.52.226.137)
Host is up (0.32s latency).

PORT    STATE  SERVICE
443/tcp closed https

Nmap scan report for node-01.do-ams3.eth.staging (167.99.19.148)
Host is up (0.031s latency).

PORT    STATE  SERVICE
443/tcp closed https

Nmap scan report for node-01.gc-us-central1-a.eth.staging (35.192.0.86)
Host is up (0.13s latency).
rDNS record for 35.192.0.86: 86.0.192.35.bc.googleusercontent.com

PORT    STATE  SERVICE
443/tcp closed https

Nmap done: 3 IP addresses (3 hosts up) scanned in 0.64 seconds
jakubgs commented 4 years ago

Even though if I go on one of the hosts the port is clearly being listened on:

admin@node-01.do-ams3.eth.staging:~ % sudo netstat -lpnt | grep 443
tcp6       0      0 :::443                  :::*                    LISTEN      7356/docker-proxy

And the ListenAddr is configured correctly:

admin@node-01.do-ams3.eth.staging:~ % grep ListenAddr /docker/statusd-whisper/config/config.json
    "ListenAddr": "0.0.0.0:443",

As can be seen within the container:

admin@node-01.do-ams3.eth.staging:~ % d exec -it statusd-whisper sh
/ # netstat -lpnt | grep 443
tcp        0      0 :::443                  :::*                    LISTEN      1/status

So why would it appear as closed...

jakubgs commented 4 years ago

I think I see the reason:

admin@node-01.do-ams3.eth.staging:~ % sudo iptables -L PREROUTING -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:30305 /* Redirect 30305 to 443 */ redir ports 443
REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:https /* Redirect 443 to 30305 */ redir ports 30305
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

The old NAT redirect rule is still there.

jakubgs commented 4 years ago

After editing the rules by hand and rebooting the host it appears as open now:

 > sudo nmap -Pn -p443 node-01.do-ams3.eth.staging  
Starting Nmap 7.80 ( https://nmap.org ) at 2020-09-23 17:02 CEST
Nmap scan report for node-01.do-ams3.eth.staging (167.99.19.148)
Host is up (0.030s latency).

PORT    STATE SERVICE
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 0.11 seconds
jakubgs commented 4 years ago

Now all of them appear as open and the canary is passing:

 > while read HOST ENODE; do echo -ne "$HOST\t"; ./build/bin/node-canary -log=ERROR -staticnode "$ENODE" && echo SUCCESS; done < <(curl -s https://fleets.status.im | jq -r '.fleets["eth.staging"].whisper | to_entries[] | "\(.key)\t\(.value)"')
node-01.ac-cn-hongkong-c.eth.staging    SUCCESS
node-01.do-ams3.eth.staging SUCCESS
node-01.gc-us-central1-a.eth.staging    SUCCESS
jakubgs commented 4 years ago

I have tested fetching historical messages with all 3 history nodes in the staging fleet and it all works. I tested with newest nightly.

jakubgs commented 4 years ago

I have replaced all whisper and mail hosts in the eth.staging fleet just to be sure:

node-01.ac-cn-hongkong-c.eth.staging    SUCCESS
node-01.do-ams3.eth.staging SUCCESS
node-01.gc-us-central1-a.eth.staging    SUCCESS
mail-01.ac-cn-hongkong-c.eth.staging    SUCCESS
mail-01.do-ams3.eth.staging SUCCESS
mail-01.gc-us-central1-a.eth.staging    SUCCESS

Lets see if that did anything. I see no issues doing manual tests on mobile.

jakubgs commented 4 years ago

In the process of re-creating nodes I also found a bug in file permissions in enode-setup due to Ansible upgrade: d22e4ae82154297689adcf06051e23d177b90c18

jakubgs commented 4 years ago

It is possible that that was the issue the whole time...

jakubgs commented 4 years ago

For now I have reverted the changes on eth.staging: https://github.com/status-im/infra-eth-cluster/commit/4179ca5c

Serhy commented 4 years ago

We have ran e2e tests suite (on eth.staging) twice and there are no issues with "Connecting..." I also checked manually on three devices switching to staging fleet and had absolutely no issues with 'Connecting...' instantly connected and fetched messages.

On eth.test fleet, however, 'Connecting...' still takes a while (takes 5+ minutes in 2of3 cases) (on three real devices ams3 mailserver used)

jakubgs commented 4 years ago

Thanks for testing this. This is really weird, but I guess the issues must be somewhere in the Go code that causes issues when listening on 443.

jakubgs commented 4 years ago

I thought of another possibility. We could run the service using the old port but map it to 443 using Docker mappings.

jakubgs commented 3 years ago

I identified an issue that causes timeouts when using node-canary:

 > for ((;;)) { ~/work/status-go/build/bin/node-canary -log=ERROR -mailserver='enode://2c8de3cbb27a3d30cbb5b3e003bc722b126f5aef82e2052aaef032ca94e0c7ad219e533ba88c70585ebd802de206693255335b100307645ab5170e88620d2a81@47.244.221.14:30504' 2> /dev/null && echo SUCCESS || echo FAILURE; sleep 10; }
SUCCESS
FAILURE
SUCCESS
FAILURE
SUCCESS
FAILURE
SUCCESS
FAILURE
SUCCESS

But considering it works just fine if I bump the delay to 30 seconds:

 > for ((;;)) {
     node-canary -mailserver='enode://2c8de3cbb27a3d30cbb5b3e003bc722b126f5aef82e2052aaef032ca94e0c7ad219e533ba88c70585ebd802de206693255335b100307645ab5170e88620d2a81@47.244.221.14:30504' 2> /dev/null \
     && echo SUCCESS \
     || echo FAILURE; \
     sleep 30;
   }                                                                                                                                                  
SUCCESS
SUCCESS
SUCCESS
SUCCESS
SUCCESS
SUCCESS
SUCCESS

This is probably due to some kind of per-IP throttling we do. But the error doesn't make sense:

ERROR[01-05|17:19:41.475] Error detected while waiting for mailserver peer to be added package=status-go/cmd/node-canary error="wait for peer: timeout"
jakubgs commented 3 years ago

Opened an issue in regards to this: https://github.com/status-im/status-go/issues/2109

jakubgs commented 3 years ago

I have switched eth.test fleet to map ports using the Docker mapping instead of prerouting iptables rules: d26ef6c0c3f003a92396ac302dcdd08099158ff4

I've tested this using netcat and this method also fixes the incorrect client IPs. Next step would be to test the fleet to see if it has same issues as before.

@Serhy would you find some time to check out eth.test fleet? No rush tho. Low priority.

jakubgs commented 3 years ago

Looks like these changes I deployed to eth.test were accidentally deployed by me to some of eth.prod hosts. It appears they caused some of the connectivity issues we've been seeing. I've rolled them back on mail-01.do-ams3.eth.prod to verify.

jakubgs commented 3 years ago

Quick script I used to check all nodes respond to canary:

curl -sL fleets.status.im | \
  jq -r '.fleets["eth.prod"].mail as $in | $in | keys[] | .+"\t"+$in[.]' | \
  xargs -I{} bash -c 'NAME=$(echo "{}" | cut -f1); ENODE=$(echo "{}" | cut -f2); echo $NAME; ~/work/status-go/build/bin/node-canary -log=WARN -mailserver="$ENODE"'

All work fine.

jakubgs commented 3 years ago

After all of the fixes mentioned in https://github.com/status-im/infra-eth-cluster/issues/39#issuecomment-772531703 being merged and deployed I tried using the Docker mapping solution again, but it seems to display the same behavior, which is timeouts and successes in same cadence:

 > for ((;;)) { node-canary -mailserver="${ENODE}" 2> /dev/null && echo SUCCESS || echo FAILURE; sleep 10; } 
SUCCESS                                                                                
FAILURE
FAILURE
SUCCESS
FAILURE
SUCCESS
FAILURE
SUCCESS
FAILURE
SUCCESS
jakubgs commented 3 years ago

I researched debugging iptables rules and I found two ways:

LOG Jump Target

You can mark packets with a LOG tag:

iptables -A INPUT -s ${MY_IP}/24 -j LOG --log-prefix 'SUSPECT: '

Which will allow them to show up in kernel logs:

admin@mail-01.do-ams3.eth.test:~ % sudo journalctl -o cat -f -t kernel
SUSPECT: IN=eth0 OUT= MAC=c6:61:d5:86:15:43:fe:00:00:00:01:01:08:00 SRC=95.160.157.69 DST=10.18.0.29 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=10750 DF PROTO=TCP SPT=22236 DPT=443 WINDOW=64240 RES=0x00 SYN URGP=0 
SUSPECT: IN=eth0 OUT= MAC=c6:61:d5:86:15:43:fe:00:00:00:01:01:08:00 SRC=95.160.157.69 DST=10.18.0.29 LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=10751 DF PROTO=TCP SPT=22236 DPT=443 WINDOW=502 RES=0x00 ACK URGP=0 
SUSPECT: IN=eth0 OUT= MAC=c6:61:d5:86:15:43:fe:00:00:00:01:01:08:00 SRC=95.160.157.69 DST=10.18.0.29 LEN=517 TOS=0x00 PREC=0x00 TTL=51 ID=10752 DF PROTO=TCP SPT=22236 DPT=443 WINDOW=502 RES=0x00 ACK PSH URGP=0
...

Links

TRACE Jump Target

You can add an entry in iptables with a target called TRACE like so:

iptables -t raw -A PREROUTING -p tcp --source ${MY_IP}/24 --dport 443 -j TRACE

In order to mark packets that reach that entry with TRACE tag which will make them appear in kernel logs like so:

admin@mail-01.do-ams3.eth.test:~ % sudo journalctl -o cat -f -t kernel
TRACE: filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced MAC=c6:61:d5:86:15:43:fe:00:00:00:01:01:08:00 SRC=95.160.157.69 DST=172.18.0.2 LEN=79 TOS=0x00 PREC=0x00 TTL=50 ID=51543 DF PROTO=TCP SPT=55485 DPT=443 SEQ=402854929 ACK=3836145038 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B5A495DEADF93A6) 
TRACE: raw:PREROUTING:policy:2 IN=eth0 OUT= MAC=c6:61:d5:86:15:43:fe:00:00:00:01:01:08:00 SRC=95.160.157.69 DST=10.18.0.29 LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=51546 DF PROTO=TCP SPT=55485 DPT=443 SEQ=402854956 ACK=3836145774 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B5A4987EADF93E9) 
TRACE: filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced MAC=c6:61:d5:86:15:43:fe:00:00:00:01:01:08:00 SRC=95.160.157.69 DST=172.18.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=51546 DF PROTO=TCP SPT=55485 DPT=443 SEQ=402854956 ACK=3836145774 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B5A4987EADF93E9) 
...

Links

jakubgs commented 3 years ago

The TRACE method gives clearly more info, specifically the rule though which the packet travels. Using this I should be able to figure out at which step the packets gets discarded which causes the sporadic timeouts.

jakubgs commented 3 years ago

Here is an example of trace of a successful node-canary call:

filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=79 TOS=0x00 PREC=0x00 TTL=50 ID=34532 DF PROTO=TCP SPT=43834 DPT=443 SEQ=3489079951 ACK=2749754797 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7AF985EB0043BF) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=34536 DF PROTO=TCP SPT=43834 DPT=443 SEQ=3489079978 ACK=2749755549 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7AF9ABEB00440D) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=34536 DF PROTO=TCP SPT=43834 DPT=443 SEQ=3489079978 ACK=2749755549 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7AF9ABEB00440D) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=34536 DF PROTO=TCP SPT=43834 DPT=443 SEQ=3489079978 ACK=2749755549 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7AF9ABEB00440D) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=84 TOS=0x00 PREC=0x00 TTL=51 ID=34537 DF PROTO=TCP SPT=43834 DPT=443 SEQ=3489079978 ACK=2749755549 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7AF9ABEB00440D) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=84 TOS=0x00 PREC=0x00 TTL=50 ID=34537 DF PROTO=TCP SPT=43834 DPT=443 SEQ=3489079978 ACK=2749755549 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7AF9ABEB00440D) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=84 TOS=0x00 PREC=0x00 TTL=50 ID=34537 DF PROTO=TCP SPT=43834 DPT=443 SEQ=3489079978 ACK=2749755549 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7AF9ABEB00440D) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=84 TOS=0x00 PREC=0x00 TTL=51 ID=34538 DF PROTO=TCP SPT=43834 DPT=443 SEQ=3489080010 ACK=2749755549 WINDOW=501 RES=0x00 ACK PSH FIN URGP=0 OPT (0101080A4B7AF9ABEB00440D) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=84 TOS=0x00 PREC=0x00 TTL=50 ID=34538 DF PROTO=TCP SPT=43834 DPT=443 SEQ=3489080010 ACK=2749755549 WINDOW=501 RES=0x00 ACK PSH FIN URGP=0 OPT (0101080A4B7AF9ABEB00440D) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=84 TOS=0x00 PREC=0x00 TTL=50 ID=34538 DF PROTO=TCP SPT=43834 DPT=443 SEQ=3489080010 ACK=2749755549 WINDOW=501 RES=0x00 ACK PSH FIN URGP=0 OPT (0101080A4B7AF9ABEB00440D) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=29205 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428440 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A4B7BC3D10000000001030307) 
nat:PREROUTING:rule:1 IN=eth0 OUT= LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=29205 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428440 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A4B7BC3D10000000001030307) 
nat:DOCKER:rule:7 IN=eth0 OUT= LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=29205 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428440 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A4B7BC3D10000000001030307) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=29205 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428440 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A4B7BC3D10000000001030307) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=29205 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428440 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A4B7BC3D10000000001030307) 
nat:POSTROUTING:policy:21 IN= OUT=br-7463352fcced LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=29205 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428440 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A4B7BC3D10000000001030307) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=29206 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428441 ACK=3805397704 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC3F2EB010E54) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=29206 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428441 ACK=3805397704 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC3F2EB010E54) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=29206 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428441 ACK=3805397704 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC3F2EB010E54) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=517 TOS=0x00 PREC=0x00 TTL=51 ID=29207 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428441 ACK=3805397704 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC3F2EB010E54) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=517 TOS=0x00 PREC=0x00 TTL=50 ID=29207 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428441 ACK=3805397704 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC3F2EB010E54) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=517 TOS=0x00 PREC=0x00 TTL=50 ID=29207 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428441 ACK=3805397704 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC3F2EB010E54) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=84 TOS=0x00 PREC=0x00 TTL=51 ID=29210 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428906 ACK=3805398239 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC412EB010E74) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=29208 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428906 ACK=3805398207 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC411EB010E74) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=29209 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428906 ACK=3805398239 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC412EB010E74) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=29212 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428938 ACK=3805398368 WINDOW=500 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC412EB010E74) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=53 TOS=0x00 PREC=0x00 TTL=51 ID=29213 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428938 ACK=3805398368 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC412EB010E74) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=29215 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429065 ACK=3805398383 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=84 TOS=0x00 PREC=0x00 TTL=50 ID=29210 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428906 ACK=3805398239 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=84 TOS=0x00 PREC=0x00 TTL=50 ID=29210 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428906 ACK=3805398239 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=29208 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428906 ACK=3805398207 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC411EB010E74) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=29208 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428906 ACK=3805398207 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC411EB010E74) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=29209 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428906 ACK=3805398239 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=29209 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428906 ACK=3805398239 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=29212 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428938 ACK=3805398368 WINDOW=500 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=29212 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428938 ACK=3805398368 WINDOW=500 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=53 TOS=0x00 PREC=0x00 TTL=50 ID=29213 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428938 ACK=3805398368 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=53 TOS=0x00 PREC=0x00 TTL=50 ID=29213 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428938 ACK=3805398368 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=29215 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429065 ACK=3805398383 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=29215 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429065 ACK=3805398383 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC412EB010E74) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=178 TOS=0x00 PREC=0x00 TTL=51 ID=29214 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428939 ACK=3805398368 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC412EB010E74) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=69 TOS=0x00 PREC=0x00 TTL=51 ID=29216 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429065 ACK=3805398383 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=178 TOS=0x00 PREC=0x00 TTL=50 ID=29214 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428939 ACK=3805398368 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=178 TOS=0x00 PREC=0x00 TTL=50 ID=29214 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495428939 ACK=3805398368 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=69 TOS=0x00 PREC=0x00 TTL=50 ID=29216 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429065 ACK=3805398383 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=69 TOS=0x00 PREC=0x00 TTL=50 ID=29216 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429065 ACK=3805398383 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC412EB010E74) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=29217 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429082 ACK=3805398399 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC412EB010E74) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=84 TOS=0x00 PREC=0x00 TTL=51 ID=29218 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429082 ACK=3805398399 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC412EB010E74) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=85 TOS=0x00 PREC=0x00 TTL=51 ID=29219 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429114 ACK=3805398399 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC412EB010E74) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=83 TOS=0x00 PREC=0x00 TTL=51 ID=29220 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429147 ACK=3805398399 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=29217 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429082 ACK=3805398399 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=29217 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429082 ACK=3805398399 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=84 TOS=0x00 PREC=0x00 TTL=50 ID=29218 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429082 ACK=3805398399 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=84 TOS=0x00 PREC=0x00 TTL=50 ID=29218 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429082 ACK=3805398399 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=85 TOS=0x00 PREC=0x00 TTL=50 ID=29219 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429114 ACK=3805398399 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=85 TOS=0x00 PREC=0x00 TTL=50 ID=29219 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429114 ACK=3805398399 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=83 TOS=0x00 PREC=0x00 TTL=50 ID=29220 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429147 ACK=3805398399 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC412EB010E74) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=83 TOS=0x00 PREC=0x00 TTL=50 ID=29220 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429147 ACK=3805398399 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC412EB010E74) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=29222 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429178 ACK=3805398432 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC432EB010E94) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=29225 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429178 ACK=3805398495 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC432EB010E95) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=29222 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429178 ACK=3805398432 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC432EB010E94) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=29222 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429178 ACK=3805398432 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC432EB010E94) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=29225 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429178 ACK=3805398495 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC432EB010E95) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=29225 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429178 ACK=3805398495 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC432EB010E95) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=84 TOS=0x00 PREC=0x00 TTL=51 ID=29226 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429178 ACK=3805398495 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC459EB010E95) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=84 TOS=0x00 PREC=0x00 TTL=51 ID=29227 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429210 ACK=3805398495 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC459EB010E95) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=84 TOS=0x00 PREC=0x00 TTL=50 ID=29226 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429178 ACK=3805398495 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC459EB010E95) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=84 TOS=0x00 PREC=0x00 TTL=50 ID=29226 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429178 ACK=3805398495 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC459EB010E95) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=84 TOS=0x00 PREC=0x00 TTL=50 ID=29227 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429210 ACK=3805398495 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC459EB010E95) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=84 TOS=0x00 PREC=0x00 TTL=50 ID=29227 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429210 ACK=3805398495 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC459EB010E95) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=84 TOS=0x00 PREC=0x00 TTL=51 ID=29228 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429242 ACK=3805398495 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC459EB010E95) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=84 TOS=0x00 PREC=0x00 TTL=50 ID=29228 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429242 ACK=3805398495 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC459EB010E95) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=84 TOS=0x00 PREC=0x00 TTL=50 ID=29228 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429242 ACK=3805398495 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC459EB010E95) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=361 TOS=0x00 PREC=0x00 TTL=51 ID=29229 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429274 ACK=3805398495 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC459EB010E95) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=79 TOS=0x00 PREC=0x00 TTL=51 ID=29230 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429583 ACK=3805398495 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC459EB010E95) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=361 TOS=0x00 PREC=0x00 TTL=50 ID=29229 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429274 ACK=3805398495 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC459EB010E95) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=361 TOS=0x00 PREC=0x00 TTL=50 ID=29229 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429274 ACK=3805398495 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC459EB010E95) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=79 TOS=0x00 PREC=0x00 TTL=50 ID=29230 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429583 ACK=3805398495 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC459EB010E95) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=79 TOS=0x00 PREC=0x00 TTL=50 ID=29230 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429583 ACK=3805398495 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC459EB010E95) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=84 TOS=0x00 PREC=0x00 TTL=51 ID=29236 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429610 ACK=3805399247 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC489EB010EEB) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=29233 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429610 ACK=3805399119 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC488EB010EEA)

And here is a failed one:

filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=84 TOS=0x00 PREC=0x00 TTL=50 ID=29236 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429610 ACK=3805399247 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC489EB010EEB) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=84 TOS=0x00 PREC=0x00 TTL=50 ID=29236 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429610 ACK=3805399247 WINDOW=501 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BC489EB010EEB) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=29233 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429610 ACK=3805399119 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC488EB010EEA) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=29233 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429610 ACK=3805399119 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC488EB010EEA) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=29235 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429610 ACK=3805399247 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC489EB010EEB) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=84 TOS=0x00 PREC=0x00 TTL=51 ID=29237 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429642 ACK=3805399247 WINDOW=501 RES=0x00 ACK PSH FIN URGP=0 OPT (0101080A4B7BC489EB010EEB) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=29235 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429610 ACK=3805399247 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC489EB010EEB) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=29235 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429610 ACK=3805399247 WINDOW=501 RES=0x00 ACK URGP=0 OPT (0101080A4B7BC489EB010EEB) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=84 TOS=0x00 PREC=0x00 TTL=50 ID=29237 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429642 ACK=3805399247 WINDOW=501 RES=0x00 ACK PSH FIN URGP=0 OPT (0101080A4B7BC489EB010EEB) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=84 TOS=0x00 PREC=0x00 TTL=50 ID=29237 DF PROTO=TCP SPT=1223 DPT=443 SEQ=1495429642 ACK=3805399247 WINDOW=501 RES=0x00 ACK PSH FIN URGP=0 OPT (0101080A4B7BC489EB010EEB) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=27569 DF PROTO=TCP SPT=49757 DPT=443 SEQ=413142264 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A4B7BF5380000000001030307) 
nat:PREROUTING:rule:1 IN=eth0 OUT= LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=27569 DF PROTO=TCP SPT=49757 DPT=443 SEQ=413142264 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A4B7BF5380000000001030307) 
nat:DOCKER:rule:7 IN=eth0 OUT= LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=27569 DF PROTO=TCP SPT=49757 DPT=443 SEQ=413142264 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A4B7BF5380000000001030307) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=27569 DF PROTO=TCP SPT=49757 DPT=443 SEQ=413142264 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A4B7BF5380000000001030307) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=27569 DF PROTO=TCP SPT=49757 DPT=443 SEQ=413142264 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A4B7BF5380000000001030307) 
nat:POSTROUTING:policy:21 IN= OUT=br-7463352fcced LEN=60 TOS=0x00 PREC=0x00 TTL=50 ID=27569 DF PROTO=TCP SPT=49757 DPT=443 SEQ=413142264 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080A4B7BF5380000000001030307) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=27570 DF PROTO=TCP SPT=49757 DPT=443 SEQ=413142265 ACK=3518957204 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080A4B7BF558EB013FBB) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=27570 DF PROTO=TCP SPT=49757 DPT=443 SEQ=413142265 ACK=3518957204 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080A4B7BF558EB013FBB) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=27570 DF PROTO=TCP SPT=49757 DPT=443 SEQ=413142265 ACK=3518957204 WINDOW=502 RES=0x00 ACK URGP=0 OPT (0101080A4B7BF558EB013FBB) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=517 TOS=0x00 PREC=0x00 TTL=51 ID=27571 DF PROTO=TCP SPT=49757 DPT=443 SEQ=413142265 ACK=3518957204 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BF559EB013FBB) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=517 TOS=0x00 PREC=0x00 TTL=50 ID=27571 DF PROTO=TCP SPT=49757 DPT=443 SEQ=413142265 ACK=3518957204 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BF559EB013FBB) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=517 TOS=0x00 PREC=0x00 TTL=50 ID=27571 DF PROTO=TCP SPT=49757 DPT=443 SEQ=413142265 ACK=3518957204 WINDOW=502 RES=0x00 ACK PSH URGP=0 OPT (0101080A4B7BF559EB013FBB) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=27572 DF PROTO=TCP SPT=49757 DPT=443 SEQ=413142730 ACK=3518957205 WINDOW=502 RES=0x00 ACK FIN URGP=0 OPT (0101080A4B7BF572EB013FD5) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=27572 DF PROTO=TCP SPT=49757 DPT=443 SEQ=413142730 ACK=3518957205 WINDOW=502 RES=0x00 ACK FIN URGP=0 OPT (0101080A4B7BF572EB013FD5) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=27572 DF PROTO=TCP SPT=49757 DPT=443 SEQ=413142730 ACK=3518957205 WINDOW=502 RES=0x00 ACK FIN URGP=0 OPT (0101080A4B7BF572EB013FD5) 
raw:PREROUTING:policy:2 IN=eth0 OUT= LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=27573 DF PROTO=TCP SPT=49757 DPT=443 SEQ=413142730 ACK=3518957205 WINDOW=502 RES=0x00 ACK FIN URGP=0 OPT (0101080A4B7BF5BAEB013FD5) 
filter:FORWARD:rule:1 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=27573 DF PROTO=TCP SPT=49757 DPT=443 SEQ=413142730 ACK=3518957205 WINDOW=502 RES=0x00 ACK FIN URGP=0 OPT (0101080A4B7BF5BAEB013FD5) 
filter:DOCKER-USER:rule:2 IN=eth0 OUT=br-7463352fcced LEN=52 TOS=0x00 PREC=0x00 TTL=50 ID=27573 DF PROTO=TCP SPT=49757 DPT=443 SEQ=413142730 ACK=3518957205 WINDOW=502 RES=0x00 ACK FIN URGP=0 OPT (0101080A4B7BF5BAEB013FD5)
jakubgs commented 3 years ago

I also found this amazing script: https://github.com/commonism/iptables-trace

It requries:

sudo apt-get install libnetfilter-conntrack3 libnetfilter-log1 libnetfilter-queue1 python3-iptables
sudo pip install libnetfilter

And provides much nicer and detailed output and allows for specifying filter using BPF Syntax:

admin@mail-01.do-ams3.eth.test:~ % sudo ./iptables-trace.py tcp dst port 443 and src 1.2.3.4
Namespace(bpf=['tcp', 'dst', 'port', '443', 'and', 'src', '1.2.3.4'], chain=['OUTPUT', 'PREROUTING'], clear_chain=False, destination=None, limit=False, protocol=None, source=None, xmark_mask='0x800001ff')
IN=eth0 OUT= SRC=1.2.3.4 DST=10.18.0.29 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=64600 PROTO=TCP SPT=46435 DPT=443 
    raw PREROUTING NFMARK=0x0 (0x1307d200)
        ACCEPT
    mangle PREROUTING NFMARK=0x0 (0x1307d200)
        ACCEPT
    nat PREROUTING (#1) NFMARK=0x0 (0x1307d200)
        ip 0.0.0.0/0.0.0.0 -> 0.0.0.0/0.0.0.0 addrtype:{'dst-type': 'LOCAL'} 
        => DOCKER 
    nat DOCKER (#7) NFMARK=0x0 (0x1307d200)
        tcp 0.0.0.0/0.0.0.0 -> 0.0.0.0/0.0.0.0 tcp:{'dport': '443'} 
        => DNAT {'to-destination': '172.18.0.2:443'}
    mangle FORWARD NFMARK=0x0 (0x1307d200)
        ACCEPT
    filter FORWARD (#1) NFMARK=0x0 (0x1307d200)
        ip 0.0.0.0/0.0.0.0 -> 0.0.0.0/0.0.0.0 
        => DOCKER-USER 
    filter DOCKER-USER (#2) /* Enable status-go */NFMARK=0x0 (0x1307d200)
        tcp 0.0.0.0/0.0.0.0 -> 0.0.0.0/0.0.0.0 tcp:{'dport': '443'} 
        => ACCEPT 
    security FORWARD NFMARK=0x0 (0x1307d200)
        ACCEPT
    mangle POSTROUTING NFMARK=0x0 (0x1307d200)
        ACCEPT
    nat POSTROUTING NFMARK=0x0 (0x1307d200)
        ACCEPT
jakubgs commented 3 years ago

Here's the full output for a node-canary call that times out:

IN=eth0 OUT= SRC=1.2.3.4 DST=10.18.0.29 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=62499 PROTO=TCP SPT=19180 DPT=443 
    raw PREROUTING NFMARK=0x0 (0x3100ae00)
        ACCEPT
    mangle PREROUTING NFMARK=0x0 (0x3100ae00)
        ACCEPT
    nat PREROUTING (#1) NFMARK=0x0 (0x3100ae00)
        ip 0.0.0.0/0.0.0.0 -> 0.0.0.0/0.0.0.0 addrtype:{'dst-type': 'LOCAL'} 
        => DOCKER 
    nat DOCKER (#7) NFMARK=0x0 (0x3100ae00)
        tcp 0.0.0.0/0.0.0.0 -> 0.0.0.0/0.0.0.0 tcp:{'dport': '443'} 
        => DNAT {'to-destination': '172.18.0.2:443'}
    mangle FORWARD NFMARK=0x0 (0x3100ae00)
        ACCEPT
    filter FORWARD (#1) NFMARK=0x0 (0x3100ae00)
        ip 0.0.0.0/0.0.0.0 -> 0.0.0.0/0.0.0.0 
        => DOCKER-USER 
    filter DOCKER-USER (#2) /* Enable status-go */NFMARK=0x0 (0x3100ae00)
        tcp 0.0.0.0/0.0.0.0 -> 0.0.0.0/0.0.0.0 tcp:{'dport': '443'} 
        => ACCEPT 
    security FORWARD NFMARK=0x0 (0x3100ae00)
        ACCEPT
    mangle POSTROUTING NFMARK=0x0 (0x3100ae00)
        ACCEPT
    nat POSTROUTING NFMARK=0x0 (0x3100ae00)
        ACCEPT
IN=eth0 OUT= SRC=1.2.3.4 DST=10.18.0.29 LEN=517 TOS=0x00 PREC=0x00 TTL=51 ID=62501 PROTO=TCP SPT=19180 DPT=443 
    raw PREROUTING NFMARK=0x0 (0x3100ae00)
        ACCEPT
    mangle PREROUTING NFMARK=0x0 (0x3100ae00)
        ACCEPT
IN=eth0 OUT= SRC=1.2.3.4 DST=10.18.0.29 LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=62500 PROTO=TCP SPT=19180 DPT=443 
    raw PREROUTING NFMARK=0x0 (0x3100ae00)
        ACCEPT
    mangle PREROUTING NFMARK=0x0 (0x3100ae00)
        ACCEPT
    mangle FORWARD NFMARK=0x0 (0x3100ae00)
        ACCEPT
    filter FORWARD (#1) NFMARK=0x0 (0x3100ae00)
        ip 0.0.0.0/0.0.0.0 -> 0.0.0.0/0.0.0.0 
        => DOCKER-USER 
    filter DOCKER-USER (#2) /* Enable status-go */NFMARK=0x0 (0x3100ae00)
        tcp 0.0.0.0/0.0.0.0 -> 0.0.0.0/0.0.0.0 tcp:{'dport': '443'} 
        => ACCEPT 
    security FORWARD NFMARK=0x0 (0x3100ae00)
        ACCEPT
    mangle POSTROUTING NFMARK=0x0 (0x3100ae00)
        ACCEPT
    mangle FORWARD NFMARK=0x0 (0x3100ae00)
        ACCEPT
    filter FORWARD (#1) NFMARK=0x0 (0x3100ae00)
        ip 0.0.0.0/0.0.0.0 -> 0.0.0.0/0.0.0.0 
        => DOCKER-USER 
    filter DOCKER-USER (#2) /* Enable status-go */NFMARK=0x0 (0x3100ae00)
        tcp 0.0.0.0/0.0.0.0 -> 0.0.0.0/0.0.0.0 tcp:{'dport': '443'} 
        => ACCEPT 
    security FORWARD NFMARK=0x0 (0x3100ae00)
        ACCEPT
    mangle POSTROUTING NFMARK=0x0 (0x3100ae00)
        ACCEPT
jakubgs commented 3 years ago

According to Wireshark we actually get a FIN/ACK from the node: image

jakubgs commented 3 years ago

And in another case we get a RST: image

jakubgs commented 3 years ago

I found something disturbing. It looks like even connections to healthy nodes end in some weird RST retransmissions:

image

But when I try to run node-canary against my node running on localhost It ends correctly:

image

Which suggestes that even our "healthy" setup isn't actually healthy.

jakubgs commented 3 years ago

This is an interesting one:

IN=eth0 OUT= SRC=95.160.157.69 DST=10.18.0.29 LEN=52 TOS=0x00 PREC=0x00 TTL=51 ID=61865 PROTO=TCP SPT=42265 DPT=30504 
    raw PREROUTING NFMARK=0x0 (0x4ff37c00)
        ACCEPT
    mangle PREROUTING NFMARK=0x0 (0x4ff37c00)
        ACCEPT
    mangle FORWARD NFMARK=0x0 (0x4ff37c00)
        ACCEPT
    filter FORWARD (#1) NFMARK=0x0 (0x4ff37c00)
        ip 0.0.0.0/0.0.0.0 -> 0.0.0.0/0.0.0.0 
        => DOCKER-USER 
    filter DOCKER-USER (#4) /* Enable status-go */NFMARK=0x0 (0x4ff37c00)
        tcp 0.0.0.0/0.0.0.0 -> 0.0.0.0/0.0.0.0 tcp:{'dport': '30504'} 
        => ACCEPT 
    security FORWARD NFMARK=0x0 (0x4ff37c00)
        ACCEPT
    mangle POSTROUTING NFMARK=0x0 (0x4ff37c00)
        ACCEPT
    mangle FORWARD NFMARK=0x0 (0x4ff37c00)
        ACCEPT
    filter FORWARD (#1) NFMARK=0x0 (0x4ff37c00)
        ip 0.0.0.0/0.0.0.0 -> 0.0.0.0/0.0.0.0 
        => DOCKER-USER 
    filter DOCKER-USER (#4) /* Enable status-go */NFMARK=0x0 (0x4ff37c00)
        tcp 0.0.0.0/0.0.0.0 -> 0.0.0.0/0.0.0.0 tcp:{'dport': '30504'} 
        => ACCEPT 
    security FORWARD NFMARK=0x0 (0x4ff37c00)
        ACCEPT
    mangle POSTROUTING NFMARK=0x0 (0x4ff37c00)
        ACCEPT
    mangle FORWARD NFMARK=0x0 (0x4ff37c00)
        ACCEPT
    filter FORWARD (#1) NFMARK=0x0 (0x4ff37c00)
        ip 0.0.0.0/0.0.0.0 -> 0.0.0.0/0.0.0.0 
        => DOCKER-USER 
    filter DOCKER-USER (#4) /* Enable status-go */NFMARK=0x0 (0x4ff37c00)
        tcp 0.0.0.0/0.0.0.0 -> 0.0.0.0/0.0.0.0 tcp:{'dport': '30504'} 
        => ACCEPT 
    security FORWARD NFMARK=0x0 (0x4ff37c00)
        ACCEPT
    mangle POSTROUTING NFMARK=0x0 (0x4ff37c00)
        ACCEPT

Why does this packet go three times through the filter DOCKER-USER (#4) rule?

jakubgs commented 3 years ago

Oh for fuck's sake:

DEBUG[02-09|14:13:34.461] Rejected inbound connnection    addr=95.160.157.69:16178 err="too many attempts"

It was rate limiting all along! IT WAS THE RATE LIMITING!

jakubgs commented 3 years ago

But it's in vendor/github.com/ethereum/go-ethereum/p2p/server.go not our own:

func (srv *Server) checkInboundConn(fd net.Conn, remoteIP net.IP) error {
    if remoteIP != nil {
        // Reject connections that do not match NetRestrict.
        if srv.NetRestrict != nil && !srv.NetRestrict.Contains(remoteIP) {
            return fmt.Errorf("not whitelisted in NetRestrict")
        }
        // Reject Internet peers that try too often.
        srv.inboundHistory.expire(time.Now())
        if !netutil.IsLAN(remoteIP) && srv.inboundHistory.contains(remoteIP.String()) {
            return fmt.Errorf("too many attempts")
        }
        srv.inboundHistory.add(remoteIP.String(), time.Now().Add(inboundThrottleTime))
    }
    return nil
}

https://github.com/status-im/status-go/blob/7c80d223/vendor/github.com/ethereum/go-ethereum/p2p/server.go#L929-L931

    // This time limits inbound connection attempts per source IP.
    inboundThrottleTime = 30 * time.Second

https://github.com/status-im/status-go/blob/7c80d223/vendor/github.com/ethereum/go-ethereum/p2p/server.go#L53-L54

jakubgs commented 3 years ago

This explains the pattern I've been seeing since https://github.com/status-im/infra-eth-cluster/issues/35#issuecomment-754745436.