ccll commented 2 years ago

I'm running a k3s cluster in my home in China, where access to gcr.io/github.com/... are blocked by The Great Fire Wall (GFW).

With the builtin Flannel network I was using a bunch of iptable rules (see below) to redirect traffic through a local transparent proxy port, which works great for both the host node and pods traffic.

After upgraded to Calico, I can't access the entire Internet from within the pods if the iptable rules are in effective, all outgoing connections from pods will hang and timeout eventually (where host node can still access the Internet without problem).

After removing the iptable rules, the pods can access the Internet again.

Any help will be appreciated!

# My iptable rules.

iptables -t nat -N PROXY

# Skip reserved and local IP.
iptables -t nat -A PROXY -d 0.0.0.0/8 -j RETURN
iptables -t nat -A PROXY -d 10.42.0.0/16 -j RETURN
iptables -t nat -A PROXY -d 10.43.0.0/16 -j RETURN
iptables -t nat -A PROXY -d 127.0.0.0/8 -j RETURN
iptables -t nat -A PROXY -d 169.254.0.0/16 -j RETURN
iptables -t nat -A PROXY -d 172.16.0.0/16 -j RETURN
iptables -t nat -A PROXY -d 172.17.0.0/16 -j RETURN
iptables -t nat -A PROXY -d 172.30.0.0/16 -j RETURN
iptables -t nat -A PROXY -d 192.168.0.0/16 -j RETURN
iptables -t nat -A PROXY -d 224.0.0.0/4 -j RETURN
iptables -t nat -A PROXY -d 240.0.0.0/4 -j RETURN

# Redirect traffic to transparent proxy.
iptables -t nat -A PROXY -p tcp -j REDIRECT --to-ports 7892

# Handle the OUTPUT chain (for host node).
iptables -t nat -A OUTPUT -p tcp -m owner ! --uid-owner clash -j PROXY

# Handle the PREROUTING chain (for docker/k8s/other hosts...).
iptables -t nat -A PREROUTING -p tcp -j PROXY

Expected Behavior

The pods should have access to the Internet.

Current Behavior

The pods' access to the Internet just hang and timeout.

Your Environment

Calico version Installed with Tigera operator.

Orchestrator version (e.g. kubernetes, mesos, rkt):

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4+k3s1", GitCommit:"43b1cb48200d8f6af85c16ed944d68fcc96b6506", GitTreeState:"clean", BuildDate:"2022-02-24T22:38:17Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4+k3s1", GitCommit:"43b1cb48200d8f6af85c16ed944d68fcc96b6506", GitTreeState:"clean", BuildDate:"2022-02-24T22:38:17Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}

Operating System and version:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 10 (buster)
Release:        10
Codename:       buster

My Tigera Installation manifest

# This section includes base Calico installation configuration.
# For more information, see: https://projectcalico.docs.tigera.io/v3.22/reference/installation/api#operator.tigera.io/v1.Installation
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
name: default
spec:
# Configures Calico networking.
calicoNetwork:
# Note: The ipPools section cannot be modified post-install.
ipPools:
  - blockSize: 26
    cidr: 10.42.0.0/16
    encapsulation: VXLANCrossSubnet
    natOutgoing: Enabled
    nodeSelector: all()
containerIPForwarding: Enabled
---
# This section configures the Calico API server.
# For more information, see: https://projectcalico.docs.tigera.io/v3.22/reference/installation/api#operator.tigera.io/v1.APIServer
apiVersion: operator.tigera.io/v1
kind: APIServer
metadata:
name: default
spec: {}

/etc/cni/net.d/10-calico.conflist

{
"name": "k8s-pod-network",
"cniVersion": "0.3.1",
"plugins": [
{
  "type": "calico",
  "datastore_type": "kubernetes",
  "mtu": 0,
  "nodename_file_optional": false,
  "log_level": "Info",
  "log_file_path": "/var/log/calico/cni/cni.log",
  "ipam": { "type": "calico-ipam", "assign_ipv4" : "true", "assign_ipv6" : "false"},
  "container_settings": {
      "allow_ip_forwarding": true
  },
  "policy": {
      "type": "k8s"
  },
  "kubernetes": {
      "k8s_api_root":"https://10.43.0.1:443",
      "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
  }
},
{
  "type": "bandwidth",
  "capabilities": {"bandwidth": true}
},
{"type": "portmap", "snat": true, "capabilities": {"portMappings": true}}
]
}

raw table


$ iptables -t raw -vnL
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target     prot opt in     out     source               destination         
16M  111G cali-PREROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:6gwbT8clXdHdC1b1 */

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination
4430K 8821M cali-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 / cali:tVnHkvAo15HuiPy0 /

Chain cali-from-host-endpoint (1 references) pkts bytes target prot opt in out source destination

Chain cali-to-host-endpoint (1 references) pkts bytes target prot opt in out source destination

Chain cali-PREROUTING (1 references) pkts bytes target prot opt in out source destination
16M 111G MARK all -- 0.0.0.0/0 0.0.0.0/0 / cali:XFX5xbM8B9qR10JG / MARK and 0xfff0ffff 126 44899 MARK all -- cali+ 0.0.0.0/0 0.0.0.0/0 / cali:EWMPb0zVROM-woQp / MARK or 0x40000 0 0 DROP all -- 0.0.0.0/0 0.0.0.0/0 / cali:mPIOOWmbH3iO0R90 / mark match 0x40000/0x40000 rpfilter validmark invert 16M 111G cali-from-host-endpoint all -- 0.0.0.0/0 0.0.0.0/0 / cali:8eOxmFpkWr0RjKXR / mark match 0x0/0x40000 0 0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 / cali:F-nExspj_POyei0v */ mark match 0x10000/0x10000

Chain cali-OUTPUT (1 references) pkts bytes target prot opt in out source destination
4430K 8821M MARK all -- 0.0.0.0/0 0.0.0.0/0 / cali:njdnLwYeGqBJyMxW / MARK and 0xfff0ffff 4430K 8821M cali-to-host-endpoint all -- 0.0.0.0/0 0.0.0.0/0 / cali:rz86uTUcEZAfFsh7 / 0 0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 / cali:pN0F5zD0b8yf9W1Z / mark match 0x10000/0x10000


* mangle table

$ iptables -t mangle -vnL

Warning: iptables-legacy tables present, use iptables-legacy to see them

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination
16M 113G cali-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0 / cali:6gwbT8clXdHdC1b1 /

Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination
16M 113G cali-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0 / cali:O3lYWMrLQYEMJtB5 /

Chain KUBE-KUBELET-CANARY (0 references) pkts bytes target prot opt in out source destination

Chain KUBE-PROXY-CANARY (0 references) pkts bytes target prot opt in out source destination

Chain cali-from-host-endpoint (1 references) pkts bytes target prot opt in out source destination

Chain cali-to-host-endpoint (1 references) pkts bytes target prot opt in out source destination

Chain cali-PREROUTING (1 references) pkts bytes target prot opt in out source destination
16M 113G ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 / cali:6BJqBjBC7crtA-7- / ctstate RELATED,ESTABLISHED 0 0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 / cali:KX7AGNd6rMcDUai6 / mark match 0x10000/0x10000 527K 70M cali-from-host-endpoint all -- 0.0.0.0/0 0.0.0.0/0 / cali:wNH7KsA3ILKJBsY9 / 0 0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 / cali:Cg96MgVuoPm7UMRo / / Host endpoint policy accepted packet. / mark match 0x10000/0x10000

Chain cali-POSTROUTING (1 references) pkts bytes target prot opt in out source destination
4 240 RETURN all -- 0.0.0.0/0 0.0.0.0/0 / cali:NX-7roTexQ3fGRfU / mark match 0x10000/0x10000 16M 113G MARK all -- 0.0.0.0/0 0.0.0.0/0 / cali:nnqPh8lh2VOogSzX / MARK and 0xfff0ffff 9354K 3851M cali-to-host-endpoint all -- 0.0.0.0/0 0.0.0.0/0 / cali:nquN8Jw8Tz72pcBW / ctstate DNAT 0 0 RETURN all -- 0.0.0.0/0 0.0.0.0/0 / cali:jWrgvDQ0xEZHmta3 / / Host endpoint policy accepted packet. / mark match 0x10000/0x10000


* nat table (ommited repeating entries of KUBE-SVC-* for clarity)

$ iptables -t nat -vnL Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination
499K 54M cali-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0 / cali:6gwbT8clXdHdC1b1 / 1052K 114M KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 / kubernetes service portals / 438 117K DOCKER all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL 418 114K CNI-HOSTPORT-DNAT all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL 863 51780 PROXY tcp -- 0.0.0.0/0 0.0.0.0/0

Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination
476K 53M cali-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0 / cali:O3lYWMrLQYEMJtB5 / 1001K 111M CNI-HOSTPORT-MASQ all -- 0.0.0.0/0 0.0.0.0/0 / CNI portfwd requiring masquerade / 999K 111M KUBE-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0 / kubernetes postrouting rules / 29 1968 MASQUERADE all -- !docker0 172.17.0.0/16 0.0.0.0/0
0 0 MASQUERADE all -- !br-63fbfcfdc361 172.18.0.0/16 0.0.0.0/0
24 1952 MASQUERADE all -- !hassio 172.30.32.0/23 0.0.0.0/0
0 0 MASQUERADE tcp -- 172.30.32.6 172.30.32.6 tcp dpt:80 0 0 MASQUERADE tcp -- 172.30.33.0 172.30.33.0 tcp dpt:8884 0 0 MASQUERADE tcp -- 172.30.33.0 172.30.33.0 tcp dpt:8883 0 0 MASQUERADE tcp -- 172.30.33.0 172.30.33.0 tcp dpt:1884 0 0 MASQUERADE tcp -- 172.30.33.0 172.30.33.0 tcp dpt:1883 930K 104M RETURN all -- 10.42.0.0/16 10.42.0.0/16 / flanneld masq / 2470 240K MASQUERADE all -- 10.42.0.0/16 !224.0.0.0/4 / flannel masq / random-fully 0 0 RETURN all -- !10.42.0.0/16 10.42.0.0/24 / flanneld masq / 0 0 MASQUERADE all -- !10.42.0.0/16 10.42.0.0/16 / flanneld masq */ random-fully

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination
25058 1519K cali-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 / cali:tVnHkvAo15HuiPy0 / 57565 3501K KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 / kubernetes service portals / 153 9360 DOCKER all -- 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL 20281 1217K CNI-HOSTPORT-DNAT all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL 11431 686K PROXY tcp -- 0.0.0.0/0 0.0.0.0/0 ! owner UID match 1002

Chain DOCKER (2 references) pkts bytes target prot opt in out source destination
0 0 RETURN all -- docker0 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all -- br-63fbfcfdc361 0.0.0.0/0 0.0.0.0/0
160 9600 RETURN all -- hassio 0.0.0.0/0 0.0.0.0/0
0 0 DNAT tcp -- !hassio 0.0.0.0/0 0.0.0.0/0 tcp dpt:4357 to:172.30.32.6:80 0 0 DNAT tcp -- !hassio 0.0.0.0/0 0.0.0.0/0 tcp dpt:8884 to:172.30.33.0:8884 0 0 DNAT tcp -- !hassio 0.0.0.0/0 0.0.0.0/0 tcp dpt:8883 to:172.30.33.0:8883 0 0 DNAT tcp -- !hassio 0.0.0.0/0 0.0.0.0/0 tcp dpt:1884 to:172.30.33.0:1884 1 60 DNAT tcp -- !hassio 0.0.0.0/0 0.0.0.0/0 tcp dpt:1883 to:172.30.33.0:1883

Chain KUBE-MARK-DROP (2 references) pkts bytes target prot opt in out source destination
0 0 MARK all -- 0.0.0.0/0 0.0.0.0/0 MARK or 0x8000

Chain KUBE-MARK-MASQ (76 references) pkts bytes target prot opt in out source destination
81 4860 MARK all -- 0.0.0.0/0 0.0.0.0/0 MARK or 0x4000

Chain KUBE-POSTROUTING (1 references) pkts bytes target prot opt in out source destination
332K 37M RETURN all -- 0.0.0.0/0 0.0.0.0/0 mark match ! 0x4000/0x4000 54 3240 MARK all -- 0.0.0.0/0 0.0.0.0/0 MARK xor 0x4000 54 3240 MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0 / kubernetes service traffic requiring SNAT / random-fully

Chain KUBE-KUBELET-CANARY (0 references) pkts bytes target prot opt in out source destination

Chain KUBE-PROXY-CANARY (0 references) pkts bytes target prot opt in out source destination

Chain CNI-HOSTPORT-SETMARK (18 references) pkts bytes target prot opt in out source destination
1645 98700 MARK all -- 0.0.0.0/0 0.0.0.0/0 / CNI portfwd masquerade mark / MARK or 0x2000

Chain CNI-HOSTPORT-MASQ (1 references) pkts bytes target prot opt in out source destination
1645 98700 MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0 mark match 0x2000/0x2000

Chain CNI-HOSTPORT-DNAT (2 references) pkts bytes target prot opt in out source destination
0 0 CNI-DN-dc797e37debf6a85448ca tcp -- 0.0.0.0/0 0.0.0.0/0 / dnat name: "cbr0" id: "be7df4ae5aa06f6524655da6eebb5602eaa9d3989a8fa31d2750cf3e0ae49fec" / multiport dports 18080,19090 (...) 0 0 CNI-DN-21b908ae30828e1659ee0 tcp -- 0.0.0.0/0 0.0.0.0/0 / dnat name: "cbr0" id: "93ee8543f0f034aa22a06f1d0f2e862efae1f53a66c2e23803cfbbb8341911e3" / multiport dports 20000

Chain CNI-DN-dc797e37debf6a85448ca (1 references) pkts bytes target prot opt in out source destination
0 0 CNI-HOSTPORT-SETMARK tcp -- 10.42.0.0/24 0.0.0.0/0 tcp dpt:18080 0 0 CNI-HOSTPORT-SETMARK tcp -- 127.0.0.1 0.0.0.0/0 tcp dpt:18080 0 0 DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:18080 to:10.42.0.54:18080 0 0 CNI-HOSTPORT-SETMARK tcp -- 10.42.0.0/24 0.0.0.0/0 tcp dpt:19090 0 0 CNI-HOSTPORT-SETMARK tcp -- 127.0.0.1 0.0.0.0/0 tcp dpt:19090 0 0 DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:19090 to:10.42.0.54:19090

Chain KUBE-SERVICES (2 references) pkts bytes target prot opt in out source destination
0 0 KUBE-SVC-RK657RLKDNVNU64O tcp -- 0.0.0.0/0 10.43.232.36 / calico-system/calico-typha:calico-typha cluster IP / tcp dpt:5473 (...) 0 0 KUBE-FW-CG5I4G2RS3ZVWGLK tcp -- 0.0.0.0/0 192.168.1.10 / ingress-nginx/ingress-nginx-controller:http loadbalancer IP / tcp dpt:9080 (...) 2126 155K KUBE-NODEPORTS all -- 0.0.0.0/0 0.0.0.0/0 / kubernetes service nodeports; NOTE: this must be the last rule in this chain / ADDRTYPE match dst-type LOCAL

Chain KUBE-NODEPORTS (1 references) pkts bytes target prot opt in out source destination
0 0 KUBE-SVC-EDNDUDH2C75GIR6O tcp -- 0.0.0.0/0 0.0.0.0/0 / ingress-nginx/ingress-nginx-controller:https / tcp dpt:31818 0 0 KUBE-SVC-CG5I4G2RS3ZVWGLK tcp -- 0.0.0.0/0 0.0.0.0/0 / ingress-nginx/ingress-nginx-controller:http / tcp dpt:31018

(... all KUBE-SVC-* rules)

Chain PROXY (2 references) pkts bytes target prot opt in out source destination
0 0 RETURN all -- 0.0.0.0/0 0.0.0.0/8
9935 596K RETURN all -- 0.0.0.0/0 10.42.0.0/16
421 25260 RETURN all -- 0.0.0.0/0 10.43.0.0/16
1436 86160 RETURN all -- 0.0.0.0/0 127.0.0.0/8
0 0 RETURN all -- 0.0.0.0/0 169.254.0.0/16
0 0 RETURN all -- 0.0.0.0/0 172.16.0.0/16
0 0 RETURN all -- 0.0.0.0/0 172.17.0.0/16
129 7740 RETURN all -- 0.0.0.0/0 172.30.0.0/16
292 17520 RETURN all -- 0.0.0.0/0 192.168.0.0/16
0 0 RETURN all -- 0.0.0.0/0 224.0.0.0/4
0 0 RETURN all -- 0.0.0.0/0 240.0.0.0/4
0 0 RETURN all -- 0.0.0.0/0 59.110.106.146
20 1200 RETURN all -- 0.0.0.0/0 45.32.127.212
61 3660 REDIRECT tcp -- 0.0.0.0/0 0.0.0.0/0 redir ports 7892


* k3s command

ExecStart=/usr/local/bin/k3s \ server \ --disable traefik \ --disable local-storage \ --cluster-init \ --node-ip 192.168.1.10 \ --node-external-ip 192.168.1.10 \ --node-name homelab-1 \ --node-label kubernetes.io/hostname=homelab-1 \ --flannel-backend=none \ --disable-network-policy \ --cluster-cidr=10.42.0.0/16 \ --kubelet-arg=max-pods=500

caseydavenport commented 2 years ago

Right, so Calico will by default attempt to put its iptables rules at the top of the chain so that nobody can sneak rules in front and bypass Calico's policy enforcement.

You can modify that behavior though using the chainInsertMode: Append option on FelixConfiguration: https://projectcalico.docs.tigera.io/reference/resources/felixconfig

That might get you what you need?

ccll commented 2 years ago

@caseydavenport Thanks for the help, unfortunately your solution does not work. Finally I gave up on fiddling with iptables and settled on a builtin 'tun' device solution provided by the proxy software Clash, which works perfect for both LOCAL and PREROUTING traffics. I'm closing this issue for now.

projectcalico / calico

How can I redirect PODs' outgoing connection through transparent proxy? #5978

Expected Behavior

Current Behavior

Your Environment

Warning: iptables-legacy tables present, use iptables-legacy to see them