search

projectcalico / calico

Cloud native networking and network security
https://docs.tigera.io/calico/latest/about/
Apache License 2.0
5.88k stars 1.31k forks source link

iptables-legacy-save command failed after retries and getHashesAndRulesFromDataplane panic #7057

Closed futurist closed 1 year ago

futurist commented 1 year ago

Expected Behavior

Current Behavior

some node panic and not ready after install:

2022-12-03 13:55:50.096 [WARNING][2072] felix/table.go 816: iptables save failed error=exit status 1
2022-12-03 13:55:50.096 [WARNING][2072] felix/table.go 765: iptables-legacy-save command failed error=exit status 1 ipVersion=0x4 stderr="" table="filter"
2022-12-03 13:55:50.299 [WARNING][2072] felix/table.go 816: iptables save failed error=exit status 1
2022-12-03 13:55:50.299 [WARNING][2072] felix/table.go 765: iptables-legacy-save command failed error=exit status 1 ipVersion=0x4 stderr="" table="filter"
2022-12-03 13:55:50.702 [WARNING][2072] felix/table.go 816: iptables save failed error=exit status 1
2022-12-03 13:55:50.702 [WARNING][2072] felix/table.go 765: iptables-legacy-save command failed error=exit status 1 ipVersion=0x4 stderr="" table="filter"
2022-12-03 13:55:50.702 [PANIC][2072] felix/table.go 771: iptables-legacy-save command failed after retries ipVersion=0x4 table="filter"
panic: (*logrus.Entry) 0xc000340620

goroutine 405 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc000848c40, 0x0, {0xc000de0040, 0x31})
    /go/pkg/mod/github.com/sirupsen/logrus@v1.9.0/entry.go:260 +0x47e
github.com/sirupsen/logrus.(*Entry).Log(0xc000848c40, 0x0, {0xc00067aa28?, 0x1?, 0x1?})
    /go/pkg/mod/github.com/sirupsen/logrus@v1.9.0/entry.go:304 +0x4f
github.com/sirupsen/logrus.(*Entry).Logf(0xc000848c40, 0x0, {0x2f7ac52?, 0x6?}, {0xc00067aaf0?, 0xc000034840?, 0xc000b75000?})
    /go/pkg/mod/github.com/sirupsen/logrus@v1.9.0/entry.go:349 +0x85
github.com/sirupsen/logrus.(*Entry).Panicf(...)
    /go/pkg/mod/github.com/sirupsen/logrus@v1.9.0/entry.go:387
github.com/projectcalico/calico/felix/iptables.(*Table).getHashesAndRulesFromDataplane(0xc0001b0c00)
    /go/src/github.com/projectcalico/calico/felix/iptables/table.go:771 +0x3db
github.com/projectcalico/calico/felix/iptables.(*Table).loadDataplaneState(0xc0001b0c00)
    /go/src/github.com/projectcalico/calico/felix/iptables/table.go:608 +0x196
github.com/projectcalico/calico/felix/iptables.(*Table).Apply(0xc0001b0c00)
    /go/src/github.com/projectcalico/calico/felix/iptables/table.go:992 +0x372
github.com/projectcalico/calico/felix/dataplane/linux.(*InternalDataplane).apply.func4(0x0?)
    /go/src/github.com/projectcalico/calico/felix/dataplane/linux/int_dataplane.go:2034 +0x52
created by github.com/projectcalico/calico/felix/dataplane/linux.(*InternalDataplane).apply
    /go/src/github.com/projectcalico/calico/felix/dataplane/linux/int_dataplane.go:2033 +0x717

Steps to Reproduce (for bugs)

  1. kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.24.5/manifests/tigera-operator.yaml
  2. with custom-resources.yaml: 
    apiVersion: crd.projectcalico.org/v1
kind: IPPool
metadata:
annotations:
projectcalico.org/metadata: '{"uid":"c383d5a0-4c10-47ab-ae69-56206b8d9530","creationTimestamp":"2022-12-03T13:25:36Z"}'
creationTimestamp: "2022-12-03T13:25:36Z"
generation: 2
name: default-ipv4-ippool
resourceVersion: "2360467"
uid: 558b14ec-e4e8-4693-9cdc-77784a6a458e
spec:
allowedUses:
- Workload
- Tunnel
blockSize: 26
cidr: 192.168.0.0/16
ipipMode: Always
natOutgoing: true
nodeSelector: all()
vxlanMode: Never

Context

Your Environment

caseydavenport commented 1 year ago

apiVersion: crd.projectcalico.org/v1

Likely not the problem, but I am obligated to reference this issue: https://github.com/projectcalico/calico/issues/6412

Generally I've seen panics like this show up when the host is missing a kernel feature / module that we require. It looks like you have the log verbosity turned down to WARNING - I believe if you set it to INFO (or DEBUG) it will give more information about the command that failed.

futurist commented 1 year ago

I followed below steps to setup calico: https://projectcalico.docs.tigera.io/getting-started/kubernetes/quickstart

Could you clarify how to set log level to INFO in my Installation CRD(the custom-resources.yaml file)?

song-jiang commented 1 year ago

To update log level, update the field logSeverityScreen in the default FelixConfiguration

tomastigera commented 1 year ago

@futurist any new information after you increased the log severity? Closing for now.

waheedi commented 1 year ago

Here is some more debugging information: I can't really tell why this is failing as I have not read the code yet :D 

 Read hashes from dataplane: map[string][]string{"FORWARD":[]string{"wUHhoiAYhphO9Mso", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "S93hcgKJrXEqnTfs", "mp77cMpurHhyjLrM"}, "INPUT":[]string{"Cz_u1IQiXIMmKD4c", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", ""}, "KUBE-EXTERNAL-SERVICES":[]string{}, "KUBE-FIREWALL":[]string{""}, "KUBE-FORWARD":[]string{"", "", ""}, "KUBE-KUBELET-CANARY":[]string{}, "KUBE-NODEPORTS":[]string{}, "KUBE-PROXY-CANARY":[]string{}, "KUBE-PROXY-FIREWALL":[]string{}, "KUBE-SERVICES":[]string{"", "", ""}, "OUTPUT":[]string{"tVnHkvAo15HuiPy0", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", ""}, "cali-FORWARD":[]string{"vjrMJCRpqwy5oRoX", "A_sPAO0mcxbT9mOV", "8ZoYfO5HKXWbB3pk", "jdEuaPBe14V2hutn", "12bc6HljsMKsmfr-", "NOSxoaGx8OIstr1z"}, "cali-INPUT":[]string{"PajejrV4aFdkZojI", "_wjq-Yrma8Ly1Svo", "Tg6S9Eo_JS75EQjk", "jS_-YNc_itpr8SQU", "gYs9tYR3n_9c6pIu", "3YW6A7LJCwdp9Flz", "3WC_LYs974WQHEC5", "2tsRNaRVV2iHkTLN"}, "cali-OUTPUT":[]string{"Mq1_rAdXXH3YkrzW", "69FkRTJDvD5Vu6Vl", "AnEsmO6bDZbQntWW", "1lPEu3eMab1b36wW", "8nTUM5jQgrHB6Eg-", "UT6kRIJ617f9pHEF", "7y7RH3HSHgRj2P5G"}, "cali-cidr-block":[]string{}, "cali-from-hep-forward":[]string{}, "cali-from-host-endpoint":[]string{}, "cali-from-wl-dispatch":[]string{"zTj6P0TIgYvgz-md"}, "cali-to-hep-forward":[]string{}, "cali-to-host-endpoint":[]string{}, "cali-to-wl-dispatch":[]string{"7KNphB1nNHw80nIO"}, "cali-wl-to-host":[]string{"Ee9Sbo10IpVujdIY", "nSZbcOoG1xPONxb8"}} ipVersion=0x4 table="filter"
2023-06-08 08:32:28.534 [DEBUG][10006] felix/table.go 934: Read rules from dataplane: map[string][]string{"FORWARD":[]string{"-A FORWARD -m comment --comment \"cali:wUHhoiAYhphO9Mso\" -j cali-FORWARD", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-A FORWARD -m comment --comment \"cali:S93hcgKJrXEqnTfs\" -m comment --comment \"Policy explicitly accepted packet.\" -m mark --mark 0x10000/0x10000 -j ACCEPT", "-A FORWARD -m comment --comment \"cali:mp77cMpurHhyjLrM\" -j MARK --set-xmark 0x10000/0x10000"}, "INPUT":[]string{"-A INPUT -m comment --comment \"cali:Cz_u1IQiXIMmKD4c\" -j cali-INPUT", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-"}, "OUTPUT":[]string{"-A OUTPUT -m comment --comment \"cali:tVnHkvAo15HuiPy0\" -j cali-OUTPUT", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-"}} ipVersion=0x4 table="filter"
2023-06-08 08:32:28.534 [DEBUG][10006] felix/table.go 667: Scanning for unexpected iptables chains ipVersion=0x4 table="filter"
2023-06-08 08:32:28.534 [DEBUG][10006] felix/table.go 672: Skipping known-dirty chain chainName="cali-to-hep-forward" ipVersion=0x4 table="filter"
2023-06-08 08:32:28.534 [DEBUG][10006] felix/table.go 672: Skipping known-dirty chain chainName="cali-to-wl-dispatch" ipVersion=0x4 table="filter"
2023-06-08 08:32:28.534 [DEBUG][10006] felix/table.go 672: Skipping known-dirty chain chainName="cali-wl-to-host" ipVersion=0x4 table="filter"
2023-06-08 08:32:28.534 [DEBUG][10006] felix/table.go 672: Skipping known-dirty chain chainName="cali-INPUT" ipVersion=0x4 table="filter"
2023-06-08 08:32:28.534 [DEBUG][10006] felix/table.go 672: Skipping known-dirty chain chainName="cali-OUTPUT" ipVersion=0x4 table="filter"
2023-06-08 08:32:28.534 [DEBUG][10006] felix/table.go 672: Skipping known-dirty chain chainName="cali-cidr-block" ipVersion=0x4 table="filter"
2023-06-08 08:32:28.534 [DEBUG][10006] felix/table.go 672: Skipping known-dirty chain chainName="cali-from-hep-forward" ipVersion=0x4 table="filter"
2023-06-08 08:32:28.534 [DEBUG][10006] felix/table.go 672: Skipping known-dirty chain chainName="FORWARD" ipVersion=0x4 table="filter"
2023-06-08 08:32:28.534 [DEBUG][10006] felix/table.go 672: Skipping known-dirty chain chainName="OUTPUT" ipVersion=0x4 table="filter"
2023-06-08 08:32:28.534 [DEBUG][10006] felix/table.go 672: Skipping known-dirty chain chainName="cali-to-host-endpoint" ipVersion=0x4 table="filter"
2023-06-08 08:32:28.534 [DEBUG][10006] felix/table.go 672: Skipping known-dirty chain chainName="INPUT" ipVersion=0x4 table="filter"
2023-06-08 08:32:28.534 [DEBUG][10006] felix/table.go 672: Skipping known-dirty chain chainName="cali-FORWARD" ipVersion=0x4 table="filter"
2023-06-08 08:32:28.534 [DEBUG][10006] felix/table.go 672: Skipping known-dirty chain chainName="cali-from-wl-dispatch" ipVersion=0x4 table="filter"
2023-06-08 08:32:28.534 [DEBUG][10006] felix/table.go 672: Skipping known-dirty chain chainName="cali-from-host-endpoint" ipVersion=0x4 table="filter"
2023-06-08 08:32:28.534 [DEBUG][10006] felix/table.go 699: Finished loading iptables state ipVersion=0x4 table="filter"
2023-06-08 08:32:28.534 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Clear:0xe0000 chain="cali-FORWARD" hash="vjrMJCRpqwy5oRoX" position=0 ruleFragment="-A cali-FORWARD HASH --jump MARK --set-mark 0/0xe0000"
2023-06-08 08:32:28.535 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Jump->cali-from-hep-forward chain="cali-FORWARD" hash="A_sPAO0mcxbT9mOV" position=1 ruleFragment="-A cali-FORWARD HASH -m mark --mark 0/0x10000 --jump cali-from-hep-forward"
2023-06-08 08:32:28.535 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Jump->cali-from-wl-dispatch chain="cali-FORWARD" hash="8ZoYfO5HKXWbB3pk" position=2 ruleFragment="-A cali-FORWARD HASH --in-interface cali+ --jump cali-from-wl-dispatch"
2023-06-08 08:32:28.535 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Jump->cali-to-wl-dispatch chain="cali-FORWARD" hash="jdEuaPBe14V2hutn" position=3 ruleFragment="-A cali-FORWARD HASH --out-interface cali+ --jump cali-to-wl-dispatch"
2023-06-08 08:32:28.535 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Jump->cali-to-hep-forward chain="cali-FORWARD" hash="12bc6HljsMKsmfr-" position=4 ruleFragment="-A cali-FORWARD HASH --jump cali-to-hep-forward"
2023-06-08 08:32:28.535 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Jump->cali-cidr-block chain="cali-FORWARD" hash="NOSxoaGx8OIstr1z" position=5 ruleFragment="-A cali-FORWARD HASH --jump cali-cidr-block"
2023-06-08 08:32:28.535 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Accept chain="cali-OUTPUT" hash="Mq1_rAdXXH3YkrzW" position=0 ruleFragment="-A cali-OUTPUT HASH -m mark --mark 0x10000/0x10000 --jump ACCEPT"
2023-06-08 08:32:28.535 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Return chain="cali-OUTPUT" hash="69FkRTJDvD5Vu6Vl" position=1 ruleFragment="-A cali-OUTPUT HASH --out-interface cali+ --jump RETURN"
2023-06-08 08:32:28.535 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Accept chain="cali-OUTPUT" hash="AnEsmO6bDZbQntWW" position=2 ruleFragment="-A cali-OUTPUT HASH -m comment --comment \"Allow IPIP packets to other Calico hosts\" -p 4 -m set --match-set cali40all-hosts-net dst -m addrtype --src-type LOCAL --jump ACCEPT"
2023-06-08 08:32:28.535 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Accept chain="cali-OUTPUT" hash="1lPEu3eMab1b36wW" position=3 ruleFragment="-A cali-OUTPUT HASH -m comment --comment \"Allow outgoing IPv4 Wireguard packets\" -p 17 -m multiport --destination-ports 51820 -m addrtype --src-type LOCAL --jump ACCEPT"
2023-06-08 08:32:28.535 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Clear:0xf0000 chain="cali-OUTPUT" hash="8nTUM5jQgrHB6Eg-" position=4 ruleFragment="-A cali-OUTPUT HASH --jump MARK --set-mark 0/0xf0000"
2023-06-08 08:32:28.535 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Jump->cali-to-host-endpoint chain="cali-OUTPUT" hash="UT6kRIJ617f9pHEF" position=5 ruleFragment="-A cali-OUTPUT HASH -m conntrack ! --ctstate DNAT --jump cali-to-host-endpoint"
2023-06-08 08:32:28.535 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Accept chain="cali-OUTPUT" hash="7y7RH3HSHgRj2P5G" position=6 ruleFragment="-A cali-OUTPUT HASH -m comment --comment \"Host endpoint policy accepted packet.\" -m mark --mark 0x10000/0x10000 --jump ACCEPT"
2023-06-08 08:32:28.535 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Drop chain="cali-from-wl-dispatch" hash="zTj6P0TIgYvgz-md" position=0 ruleFragment="-A cali-from-wl-dispatch HASH -m comment --comment \"Unknown interface\" --jump DROP"
2023-06-08 08:32:28.535 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Drop chain="cali-to-wl-dispatch" hash="7KNphB1nNHw80nIO" position=0 ruleFragment="-A cali-to-wl-dispatch HASH -m comment --comment \"Unknown interface\" --jump DROP"
2023-06-08 08:32:28.536 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Jump->cali-from-wl-dispatch chain="cali-wl-to-host" hash="Ee9Sbo10IpVujdIY" position=0 ruleFragment="-A cali-wl-to-host HASH --jump cali-from-wl-dispatch"
2023-06-08 08:32:28.536 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Accept chain="cali-wl-to-host" hash="nSZbcOoG1xPONxb8" position=1 ruleFragment="-A cali-wl-to-host HASH -m comment --comment \"Configured DefaultEndpointToHostAction\" --jump ACCEPT"
2023-06-08 08:32:28.536 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Accept chain="cali-INPUT" hash="PajejrV4aFdkZojI" position=0 ruleFragment="-A cali-INPUT HASH -m comment --comment \"Allow IPIP packets from Calico hosts\" -p 4 -m set --match-set cali40all-hosts-net src -m addrtype --dst-type LOCAL --jump ACCEPT"
2023-06-08 08:32:28.536 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Drop chain="cali-INPUT" hash="_wjq-Yrma8Ly1Svo" position=1 ruleFragment="-A cali-INPUT HASH -m comment --comment \"Drop IPIP packets from non-Calico hosts\" -p 4 --jump DROP"
2023-06-08 08:32:28.536 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Accept chain="cali-INPUT" hash="Tg6S9Eo_JS75EQjk" position=2 ruleFragment="-A cali-INPUT HASH -m comment --comment \"Allow incoming IPv4 Wireguard packets\" -p 17 -m multiport --destination-ports 51820 -m addrtype --dst-type LOCAL --jump ACCEPT"
2023-06-08 08:32:28.536 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Goto->cali-wl-to-host chain="cali-INPUT" hash="jS_-YNc_itpr8SQU" position=3 ruleFragment="-A cali-INPUT HASH --in-interface cali+ --goto cali-wl-to-host"
2023-06-08 08:32:28.536 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Accept chain="cali-INPUT" hash="gYs9tYR3n_9c6pIu" position=4 ruleFragment="-A cali-INPUT HASH -m mark --mark 0x10000/0x10000 --jump ACCEPT"
2023-06-08 08:32:28.536 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Clear:0xf0000 chain="cali-INPUT" hash="3YW6A7LJCwdp9Flz" position=5 ruleFragment="-A cali-INPUT HASH --jump MARK --set-mark 0/0xf0000"
2023-06-08 08:32:28.536 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Jump->cali-from-host-endpoint chain="cali-INPUT" hash="3WC_LYs974WQHEC5" position=6 ruleFragment="-A cali-INPUT HASH --jump cali-from-host-endpoint"
2023-06-08 08:32:28.536 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Accept chain="cali-INPUT" hash="2tsRNaRVV2iHkTLN" position=7 ruleFragment="-A cali-INPUT HASH -m comment --comment \"Host endpoint policy accepted packet.\" -m mark --mark 0x10000/0x10000 --jump ACCEPT"
2023-06-08 08:32:28.536 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Jump->cali-INPUT chain="INPUT" hash="Cz_u1IQiXIMmKD4c" position=0 ruleFragment="-A INPUT HASH --jump cali-INPUT"
2023-06-08 08:32:28.536 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Jump->cali-FORWARD chain="FORWARD" hash="wUHhoiAYhphO9Mso" position=0 ruleFragment="-A FORWARD HASH --jump cali-FORWARD"
2023-06-08 08:32:28.536 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Accept chain="FORWARD*appends*" hash="S93hcgKJrXEqnTfs" position=0 ruleFragment="-A FORWARD*appends* HASH -m comment --comment \"Policy explicitly accepted packet.\" -m mark --mark 0x10000/0x10000 --jump ACCEPT"
2023-06-08 08:32:28.536 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Set:0x10000 chain="FORWARD*appends*" hash="mp77cMpurHhyjLrM" position=1 ruleFragment="-A FORWARD*appends* HASH --jump MARK --set-mark 0x10000/0x10000"
2023-06-08 08:32:28.537 [DEBUG][10006] felix/rules.go 169: Hashed rule action=Jump->cali-OUTPUT chain="OUTPUT" hash="tVnHkvAo15HuiPy0" position=0 ruleFragment="-A OUTPUT HASH --jump cali-OUTPUT"
2023-06-08 08:32:28.537 [DEBUG][10006] felix/table.go 1263: Update ended up being no-op, skipping call to ip(6)tables-restore. ipVersion=0x4 table="filter"
2023-06-08 08:32:28.604 [DEBUG][10006] felix/table.go 933: Read hashes from dataplane: map[string][]string{} ipVersion=0x4 table="raw"
2023-06-08 08:32:28.604 [DEBUG][10006] felix/table.go 934: Read rules from dataplane: map[string][]string{} ipVersion=0x4 table="raw"
2023-06-08 08:32:28.604 [WARNING][10006] felix/table.go 816: iptables save failed error=exit status 1
2023-06-08 08:32:28.604 [WARNING][10006] felix/table.go 765: iptables-legacy-save command failed error=exit status 1 ipVersion=0x4 stderr="" table="raw"
2023-06-08 08:32:28.807 [DEBUG][10006] felix/table.go 933: Read hashes from dataplane: map[string][]string{} ipVersion=0x4 table="raw"
2023-06-08 08:32:28.807 [DEBUG][10006] felix/table.go 934: Read rules from dataplane: map[string][]string{} ipVersion=0x4 table="raw"
2023-06-08 08:32:28.808 [WARNING][10006] felix/table.go 816: iptables save failed error=exit status 1
2023-06-08 08:32:28.808 [WARNING][10006] felix/table.go 765: iptables-legacy-save command failed error=exit status 1 ipVersion=0x4 stderr="" table="raw"
2023-06-08 08:32:29.211 [DEBUG][10006] felix/table.go 933: Read hashes from dataplane: map[string][]string{} ipVersion=0x4 table="raw"
2023-06-08 08:32:29.211 [DEBUG][10006] felix/table.go 934: Read rules from dataplane: map[string][]string{} ipVersion=0x4 table="raw"
2023-06-08 08:32:29.211 [WARNING][10006] felix/table.go 816: iptables save failed error=exit status 1
2023-06-08 08:32:29.211 [WARNING][10006] felix/table.go 765: iptables-legacy-save command failed error=exit status 1 ipVersion=0x4 stderr="" table="raw"
2023-06-08 08:32:29.212 [PANIC][10006] felix/table.go 771: iptables-legacy-save command failed after retries ipVersion=0x4 table="raw"
panic: (*logrus.Entry) 0xc00025ee70

goroutine 390 [running]:
github.com/sirupsen/logrus.(*Entry).log(0xc000269dc0, 0x0, {0xc000ab6e00, 0x31})
        /go/pkg/mod/github.com/sirupsen/logrus@v1.9.0/entry.go:260 +0x4d6
github.com/sirupsen/logrus.(*Entry).Log(0xc000269dc0, 0x0, {0xc00161ca10?, 0x1?, 0x1?})
        /go/pkg/mod/github.com/sirupsen/logrus@v1.9.0/entry.go:304 +0x4f
github.com/sirupsen/logrus.(*Entry).Logf(0xc000269dc0, 0x0, {0x31c6768?, 0x6?}, {0xc00161cad8?, 0xffffffffffffffff?, 0xc00161cbc8?})
        /go/pkg/mod/github.com/sirupsen/logrus@v1.9.0/entry.go:349 +0x85
github.com/sirupsen/logrus.(*Entry).Panicf(...)
        /go/pkg/mod/github.com/sirupsen/logrus@v1.9.0/entry.go:387
github.com/projectcalico/calico/felix/iptables.(*Table).getHashesAndRulesFromDataplane(0xc0001a8fc0)
        /go/src/github.com/projectcalico/calico/felix/iptables/table.go:771 +0x3db
github.com/projectcalico/calico/felix/iptables.(*Table).loadDataplaneState(0xc0001a8fc0)
        /go/src/github.com/projectcalico/calico/felix/iptables/table.go:608 +0x1a5
github.com/projectcalico/calico/felix/iptables.(*Table).Apply(0xc0001a8fc0)
        /go/src/github.com/projectcalico/calico/felix/iptables/table.go:992 +0x371
github.com/projectcalico/calico/felix/dataplane/linux.(*InternalDataplane).apply.func4(0xc0009a1800?)
        /go/src/github.com/projectcalico/calico/felix/dataplane/linux/int_dataplane.go:2120 +0x65
created by github.com/projectcalico/calico/felix/dataplane/linux.(*InternalDataplane).apply
        /go/src/github.com/projectcalico/calico/felix/dataplane/linux/int_dataplane.go:2119 +0x12d6
W0608 08:32:29.248448   10082 feature_gate.go:241] Setting GA feature gate ServiceInternalTrafficPolicy=true. It will be removed in a future release.
2023-06-08 08:32:29.250 [INFO][10082] felix/logutils.go 82: Early screen log level set to debug
2023-06-08 08:32:29.250 [DEBUG][10082] felix/daemon.go 132: No GOGC value set, defaulting to 20%.
2023-06-08 08:32:29.251 [INFO][10082] felix/daemon.go 148: Felix starting up GOMAXPROCS=16 builddate="2023-05-26T22:15:13+0000" gitcommit="8b103f46fbdc989e59d81e08d215ab4a59fa6cec" version="v3.26.0"
waheedi commented 1 year ago

So lets try to solve it, can we debug what is happening here: I guess this should be updated :

log.WithError(err).Warnf("Failed to get stdout pipe for %s", t.iptablesSaveCmd) With this as it actually gives the full cmd its trying to execute, me know, no go ,but I think this:

log.WithError(err).Warnf("Failed to get stdout pipe for %s", cmd) Then I can try to execute this failing cmd manually

lotthuang commented 9 months ago

Hi ,Is this issue solved? Where is the solution？

lotthuang commented 8 months ago

Hi ,Is this issue solved? Where is the solution？

WX001

xdsdmg commented 3 months ago

Hi ,Is this issue solved? Where is the solution？

  • I Have sloved this problem by this way：
  • When you login into the caclico pod，running the iptables-save command , you will find the root cause:

WX001

  • Then you can disable the tcpcsum module in the host or install it in the pod.

Hello, I had the same problem, but I do not know how to disable the tcpcsum module in the host.

waheedi commented 3 months ago

What was mainly missing is a kernel module that was needed to run some backend network for calcio. vxlan or something else. I recompiled the kernel with that new module which totally forgot what it was but it also can be a different module in your case. Try to observe the kernel log.

Anyway the host machine should mostly have that kernel module but it also depends what is running on that host machine :)

All my best,

On Thu, Jun 13, 2024 at 9:50 AM ZhangChi @.***> wrote:

Hi ,Is this issue solved? Where is the solution？

  • I Have sloved this problem by this way：
  • When you login into the caclico pod，running the iptables-save command , you will find the root cause:

[image: WX001] https://private-user-images.githubusercontent.com/28944052/292759939-592ccff4-5968-4d3b-a9e6-fc8a7ce1791c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTgyNjQyNDcsIm5iZiI6MTcxODI2Mzk0NywicGF0aCI6Ii8yODk0NDA1Mi8yOTI3NTk5MzktNTkyY2NmZjQtNTk2OC00ZDNiLWE5ZTYtZmM4YTdjZTE3OTFjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA2MTMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNjEzVDA3MzIyN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTNlNjJhYmRlYmFkZTkxNDVhYzljNWUyOTgyZjE1ODdlY2I5OTZhNjgzYmQ5M2ZlMThjODYyMjIxYzBhOGY0YmYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.snT4Lrb6gp_InAhhAgdLDIyi57u94NPEphZ_zW_u9BE

  • Then you can disable the tcpcsum module in the host or install it in the pod.

Hello, I had the same problem, but I do not know how to disable the tcpcsum module in the host.

— Reply to this email directly, view it on GitHub https://github.com/projectcalico/calico/issues/7057#issuecomment-2164886885, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAATOGYUDKIJFSJJO3E74Q3ZHFFK7AVCNFSM6AAAAABJH2LT56VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRUHA4DMOBYGU . You are receiving this because you commented.Message ID: @.***>

lotthuang commented 3 months ago

@xdsdmg Make sure no iptables rule includes the tcpcsum module.