Closed Farhanec07 closed 1 month ago
Please share your thoughts on this. We are currently blocked from upgrading to EKS 1.29 due to this issue.
What Linux distro/version do you use? Does it have (proper) support for iptables?
What Linux distro/version do you use? Does it have (proper) support for iptables?
we create cluster on ami which is amazon-linux-2-arm64 AMI on which we faced above issue
The actual AMIs which are in question here are the Optimized EKS ones (such as amazon-eks-arm64-node-1.26
and amazon-eks-arm64-node-1.29
).
All versions of these AMIs (even the x86/AMD64 ones) have the same version of iptables (v1.8.4):
rpm -q iptables nftables firewalld
iptables-1.8.4-10.amzn2.1.2.aarch64
package nftables is not installed
package firewalld is not installed
So I don't think this would be related to the version of iptables. The same commands work on much older 1.26 ARM instances (which work with earlier versions of Calico).
# iptables -A INPUT -s 1.2.3.4 -j DROP
# iptables-save
# Generated by iptables-save v1.8.4 on Fri Jun 14 19:59:06 2024
*filter
:INPUT ACCEPT [65:3736]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [34:2728]
-A INPUT -s 1.2.3.4/32 -j DROP
COMMIT
# Completed on Fri Jun 14 19:59:06 2024
@jonathan-hurley the function in question (xtables_strdup()
) is present in iptables v1.8.8 (which is what calico v3.27.2+ uses): https://github.com/PKRoma/iptables/blob/v1.8.8/libxtables/xtables.c#L463, but it doesn't seem to be there in the version you mentioned (v1.8.4): https://github.com/PKRoma/iptables/blob/v1.8.4/libxtables/xtables.c
Would it be possible to upgrade iptables to v1.8.8 in your instances? Alternatively, calico pre-v3.27.2 should be using iptables v1.8.4, could you try that and see if the issue is resolved? (not ideal, but this would at least help diagnose this)
Amazon EKS optimized images have always used 1.8.4; we do not have the option to change this.
We must use the latest versions of Calico in order to resolve CVEs.
@jonathan-hurley the function in question (
xtables_strdup()
) is present in iptables v1.8.8 (which is what calico v3.27.2+ uses): PKRoma/iptables@v1.8.8
/libxtables/xtables.c#L463, but it doesn't seem to be there in the version you mentioned (v1.8.4): PKRoma/iptables@v1.8.4
/libxtables/xtables.cWould it be possible to upgrade iptables to v1.8.8 in your instances? Alternatively, calico pre-v3.27.2 should be using iptables v1.8.4, could you try that and see if the issue is resolved? (not ideal, but this would at least help diagnose this)
Hi @coutinhop I got seg fault
2024-05-23 10:22:44.982 [WARNING][77] felix/table.go 816: iptables save failed error=signal: segmentation fault (core dumped)
2024-05-23 10:22:44.982 [WARNING][77] felix/table.go 765: iptables-nft-save command failed error=signal: segmentation fault (core dumped) ipVersion=0x4 stderr="" table="filter"
when using calico 3.26.3 on a Photon OS with iptables 1.8.9. Do you mean the iptables on the OS should also use 1.8.4 version to avoid issue? I wish calico prints more than just seg fault. Looks like hashes does contain stdout, maybe it could be printed in debug mode. Just found out it already in if debug mode
root [ /home/capv ]# iptables --version
iptables v1.8.9 (nf_tables)
If I do iptables -L
inside the calico pod, it's empty, but I got valid output if I run it inside the VM directly.
Is it because calico is still iptables-nft to call nftables API, so the iptables command on the calico pod which only talks to legacy API can't get correct information?
@coutinhop This change breaks Calico 3.27.2+ on every version of Amazon Linux 2 and every Optimized EKS AMI based on it. The latest versions of Amazon Linux 2 still only support iptables 1.8.4 are are not EOL for another entire year.
What is the possibility that the Felix change which caused this can be reverted and the xtables_strdup()
can not be used? It seems like this is just a wrapper function anyway.
@coutinhop I don't think this has anything to do with the AMI / version of iptables shipped on the Amazon VM instances. Calico packages the necessary libraries for the binaries it uses inside the container, so it sounds like we have published ARM images that are missing a necessary lib (I'm not able to reproduce this on amd64)
If I had to guess, we update the version of iptables to v1.8.8 which introduced a dependency on xtables_strdup
but somehow have not included that symbol in the shared libraries we pacakge into the container.
This commit is likely to be the one that introduced the problem: https://github.com/projectcalico/calico/pull/8485/commits/c053d1c72e29ad9046c0be97bce538b08f20807c
so it sounds like we have published ARM images that are missing a necessary lib (I'm not able to reproduce this on amd64)
@caseydavenport - I can reproduce this on AMD64 as well using an m5.large
instance type running Amazon Linux 2. It appears as though the architecture doesn't matter here.
Well, that certainly changes things. Perhaps there is some interaction with the host packages that I wasn't expecting.
Here are the details of the VM which I tried:
# uname -p
x86_64
# cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
SUPPORT_END="2025-06-30"
And when I try to invoke iptables from the Calico pod on this machine, I get the same xtables_strdup
message:
# iptables --version
iptables: symbol lookup error: iptables: undefined symbol: xtables_strdup
I managed to push a PR with a tentative fix, see here for more details: https://github.com/projectcalico/calico/pull/9022
@jonathan-hurley or @Farhanec07, would any of you be able to test this with an image I built locally with the fix and let us know if that gets rid of the problem?
amd64: quay.io/coutinho/calico-node:v3.28.0-amd64
arm64: quay.io/coutinho/calico-node:v3.28.0-arm64
Thanks!
@coutinhop - that fixed it!
calico-node 2024-07-18 15:23:34.839 [INFO][66] felix/int_dataplane.go 1360: Linux interface addrs changed. addrs=set.Set{} ifaceName="calico_tmp_B"
calico-node 2024-07-18 15:23:34.839 [INFO][66] felix/int_dataplane.go 1316: Linux interface state changed. ifIndex=91563 ifaceName="calico_tmp_A" state="down"
calico-node 2024-07-18 15:23:34.839 [INFO][66] felix/int_dataplane.go 1360: Linux interface addrs changed. addrs=set.Set{} ifaceName="calico_tmp_A"
calico-node 2024-07-18 15:23:34.850 [INFO][66] felix/wireguard.go 1704: Trying to connect to linkClient ipVersion=0x4
install-cni {
calico-node 2024-07-18 15:23:34.850 [INFO][66] felix/route_rule.go 189: Trying to connect to netlink
calico-node 2024-07-18 15:23:34.850 [INFO][66] felix/wireguard.go 635: Public key out of sync or updated ipVersion=0x4 ourPublicKey=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
calico-node 2024-07-18 15:23:34.869 [INFO][66] felix/int_dataplane.go 1857: Completed first update to dataplane. secsSinceStart=0.459817401
calico-node 2024-07-18 15:23:34.872 [INFO][66] felix/health.go 206: Health of component changed name="InternalDataplaneMainLoop" newReport="live,ready" oldReport="live,non-ready"
install-cni "type": "portmap",
install-cni "snat": true,
install-cni "capabilities": {"portMappings": true}
install-cni },
install-cni {
install-cni "type": "bandwidth",
install-cni "capabilities": {"bandwidth": true}
install-cni }
install-cni ]
calico-node 2024-07-18 15:23:34.872 [INFO][66] felix/int_dataplane.go 1957: Received interface update msg=&intdataplane.ifaceStateUpdate{Name:"calico_tmp_B", State:"down", Index:91562}
install-cni }
calico-node 2024-07-18 15:23:34.872 [INFO][66] felix/int_dataplane.go 1981: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"calico_tmp_B", Addrs:set.Typed[string]{}}
calico-node 2024-07-18 15:23:34.872 [INFO][66] felix/hostip_mgr.go 84: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"calico_tmp_B", Addrs:set.Typed[string]{}}
calico-node 2024-07-18 15:23:34.872 [INFO][66] felix/int_dataplane.go 1957: Received interface update msg=&intdataplane.ifaceStateUpdate{Name:"calico_tmp_A", State:"down", Index:91563}
calico-node 2024-07-18 15:23:34.872 [INFO][66] felix/int_dataplane.go 1981: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"calico_tmp_A", Addrs:set.Typed[string]{}}
calico-node 2024-07-18 15:23:34.872 [INFO][66] felix/hostip_mgr.go 84: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"calico_tmp_A", Addrs:set.Typed[string]{}}
calico-node 2024-07-18 15:23:34.914 [INFO][66] felix/int_dataplane.go 1316: Linux interface state changed. ifIndex=91563 ifaceName="calico_tmp_A" state=""
calico-node 2024-07-18 15:23:34.914 [INFO][66] felix/int_dataplane.go 1360: Linux interface addrs changed. addrs=<nil> ifaceName="calico_tmp_A"
calico-node 2024-07-18 15:23:34.914 [INFO][66] felix/int_dataplane.go 1957: Received interface update msg=&intdataplane.ifaceStateUpdate{Name:"calico_tmp_A", State:"", Index:91563}
calico-node 2024-07-18 15:23:34.914 [INFO][66] felix/int_dataplane.go 1981: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"calico_tmp_A", Addrs:set.Set[string](nil)}
calico-node 2024-07-18 15:23:34.914 [INFO][66] felix/hostip_mgr.go 84: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"calico_tmp_A", Addrs:set.Set[string](nil)}
calico-node 2024-07-18 15:23:34.915 [INFO][66] felix/int_dataplane.go 1316: Linux interface state changed. ifIndex=91562 ifaceName="calico_tmp_B" state=""
calico-node 2024-07-18 15:23:34.915 [INFO][66] felix/int_dataplane.go 1360: Linux interface addrs changed. addrs=<nil> ifaceName="calico_tmp_B"
calico-node 2024-07-18 15:23:34.915 [INFO][66] felix/int_dataplane.go 1957: Received interface update msg=&intdataplane.ifaceStateUpdate{Name:"calico_tmp_B", State:"", Index:91562}
calico-node 2024-07-18 15:23:34.915 [INFO][66] felix/int_dataplane.go 1981: Received interface addresses update msg=&intdataplane.ifaceAddrsUpdate{Name:"calico_tmp_B", Addrs:set.Set[string](nil)}
calico-node 2024-07-18 15:23:34.916 [INFO][66] felix/hostip_mgr.go 84: Interface addrs changed. update=&intdataplane.ifaceAddrsUpdate{Name:"calico_tmp_B", Addrs:set.Set[string](nil)}
calico-node 2024-07-18 15:23:34.916 [INFO][66] felix/int_dataplane.go 1867: Dataplane updates throttled
calico-node bird: device1: Initializing
calico-node bird: direct1: Initializing
calico-node bird: device1: Starting
calico-node bird: device1: Connected to table master
calico-node bird: device1: State changed to feed
calico-node bird: direct1: Starting
calico-node bird: direct1: Connected to table master
calico-node bird: direct1: State changed to feed
calico-node bird: Graceful restart started
calico-node bird: Graceful restart done
calico-node bird: Started
calico-node bird: device1: State changed to up
calico-node bird: direct1: State changed to up
calico-node bird: device1: Initializing
calico-node bird: direct1: Initializing
calico-node bird: Mesh_10_80_130_113: Initializing
calico-node bird: Mesh_10_80_160_114: Initializing
calico-node bird: device1: Starting
calico-node bird: device1: Connected to table master
calico-node bird: device1: State changed to feed
calico-node bird: direct1: Starting
calico-node bird: direct1: Connected to table master
calico-node bird: direct1: State changed to feed
calico-node bird: Mesh_10_80_130_113: Starting
calico-node bird: Mesh_10_80_130_113: State changed to start
calico-node bird: Mesh_10_80_160_114: Starting
calico-node bird: Mesh_10_80_160_114: State changed to start
calico-node bird: Graceful restart started
calico-node bird: Started
calico-node bird: device1: State changed to up
calico-node bird: direct1: State changed to up
calico-node bird: Mesh_10_80_160_114: Connected to table master
calico-node bird: Mesh_10_80_160_114: State changed to feed
calico-node bird: Mesh_10_80_160_114: State changed to up
calico-node 2024-07-18 15:23:39.866 [INFO][66] felix/int_dataplane.go 1826: Dataplane updates no longer throttled
Stream closed EOF for kube-system/calico-node-mbblk (install-cni)
Stream closed EOF for kube-system/calico-node-mbblk (mount-bpffs)
Stream closed EOF for kube-system/calico-node-mbblk (upgrade-ipam)
calico-node 2024-07-18 15:23:52.000 [INFO][66] felix/health.go 336: Overall health status changed: live=true ready=true
calico-node +---------------------------+---------+----------------+-----------------+--------+
calico-node | COMPONENT | TIMEOUT | LIVENESS | READINESS | DETAIL |
calico-node +---------------------------+---------+----------------+-----------------+--------+
calico-node | CalculationGraph | 30s | reporting live | reporting ready | |
calico-node | FelixStartup | - | reporting live | reporting ready | |
calico-node | InternalDataplaneMainLoop | 1m30s | reporting live | reporting ready | |
calico-node +---------------------------+---------+----------------+-----------------+--------+
Containers:
calico-node:
Container ID: containerd://8a230a9241a8bf8c09926d423e7f17fd2c1eeef30b7e9416011e9201f36d3b39
Image: quay.io/coutinho/calico-node:v3.28.0-arm64
Image ID: quay.io/coutinho/calico-node@sha256:6f30a1f29edf83f7e04626d883a936259a76bec9cde60f6945ce7e3a6d251439
Port: <none>
Host Port: <none>
State: Running
Started: Thu, 18 Jul 2024 11:23:33 -0400
Ready: True
@jonathan-hurley that is great news! I'm still having a bit of trouble reproducing the issue myself, could you share some more details about your setup? Like, inside a calico-node pod WITH the problem, could you run env
and ls -l /lib64/libxtables.so.12*
, cat /etc/ld.so.conf
, cat /etc/ld.so.conf.d/*
?
Also, it seems like it didn't happen consistently on all pods, did it? Anything in particular in your setup that could be a cause?
I've tried spinning up amazon linux 2 VMs and an EKS cluster, but to no avail iptables
and iptables-legacy-save
don't fail with that error for me...
I'm glad the issue seems to be gone, but trying to understand better what caused it, appreciate your help. Thanks!
Yes, the problem happened consistently on all pods, on both AMD64/x86 and ARM64 architectures. We use the Optimized EKS AMI images (which are based off of Amazon Linux 2).
# ls -l /lib64/libxtables.so.12*
lrwxrwxrwx 1 root root 20 Nov 16 2023 /lib64/libxtables.so.12 -> libxtables.so.12.3.0
-rwxr-xr-x 1 root root 70200 Nov 16 2023 /lib64/libxtables.so.12.3.0
# env
KUBE_DNS_SERVICE_PORT=53
KUBE_DNS_PORT_53_TCP_PROTO=tcp
KUBED_SERVICE_PORT=443
CALICO_TYPHA_SERVICE_PORT_CALICO_TYPHA=5473
IP_AUTODETECTION_METHOD=first-found
FELIX_TYPHAK8SSERVICENAME=calico-typha
KUBED_SERVICE_PORT_API=443
SVDIR=/etc/service/enabled
KUBED_PORT_443_TCP=tcp://172.20.136.172:443
CALICO_IPV4POOL_CIDR=100.100.0.0/16
LANG=C.utf8
HOSTNAME=ip-10-80-1-1.us-west-2.compute.internal
KUBED_PORT_443_TCP_PORT=443
CALICO_TYPHA_SERVICE_PORT=5473
FELIX_WIREGUARDMTU=0
KUBE_DNS_PORT_53_UDP=udp://172.20.0.10:53
FELIX_VXLANMTU=0
CALICO_TYPHA_SERVICE_HOST=172.20.122.83
CALICO_IPV4POOL_VXLAN=Never
KUBE_DNS_PORT_53_UDP_PROTO=udp
FELIX_HEALTHENABLED=true
CALICO_DISABLE_FILE_LOGGING=true
which_declare=declare -f
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_ADDR=172.20.0.1
container=oci
FELIX_USAGEREPORTINGENABLED=false
CALICO_TYPHA_PORT=tcp://172.20.122.83:5473
KUBE_DNS_PORT_53_UDP_ADDR=172.20.0.10
KUBERNETES_PORT=tcp://172.20.0.1:443
PWD=/
HOME=/root
CALICO_TYPHA_PORT_5473_TCP_PROTO=tcp
KUBE_DNS_PORT_53_TCP_ADDR=172.20.0.10
NODENAME=ip-10-80-1-1.us-west-2.compute.internal
KUBE_DNS_PORT=udp://172.20.0.10:53
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP_PORT=443
KUBED_PORT_443_TCP_PROTO=tcp
WAIT_FOR_DATASTORE=true
KUBED_PORT_443_TCP_ADDR=172.20.136.172
KUBERNETES_PORT_443_TCP=tcp://172.20.0.1:443
KUBE_DNS_PORT_53_TCP_PORT=53
CLUSTER_TYPE=k8s,bgp
CALICO_TYPHA_PORT_5473_TCP_ADDR=172.20.122.83
IP=autodetect
TERM=xterm
FELIX_DEFAULTENDPOINTTOHOSTACTION=ACCEPT
CALICO_IPV4POOL_IPIP=CrossSubnet
KUBE_DNS_PORT_53_TCP=tcp://172.20.0.10:53
CALICO_TYPHA_PORT_5473_TCP=tcp://172.20.122.83:5473
FELIX_IPINIPMTU=0
FELIX_LOGSEVERITYSCREEN=info
CALICO_IPV6POOL_VXLAN=Never
SHLVL=1
KUBERNETES_SERVICE_PORT=443
CALICO_NETWORKING_BACKEND=bird
KUBE_DNS_SERVICE_PORT_DNS=53
CALICO_TYPHA_PORT_5473_TCP_PORT=5473
KUBE_DNS_SERVICE_PORT_DNS_TCP=53
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DATASTORE_TYPE=kubernetes
KUBED_SERVICE_HOST=172.20.136.172
FELIX_IPV6SUPPORT=false
KUBERNETES_SERVICE_HOST=172.20.0.1
KUBE_DNS_PORT_53_UDP_PORT=53
KUBE_DNS_SERVICE_HOST=172.20.0.10
KUBED_PORT=tcp://172.20.136.172:443
BASH_FUNC_which%%=() { ( alias;
eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@
}
_=/usr/bin/env
# cat /etc/ld.so.conf
include ld.so.conf.d/*.conf
# ls -l /etc/ld.so.conf.d
total 0
Thank you @jonathan-hurley!
Just to clarify, is this in the calico-node pod? What image are you using?
This looks very weird:
# ls -l /lib64/libxtables.so.12*
lrwxrwxrwx 1 root root 20 Nov 16 2023 /lib64/libxtables.so.12 -> libxtables.so.12.3.0
-rwxr-xr-x 1 root root 70200 Nov 16 2023 /lib64/libxtables.so.12.3.0
As 'libxtables.so.12.3.0' is the "outdated" version of the lib which won't contain the xtables_strdup
symbol... Are you building a custom calico-node image? If so, can you share the steps/modifications? If not, can you share the image url?
Would you be willing to hit me up in the Calico Users slack? We could make this conversation a lot more real-time if so: https://slack.projectcalico.org/
Yes, these commands were being run from the calico-node pod before it died due to Felix crashing. The image for this run was ami-0be1daad79c89dd0a with a build ID of amazon/amazon-eks-node-1.28-v20240703
.
Sure, let me hop on slack ...
Resolution was that the user was using an incorrectly built external image from Iron Bank. It had incorrect versions of the libraries.
Expected Behavior
Current Behavior
panic which i observed that its failing to save iptables rules causing pods to crash. calico-node pod log -
checked cni.log . could see only below error are
while exec into pod iptables cmd is not executing
Possible Solution
Steps to Reproduce (for bugs)
1. 2. 3. 4.
Context
Your Environment