Closed defo89 closed 8 months ago
The latest release of kube-proxy uses iptables v1.8.9, I believe bumping iptables to v1.8.9 is meaningful.
root@10-20-1-20:~# kubectl exec -it -n kube-system kube-proxy-hxkrd bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "981dab023a784a4694230dcd3becc8596f77b285843b220fdab80c0636fd4d55": OCI runtime exec failed: exec failed: unable to start container process: exec: "bash": executable file not found in $PATH: unknown
root@10-20-1-20:~# kubectl exec -it -n kube-system kube-proxy-hxkrd sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
# iptables -V
iptables v1.8.9 (nf_tables)
# exit
root@10-20-1-20:~# kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.0
I would like to work on this.
/cc @caseydavenport
I have played a bit with node/Dockerfile.amd64
and was able to confirm that the error is not happening with iptables ≥1.8.8. This of course is not a full fix since legacy case is not covered.
diff --git a/node/Dockerfile.amd64 b/node/Dockerfile.amd64
index b3f47e7b7..9941ba9f1 100644
--- a/node/Dockerfile.amd64
+++ b/node/Dockerfile.amd64
@@ -13,7 +13,7 @@
# limitations under the License.
ARG ARCH=x86_64
ARG GIT_VERSION=unknown
-ARG IPTABLES_VER=1.8.4-17
+ARG IPTABLES_VER=1.8.8-6
ARG LIBNFTNL_VER=1.1.5-4
ARG IPSET_VER=7.11-6
ARG RUNIT_VER=2.1.2
@@ -35,7 +35,7 @@ ARG IPSET_VER
ARG RUNIT_VER
ARG CENTOS_MIRROR_BASE_URL=http://linuxsoft.cern.ch/centos-vault/8.4.2105
ARG LIBNFTNL_SOURCERPM_URL=${CENTOS_MIRROR_BASE_URL}/BaseOS/Source/SPackages/libnftnl-${LIBNFTNL_VER}.el8.src.rpm
-ARG IPTABLES_SOURCERPM_URL=${CENTOS_MIRROR_BASE_URL}/BaseOS/Source/SPackages/iptables-${IPTABLES_VER}.el8.src.rpm
+ARG IPTABLES_SOURCERPM_URL=https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/source/tree/Packages/iptables-${IPTABLES_VER}.el9.src.rpm
ARG STREAM9_MIRROR_BASE_URL=https://iad.mirror.rackspace.com/centos-stream/9-stream
ARG IPSET_SOURCERPM_URL=${STREAM9_MIRROR_BASE_URL}/BaseOS/source/tree/Packages/ipset-${IPSET_VER}.el9.src.rpm
@@ -165,7 +165,7 @@ RUN rm /etc/yum.repos.d/ubi.repo && \
rpm --force -i /tmp/rpms/iptables-libs-${IPTABLES_VER}.el8.${ARCH}.rpm && \
# Install compatible libnftnl version with selected iptables version
rpm --force -i /tmp/rpms/libnftnl-${LIBNFTNL_VER}.el8.${ARCH}.rpm && \
- rpm -i /tmp/rpms/iptables-${IPTABLES_VER}.el8.${ARCH}.rpm && \
+ rpm -i /tmp/rpms/iptables-nft-${IPTABLES_VER}.el8.${ARCH}.rpm && \
# Install ipset version
rpm --force -i /tmp/rpms/ipset-libs-${IPSET_VER}.el8.x86_64.rpm && \
rpm -i /tmp/rpms/ipset-${IPSET_VER}.el8.x86_64.rpm && \
@@ -221,8 +221,8 @@ RUN chmod u+s /bin/mountns
# Clean out as many files as we can from the filesystem. We no longer need dnf or the platform python install
# or any of its dependencies.
-ADD clean-up-filesystem.sh /
-RUN /clean-up-filesystem.sh
Log:
2024-01-16T10:43:52.162002831Z 2024-01-16 10:43:52.161 [INFO][80] felix/feature_detect_linux.go 170: Updating detected iptables features features=environment.Features{SNATFullyRandom:true, MASQFullyRandom:true, RestoreSupportsLock:true, ChecksumOffloadBroken:true, IPIPDeviceIsL3:true, KernelSideRouteFiltering:true} iptablesVersion=1.8.8 kernelVersion=6.1.66
2024-01-16T10:43:52.162141257Z 2024-01-16 10:43:52.161 [INFO][80] felix/table.go 344: Calculated old-insert detection regex. pattern="(?:-j|--jump) cali-|(?:-j|--jump) califw-|(?:-j|--jump) calitw-|(?:-j|--jump) califh-|(?:-j|--jump) calith-|(?:-j|--jump) calipi-|(?:-j|--jump) calipo-|(?:-j|--jump) felix-"
2024-01-16T10:43:52.162284671Z 2024-01-16 10:43:52.162 [INFO][80] felix/table.go 462: Enabling iptables-in-nftables-mode workarounds.
2024-01-16T10:43:52.162292968Z 2024-01-16 10:43:52.162 [INFO][80] felix/feature_detect_linux.go 410: Looked up iptables command backendMode="nft" candidates=[]string{"iptables-nft-restore", "iptables-restore"} command="iptables-nft-restore" ipVersion=0x4 saveOrRestore="restore"
2024-01-16T10:43:52.162432708Z 2024-01-16 10:43:52.162 [INFO][80] felix/feature_detect_linux.go 410: Looked up iptables command backendMode="nft" candidates=[]string{"iptables-nft-save", "iptables-save"} command="iptables-nft-save" ipVersion=0x4 saveOrRestore="save"
Pod (my PREROUTING rule is also there):
› k exec -it calico-node-mhn7f bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Defaulted container "calico-node" out of: calico-node, upgrade-ipam (init), install-cni (init), mount-bpffs (init)
[root@node /]# iptables -V
iptables v1.8.8 (nf_tables)
# iptables-nft-save -t raw
# Generated by iptables-nft-save v1.8.8 (nf_tables) on Tue Jan 16 10:48:00 2024
*raw
:PREROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:cali-OUTPUT - [0:0]
:cali-PREROUTING - [0:0]
:cali-from-host-endpoint - [0:0]
:cali-rpf-skip - [0:0]
:cali-to-host-endpoint - [0:0]
-A PREROUTING -m comment --comment "cali:6gwbT8clXdHdC1b1" -j cali-PREROUTING
-A PREROUTING -i <uplink> -p icmp -m icmp --icmp-type 3/4 -j NFLOG --nflog-group 33
-A OUTPUT -m comment --comment "cali:tVnHkvAo15HuiPy0" -j cali-OUTPUT
-A cali-OUTPUT -m comment --comment "cali:njdnLwYeGqBJyMxW" -j MARK --set-xmark 0x0/0xf0000
-A cali-OUTPUT -m comment --comment "cali:rz86uTUcEZAfFsh7" -j cali-to-host-endpoint
-A cali-OUTPUT -m comment --comment "cali:pN0F5zD0b8yf9W1Z" -m mark --mark 0x10000/0x10000 -j ACCEPT
-A cali-PREROUTING -m comment --comment "cali:XFX5xbM8B9qR10JG" -j MARK --set-xmark 0x0/0xf0000
-A cali-PREROUTING -i cali+ -m comment --comment "cali:EWMPb0zVROM-woQp" -j MARK --set-xmark 0x40000/0x40000
-A cali-PREROUTING -m comment --comment "cali:PWuxTAIaFCtsg5Qa" -m mark --mark 0x40000/0x40000 -j cali-rpf-skip
-A cali-PREROUTING -m comment --comment "cali:fSSbGND7dgyemWU7" -m mark --mark 0x40000/0x40000 -m rpfilter --validmark --invert -j DROP
-A cali-PREROUTING -m comment --comment "cali:ImU0-4Rl2WoOI9Ou" -m mark --mark 0x0/0x40000 -j cali-from-host-endpoint
-A cali-PREROUTING -m comment --comment "cali:lV4V2MPoMBf0hl9T" -m mark --mark 0x10000/0x10000 -j ACCEPT
COMMIT
# Completed on Tue Jan 16 10:48:00 2024
[root@node /]#
may also relate to https://github.com/projectcalico/calico/issues/8025
I would like to work on this. I'd be happy to take a PR for this @cyclinder - please give me a shout if you hit any unexpected problems (e.g. base image issues or similar). @mazdakn looked at a similar bump for ipset recently and would also be a good point of contact.
For ipset issue (https://github.com/projectcalico/calico/issues/8372), we ended up changing the logic of parsing the ipset output instead of bumping ipset version, in order to prevent dealing with the same issue in future. I am not sure if this is doable here, but if possible, it would be a better option.
please give me a shout if you hit any unexpected problems
Thanks @matthewdupre @mazdakn, I'll start looking into this. As far as I know, kube-proxy keeps upgrading the version of iptables based on changing the base image, and you can find more details on https://github.com/kubernetes/release/tree/master/images/build/distroless-iptables, The latest release of kube-proxy uses iptables v1.8.9. It's good for us if we bump iptables to the latest.
I'm try to building calico-node for amd64 in my local machine, but it failed due to clean-up-filesystem.sh
, any advices? thanks!
#0 50.82 warning: file /usr/share/locale/en_GB/LC_MESSAGES/json-glib-1.0.mo: remove failed: No such file or directory
#0 51.08 Failed to disable unit, unit systemd-readahead-replay.service does not exist.
#0 51.08 Failed to disable unit, unit systemd-readahead-collect.service does not exist.
#0 51.14 warning: file /etc/rc.local: remove failed: No such file or directory
#0 51.38 install-info: No such file or directory for /usr/share/info/nettle.info
#0 51.73 install-info: No such file or directory for /usr/share/info/history.info
#0 51.73 install-info: No such file or directory for /usr/share/info/rluserman.info
#0 51.79 warning: file /usr/share/locale/en_GB/LC_MESSAGES/p11-kit.mo: remove failed: No such file or directory
#0 52.70 Binary is missing after RPM cleanup: /usr/sbin/ip6tables
------
Dockerfile.amd64:224
--------------------
222 | # or any of its dependencies.
223 | ADD clean-up-filesystem.sh /
224 | >>> RUN /clean-up-filesystem.sh
225 |
226 | # Copy everything into a fresh scratch image so that naive CVE scanners don't pick up binaries and libraries
--------------------
ERROR: failed to solve: process "/bin/sh -c /clean-up-filesystem.sh" did not complete successfully: exit code: 1
make: *** [Makefile:297: .calico_node.created-amd64] Error 1
What are you trying to run? make image
? Most of the builds are intended to be ran by the go-build container, it should hopefully be straightforward if you start from the right make targets. make ut
should also be useful (and should have enough dependencies to rebuild everything).
@cyclinder clean-up-filesystem.sh
is executed to clean up of the file system of the Calico node image. There is a list of expected binaries in that script which should match with the binaries expressed in the Docker file. In this output it seems it's complaining about ip6tables
not being present after executing of the script. Do you have a PR to share to check your changes?
@matthewdupre Yes, I updated the Dockerfile.amd64 and ran make node
.
@mazdakn Thanks for the details, I opened a draft PR, please see https://github.com/projectcalico/calico/pull/8416
CC @hjiawei
Current Behavior
Due to custom rule calico/node fails to execute
iptables-nft-save
command and is failing readiness probe:Assumption is that is due to the output of
iptables-nft-save -t raw
command in calico/node pod:Possible Solution
Bump iptables to ≥1.8.8. In these versions I am able to execute the command
iptables-nft-save -t raw
in a hostnetwork pod and get same rules output as on host.Steps to Reproduce (for bugs)
calico/node Pod:
OS:
Context
Rule in question is added by another daemon (for path mtu discovery):
Your Environment