Built-in Traefik crashes

[AllardKrings]

Hello,

I have installed the kubernetes binary on my 2 riscv-sbc’s. Fantastic that you have built them!

However I have noticed a problem with the built-in traefic.

The traefik pod itself runs fine, however the loadbalancers crash.

kube-system helm-install-traefik-crd-989n8 0/1 Completed 0 127m kube-system helm-install-traefik-wcfzz 0/1 Completed 2 127m kube-system traefik-8657d6b9f4-zzbhb 1/1 Running 1 (110m ago) 124m kube-system coredns-97b598894-gxwfr 1/1 Running 1 (110m ago) 127m kube-system local-path-provisioner-6d44f4f9d7-2tvz5 1/1 Running 2 (109m ago) 127m kube-system metrics-server-7c55d89d5d-zmqt8 1/1 Running 2 (109m ago) 127m kube-system svclb-traefik-c167a2e3-f2wsl 0/2 CrashLoopBackOff 44 (4m52s ago) 93m

When I look in the logs of the containers it says:

• trap exit TERM INT
• BIN_DIR=/sbin
• check_iptables_mode
• set +e
• lsmod
• grep nf_tables
• '[' 1 '=' 0 ]
• mode=legacy
• set -e
• info 'legacy mode detected'
• echo '[INFO] ' 'legacy mode detected'
• set_legacy
• ln -sf /sbin/xtables-legacy-multi /sbin/iptables [INFO] legacy mode detected
• ln -sf /sbin/xtables-legacy-multi /sbin/iptables-save
• ln -sf /sbin/xtables-legacy-multi /sbin/iptables-restore
• ln -sf /sbin/xtables-legacy-multi /sbin/ip6tables
• start_proxy
• echo 0.0.0.0/0
• grep -Eq :
• iptables -t filter -I FORWARD -s 0.0.0.0/0 -p TCP --dport 80 -j ACCEPT /usr/bin/entry: line 46: iptables: not found

I am running

DISTRIB_ID=Ubuntu DISTRIB_RELEASE=23.10 DISTRIB_CODENAME=mantic DISTRIB_DESCRIPTION="Ubuntu 23.10"

On 2 Starfive Visionfive SBC’s

Both have the same problem.

Maybe it has nothing to do with the K3S-binaries but with my system-configuration.

I would apprectie some tips/help.

Kind Regards Allard Krings

[chazapis]

@AllardKrings, thanks for the feedback. I will try this out and let you know if it crashes. Can you check whether the svclb-traefik-c167a2e3-f2wsl mounts the iptables binary from the system or it is supposed to include it and it is missing?

[AllardKrings]

hello Anthony,

issuing kubectl get pod svclb-traefik-82f5b39b-kmvj5 -n kube-system -o yaml

gives:

apiVersion: v1 kind: Pod metadata: creationTimestamp: "2024-04-19T07:41:16Z" generateName: svclb-traefik-82f5b39b- labels: app: svclb-traefik-82f5b39b controller-revision-hash: 8666c56fb8 pod-template-generation: "1" svccontroller.k3s.cattle.io/svcname: traefik svccontroller.k3s.cattle.io/svcnamespace: kube-system name: svclb-traefik-82f5b39b-kmvj5 namespace: kube-system ownerReferences:

• apiVersion: apps/v1 blockOwnerDeletion: true controller: true kind: DaemonSet name: svclb-traefik-82f5b39b uid: 3eec9cbd-a16c-4eb3-92f8-dcd2bfcb1b18 resourceVersion: "784" uid: f4be8419-82d7-48cd-a0c4-68c00c595a9d spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms:
  • matchFields:
    • key: metadata.name operator: In values:
      • rvsvrwsv02 automountServiceAccountToken: false containers:
• env:
  • name: SRC_PORT value: "80"
  • name: SRC_RANGES value: 0.0.0.0/0
  • name: DEST_PROTO value: TCP
  • name: DEST_PORT value: "80"
  • name: DEST_IPS value: 10.43.226.66 image: carvicsforth/klipper-lb:v0.4.4 imagePullPolicy: IfNotPresent name: lb-tcp-80 ports:
  • containerPort: 80 hostPort: 80 name: lb-tcp-80 protocol: TCP resources: {} securityContext: capabilities: add:
    • NET_ADMIN terminationMessagePath: /dev/termination-log terminationMessagePolicy: File
• env:
  • name: SRC_PORT value: "443"
  • name: SRC_RANGES value: 0.0.0.0/0
  • name: DEST_PROTO value: TCP
  • name: DEST_PORT value: "443"
  • name: DEST_IPS value: 10.43.226.66 image: carvicsforth/klipper-lb:v0.4.4 imagePullPolicy: IfNotPresent name: lb-tcp-443 ports:
  • containerPort: 443 hostPort: 443 name: lb-tcp-443 protocol: TCP resources: {} securityContext: capabilities: add:
    • NET_ADMIN terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst enableServiceLinks: true nodeName: rvsvrwsv02 preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Always schedulerName: default-scheduler securityContext: sysctls:
  • name: net.ipv4.ip_forward value: "1" serviceAccount: svclb serviceAccountName: svclb terminationGracePeriodSeconds: 30 tolerations:
• effect: NoSchedule key: node-role.kubernetes.io/master operator: Exists
• effect: NoSchedule key: node-role.kubernetes.io/control-plane operator: Exists
• key: CriticalAddonsOnly operator: Exists
• effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists
• effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists
• effect: NoSchedule key: node.kubernetes.io/disk-pressure operator: Exists
• effect: NoSchedule key: node.kubernetes.io/memory-pressure operator: Exists
• effect: NoSchedule key: node.kubernetes.io/pid-pressure operator: Exists
• effect: NoSchedule key: node.kubernetes.io/unschedulable operator: Exists status: conditions:
• lastProbeTime: null lastTransitionTime: "2024-04-19T07:41:16Z" status: "True" type: Initialized
• lastProbeTime: null lastTransitionTime: "2024-04-19T07:41:16Z" message: 'containers with unready status: [lb-tcp-80 lb-tcp-443]' reason: ContainersNotReady status: "False" type: Ready
• lastProbeTime: null lastTransitionTime: "2024-04-19T07:41:16Z" message: 'containers with unready status: [lb-tcp-80 lb-tcp-443]' reason: ContainersNotReady status: "False" type: ContainersReady
• lastProbeTime: null lastTransitionTime: "2024-04-19T07:41:16Z" status: "True" type: PodScheduled containerStatuses:
• containerID: containerd://75ab4e8a9fea5b582acb1783cc195c81f3033c97be3e5472055c59b775a18130 image: docker.io/carvicsforth/klipper-lb:v0.4.4 imageID: docker.io/carvicsforth/klipper-lb@sha256:d7cb10c117e39056f577ddbc8cca0af1110b5c1c4a8dd3150a51cf9413bf51a9 lastState: terminated: containerID: containerd://75ab4e8a9fea5b582acb1783cc195c81f3033c97be3e5472055c59b775a18130 exitCode: 127 finishedAt: "2024-04-19T07:47:16Z" reason: Error startedAt: "2024-04-19T07:47:16Z" name: lb-tcp-443 ready: false restartCount: 6 started: false state: waiting: message: back-off 5m0s restarting failed container=lb-tcp-443 pod=svclb-traefik-82f5b39b-kmvj5_kube-system(f4be8419-82d7-48cd-a0c4-68c00c595a9d) reason: CrashLoopBackOff
• containerID: containerd://35e2a17cf6032868e0648484db0bc2b7b6b1b0a67f9ed72e67df11d2c91813e9 image: docker.io/carvicsforth/klipper-lb:v0.4.4 imageID: docker.io/carvicsforth/klipper-lb@sha256:d7cb10c117e39056f577ddbc8cca0af1110b5c1c4a8dd3150a51cf9413bf51a9 lastState: terminated: containerID: containerd://35e2a17cf6032868e0648484db0bc2b7b6b1b0a67f9ed72e67df11d2c91813e9 exitCode: 127 finishedAt: "2024-04-19T07:47:16Z" reason: Error startedAt: "2024-04-19T07:47:16Z" name: lb-tcp-80 ready: false restartCount: 6 started: false state: waiting: message: back-off 5m0s restarting failed container=lb-tcp-80 pod=svclb-traefik-82f5b39b-kmvj5_kube-system(f4be8419-82d7-48cd-a0c4-68c00c595a9d) reason: CrashLoopBackOff hostIP: 192.168.2.27 phase: Running podIP: 10.42.0.7 podIPs:
• ip: 10.42.0.7 qosClass: BestEffort startTime: "2024-04-19T07:41:16Z"

The host is running: iptables v1.8.9 (legacy)

lsmod gives:

Module Size Used by xt_limit 12288 5 xt_NFLOG 12288 5 nfnetlink_log 28672 5 xt_physdev 12288 10 xt_multiport 20480 3 ip_set 86016 0 ipt_REJECT 12288 5 nf_reject_ipv4 16384 1 ipt_REJECT ip6table_filter 12288 1 ip6table_nat 12288 1 ip6table_mangle 12288 1 ip6_tables 36864 3 ip6table_filter,ip6table_nat,ip6table_mangle xt_comment 12288 170 tls 208896 0 ip_vs_rr 12288 23 xt_ipvs 16384 4 ip_vs 294912 26 ip_vs_rr,xt_ipvs xt_REDIRECT 16384 8 xt_nat 16384 69 veth 53248 0 vxlan 204800 0 ip6_udp_tunnel 12288 1 vxlan udp_tunnel 36864 1 vxlan xt_policy 16384 0 iptable_mangle 12288 5 xt_mark 12288 69 xt_u32 12288 0 xt_tcpudp 20480 144 rpcsec_gss_krb5 49152 0 xt_conntrack 16384 59 xt_MASQUERADE 16384 10 nf_conntrack_netlink 77824 0 nfnetlink 24576 4 nf_conntrack_netlink,ip_set,nfnetlink_log xfrm_user 73728 1 xfrm_algo 20480 1 xfrm_user xt_addrtype 16384 16 iptable_filter 12288 7 iptable_nat 12288 12 nf_nat 86016 5 ip6table_nat,xt_nat,iptable_nat,xt_MASQUERADE,xt_REDIRECT nf_conntrack 270336 7 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE,ip_vs,xt_REDIRECT nf_defrag_ipv6 36864 2 nf_conntrack,ip_vs nf_defrag_ipv4 12288 1 nf_conntrack bpfilter 12288 0 nfsv4 1392640 1 nfs 774144 2 nfsv4 fscache 434176 1 nfs netfs 86016 2 fscache,nfs overlay 278528 16 rtw88_8821cu 12288 0 rtw88_8821c 90112 1 rtw88_8821cu rtw88_usb 28672 1 rtw88_8821cu rtw88_core 421888 2 rtw88_8821c,rtw88_usb mac80211 2019328 2 rtw88_core,rtw88_usb btusb 118784 0 btrtl 53248 1 btusb btbcm 28672 1 btusb btintel 90112 1 btusb btmtk 16384 1 btusb binfmt_misc 36864 1 bluetooth 1859584 6 btrtl,btmtk,btintel,btbcm,btusb cfg80211 1634304 2 rtw88_core,mac80211 cdns3 200704 0 ecdh_generic 16384 1 bluetooth libarc4 12288 1 mac80211 ecc 57344 1 ecdh_generic cdns_usb_common 40960 1 cdns3 udc_core 110592 1 cdns3 ofpart 20480 0 cmdlinepart 16384 0 jh7110_tdm 20480 0 snd_soc_core 512000 1 jh7110_tdm spi_nor 196608 0 nls_iso8859_1 12288 1 snd_compress 40960 1 snd_soc_core cdns3_starfive 16384 0 ac97_bus 12288 1 snd_soc_core dw_axi_dmac_platform 45056 4 mtd 143360 7 spi_nor,cmdlinepart,ofpart axp20x_pek 16384 0 snd_pcm_dmaengine 20480 1 snd_soc_core snd_pcm 233472 4 snd_compress,snd_soc_core,jh7110_tdm,snd_pcm_dmaengine sfctemp 16384 0 snd_timer 65536 1 snd_pcm pwm_starfive 16384 0 snd 167936 4 snd_timer,snd_compress,snd_soc_core,snd_pcm soundcore 20480 1 snd uio_pdrv_genirq 20480 0 uio 32768 1 uio_pdrv_genirq nfsd 1105920 5 dm_multipath 61440 0 br_netfilter 40960 0 bridge 544768 1 br_netfilter auth_rpcgss 225280 2 nfsd,rpcsec_gss_krb5 stp 12288 1 bridge llc 16384 2 bridge,stp nfs_acl 16384 1 nfsd drm 946176 0 lockd 208896 2 nfsd,nfs grace 16384 2 nfsd,lockd efi_pstore 16384 0 backlight 36864 1 drm sunrpc 1036288 24 nfsd,nfsv4,auth_rpcgss,lockd,rpcsec_gss_krb5,nfs_acl,nfs ip_tables 36864 3 iptable_filter,iptable_nat,iptable_mangle x_tables 65536 24 ip6table_filter,xt_conntrack,iptable_filter,ip6table_nat,xt_multiport,xt_NFLOG,xt_tcpudp,xt_addrtype,xt_physdev,xt_nat,xt_ipvs,xt_comment,xt_policy,ip6_tables,xt_u32,ipt_REJECT,ip_tables,iptable_nat,xt_limit,ip6table_mangle,xt_MASQUERADE,iptable_mangle,xt_REDIRECT,xt_mark autofs4 94208 2 btrfs 2924544 0 blake2b_generic 24576 0 raid10 110592 0 raid456 356352 0 async_raid6_recov 24576 1 raid456 async_memcpy 16384 2 raid456,async_raid6_recov async_pq 24576 2 raid456,async_raid6_recov async_xor 24576 3 async_pq,raid456,async_raid6_recov async_tx 20480 5 async_pq,async_memcpy,async_xor,raid456,async_raid6_recov xor 20480 2 async_xor,btrfs raid6_pq 102400 4 async_pq,btrfs,raid456,async_raid6_recov libcrc32c 12288 5 nf_conntrack,nf_nat,btrfs,raid456,ip_vs raid1 90112 0 raid0 40960 0 multipath 24576 0 linear 20480 0 motorcomm 36864 1 axp20x_regulator 65536 6 xhci_pci 32768 0 dwmac_starfive 12288 0 nvme 73728 3 stmmac_platform 40960 1 dwmac_starfive stmmac 471040 4 dwmac_starfive,stmmac_platform xhci_pci_renesas 32768 1 xhci_pci nvme_core 282624 4 nvme dw_mmc_starfive 20480 0 nvme_common 28672 1 nvme_core pcs_xpcs 28672 1 stmmac axp20x_i2c 12288 0 dw_mmc_pltfm 12288 1 dw_mmc_starfive phylink 106496 2 stmmac,pcs_xpcs axp20x 49152 1 axp20x_i2c clk_starfive_jh7110_vout 16384 0 clk_starfive_jh7110_aon 12288 3 pinctrl_starfive_jh7110_aon 12288 0 clk_starfive_jh7110_isp 16384 0 dw_mmc 77824 1 dw_mmc_pltfm spi_cadence_quadspi 49152 0 jh7110_trng 16384 0 phy_jh7110_usb 20480 1

If you need any more info just ask

[AllardKrings]

If I run:

docker run --cap-add SYS_ADMIN -it docker.io/carvicsforth/klipper-lb:v0.4.4 /bin/sh

and then issue “iptables”

I get:

/ # iptables iptables v1.8.9 (nf_tables): no command specified Try `iptables -h' or 'iptables --help' for more information. / #

So apparently the container default runs in nf-tables mode, the host runs in legacy mode. I am not a netwrok expert but maybe this info helps.

[AllardKrings]

Hello,

I did some more research. I found out that both hosts were running in LEGACY-mode. Switching to NFT-mode resulted in the loadbalancers running fine. Apparently from within the container the softlinks to iptables on the host work fine now.

[chazapis]

Thanks @AllardKrings for figuring this out. I'll update the documentation and link to the issue.