search

coreos / fedora-coreos-tracker

Issue tracker for Fedora CoreOS
https://fedoraproject.org/coreos/
264 stars 59 forks source link

kernel bug in 6.11.5 stable 41.20241027.3.0 #1834

Open HaveFun83 opened 4 hours ago

HaveFun83 commented 4 hours ago

Describe the bug

fcos stable 41.20241027.3.0 with kernel 6.11.5 was rolled out today morning on our k8s nodes which broke our CNI (calico). We need to manually rollback to "40.20241019.3.0"

We found out that kernel 6.11.5 has a bug which broke the CNI operation

Ref: https://github.com/tailscale/tailscale/issues/13863 https://github.com/torvalds/linux/commit/306ed1728e8438caed30332e1ab46b28c25fe3d8

Reproduction steps

Deploy calico CNI on fcos 41.20241027.3.0 k8s nodes

Expected behavior

calico-node pods can operate without any errors

Actual behavior

calico-nodes pods are unable to operate and never get ready

2024-11-14 09:47:29.537 [WARNING][2938083] felix/table.go 1454: Failed to execute ip(6)tables-restore command error=exit status 2 errorOutput="Warning: Extension MARK revision 0 not supported, missing kernel module?

System details

stable 41.20241027.3.0

Butane or Ignition config

No response

Additional information

looks like the bug was fixed in 6.11.6

HaveFun83 commented 3 hours ago

@m4us1ne thanks for drill down this issue ;)

dustymabe commented 3 hours ago

looks like https://github.com/torvalds/linux/commit/e92e991748484a8b158cd05e348002d2eeefd1f0 is already in kernel v6.11.6 and newer so this should already be fixed in testing. Can you test testing stream 41.20241109.2.0 that was released this week?