Closed dedene closed 1 month ago
I'd guess it's Cilium/BPF issue (might go deeper into Linux kernel), but not something we can fix exactly on Talos side.
Thanks! I'm also opening a Github issue on Cilium. For now I'll stick to Talos 1.6.7. I'll update here when there is more information.
No its an upstream Kernel issue.
See patchwork and mailinglist
I have applied it in my custom talos kernel and this fixes it. So I guess in order to fix this in 1.7 and newer versions is to disable UBSAN as a kernel parameter. Or compile your own kernel
Nice find, I hope it will get backported to 6.6.x
With Talos 1.7.6
and Linux6.6.43
this should be fixed, right? This doesn't seem to be the case for me at the moment:
kern: err: [2024-08-12T15:18:50.641243422Z]: ================================================================================
kern: err: [2024-08-12T15:18:50.641247422Z]: UBSAN: array-index-out-of-bounds in kernel/bpf/lpm_trie.c:194:14
kern: err: [2024-08-12T15:18:50.641252422Z]: index 8 is out of range for type '__u8 [*]'
kern: warning: [2024-08-12T15:18:50.641255422Z]: CPU: 0 PID: 5240 Comm: cilium-agent Not tainted 6.6.43-talos #1
kern: warning: [2024-08-12T15:18:50.641259422Z]: Hardware name: Hetzner vServer/KVM Virtual Machine, BIOS 20171111 11/11/2017
kern: warning: [2024-08-12T15:18:50.641261422Z]: Call trace:
kern: warning: [2024-08-12T15:18:50.641262422Z]: dump_backtrace+0x9c/0x100
kern: warning: [2024-08-12T15:18:50.641269422Z]: show_stack+0x34/0x50
kern: warning: [2024-08-12T15:18:50.641272422Z]: dump_stack_lvl+0x78/0xd0
kern: warning: [2024-08-12T15:18:50.641278422Z]: dump_stack+0x1c/0x30
kern: warning: [2024-08-12T15:18:50.641281422Z]: __ubsan_handle_out_of_bounds+0xc0/0x100
kern: warning: [2024-08-12T15:18:50.641286422Z]: longest_prefix_match.isra.0+0x200/0x258
kern: warning: [2024-08-12T15:18:50.641290422Z]: trie_update_elem+0x160/0x3a0
kern: warning: [2024-08-12T15:18:50.641292422Z]: bpf_map_update_value+0xcc/0x2c8
kern: warning: [2024-08-12T15:18:50.641295422Z]: map_update_elem+0x19c/0x328
kern: warning: [2024-08-12T15:18:50.641299422Z]: __sys_bpf+0x834/0x1c48
kern: warning: [2024-08-12T15:18:50.641303422Z]: __arm64_sys_bpf+0x34/0x58
kern: warning: [2024-08-12T15:18:50.641307422Z]: invoke_syscall+0x90/0x128
kern: warning: [2024-08-12T15:18:50.641309422Z]: el0_svc_common.constprop.0+0xec/0x118
kern: warning: [2024-08-12T15:18:50.641311422Z]: do_el0_svc+0x34/0x50
kern: warning: [2024-08-12T15:18:50.641313422Z]: el0_svc+0x4c/0x178
kern: warning: [2024-08-12T15:18:50.641315422Z]: el0t_64_sync_handler+0x128/0x138
kern: warning: [2024-08-12T15:18:50.641317422Z]: el0t_64_sync+0x1bc/0x1c0
kern: err: [2024-08-12T15:18:50.641320422Z]: ================================================================================
Whereby the trace seems to be somewhat different.
It was merged to pkg:main, and not backported (yet) to pkg:release-1.7 branch. That means it will be in Talos 1.8.
Bug Report
Description
After upgrading from Talos v1.6.7 to v.1.7.x a kernel error is logged after booting:
UBSAN: array-index-out-of-bounds in kernel/bpf/lpm_trie.c:194:14
. The node seems perfectly functioning however, but this message is not appearing on v1.6.7 or lower. I'm running Cilium 1.15.5 as CNI.I'm not sure if this is a Cilium issue or Talos Linux related issue, Or even upstream due to upgrading Linux from 6.1.x to 6.6.x. But I'm hoping for some help or advice from both communities. Feel free to close this issue if not applicable.
Many thanks in advance!
Logs
Environment