This draft PR captures most of the work needed for Retina to plug into all of Linux kernel's packet drop reasons by using a kprobe at function kfree_skb_reason().
We are now shelving this work since the number of drop reasons available in kernel 5.15 LTS used by Azure Linux and AKS, is not much greater than what Retina has and so not very useful. Azure Linux 3.0 will run on kernel 6.6 which contains a much expanded version of enum skb_drop_reason, and also allows us to bypass the problems resulting from it having been prepended to after it was initially introduced.
Changes:
added eBPF program kprobe/kfree_skb_reason
generated updated vmlinux.h file from a v6.8 kernel image
bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h
updated the .proto and .pb.go files with about 70 new drop reasons
provide larger eBPF verifier buffer in Cilium loader config to overcome program load failures we've seen in the past
amend the definition of the dropreason_events map to allow compiling with by hand with clang (bpf2go deduces key/value size fields, but they're necessary without it)
Validation done:
can observe the kprobe being hit cat /sys/kernel/debug/tracing/trace_pipe both locally and in an AKS node
can correlate crafted packet drops (iptables -A INPUT -p icmp -s 127.0.0.1 -j DROP) with eBPF map values (metrics_map) and HTTP metric endpoint results ( curl http://10.0.0.4:10093/metrics)
Below, we can see one dropped ping packet being recorded in the drop count metric - it changes from 19 to 20. Its kernel drop reason value is _NETFILTER_DROP = 6, which currently maps to Retina's UNKNOWN_DROP. We can also see Retina's IPTABLE_RULE_DROP go up in tandem, since the image deployed also contained the existing nf_hook_slow kprobe.
Remaning work:
remove our existing drop reason kprobes to avoid duplicate drop counts (new kprobe will eventually be called for all packet drop sites in the kernel, so double-check which ones have been updated in the kernel to use kfree_skb_reason, when we complete this PR in the future)
either map our existing drop reason enum (5 values) to the Linux kernel ones, or make a breaking change and just use the new enum without modifications - existing drop reason metrics will map to different values.
Currently, they are commented which is why the CI job failed.
decide how to interpret and present the disproportionately large number of spontaneous SKB_DROP_REASON_NO_SOCKET drops (showing up as IPTABLE_NAT_DROP in the image above due to clashing of enum values 1). Retina currently doesn't capture these: could they be confusing and unremarkable for an AKS cluster?
Description
This draft PR captures most of the work needed for Retina to plug into all of Linux kernel's packet drop reasons by using a kprobe at function
kfree_skb_reason().We are now shelving this work since the number of drop reasons available in kernel 5.15 LTS used by Azure Linux and AKS, is not much greater than what Retina has and so not very useful. Azure Linux 3.0 will run on kernel 6.6 which contains a much expanded version of
enum skb_drop_reason, and also allows us to bypass the problems resulting from it having been prepended to after it was initially introduced.Changes:
kprobe/kfree_skb_reasonvmlinux.hfile from a v6.8 kernel imagebpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.hdropreason_eventsmap to allow compiling with by hand with clang (bpf2go deduces key/value size fields, but they're necessary without it)Validation done:
cat /sys/kernel/debug/tracing/trace_pipeboth locally and in an AKS nodeiptables -A INPUT -p icmp -s 127.0.0.1 -j DROP) with eBPF map values (metrics_map) and HTTP metric endpoint results ( curl http://10.0.0.4:10093/metrics)Below, we can see one dropped ping packet being recorded in the drop count metric - it changes from 19 to 20. Its kernel drop reason value is
_NETFILTER_DROP = 6, which currently maps to Retina'sUNKNOWN_DROP. We can also see Retina'sIPTABLE_RULE_DROPgo up in tandem, since the image deployed also contained the existingnf_hook_slowkprobe.Remaning work:
SKB_DROP_REASON_NO_SOCKETdrops (showing up asIPTABLE_NAT_DROPin the image above due to clashing of enum values1). Retina currently doesn't capture these: could they be confusing and unremarkable for an AKS cluster?Related Issue
Fixes #367