Allow libbpf-tools to run on old kernels

[chenhengqi]

We have a bunch of libbpf tools require features like fentry / tp_btf which only available on kernel v5.5+. Running these tools on old kernels result in EINVAL. Instead of error out, we can fallback to kprobe. Open this ticket to keep track of this process:

• [x] biolatency
• [ ] biosnoop
• [ ] biostacks
• [ ] bitesize
• [ ] cachestat
• [ ] cpudist
• [ ] cpufreq
• [x] drsnoop
• [x] hardirqs
• [x] mdflush
• [x] numamove
• [ ] offcputime
• [ ] readahead
• [x] softirqs

If you are interested, please feel free to pick it up.

[eiffel-fl]

biolatency

I wanted to give a shot at biolatency but I hit a wall as this is not easy to replace the tracepoint by kprobes. Indeed, the three tracepoints of this tools are called from different kernel functions, so this is not trivial to replace it with kprobes.

I suggest more using raw_tp instead (which I was able to make work). But I am not sure when the raw_tp were introduced... Were they in 4.17 with this commit?

[chenhengqi]

Using raw_tp is enough, this will make the tools running on older LTS kernels (like v4.19/v5.4).

[woodpenker]

Do we consider providing a cmd flag such as -btf to let user provide their BTF file for the kernel, so that we can use tools even on 4.x kernels?

[chenhengqi]

Do we consider providing a cmd flag such as -btf to let user provide their BTF file for the kernel, so that we can use tools even on 4.x kernels?

We have such support at build time. See https://github.com/iovisor/bcc/pull/3889 .

[woodpenker]

Do we consider providing a cmd flag such as -btf to let user provide their BTF file for the kernel, so that we can use tools even on 4.x kernels?

We have such support at build time. See #3889 .

It seems to rely on the BTF hub and won't support self-built kernels or some old kernels which can't be found in the hub, eg: Ubuntu 4.18.0-22. Since LIBBPF_OPTS is supported, how about letting the user make a choice? example:

if (env.btf_path) {
  LIBBPF_OPTS(bpf_object_open_opts, bpf_opts, .btf_custom_path = env.btf_path);
  skel = xxx_bpf__open_opts(&bpf_opts);
} else {
  skel = xxx_bpf__open();
}

[chenhengqi]

Instead of touching every tool, I'd suggest providing the BTF path through environment variable and teach ensure_core_btf() to recognize it.

[woodpenker]

Instead of touching every tool, I'd suggest providing the BTF path through environment variable and teach ensure_core_btf() to recognize it.

Thanks for explaining, I may try that. But I realize another problem with global values, only legacy codes are supported on old kernels like this. To support them, maybe we need some mechanisms to auto-generate legacy codes instead of global values?

[woodpenker]

BTW, some tools such as softirqs seems not using ensure_core_btf(), is it intended?

[chenhengqi]

Yes, tp_btf relies on BPF trampoline which support /sys/kernel/btf/vmlinux only.

[woodpenker]

Yes, tp_btf relies on BPF trampoline which support /sys/kernel/btf/vmlinux only.

Thanks, I want to try to teach some tools listed above to fall back to raw_tp or Kprobe, maybe start from offcputime which I am familiar with. Another question, it seems kernel support auto falls back to tp now after this patch, am I right?

[chenhengqi]

Another question, it seems kernel support auto falls back to tp now after this patch, am I right?

No.

[woodpenker]

Since cpufreq need Memory-mapping BPF maps support starting from v5.7(b97e12e594eb & 333291ce5055), I think there is no need to change it.