search

iovisor / bcc

BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more
Apache License 2.0
20.64k stars 3.89k forks source link

Error in bcc-tool execution #3018

Open harii9 opened 4 years ago

harii9 commented 4 years ago

Hello Team,

I have successfully installed bcc-tools on the CoreOS toolbox. I am getting error when executing tools:

sh-4.4# ./nodestat Tracing... Output every 1 secs. Hit Ctrl-C to end sh: modprobe: command not found Unable to find kernel headers. Try rebuilding kernel with CONFIG_IKHEADERS=m (module) chdir(/lib/modules/4.18.0-147.8.1.el8_1.x86_64/build): No such file or directory Traceback (most recent call last): File "./lib/ustat", line 305, in Tool().run() File "./lib/ustat", line 297, in run self._loop_iter() File "./lib/ustat", line 246, in _loop_iter self._attach_probes() File "./lib/ustat", line 224, in _attach_probes self.bpf = BPF(text=program) File "/usr/lib/python3.6/site-packages/bcc/init.py", line 347, in init raise Exception("Failed to compile BPF module %s" % (src_file or "")) Exception: Failed to compile BPF module

Thanks

alban commented 4 years ago

You have 2 options to resolve this issue in toolbox:

  1. Get a kernel with CONFIG_IKHEADERS. It seems your version of CoreOS does not have it. If you file an issue for it, you can take example of similar issues in other distros (https://github.com/flatcar-linux/Flatcar/issues/152, https://github.com/kubernetes/minikube/issues/8556, https://github.com/gardenlinux/gardenlinux/issues/76).

  2. Install kernel headers on the host and make them available in your toolbox container.

Do you have it already installed on the host? You can check with:

$ ls -l /lib/modules/4.18.0-147.8.1.el8_1.x86_64/

I used to pick the kernel headers from CentOS at http://mirror.centos.org/centos/8/BaseOS/x86_64/os/Packages/ but your specific kernel version (.18.0-147.8.1.el8_1) does not seem to be there.

I see that the toolbox script does not make /lib/modules available in the container: https://github.com/coreos/toolbox/blob/master/toolbox#L24

TOOLBOX_BIND="--bind=/:/media/root --bind=/usr:/media/root/usr --bind=/run:/media/root/run"

Could you try to add --bind=/lib/modules:/lib/modules in the list?

Another option could be to run bcc tools from a docker image (from here).

You can give it a try with something like:

docker run -ti --rm --privileged -v /lib/modules:/lib/modules -v /sys/fs/bpf:/sys/fs/bpf --pid=host kinvolk/bcc:latest /usr/share/bcc/tools/execsnoop
anakryiko commented 4 years ago

You also have another option, use BPF CO-RE-based libbpf-tools. I don't know anything about CoreOS, though, what version of Linux kernel is it based on?

If you have recent enough kernel and can build with CONFIG_DEBUG_INFO_BTF=y, then libbpf-tools and BPF CO-RE become a viable choice. They don't require kernel headers or CONFIG_IKHEADERS, because BTF in vmlinux will have all the internal types described and exposed to BPF programs.

Please check out https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portability-and-co-re.html and let me know if you have any questions. Thanks!