weaveworks-plugins / scope-http-statistics

Plugin for Weave Scope which implements various HTTP statistics
Apache License 2.0
40 stars 19 forks source link

Failed to load BPF program kprobe__copy_from_iter: Invalid argument #11

Closed desponda closed 7 years ago

desponda commented 7 years ago

Trying to use the scope-http-statistics plug in I am getting:

In file included from /virtual/main.c:1:
In file included from /lib/modules/4.4.51-40.60.amzn1.x86_64/build/include/linux/skbuff.h:29:
In file included from /lib/modules/4.4.51-40.60.amzn1.x86_64/build/include/linux/net.h:28:
/lib/modules/4.4.51-40.60.amzn1.x86_64/build/include/linux/fs.h:2693:9: warning: comparison of unsigned enum expression < 0 is always false [-Wtautological-compare]
        if (id < 0 || id >= READING_MAX_ID)
            ~~ ^ ~
1 warning generated.
bpf: Invalid argument

Traceback (most recent call last):
  File "/usr/bin/http-statistics.py", line 294, in <module>
    kernel_inspector = KernelInspector()
  File "/usr/bin/http-statistics.py", line 57, in __init__
    self.bpf = bcc.BPF(EBPF_PROGRAM)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 291, in __init__
    self._trace_autoload()
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 892, in _trace_autoload
    fn = self.load_func(func_name, BPF.KPROBE)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 335, in load_func
    (func_name, errstr))
Exception: Failed to load BPF program kprobe__copy_from_iter: Invalid argument
alban commented 7 years ago

@desponda what Linux distribution do you use, and which kernel (uname -a)? Does it match the version listed in your logs (4.4.51-40.60.amzn1.x86_64)?

desponda commented 7 years ago

@alban ok I was able to fix that after getting the correct linux kernel version but now I'm getting the following even though this file exists in the path:

open(/sys/kernel/debug/tracing/kprobe_events): No such file or directory
Traceback (most recent call last):
  File "/usr/bin/http-statistics.py", line 294, in <module>
    kernel_inspector = KernelInspector()
  File "/usr/bin/http-statistics.py", line 57, in __init__
    self.bpf = bcc.BPF(EBPF_PROGRAM)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 291, in __init__
    self._trace_autoload()
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 893, in _trace_autoload
    self.attach_kprobe(event=fn.name[8:], fn_name=fn.name)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 511, in attach_kprobe
    raise Exception("Failed to attach BPF to kprobe")
Exception: Failed to attach BPF to kprobe

I'm getting this even though I mounted debugfs to /sys/kernel/debug , I attached /sys/kernel/debug as a volume to the container as I'm just copying and pasting the suggested command to run the container

alban commented 7 years ago

@desponda There was an Amazon Linux issue described on https://forums.aws.amazon.com/thread.jspa?threadID=240916

Could you try by mounting the debugfs before starting the Docker daemon, in this order?

$ sudo mount -t debugfs debugfs /sys/kernel/debug
$ sudo /etc/init.d/docker start
desponda commented 7 years ago

@alban and that did it! Thank's for your help, awesome plugin!