error: use of undeclared identifier 'BPF_TAG_SIZE'

[errordeveloper]

Just tried running it on Kubernetes 1.9 in GKE with Ubuntu node OS.

ilya@gke-istio-demo-default-pool-11532261-j543:~$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="16.04.4 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.4 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
ilya@gke-istio-demo-default-pool-11532261-j543:~$ uname -a
Linux gke-istio-demo-default-pool-11532261-j543 4.13.0-1011-gcp #15-Ubuntu SMP Mon Feb 12 16:29:04 UTC 2018 x86_64 
x86_64 x86_64 GNU/Linux
ilya@gke-istio-demo-default-pool-11532261-j543:~$ 

ilya@dx-general:~$ kubectl logs weavescope-http-statistics-plugin-9hkmm
In file included from /virtual/main.c:2:
In file included from include/net/sock.h:58:
include/linux/memcontrol.h:580:31: warning: implicit conversion from enumeration type 'enum node_stat_item' to different enumeration type 'enum memcg_stat_item' [-Wenum-conversion]
        __mod_memcg_state(pn->memcg, idx, val);
        ~~~~~~~~~~~~~~~~~            ^~~
include/linux/memcontrol.h:593:29: warning: implicit conversion from enumeration type 'enum node_stat_item' to different enumeration type 'enum memcg_stat_item' [-Wenum-conversion]
        mod_memcg_state(pn->memcg, idx, val);
        ~~~~~~~~~~~~~~~            ^~~
include/linux/memcontrol.h:605:38: warning: implicit conversion from enumeration type 'enum node_stat_item' to different enumeration type 'enum memcg_stat_item' [-Wenum-conversion]
        __mod_memcg_state(page->mem_cgroup, idx, val);
        ~~~~~~~~~~~~~~~~~                   ^~~
include/linux/memcontrol.h:618:36: warning: implicit conversion from enumeration type 'enum node_stat_item' to different enumeration type 'enum memcg_stat_item' [-Wenum-conversion]
        mod_memcg_state(page->mem_cgroup, idx, val);
        ~~~~~~~~~~~~~~~                   ^~~
include/linux/memcontrol.h:639:40: warning: implicit conversion from enumeration type 'enum memcg_stat_item' to different enumeration type 'enum vm_event_item' [-Wenum-conversion]
                count_memcg_events(page->mem_cgroup, idx, 1);
                ~~~~~~~~~~~~~~~~~~                   ^~~
        ~~~~~~~~~~~~~~~~~            ^~~
In file included from /virtual/main.c:2:
In file included from include/net/sock.h:64:
include/linux/filter.h:466:11: error: use of undeclared identifier 'BPF_TAG_SIZE'
        u8                      tag[BPF_TAG_SIZE];
                                    ^
5 warnings and 1 error generated.
Traceback (most recent call last):
  File "/usr/bin/http-statistics.py", line 294, in <module>
    kernel_inspector = KernelInspector()
  File "/usr/bin/http-statistics.py", line 57, in __init__
    self.bpf = bcc.BPF(EBPF_PROGRAM)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 284, in __init__
    raise Exception("Failed to compile BPF module %s" % src_file)
Exception: Failed to compile BPF module /usr/bin/ebpf-http-statistics.c
ilya@dx-general:~$

[errordeveloper]

From what I gathered (e.g. https://github.com/iovisor/bcc/issues/1407) updating the dependencies the right thing to do in the attempt to fix the original problem, but that breaks other things, e.g. I'm getting this:

Traceback (most recent call last):
  File "/usr/bin/http-statistics.py", line 294, in <module>
    kernel_inspector = KernelInspector()
  File "/usr/bin/http-statistics.py", line 57, in __init__
    self.bpf = bcc.BPF(EBPF_PROGRAM)
  File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 299, in __init__
    src_file = BPF._find_file(src_file)
  File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 211, in _find_file
    t = b"/".join([os.path.abspath(os.path.dirname(argv0)), filename])
  File "/usr/lib/python3.5/posixpath.py", line 148, in dirname
    i = p.rfind(sep) + 1
AttributeError: 'ArgString' object has no attribute 'rfind'

[errordeveloper]

I did this:

diff --git a/Dockerfile b/Dockerfile
index 1309212..dff6ec2 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -7,5 +7,7 @@ RUN echo "deb [trusted=yes] http://repo.iovisor.org/apt/xenial xenial-nightly ma
 RUN apt-get update && apt-get install -y libbcc libbcc-examples python-bcc

 # Add our plugin
-ADD ./ebpf-http-statistics.c ./http-statistics.py /usr/bin/
-ENTRYPOINT ["/usr/bin/http-statistics.py"]
+RUN mkdir /src/
+WORKDIR /src/
+ADD ./ebpf-http-statistics.c ./http-statistics.py /src/
+ENTRYPOINT ["/src/http-statistics.py"]

And now it gets past the ArgString error, and this is what I get now:

Traceback (most recent call last):
  File "/src/http-statistics.py", line 294, in <module>
    kernel_inspector = KernelInspector()
  File "/src/http-statistics.py", line 57, in __init__
    self.bpf = bcc.BPF(EBPF_PROGRAM)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 315, in __init__
    self._trace_autoload()
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 936, in _trace_autoload
    self.attach_kprobe(event=fn.name[8:], fn_name=fn.name)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 547, in attach_kprobe
    raise Exception("Failed to attach BPF to kprobe")
Exception: Failed to attach BPF to kprobe

[alban]

I tried the same on Fedora 27 with Linux 4.15.14-300.fc27.x86_64 and I have the same behaviour.

I changed the entrypoint to:

+ENTRYPOINT ["/bin/sh", "-c", "echo sleeping... ; sleep 20 ; /src/http-statistics.py"]

So I have time to get the pid of the shell in the container, and run sudo strace -p $PID -f -s 512. I noticed:

[pid 16946] open("/sys/kernel/debug/tracing/kprobe_events", O_WRONLY|O_APPEND) = 9
[pid 16946] write(9, "p:kprobes/p_copy_from_iter_bcc_16946 copy_from_iter", 51) = -1 ENOENT (No such file or directory)
[pid 16946] close(9)                    = 0
[pid 16946] write(2, "Traceback (most recent call last):\n", 35) = 35

So the issue does not seem to be about the BPF program itself but when creating the kprobe for copy_from_iter...

[errordeveloper]

I've put strace in Dockerfile instead, and this is what I'm seeing after many header file reads, followed by many munmaps:

chdir("/src")                           = 0
stat("/src", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat(".", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f49ce5f5000
mmap(0x7f49ce5f6000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f49ce5f4000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f49ce52e000
mprotect(0x7f49ce5f5000, 4096, PROT_READ|PROT_EXEC) = 0
mprotect(0x7f49ce5f5000, 4096, PROT_READ|PROT_EXEC) = 0
mprotect(0x7f49ce5f4000, 4096, PROT_READ|PROT_EXEC) = 0
mprotect(0x7f49ce5f4000, 4096, PROT_READ|PROT_EXEC) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=37, insns=0x7f49ce5f57d0, license="GPL", log_level=0, log_size=0, log_buf=0, kern_version=265485}, 72) = -1 E2BIG (Argument list too long)
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=37, insns=0x7f49ce5f57d0, license="GPL", log_level=0, log_size=0, log_buf=0, kern_version=265485}, 72) = 8
open("/sys/bus/event_source/devices/kprobe/type", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/sys/bus/event_source/devices/kprobe/format/retprobe", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/sys/kernel/debug/tracing/kprobe_events", O_WRONLY|O_APPEND) = 9
write(9, "p:kprobes/p_copy_from_iter_bcc_2"..., 51) = -1 ENOENT (No such file or directory)
close(9)                                = 0
write(2, "Traceback (most recent call last"..., 35Traceback (most recent call last):
) = 35
write(2, "  File \"/src/http-statistics.py\""..., 56  File "/src/http-statistics.py", line 294, in <module>
) = 56
open("/src/http-statistics.py", O_RDONLY) = 9
fstat(9, {st_mode=S_IFREG|0775, st_size=11484, ...}) = 0
read(9, "#!/usr/bin/env python\nimport bcc"..., 4096) = 4096
read(9, "lf.http_resp_code_rate_per_pid ="..., 4096) = 4096
read(9, "                  'label':    'H"..., 4096) = 3292
write(2, "    ", 4    )                     = 4
write(2, "kernel_inspector = KernelInspect"..., 37kernel_inspector = KernelInspector()
) = 37
close(9)                                = 0
write(2, "  File \"/src/http-statistics.py\""..., 55  File "/src/http-statistics.py", line 57, in __init__
) = 55
open("/src/http-statistics.py", O_RDONLY) = 9
fstat(9, {st_mode=S_IFREG|0775, st_size=11484, ...}) = 0
read(9, "#!/usr/bin/env python\nimport bcc"..., 4096) = 4096
write(2, "    ", 4    )                     = 4
write(2, "self.bpf = bcc.BPF(EBPF_PROGRAM)"..., 33self.bpf = bcc.BPF(EBPF_PROGRAM)
) = 33
close(9)                                = 0
write(2, "  File \"/usr/lib/python2.7/dist-"..., 81  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 315, in __init__
) = 81
open("/usr/lib/python2.7/dist-packages/bcc/__init__.py", O_RDONLY) = 9
fstat(9, {st_mode=S_IFREG|0644, st_size=43611, ...}) = 0
read(9, "# Copyright 2015 PLUMgrid\n#\n# Li"..., 4096) = 4096
read(9, "FILTER = 1\n    KPROBE = 2\n    SC"..., 4096) = 4096
read(9, " BPF module with the given sourc"..., 4096) = 4096
write(2, "    ", 4    )                     = 4
write(2, "self._trace_autoload()\n", 23self._trace_autoload()
) = 23
close(9)                                = 0
write(2, "  File \"/usr/lib/python2.7/dist-"..., 88  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 936, in _trace_autoload
) = 88
open("/usr/lib/python2.7/dist-packages/bcc/__init__.py", O_RDONLY) = 9
fstat(9, {st_mode=S_IFREG|0644, st_size=43611, ...}) = 0
read(9, "# Copyright 2015 PLUMgrid\n#\n# Li"..., 4096) = 4096
read(9, "FILTER = 1\n    KPROBE = 2\n    SC"..., 4096) = 4096
read(9, " BPF module with the given sourc"..., 4096) = 4096
read(9, "nothing)\n            if ct.get_e"..., 4096) = 4096
read(9, "\n        del self.tables[key]\n\n "..., 4096) = 4096
read(9, "retprobe\")\n        self._add_kpr"..., 4096) = 4096
read(9, "kernel tracepoint\n        specif"..., 4096) = 4096
read(9, "ol names because it\n        turn"..., 4096) = 4096
read(9, "     for sym_addr in BPF.get_use"..., 4096) = 4096
write(2, "    ", 4    )                     = 4
write(2, "self.attach_kprobe(event=fn.name"..., 55self.attach_kprobe(event=fn.name[8:], fn_name=fn.name)
) = 55
close(9)                                = 0
write(2, "  File \"/usr/lib/python2.7/dist-"..., 86  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 547, in attach_kprobe
) = 86
open("/usr/lib/python2.7/dist-packages/bcc/__init__.py", O_RDONLY) = 9
fstat(9, {st_mode=S_IFREG|0644, st_size=43611, ...}) = 0
read(9, "# Copyright 2015 PLUMgrid\n#\n# Li"..., 4096) = 4096
read(9, "FILTER = 1\n    KPROBE = 2\n    SC"..., 4096) = 4096
read(9, " BPF module with the given sourc"..., 4096) = 4096
read(9, "nothing)\n            if ct.get_e"..., 4096) = 4096
read(9, "\n        del self.tables[key]\n\n "..., 4096) = 4096
write(2, "    ", 4    )                     = 4
write(2, "raise Exception(\"Failed to attac"..., 50raise Exception("Failed to attach BPF to kprobe")
) = 50
close(9)                                = 0
write(2, "Exception", 9Exception)                = 9
write(2, ": ", 2: )                       = 2
write(2, "Failed to attach BPF to kprobe", 30Failed to attach BPF to kprobe) = 30
write(2, "\n", 1
)                       = 1
munmap(0x7f49ce5f5000, 4096)            = 0
munmap(0x7f49ce5f4000, 4096)            = 0
munmap(0x7f49ce52e000, 4096)            = 0
close(5)                                = 0
close(3)                                = 0
close(4)                                = 0
close(6)                                = 0
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7f49ce1c6390}, {0x534a20, [], SA_RESTORER, 0x7f49ce1c6390}, 8) = 0
close(7)                                = 0
exit_group(1)                           = ?
+++ exited with 1 +++

[alban]

In my kernel version, copy_from_iter() became a __always_inline (see include/linux/uio.h). So, no symbol to attach a kprobe to.

Since Linux v4.13 https://github.com/torvalds/linux/commit/aa28de275a248879f9828cb9f7ee7e119c72ff96

The BPF will need to be rewritten differently for those new kernels...

[alban]

This could be updated to use eBPF socket filter instead of eBPF kprobes (idea from bcc's http_filter). In that way, we would use a stable kernel API. But then, we would not get the process IDs: the view in Scope would be per network namespace (per pod or container) instead of per-process. That's probably fine (or even better?) for an UI perspective. And we would have to parse the TCP/IP packets instead of just the payload. The python script would need to connect to the Docker socket to get the list of containers (the scope-traffic-control plugin does that already) and to add a tracer in each network namespace.

[StiviiK]

Are there any updates on this issue yet?

[errordeveloper]

Nobody is working on this at the moment. If this is critical to you, please let us know.

On Wed, 6 Feb 2019, 11:14 am Stefan Kürzeder <notifications@github.com wrote:

Are there any updates on this yet?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/weaveworks-plugins/scope-http-statistics/issues/13#issuecomment-460987047, or mute the thread https://github.com/notifications/unsubscribe-auth/AAPWS6vKiCWdOSmREdsxZnsK0WxI7cByks5vKrkugaJpZM4TboGC .

[StiviiK]

No it isn't critical, i just wanted to try the plugin.

[mJace]

+1 Want to try this too.