Failed to use sysdig build from source in 5.19.11-arch1-1

[tarihub]

[OS Version] Linux archlinux 5.19.11-arch1-1 #1 SMP PREEMPT_DYNAMIC Sat, 24 Sep 2022 18:24:15 +0000 x86_64 GNU/Linux

[Build cmd]

git clone https://github.com/draios/sysdig.git
mkdir -p sysdig/build
cd sysdig/build
cmake -DBUILD_BPF=ON ..
make
make install

[Details] When I run sysdig use sysdig, I get an error below

root@archlinux ~/s/build# sysdig                                                                  dev
Unable to load the driver
unable to open '/sys/module/scap/parameters/g_buffer_bytes_dim': Errno 2. Please ensure the kernel module is already loaded

When I run sysdig with --bpf parameter: ./userspace/sysdig/sysdig --bpf=driver/bpf/probe.o, It works

root@archlinux ~/s/build# ./userspace/sysdig/sysdig --bpf=driver/bpf/probe.o                      dev
25 08:03:09.929858567 3 sysdig (251544.251544) > switch next=0 pgft_maj=0 pgft_min=1073 vm_size=175492 vm_rss=17988 vm_swap=0
26 08:03:09.929909047 3 <NA> (<NA>.0) > switch next=251544(sysdig) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0
.....

When I want to write my eBPF program, I add the code in the end of driver-repo/driver-prefix/src/driver/bpf/probe.c

__attribute__((section("raw_tracepoint/sys_enter"), used))
int bpf_openat_parser()
{
    return 0;
}

And rebuild

make && make install

And then run it, but it doesn't work

root@archlinux ~/s/build# ./userspace/sysdig/sysdig --bpf=driver/bpf/probe.o                      dev
-- BEGIN PROG LOAD LOG --
unreachable insn 442
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

-- END PROG LOAD LOG --
* Running scap-driver-loader for: driver version=3d69d4b515891101323154eea0dac6fed0f8b065
* Running scap-driver-loader with: driver=bpf, compile=yes, download=yes
* Mounting debugfs
* Trying to download a prebuilt eBPF probe from https://download.sysdig.com/scap-drivers/3d69d4b515891101323154eea0dac6fed0f8b065/scap_arch_5.19.11-arch1-1_1.o
curl: (22) The requested URL returned error: 404
Unable to find a prebuilt scap eBPF probe
* Trying to compile the eBPF probe (scap_arch_5.19.11-arch1-1_1.o)
make: *** /usr/src/scap-3d69d4b515891101323154eea0dac6fed0f8b065/bpf: No such file or directory.  Stop.
mv: cannot stat '/usr/src/scap-3d69d4b515891101323154eea0dac6fed0f8b065/bpf/probe.o': No such file or directory
Unable to load the scap eBPF probe
Unable to load the BPF probe
-- BEGIN PROG LOAD LOG --
unreachable insn 442
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

-- END PROG LOAD LOG --
libscap: bpf_load_program() err=22 event=sys_enter

I try to fix /usr/src/scap-3d69d4b515891101323154eea0dac6fed0f8b065 not fond bug

ln -s /usr/local/src/scap-3d69d4b515891101323154eea0dac6fed0f8b065 /usr/src

And run sysdig twice, the last one print

root@archlinux ~/s/build# ./userspace/sysdig/sysdig --bpf=driver/bpf/probe.o                    1 dev
-- BEGIN PROG LOAD LOG --
unreachable insn 442
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

-- END PROG LOAD LOG --
* Running scap-driver-loader for: driver version=3d69d4b515891101323154eea0dac6fed0f8b065
* Running scap-driver-loader with: driver=bpf, compile=yes, download=yes
* Mounting debugfs
* Skipping download, eBPF probe is already present in /root/.scap/scap_arch_5.19.11-arch1-1_1.o
* Skipping compilation, eBPF probe is already present in /root/.scap/scap_arch_5.19.11-arch1-1_1.o
* eBPF probe located in /root/.scap/scap_arch_5.19.11-arch1-1_1.o
* Success: eBPF probe symlinked to /root/.scap/scap-bpf.o
-- BEGIN PROG LOAD LOG --
unreachable insn 442
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

-- END PROG LOAD LOG --
libscap: bpf_load_program() err=22 event=sys_enter

My issue is

1. Why can't I run sysdig directly
2. Why can't I write my eBPF program

Looking forward for your reply!

[jasondellaluce]

Hi @TARI0510. Trying to respond point by point:

1. You're building sysdig from dev, and we're currently working on bumping the falcosecurity/libs version currently so we can expect some errors until the release is stable. I would suggest sticking to the latest release tag (0.29.3).
2. If you don't use the --bpf option, Sysdig attempts using the kernel module, which you are not building/installing. You can solve this with make driver && insmod driver/scap.ko
3. This is mostly an eBPF-related question, from the example you've provided I'm not sure what you're trying to attain here. What I can say is that our bpf probe should use the sys_enter/sys_exit tracepoint for collecting syscalls, and that your program is rejected by the eBPF verifier (not by sysdig itself)

[tarihub]

Hi @TARI0510. Trying to respond point by point:

1. You're building sysdig from dev, and we're currently working on bumping the falcosecurity/libs version currently so we can expect some errors until the release is stable. I would suggest sticking to the latest release tag (0.29.3).
2. If you don't use the --bpf option, Sysdig attempts using the kernel module, which you are not building/installing. You can solve this with make driver && insmod driver/scap.ko
3. This is mostly an eBPF-related question, from the example you've provided I'm not sure what you're trying to attain here. What I can say is that our bpf probe should use the sys_enter/sys_exit tracepoint for collecting syscalls, and that your program is rejected by the eBPF verifier (not by sysdig itself)

@jasondellaluce Thanks for your reply. [For point 1] if I git checkout to 0.29.3 and make, I will get an error

[ 21%] Built target curl
[ 24%] Built target valijson
[ 27%] Built target c-ares
[ 30%] Built target yaml-cpp
[ 33%] Built target njson
Consolidate compiler generated dependencies of target scap
[ 39%] Built target scap
ERROR: modpost: "cc_mkenc" [/root/sysdig/build/driver/src/scap.ko] undefined!
make[5]: *** [scripts/Makefile.modpost:128: /root/sysdig/build/driver/src/Module.symvers] Error 1
make[4]: *** [Makefile:1760: modules] Error 2
make[3]: *** [Makefile:16: all] Error 2
make[2]: *** [driver/CMakeFiles/driver.dir/build.make:70: driver/CMakeFiles/driver] Error 2
make[1]: *** [CMakeFiles/Makefile2:574: driver/CMakeFiles/driver.dir/all] Error 2
make: *** [Makefile:156: all] Error 2

So I use dev branch for compile.

[For point 2] It solves my problem when I run sysdig directly

[For point 3] I try to learn to write eBPF program from https://sysdig.com/blog/the-art-of-writing-ebpf-programs-a-primer/, which finally Implement such a function

__attribute__((section("maps"), used))
struct bpf_map_def tmp_storage_map = {
    .type = BPF_MAP_TYPE_PERCPU_ARRAY,
    .key_size = sizeof(u32),
    .value_size = PATH_MAX,
    .max_entries = 1,
};

__attribute__((section("raw_tracepoint/sys_enter"), used))
int bpf_openat_parser(struct bpf_raw_tracepoint_args *ctx)
{
    unsigned long syscall_id = ctx->args[1];
    struct pt_regs *regs;
    const char *pathname;
    char *map_value;
    u32 map_id;
    int res;

    if (syscall_id != __NR_openat)
        return 0;

    regs = (struct pt_regs *)ctx->args[0];

    res = bpf_probe_read(&pathname, sizeof(pathname), &regs->si);

    map_id = 0;
    map_value = bpf_map_lookup_elem(&tmp_storage_map, &map_id);
    if (!map_value)
        return 0;

    res = bpf_probe_read_str(map_value, PATH_MAX, pathname);

    return 0;
}

It gets the same error with the code above

__attribute__((section("raw_tracepoint/sys_enter"), used))
int bpf_openat_parser()
{
    return 0;
}

Error:

root@archlinux ~/s/build# ./userspace/sysdig/sysdig --bpf=driver/bpf/probe.o                      dev
-- BEGIN PROG LOAD LOG --
unreachable insn 442
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

-- END PROG LOAD LOG --
* Running scap-driver-loader for: driver version=3d69d4b515891101323154eea0dac6fed0f8b065
* Running scap-driver-loader with: driver=bpf, compile=yes, download=yes
* Mounting debugfs
* Skipping download, eBPF probe is already present in /root/.scap/scap_arch_5.19.11-arch1-1_1.o
* Skipping compilation, eBPF probe is already present in /root/.scap/scap_arch_5.19.11-arch1-1_1.o
* eBPF probe located in /root/.scap/scap_arch_5.19.11-arch1-1_1.o
* Success: eBPF probe symlinked to /root/.scap/scap-bpf.o
-- BEGIN PROG LOAD LOG --
unreachable insn 442
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

-- END PROG LOAD LOG --
libscap: bpf_load_program() err=22 event=sys_enter

Could you get some advice to solve Point 1 or Point 2?

[jasondellaluce]

• Point 1: IIRC correctly, this was fixed by @FedeDP some months ago, and the fix should be rolled-out in the next Sysdig 0.30 release (0.29.3 should still be affected). It's related to some changes in the kernel headers in the most recent kernel versions.
• Point 2: Will take a deeper look, but all I can say is that our eBPF probe changed a lot over the past few months mostly thanks to all the great work from @Andreagit97, and I'm not sure that tutorial is still 100% reliable

[tarihub]

OK, I will wait for 0.30 release ~

[longpi1]

I also met this problem. Is there a solution now?

[longpi1]

I have tried 0.30.0, 0.30.1 and 0.30.2 and found this problem exists

[jasondellaluce]

@longpi1 Can you expanding on what your problem looks like in the specific?

[QGB]

@jasondellaluce
Linux ubuntu 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux PRETTY_NAME="Ubuntu 22.04.1 LTS"

docker run --rm -i -t --privileged --net=host \
    -v /var/run/docker.sock:/host/var/run/docker.sock \
    -v /dev:/host/dev \
    -v /proc:/host/proc:ro \
    -v /boot:/host/boot:ro \
    -v /src:/src \
    -v /lib/modules:/host/lib/modules:ro \
    -v /usr:/host/usr:ro \
    -v /etc:/host/etc:ro \
    docker.io/sysdig/sysdig

[root@ubuntu /]# sysdig -c netstat
Unable to load the driver
unable to open '/sys/module/scap/parameters/g_buffer_bytes_dim': Errno 2. Please ensure the kernel module is already loaded.
[root@ubuntu /]# csysdig
unable to open '/sys/module/scap/parameters/g_buffer_bytes_dim': Errno 2. Please ensure the kernel module is already loaded.

[jasondellaluce]

That error indicates that sysdig is not able to reach the kernel module. This happens either because the sysdig process doesn't have enough privileges, or because the module has not been injected (which is likely your case). Did you inject the kernel module with the scap-driver-loader? What was the output of that script?

[therealbobo]

Any update on this issue? 😄 @tarimoe @QGB @longpi1

[QGB]

give up

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[longpi1]

Any update on this issue? 😄 @tarimoe @QGB @longpi1

Any update on this issue? 😄 @tarimoe @QGB @longpi1

Sorry, I haven't been aware of this problem, I have previously solved a similar problem by recompiling the sysdig source code