Closed guolifu closed 2 months ago
inge4pres
commented
2 months ago hey @guolifu 😄 This line
load program: no space left on device: 174:seems to indicate that the BPF programs cannot be loaded due to limited resources. Not sure exactly where the shortage is, pretty sure it's not on disk and the message is misleading.
I tested on three different kernels
Are you using VMs? If yes, how are they configured?
athre0z
commented
2 months ago That's weird. The bpf syscall is documented to return that status code under the following circumstances:
The eBPF program is too large or a map reached the
max_entries limit (maximum number of elements).All kernel versions that you listed should have a limit of 1M instructions whereas we are trying to stay below 4K instructions (for compat with old kernels), so this is probably not it. At the same time, the map sizes that are being printed in your verbose output also don't look outrageously large.
What architecture is this? How many cores does the machine have? How did you build, make agent (uses Docker) or just plain make (uses host LLVM toolchain)?
guolifu
commented
2 months ago That's weird. The
bpfsyscall is documented to return that status code under the following circumstances:The eBPF program is too large or a map reached the max_entries limit (maximum number of elements).All kernel versions that you listed should have a limit of 1M instructions whereas we are trying to stay below 4K instructions (for compat with old kernels), so this is probably not it. At the same time, the map sizes that are being printed in your verbose output also don't look outrageously large.
What architecture is this? How many cores does the machine have? How did you build,
make agent(uses Docker) or just plainmake(uses host LLVM toolchain)?
One of them is this
root@ubuntu2204:/opt/github/euspace# uname -a
Linux ubuntu2204 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linuxcpu cores : 2root@ubuntu2204:/opt/github/euspace# ulimit -a
real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 7373
max locked memory (kbytes, -l) 250420
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 7373
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimitedmake agent (uses Docker)
rockdaboot
commented
2 months ago @guolifu I could reproduce the issue in a fresh Ubuntu22.04 VM.
The work-around is to drop the -bpf-log-level=2 option.
We have to investigate what exactly happens when the log level is set.
guolifu
commented
2 months ago @guolifu我可以在新的 Ubuntu22.04 VM 中重现该问题。 解决方法是删除该
-bpf-log-level=2选项。 我们必须调查设置日志级别时到底会发生什么。
You are right. Setting -bpf-log-level to either 1 or 2 will cause problems, while 0 will not cause any problems.
It's strange.😂
rockdaboot
commented
2 months ago You have to set the -bpf-log-size values high enough (possibly depends on kernel version).
Works here on 6.7.9-amd64:
-bpf-log-level=1 -bpf-log-size=524288
and
-bpf-log-level=2 -bpf-log-size=8388608
guolifu
commented
2 months ago You have to set the
-bpf-log-sizevalues high enough (possibly depends on kernel version).Works here on 6.7.9-amd64:
-bpf-log-level=1 -bpf-log-size=524288and-bpf-log-level=2 -bpf-log-size=8388608
Your help was very much appreciated,I'll give it a try👍
I tried to run it, but the program reported an error. I tested on three different kernels and found that this issue exists in all of them. centos7.9（5.4.219）Ubuntu（5.15-101、5.15-102）