Closed vincentmli closed 5 months ago
I first tried to start loxilb, but I get error below, that is why I tried ntc manually, my attempt is to attach to TC hook, it appears loxilb trying XDP hook first?
[root@bpfire-3 ~]# ./loxilb
loxilb start
INFO: xdp_link_detach() no curr XDP prog on ifindex:6
09:38:04 DEBUG loxilb_libdp.c:2682: exec: ntc filter del dev llb0 ingress 2>&1 > /dev/null
Error: Parent Qdisc doesn't exists.
We have an error talking to the kernel
09:38:04 DEBUG loxilb_libdp.c:2743: /opt/loxilb/llb_xdp_main.o: nr 0 psection xdp_packet_hook
libbpf: map 'cpu_map': failed to create: Argument list too long(-7)
libbpf: failed to load object '/opt/loxilb/llb_xdp_main.o'
ERR: loading BPF-OBJ file(/opt/loxilb/llb_xdp_main.o) (-7): Argument list too long
ERR: loading file: /opt/loxilb/llb_xdp_main.o
Yes you are right. First it loads its XDP program which also creates/pins its map. The same maps are then shared across loxilb's ebpf programs as well.
thanks for the quick response, is there option to allow loxilb load and attach in TC, skip XDP? I already have other XDP program attached to the interface by xdp-tools xdp-loader, I prefer loxilb attach program to TC
loxilb only loads its xdp in its own interface "llb0" and does not use other system interfaces. If you can prevent loading other xdp programs in llb0, probably sudo ./loxilb
would work.
I see, I am not loading other xdp program in llb0 since I don't know that llb0 even exists :). I already have other xdp program load and attached to my red0 interface which is a real system interface in BPFire by xdp-loader. I read somewhere that it says loxilb will try to load program to each system interfaces and cause conflict? maybe I read it wrong. anyway any clue on https://github.com/loxilb-io/loxilb/issues/661#issuecomment-2090974974 error ? I think it is nice still for loxilb to have options to load the program to specific system interface and on specific hook like TC, that is flexible in my opinion :)
No it is not true that loxilb will try to load in all interfaces. loxilb has an option "--blacklist=
This is a close to manual way of loading loxilb xdp/ebpf
loxilb-ebpf/utils/mkllb_bpffs.sh
sudo ip link add tap0 type veth peer name tap1
sudo ifconfig tap0 up
sudo ifconfig tap1 up
sudo mkdir -p /opt/fs/bpf
sudo bpftool -d prog load /opt/loxilb/llb_xdp_main.o /opt/fs/bpf/xdp_packet_hook type xdp pinmaps /opt/loxilb/dp/bpf
sudo ntc filter add dev tap0 egress bpf da obj /opt/loxilb/llb_ebpf_main.o sec tc_packet_hook0
Last step will fail because some maps are not initialized fully. Those maps can be filled using bpftool as well but need to check more.
ok, I got it, do you want me to repeat above manual steps to figure out the missing maps ? and that is the possible reason to cause verifier complains ?
ok, I got it, do you want me to repeat above manual steps to figure out the missing maps ? and that is the possible reason to cause verifier complains ?
Yes kindly try the manual steps once.
And to remove xdp loading you can use the following patch:
diff --git a/common/common_user_bpf_xdp.c b/common/common_user_bpf_xdp.c
index 7e7e74e..496554f 100644
--- a/common/common_user_bpf_xdp.c
+++ b/common/common_user_bpf_xdp.c
@@ -283,6 +283,7 @@ struct bpf_object *load_bpf_and_xdp_attach(struct config *cfg)
exit(EXIT_FAIL_BPF);
}
+#if 0
/* At this point: BPF-progs are (only) loaded by the kernel, and prog_fd
* is our select file-descriptor handle. Next step is attaching this FD
* to a kernel hook point, in this case XDP net_device link-level hook.
@@ -293,6 +294,7 @@ struct bpf_object *load_bpf_and_xdp_attach(struct config *cfg)
/* Consider this soft error */
/* exit(err); */
}
+#endif
return bpf_obj;
}
Kindly recompile and recheck !!
bpftool -d prog load /opt/loxilb/llb_xdp_main.o /opt/fs/bpf/xdp_packet_hook type xdp pinmaps /opt/loxilb/dp/bpf
ok, here is the manual step output from bpftool, it looks the problem is in [ ](libbpf: map 'cpu_map': failed to create: Argument list too long(-7))
libbpf: map 'ct_ctr': found max_entries = 1.
libbpf: prog 'xdp_packet_func': unrecognized ELF section name 'xdp_packet_hook'
libbpf: prog 'xdp_pass_func': unrecognized ELF section name 'xdp_pass'
libbpf: sec '.relxdp_packet_hook': collecting relocation for section(3) 'xdp_packet_hook'
libbpf: sec '.relxdp_packet_hook': relo #0: insn #5 against 'xfis'
libbpf: prog 'xdp_packet_func': found map 0 (xfis, sec 6, off 0) for insn #5
libbpf: sec '.relxdp_packet_hook': relo #1: insn #572 against 'live_cpu_map'
libbpf: prog 'xdp_packet_func': found map 1 (live_cpu_map, sec 6, off 32) for insn #572
libbpf: sec '.relxdp_packet_hook': relo #2: insn #596 against 'cpu_map'
libbpf: prog 'xdp_packet_func': found map 2 (cpu_map, sec 6, off 64) for insn #596
libbpf: sec '.relxdp_packet_hook': relo #3: insn #1060 against 'gparser'
libbpf: prog 'xdp_packet_func': found map 40 (gparser, sec 6, off 1296) for insn #1060
libbpf: map 'xfis': created successfully, fd=4
libbpf: map 'live_cpu_map': created successfully, fd=5
libbpf: map 'cpu_map': failed to create: Argument list too long(-7)
libbpf: failed to load object '/opt/loxilb/llb_xdp_main.o'
Error: failed to load object file
just fyi, I have libbpf 1.3.0 installed in BPFire system
Got it. It seems libbpf version mismatch. By the way latest loxilb does not use the shared libbpf any more. You can check -
ldd ./loxilb
ldd /usr/local/sbin/ntc
yes, loxilb is compiled with static libbpf 0.8
[root@bpfire-3 ~]# ldd loxilb
linux-vdso.so.1 (0x00007fffd0b79000)
libelf.so.1 => /usr/lib/libelf.so.1 (0x00007fd6f41b2000)
libz.so.1 => /lib64/libz.so.1 (0x00007fd6f4198000)
libc.so.6 => /lib64/libc.so.6 (0x00007fd6f3fb7000)
libzstd.so.1 => /usr/lib/libzstd.so.1 (0x00007fd6f3ef5000)
liblzma.so.5 => /usr/lib/liblzma.so.5 (0x00007fd6f3ec5000)
libbz2.so.1.0 => /lib/libbz2.so.1.0 (0x00007fd6f3eb2000)
/lib64/ld-linux-x86-64.so.2 (0x00007fd6f41e3000)
[root@bpfire-3 ~]# ldd /usr/local/bin/ntc
linux-vdso.so.1 (0x00007ffed4590000)
libelf.so.1 => /usr/lib/libelf.so.1 (0x00007fd8a1d53000)
libmnl.so.0 => /usr/lib/libmnl.so.0 (0x00007fd8a1d4c000)
libm.so.6 => /lib/libm.so.6 (0x00007fd8a1c60000)
libc.so.6 => /lib/libc.so.6 (0x00007fd8a1a7f000)
libzstd.so.1 => /usr/lib/libzstd.so.1 (0x00007fd8a19bd000)
liblzma.so.5 => /usr/lib/liblzma.so.5 (0x00007fd8a198d000)
libbz2.so.1.0 => /lib/libbz2.so.1.0 (0x00007fd8a1978000)
/lib64/ld-linux-x86-64.so.2 (0x00007fd8a1e1c000)
my bpftool version
[root@bpfire-3 ~]# bpftool -V
bpftool v7.3.0
using libbpf v1.3
features: llvm, skeletons
[root@bpfire-3 ~]# ldd /usr/sbin/bpftool
linux-vdso.so.1 (0x00007ffe74b5e000)
libelf.so.1 => /usr/lib/libelf.so.1 (0x00007237c3688000)
libcap.so.2 => /usr/lib/libcap.so.2 (0x00007237c367c000)
libm.so.6 => /lib/libm.so.6 (0x00007237c3590000)
libzstd.so.1 => /usr/lib/libzstd.so.1 (0x00007237c34ce000)
libncursesw.so.6 => /usr/lib/libncursesw.so.6 (0x00007237c3455000)
libxml2.so.2 => /usr/lib/libxml2.so.2 (0x00007237c32da000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007237c3000000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007237c32b3000)
libc.so.6 => /lib/libc.so.6 (0x00007237c2e1f000)
/lib64/ld-linux-x86-64.so.2 (0x00007237c36b9000)
liblzma.so.5 => /usr/lib/liblzma.so.5 (0x00007237c2def000)
libbz2.so.1.0 => /lib/libbz2.so.1.0 (0x00007237c32a0000)
@UltraInstinct14 I have tried to build a few bpftool versions and all failed with same error with bpftool -d prog load /opt/loxilb/llb_xdp_main.o ...
, could you let me know what bpftool version that has been working for you?
@UltraInstinct14 I have tried to build a few bpftool versions and all failed with same error with
bpftool -d prog load /opt/loxilb/llb_xdp_main.o ...
, could you let me know what bpftool version that has been working for you?
now I am not sure if it bpftool or libbpf version issue since I build bpftool v6.8.0 with libbpf v0.8, the issue is still the same
[root@bpfire-2 /]# /tmp/bpftool -V
/tmp/bpftool v6.8.0
using libbpf v0.8
features: libbpf_strict, skeletons
libbpf: map 'live_cpu_map': created successfully, fd=5
libbpf: map 'cpu_map': failed to create: Argument list too long(-7)
libbpf: failed to load object '/opt/loxilb/llb_xdp_main.o'
Error: failed to load object file
Warning: bpftool is now running in libbpf strict mode and has more stringent requirements about BPF programs.
If it used to work for this object file but now doesn't, see --legacy option for more details.
I wonder why live_cpu_map
succeeded, but not cpu_map
, the difference is the map type is BPF_MAP_TYPE_ARRAY
vs BPF_MAP_TYPE_CPUMAP
struct {
__uint(type, BPF_MAP_TYPE_CPUMAP);
__type(key, __u32);
__type(value, __u32);
__uint(max_entries, MAX_CPUS);
} cpu_map SEC(".maps");
struct {
__uint(type, BPF_MAP_TYPE_ARRAY);
__type(key, __u32);
__type(value, __u32);
__uint(max_entries, MAX_CPUS);
} live_cpu_map SEC(".maps");
I wonder why
live_cpu_map
succeeded, but notcpu_map
, the difference is the map type isBPF_MAP_TYPE_ARRAY
vsBPF_MAP_TYPE_CPUMAP
struct { __uint(type, BPF_MAP_TYPE_CPUMAP); __type(key, __u32); __type(value, __u32); __uint(max_entries, MAX_CPUS); } cpu_map SEC(".maps"); struct { __uint(type, BPF_MAP_TYPE_ARRAY); __type(key, __u32); __type(value, __u32); __uint(max_entries, MAX_CPUS); } live_cpu_map SEC(".maps");
so I hacked the code to change cpu_map
to BPF_MAP_TYPE_ARRAY
type, libbpf get passed ok
libbpf: map 'live_cpu_map': created successfully, fd=5
libbpf: map 'cpu_map': created successfully, fd=6
but of course I get error below since cpu_map type does not match
599: (b7) r3 = 0 ; R3_w=0
600: (85) call bpf_redirect_map#51
cannot pass map_type 2 into func bpf_redirect_map#51
verification time 2872 usec
stack depth 40
the problem seems related to BPF_MAP_TYPE_CPUMAP
and libbpf, some sort of limitation there.
ok I made progress, if I change cpu_map
below from MAX_CPUS
to 2
, it seems the map get pinned ok
struct {
__uint(type, BPF_MAP_TYPE_CPUMAP);
__type(key, __u32);
__type(value, __u32);
__uint(max_entries, 2);
} cpu_map SEC(".maps");
bpftool -d prog load /opt/loxilb/llb_xdp_main.o /opt/fs/bpf/xdp_packet_hook type xdp pinmaps /opt/loxilb/dp/bpf
libbpf: prog 'xdp_pass_func': -- BEGIN PROG LOAD LOG --
func#0 @0
0: R1=ctx(off=0,imm=0) R10=fp0
; return dp_ing_pass_main(ctx);
0: (b7) r0 = 2 ; R0_w=2
1: (95) exit
verification time 51 usec
stack depth 0
processed 2 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
-- END PROG LOAD LOG --
libbpf: pinned map '/opt/loxilb/dp/bpf/xfis'
libbpf: pinned map '/opt/loxilb/dp/bpf/live_cpu_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/cpu_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/intf_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/intf_stats_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/bd_stats_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/pkt_ring'
libbpf: pinned map '/opt/loxilb/dp/bpf/cp_ring'
libbpf: pinned map '/opt/loxilb/dp/bpf/pkts'
libbpf: pinned map '/opt/loxilb/dp/bpf/fcas'
libbpf: pinned map '/opt/loxilb/dp/bpf/tx_intf_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/tx_intf_stats_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/tx_bd_stats_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/smac_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/dmac_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/tmac_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/tmac_stats_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/nh_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/ct_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/ct_stats_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/nat_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/nat_stats_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/nat_ep_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/rt_v4_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/rt_v4_stats_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/rt_v6_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/rt_v6_stats_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/mirr_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/sess_v4_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/sess_v4_stats_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/fc_v4_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/fc_v4_stats_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/fw_v4_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/fw_v4_stats_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/pgm_tbl'
libbpf: pinned map '/opt/loxilb/dp/bpf/polx_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/xfck'
libbpf: pinned map '/opt/loxilb/dp/bpf/crc32c_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/pplat_map'
libbpf: pinned map '/opt/loxilb/dp/bpf/xctk'
libbpf: pinned map '/opt/loxilb/dp/bpf/gparser'
libbpf: pinned map '/opt/loxilb/dp/bpf/ct_ctr'
ok, I think I found the problem, loxilb has MAX_CPUS
set to 128
that is greater than BPFire NR_CPUS
which is 64
, I can adjust BPFire NR_CPUS
to values bigger than 128
kernel/bpf/cpumap.c:
static struct bpf_map *cpu_map_alloc(union bpf_attr *attr)
{
u32 value_size = attr->value_size;
struct bpf_cpu_map *cmap;
/* check sanity of attributes */
if (attr->max_entries == 0 || attr->key_size != 4 ||
(value_size != offsetofend(struct bpf_cpumap_val, qsize) &&
value_size != offsetofend(struct bpf_cpumap_val, bpf_prog.fd)) ||
attr->map_flags & ~BPF_F_NUMA_NODE)
return ERR_PTR(-EINVAL);
/* Pre-limit array size based on NR_CPUS, not final CPU check */
if (attr->max_entries > NR_CPUS)
return ERR_PTR(-E2BIG); <======
include/linux/threads.h
/* Places which use this should consider cpumask_var_t. */
#define NR_CPUS CONFIG_NR_CPUS
and BPFire/IPFire config/kernel/kernel.config.x86_64-ipfire
has
CONFIG_NR_CPUS_RANGE_BEGIN=2
CONFIG_NR_CPUS_RANGE_END=512
CONFIG_NR_CPUS_DEFAULT=64
CONFIG_NR_CPUS=64 <====
Hi @vincentmli Great find !!
Describe the bug I am following https://loxilb-io.github.io/loxilbdocs/loxilbebpf/ to use ntc to manually load and attach program
ntc filter add dev red0 ingress bpf da obj /opt/loxilb/llb_ebpf_main.o sec tc_packet_hook0 verbos
verifier complains error below, is it because the map is not loaded?
To Reproduce
Expected behavior
expect program load and attach
Screenshots If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):
Additional context
trying to port loxilb to open source firewall BPFire OS https://github.com/vincentmli/BPFire/issues/12