search

gamemann / XDP-Firewall

A firewall that utilizes the Linux kernel's XDP hook. The XDP hook allows for very fast network processing on Linux systems. This is great for dropping malicious traffic from a (D)DoS attack. IPv6 is supported with this firewall! I hope this helps network engineers/programmers interested in utilizing XDP!
https://deaconn.net/
MIT License
493 stars 87 forks source link

successful compilation, but xdpfw output errors then starting #43

Closed pettai closed 8 months ago

pettai commented 8 months ago

Hi,

After successful build and installation on a vanilla Ubuntu 22.04, running xdpfw doesn't work as intended:

Sys info:

root@bygg-u2204:/home/ubuntu# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:    22.04
Codename:   jammy

root@bygg-u2204:/home/ubuntu# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether fa:16:3e:88:7b:5f brd ff:ff:ff:ff:ff:ff
    altname enp0s3

Build info:

ubuntu@bygg-u2204:~/XDP-Firewall$ make
mkdir -p build/
clang -O2 -c -o build/config.o src/config.c
clang -O2 -c -o build/cmdline.o src/cmdline.c
make -C modules/xdp-tools
make[1]: Entering directory '/home/ubuntuXDP-Firewall/modules/xdp-tools'
sh configure
Found clang binary 'clang' with version 14 (from 'Ubuntu clang version 14.0.0-1ubuntu1.1')
using /usr/lib/linux-tools/5.15.0-86-generic/bpftool v5.15.122
libbpf support: submodule v1.0.0
  perf_buffer__consume support: yes (submodule)
  btf__load_from_kernel_by_id support: yes (submodule)
  btf__type_cnt support: yes (submodule)
  bpf_object__next_map support: yes (submodule)
  bpf_object__next_program support: yes (submodule)
  bpf_program__insn_cnt support: yes (submodule)
  bpf_program__type support: yes (submodule)
  bpf_program__flags support: yes (submodule)
  bpf_program__expected_attach_type support: yes (submodule)
  bpf_map_create support: yes (submodule)
  perf_buffer__new_raw support: yes (submodule)
  bpf_xdp_attach support: yes (submodule)
  bpf_map__set_autocreate support: yes (submodule)
  bpf_prog_test_run_opts support: yes (submodule)
zlib support: yes
ELF support: yes
pcap support: yes
secure_getenv support: yes

[...]

  INSTALL  ./libbpf.a ./libbpf.so ./libbpf.so.1 ./libbpf.so.1.0.0
make: Leaving directory '/home/ubuntu/XDP-Firewall/modules/xdp-tools/lib/libbpf/src'
sudo make -C modules/xdp-tools/lib/libxdp install
make: Entering directory '/home/ubuntu/XDP-Firewall/modules/xdp-tools/lib/libxdp'
make: Leaving directory '/home/ubuntu/XDP-Firewall/modules/xdp-tools/lib/libxdp'
mkdir -p build/
clang -lconfig -lelf -lz -lxdp -I modules/xdp-tools/lib/libbpf/src -I /usr/include -I /usr/local/include -o build/xdpfw modules/xdp-tools/lib/libbpf/src/staticobjs/bpf_prog_linfo.o modules/xdp-tools/lib/libbpf/src/staticobjs/bpf.o modules/xdp-tools/lib/libbpf/src/staticobjs/btf_dump.o modules/xdp-tools/lib/libbpf/src/staticobjs/btf.o modules/xdp-tools/lib/libbpf/src/staticobjs/gen_loader.o   modules/xdp-tools/lib/libbpf/src/staticobjs/hashmap.o modules/xdp-tools/lib/libbpf/src/staticobjs/libbpf_errno.o modules/xdp-tools/lib/libbpf/src/staticobjs/libbpf_probes.o  modules/xdp-tools/lib/libbpf/src/staticobjs/libbpf.o modules/xdp-tools/lib/libbpf/src/staticobjs/linker.o modules/xdp-tools/lib/libbpf/src/staticobjs/netlink.o  modules/xdp-tools/lib/libbpf/src/staticobjs/nlattr.o modules/xdp-tools/lib/libbpf/src/staticobjs/relo_core.o modules/xdp-tools/lib/libbpf/src/staticobjs/ringbuf.o modules/xdp-tools/lib/libbpf/src/staticobjs/str_error.o modules/xdp-tools/lib/libbpf/src/staticobjs/strset.o modules/xdp-tools/lib/libbpf/src/staticobjs/usdt.o modules/xdp-tools/lib/libxdp/sharedobjs/xsk.o modules/xdp-tools/lib/libxdp/sharedobjs/libxdp.o build/config.o build/cmdline.o src/xdpfw.c
mkdir -p build/
clang -I modules/xdp-tools/lib/libbpf/src -I /usr/include -I /usr/local/include -D__BPF__ -D __BPF_TRACING__ -Wno-unused-value -Wno-pointer-sign -Wno-compare-distinct-pointer-types -O2 -emit-llvm -c -g -o build/xdpfw_kern.ll src/xdpfw_kern.c
/usr/lib/llvm-14/bin/llc -march=bpf -filetype=obj -o build/xdpfw_kern.o build/xdpfw_kern.ll

XDPFW info:

root@bygg-u2204:/home/ubuntu# cat /etc/xdpfw/xdpfw.conf
interface = "ens3";
updatetime = 0;

filters = (
    {
        enabled = true,
        action = 1
    }
);

root@bygg-u2204:/home/ubuntu# xdpfw -c /etc/xdpfw/xdpfw.conf
libbpf: elf: skipping unrecognized data section(7) .xdp_run_config
libbpf: elf: skipping unrecognized data section(8) xdp_metadata
libbpf: elf: skipping unrecognized data section(23) .eh_frame
libbpf: elf: skipping relo section(24) .rel.eh_frame for section(23) .eh_frame
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libbpf: prog 'xdp_pass': BPF program load failed: Invalid argument
libbpf: prog 'xdp_pass': failed to load: -22
libbpf: failed to load object '/usr/local/lib/bpf/xdp-dispatcher.o'
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
^C
pettai commented 8 months ago

Well, segfaults is the more correct term I guess... 

# dmesg
…
[ 1139.024355] show_signal_msg: 21 callbacks suppressed
[ 1139.024359] xdpfw[6125]: segfault at ebb0 ip 000000000000ebb0 sp 00007ffd67094f48 error 14 in xdpfw[562cafcab000+c000]
[ 1139.024373] Code: Unable to access opcode bytes at RIP 0xeb86.
[ 1185.264171] xdpfw[6562]: segfault at ebb0 ip 000000000000ebb0 sp 00007ffe53199b68 error 14 in xdpfw[5603b2226000+c000]
[ 1185.264184] Code: Unable to access opcode bytes at RIP 0xeb86.
gamemann commented 8 months ago

I just spun up a vanilla Ubuntu 22.04 VM and had the following error.


BPF program is too large. Processed 1000001 insn
processed 1000001 insns (limit 1000000) max_states_per_insn 213 total_states 32198 peak_states 4041 mark_read 324
-- END PROG LOAD LOG --
libbpf: prog 'xdp_prog_main': failed to load: -7
libbpf: failed to load object '/etc/xdpfw/xdpfw_kern.o'
libxdp: Falling back to loading single prog without dispatcher

This is due to the MAX_FILTERS constant being too high for the BPF program. When I set it to 80, it works fine.

This has happened before, but what I don't understand is it works fine for a while beforehand and nothing in the code changes before it starts outputting this error. There must be something changing on the Linux kernel/BPF's side and the amount of insns allowed to be processed has always been one million. The limit can also be raised if you rebuild the kernel and I made public patches on another XDP program I made here for it.

I'll be making a commit soon and thank you for the report :smile:

gamemann commented 8 months ago

Commit c12496a adjusts the MAX_FILTERS constant to 80.

Can you try loading the firewall after pulling the newer commits via git pull and rebuilding?

pettai commented 8 months ago

Good catch! After pulling your update & recompiling/reinstalling, it works 👍

(But I see crap output from libbpf still...


ubuntu@bygg-u2204:~/XDP-Firewall$ sudo /usr/bin/xdpfw
libbpf: elf: skipping unrecognized data section(7) .xdp_run_config
libbpf: elf: skipping unrecognized data section(8) xdp_metadata
libbpf: elf: skipping unrecognized data section(23) .eh_frame
libbpf: elf: skipping relo section(24) .rel.eh_frame for section(23) .eh_frame
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libbpf: prog 'xdp_pass': BPF program load failed: Invalid argument
libbpf: prog 'xdp_pass': failed to load: -22
libbpf: failed to load object '/usr/local/lib/bpf/xdp-dispatcher.o'
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
Loaded XDP program on mode DRV/native.
Packets Allowed: 399 | Packets Dropped: 0Loaded XDP program on mode DRV/native.
^C
``` )
pettai commented 8 months ago

(Ok, now I also see the same error you mention then looking into journalctl -u xdpfw

Oct 11 21:25:02 bygg-u2204 xdpfw[9738]: BPF program is too large. Processed 1000001 insn
Oct 11 21:25:02 bygg-u2204 xdpfw[9738]: processed 1000001 insns (limit 1000000) max_states_per_insn 213 total_states 32198 peak>
Oct 11 21:25:02 bygg-u2204 xdpfw[9738]: -- END PROG LOAD LOG --
Oct 11 21:25:02 bygg-u2204 xdpfw[9738]: libbpf: prog 'xdp_prog_main': failed to load: -7
Oct 11 21:25:02 bygg-u2204 xdpfw[9738]: libbpf: failed to load object '/etc/xdpfw/xdpfw_kern.o'
Oct 11 21:25:02 bygg-u2204 xdpfw[9738]: libxdp: Falling back to loading single prog without dispatcher
Oct 11 21:25:02 bygg-u2204 xdpfw[9738]: libbpf: elf: skipping unrecognized data section(7) xdp_metadata
Oct 11 21:25:02 bygg-u2204 xdpfw[9738]: libbpf: prog 'xdp_pass': BPF program load failed: Invalid argument
Oct 11 21:25:02 bygg-u2204 xdpfw[9738]: libbpf: prog 'xdp_pass': failed to load: -22
Oct 11 21:25:02 bygg-u2204 xdpfw[9738]: libbpf: failed to load object '/usr/local/lib/bpf/xdp-dispatcher.o'
Oct 11 21:25:02 bygg-u2204 xdpfw[9738]: Could not attach with mode DRV/native (Bad file descriptor) (9).
Oct 11 21:25:02 bygg-u2204 xdpfw[9738]: libbpf: elf: skipping unrecognized data section(7) xdp_metadata
Oct 11 21:25:02 bygg-u2204 xdpfw[9738]: libbpf: prog 'xdp_pass': BPF program load failed: Invalid argument
Oct 11 21:25:02 bygg-u2204 xdpfw[9738]: libbpf: prog 'xdp_pass': failed to load: -22
Oct 11 21:25:02 bygg-u2204 xdpfw[9738]: libbpf: failed to load object '/usr/local/lib/bpf/xdp-dispatcher.o'
Oct 11 21:25:02 bygg-u2204 xdpfw[9738]: libbpf: elf: skipping unrecognized data section(7) xdp_metadata
Oct 11 21:25:02 bygg-u2204 xdpfw[9738]: Could not attach with mode SKB/generic (Invalid argument) (22).
Oct 11 21:25:02 bygg-u2204 xdpfw[9738]: Error attaching XDP program.

I never got that on stdout/stderr in my terminal

)

pettai commented 8 months ago

it works now, so I'll close this now