search

daeuniverse / dae

eBPF-based Linux high-performance transparent proxy solution.
GNU Affero General Public License v3.0
3.05k stars 191 forks source link

[Bug Report] Failed to start dae on kernel 6.9rc1 #482

Closed sakarie9 closed 6 months ago

sakarie9 commented 6 months ago

Checks

Current Behavior

3月 26 09:49:46 systemd[1]: Starting dae Service...
3月 26 09:49:46 dae[21143]: level=info msg="Include config files: [/etc/dae/config.dae]"
3月 26 09:49:46 dae[21143]: level=warning msg="No node found."
3月 26 09:49:46 dae[21143]: level=warning msg="No interface to bind."
3月 26 09:49:46 dae[21143]: level=info msg="Loading eBPF programs and maps into the kernel..."
3月 26 09:49:46 dae[21143]: level=info msg="The loading process takes about 120MB free memory, which will be released after loading. Insufficient memory will cause loading failure."
3月 26 09:49:46 dae[21143]: time="2024-03-26T09:49:46+08:00" level=fatal msg="load eBPF objects: field TproxyLanIngress: program tproxy_lan_ingress: load program: invalid argument: invalid func unknown#195896080 (376 line(s) omitted)"
3月 26 09:49:46 systemd[1]: dae.service: Main process exited, code=exited, status=1/FAILURE
3月 26 09:49:46 systemd[1]: dae.service: Failed with result 'exit-code'.
3月 26 09:49:46 systemd[1]: Failed to start dae Service.

Expected Behavior

Succeeded

Steps to Reproduce

  1. Use kernel 6.9.0-rc1
  2. Run dae

Environment

Anything else?

minimal bootable config and working config pre 6.9 both crashed

dae-prow[bot] commented 6 months ago

Thanks for opening this issue!

mzz2017 commented 6 months ago

@sakarie9 按照文档检查一下内核选项

sakarie9 commented 6 months ago

@sakarie9 按照文档检查一下内核选项

CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_CGROUPS=y
CONFIG_KPROBES=y
CONFIG_NET_INGRESS=y
CONFIG_NET_EGRESS=y
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_CLS_BPF=m
CONFIG_NET_CLS_ACT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
CONFIG_DEBUG_INFO_BTF=y
CONFIG_KPROBE_EVENTS=y
CONFIG_BPF_EVENTS=y
mzz2017 commented 6 months ago

@sakarie9 我（怀疑）是 bpf_printk 的问题，这个需要一个内核选项，你查一下看看有没有

sakarie9 commented 6 months ago

@sakarie9 我（怀疑）是 bpf_printk 的问题，这个需要一个内核选项，你查一下看看有没有

这些吗

CONFIG_PRINTK_INDEX=y
CONFIG_PRINTK=y
CONFIG_SND_VERBOSE_PRINTK=y
# CONFIG_USB_G_DBGP_PRINTK is not set
CONFIG_PRINTK_TIME=y
# CONFIG_PRINTK_CALLER is not set
CONFIG_BOOT_PRINTK_DELAY=y
CONFIG_EARLY_PRINTK_USB=y
CONFIG_EARLY_PRINTK=y
CONFIG_EARLY_PRINTK_DBGP=y
CONFIG_EARLY_PRINTK_USB_XDBC=y
CONFIG_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y
# BPF subsystem
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_BPF_UNPRIV_DEFAULT_OFF=y
# CONFIG_BPF_PRELOAD is not set
CONFIG_BPF_LSM=y
# end of BPF subsystem
CONFIG_CGROUP_BPF=y
CONFIG_IPV6_SEG6_BPF=y
CONFIG_NETFILTER_BPF_LINK=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_STREAM_PARSER=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_BPF_LIRC_MODE2=y
# HID-BPF support
CONFIG_HID_BPF=y
# end of HID-BPF support
CONFIG_BPF_EVENTS=y
CONFIG_BPF_KPROBE_OVERRIDE=y
# CONFIG_TEST_BPF is not set
jschwinger233 commented 6 months ago

能把 log_level 换成 fatal 看看 verifier 日志吗？

sakarie9 commented 6 months ago 
3月 26 10:53:23 systemd[1]: Starting dae Service...
3月 26 10:53:23 dae[621]: level=fatal msg="0: R1=ctx() R10=fp0
3月 26 10:53:23 dae[621]: ; int tproxy_lan_ingress(struct __sk_buff *skb) { @ tproxy.c:854
3月 26 10:53:23 dae[621]: 0: (bf) r6 = r1                       ; R1=ctx() R6_w=ctx()
3月 26 10:53:23 dae[621]: ; if (get_link_h_len(skb->ifindex, &link_h_len)) { @ tproxy.c:864
3月 26 10:53:23 dae[621]: 1: (61) r1 = *(u32 *)(r6 +40)         ; R1_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R6_w=ctx()
3月 26 10:53:23 dae[621]: 2: (63) *(u32 *)(r10 -176) = r1       ; R1_w=scalar(id=1,smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R10=fp0 fp-176=????scalar(id=1,smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
3月 26 10:53:23 dae[621]: 3: (bf) r2 = r10                      ; R2_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 4: (07) r2 += -176                    ; R2_w=fp-176
3月 26 10:53:23 dae[621]: ; __u32 *plink_h_len = bpf_map_lookup_elem(&linklen_map, &ifindex); @ tproxy.c:806
3月 26 10:53:23 dae[621]: 5: (18) r1 = 0xffff8ec815422400       ; R1_w=map_ptr(map=linklen_map,ks=4,vs=4)
3月 26 10:53:23 dae[621]: 7: (85) call bpf_map_lookup_elem#1    ; R0_w=map_value_or_null(id=2,map=linklen_map,ks=4,vs=4)
3月 26 10:53:23 dae[621]: ; if (!plink_h_len) { @ tproxy.c:807
3月 26 10:53:23 dae[621]: 8: (55) if r0 != 0x0 goto pc+2 11: R0_w=map_value(map=linklen_map,ks=4,vs=4) R6_w=ctx() R10=fp0 fp-176=????mmmm
3月 26 10:53:23 dae[621]: ; *link_h_len = *plink_h_len; @ tproxy.c:810
3月 26 10:53:23 dae[621]: 11: (61) r1 = *(u32 *)(r0 +0)         ; R0_w=map_value(map=linklen_map,ks=4,vs=4) R1_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
3月 26 10:53:23 dae[621]: 12: (63) *(u32 *)(r10 -124) = r1      ; R1_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R10=fp0 fp-128=mmmm????
3月 26 10:53:23 dae[621]: ; if (link_h_len == ETH_HLEN) { @ tproxy.c:481
3月 26 10:53:23 dae[621]: 13: (55) if r1 != 0xe goto pc+13      ; R1_w=14
3月 26 10:53:23 dae[621]: 14: (b7) r7 = 0                       ; R7_w=0
3月 26 10:53:23 dae[621]: 15: (bf) r3 = r10                     ; R3_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; ret = bpf_skb_load_bytes(skb, offset, ethh, sizeof(struct ethhdr)); @ tproxy.c:482
3月 26 10:53:23 dae[621]: 16: (07) r3 += -16                    ; R3_w=fp-16
3月 26 10:53:23 dae[621]: 17: (b7) r9 = 14                      ; R9_w=14
3月 26 10:53:23 dae[621]: 18: (bf) r1 = r6                      ; R1_w=ctx() R6_w=ctx()
3月 26 10:53:23 dae[621]: 19: (b7) r2 = 0                       ; R2_w=0
3月 26 10:53:23 dae[621]: 20: (b7) r4 = 14                      ; R4_w=14
3月 26 10:53:23 dae[621]: 21: (85) call bpf_skb_load_bytes#26   ; R0=scalar() fp-8=??mmmmmm fp-16=mmmmmmmm
3月 26 10:53:23 dae[621]: ; if (ret) { @ tproxy.c:483
3月 26 10:53:23 dae[621]: 22: (67) r0 <<= 32                    ; R0_w=scalar(smax=0x7fffffff00000000,umax=0xffffffff00000000,smin32=0,smax32=umax32=0,var_off=(0x0; 0xffffffff00000000))
3月 26 10:53:23 dae[621]: 23: (77) r0 >>= 32                    ; R0_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
3月 26 10:53:23 dae[621]: 24: (15) if r0 == 0x0 goto pc+10      ; R0_w=0 R6=ctx() R7=0 R9=14 R10=fp0 fp-8=??mmmmmm fp-16=mmmmmmmm fp-128=mmmm???? fp-176=????mmmm
3月 26 10:53:23 dae[621]: ; ethh->h_proto = skb->protocol; @ tproxy.c:491
3月 26 10:53:23 dae[621]: 35: (b7) r7 = 0                       ; R7_w=0
3月 26 10:53:23 dae[621]: ; __builtin_memset(iph, 0, sizeof(struct iphdr)); @ tproxy.c:496
3月 26 10:53:23 dae[621]: 36: (63) *(u32 *)(r10 -24) = r7       ; R7_w=0 R10=fp0 fp-24=????0
3月 26 10:53:23 dae[621]: 37: (7b) *(u64 *)(r10 -32) = r7       ; R7_w=0 R10=fp0 fp-32_w=0
3月 26 10:53:23 dae[621]: 38: (7b) *(u64 *)(r10 -40) = r7       ; R7_w=0 R10=fp0 fp-40_w=0
3月 26 10:53:23 dae[621]: ; __builtin_memset(ipv6h, 0, sizeof(struct ipv6hdr)); @ tproxy.c:497
3月 26 10:53:23 dae[621]: 39: (7b) *(u64 *)(r10 -80) = r7       ; R7_w=0 R10=fp0 fp-80_w=0
3月 26 10:53:23 dae[621]: 40: (7b) *(u64 *)(r10 -72) = r7       ; R7_w=0 R10=fp0 fp-72_w=0
3月 26 10:53:23 dae[621]: 41: (7b) *(u64 *)(r10 -64) = r7       ; R7_w=0 R10=fp0 fp-64_w=0
3月 26 10:53:23 dae[621]: 42: (7b) *(u64 *)(r10 -56) = r7       ; R7_w=0 R10=fp0 fp-56_w=0
3月 26 10:53:23 dae[621]: 43: (7b) *(u64 *)(r10 -48) = r7       ; R7_w=0 R10=fp0 fp-48_w=0
3月 26 10:53:23 dae[621]: ; __builtin_memset(icmp6h, 0, sizeof(struct icmp6hdr)); @ tproxy.c:498
3月 26 10:53:23 dae[621]: 44: (7b) *(u64 *)(r10 -88) = r7       ; R7_w=0 R10=fp0 fp-88_w=0
3月 26 10:53:23 dae[621]: ; __builtin_memset(tcph, 0, sizeof(struct tcphdr)); @ tproxy.c:499
3月 26 10:53:23 dae[621]: 45: (7b) *(u64 *)(r10 -112) = r7      ; R7_w=0 R10=fp0 fp-112_w=0
3月 26 10:53:23 dae[621]: 46: (7b) *(u64 *)(r10 -104) = r7      ; R7_w=0 R10=fp0 fp-104_w=0
3月 26 10:53:23 dae[621]: 47: (63) *(u32 *)(r10 -96) = r7       ; R7_w=0 R10=fp0 fp-96=????0
3月 26 10:53:23 dae[621]: ; __builtin_memset(udph, 0, sizeof(struct udphdr)); @ tproxy.c:500
3月 26 10:53:23 dae[621]: 48: (7b) *(u64 *)(r10 -120) = r7      ; R7_w=0 R10=fp0 fp-120_w=0
3月 26 10:53:23 dae[621]: 49: (bf) r1 = r10                     ; R1_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 50: (07) r1 += -16                    ; R1_w=fp-16
3月 26 10:53:23 dae[621]: ; if (ethh->h_proto == bpf_htons(ETH_P_IP)) { @ tproxy.c:504
3月 26 10:53:23 dae[621]: 51: (69) r2 = *(u16 *)(r1 +12)        ; R1_w=fp-16 R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) fp-8=??mmmmmm
3月 26 10:53:23 dae[621]: 52: (55) if r2 != 0x8 goto pc+28      ; R2_w=8
3月 26 10:53:23 dae[621]: 53: (bf) r3 = r10                     ; R3_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; if ((ret = bpf_skb_load_bytes(skb, offset, iph, sizeof(struct iphdr)))) { @ tproxy.c:506
3月 26 10:53:23 dae[621]: 54: (07) r3 += -40                    ; R3_w=fp-40
3月 26 10:53:23 dae[621]: 55: (bf) r1 = r6                      ; R1_w=ctx() R6=ctx()
3月 26 10:53:23 dae[621]: 56: (bf) r2 = r9                      ; R2_w=14 R9=14
3月 26 10:53:23 dae[621]: 57: (b7) r4 = 20                      ; R4_w=20
3月 26 10:53:23 dae[621]: 58: (85) call bpf_skb_load_bytes#26   ; R0=scalar() fp-24=????mmmm fp-32=mmmmmmmm fp-40=mmmmmmmm
3月 26 10:53:23 dae[621]: 59: (67) r0 <<= 32                    ; R0_w=scalar(smax=0x7fffffff00000000,umax=0xffffffff00000000,smin32=0,smax32=umax32=0,var_off=(0x0; 0xffffffff00000000))
3月 26 10:53:23 dae[621]: 60: (77) r0 >>= 32                    ; R0_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
3月 26 10:53:23 dae[621]: 61: (55) if r0 != 0x0 goto pc-37      ; R0_w=0
3月 26 10:53:23 dae[621]: 62: (bf) r1 = r10                     ; R1_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; offset += iph->ihl * 4; @ tproxy.c:510
3月 26 10:53:23 dae[621]: 63: (07) r1 += -40                    ; R1_w=fp-40
3月 26 10:53:23 dae[621]: 64: (71) r2 = *(u8 *)(r1 +0)          ; R1_w=fp-40 R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) fp-40=mmmmmmmm
3月 26 10:53:23 dae[621]: 65: (67) r2 <<= 2                     ; R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=1020,var_off=(0x0; 0x3fc))
3月 26 10:53:23 dae[621]: 66: (57) r2 &= 60                     ; R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=60,var_off=(0x0; 0x3c))
3月 26 10:53:23 dae[621]: 67: (0f) r9 += r2                     ; R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=60,var_off=(0x0; 0x3c)) R9_w=scalar(smin=umin=smin32=umin32=14,smax=umax=smax32=umax32=74,var_off=(0x2; 0x7c))
3月 26 10:53:23 dae[621]: ; *l4proto = iph->protocol; @ tproxy.c:513
3月 26 10:53:23 dae[621]: 68: (71) r8 = *(u8 *)(r1 +9)          ; R1_w=fp-40 R8_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) fp-32=mmmmmmmm
3月 26 10:53:23 dae[621]: ; switch (iph->protocol) { @ tproxy.c:514
3月 26 10:53:23 dae[621]: 69: (15) if r8 == 0x11 goto pc+56     ; R8_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff))
3月 26 10:53:23 dae[621]: 70: (55) if r8 != 0x6 goto pc-46      ; R8=6
3月 26 10:53:23 dae[621]: 71: (bf) r3 = r10                     ; R3_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; bpf_skb_load_bytes(skb, offset, tcph, sizeof(struct tcphdr)))) { @ tproxy.c:517
3月 26 10:53:23 dae[621]: 72: (07) r3 += -112                   ; R3_w=fp-112
3月 26 10:53:23 dae[621]: 73: (bf) r1 = r6                      ; R1_w=ctx() R6=ctx()
3月 26 10:53:23 dae[621]: 74: (bf) r2 = r9                      ; R2_w=scalar(id=3,smin=umin=smin32=umin32=14,smax=umax=smax32=umax32=74,var_off=(0x2; 0x7c)) R9=scalar(id=3,smin=umin=smin32=umin32=14,smax=umax=smax32=umax32=74,var_off=(0x2; 0x7c))
3月 26 10:53:23 dae[621]: 75: (b7) r4 = 20                      ; R4_w=20
3月 26 10:53:23 dae[621]: 76: (85) call bpf_skb_load_bytes#26   ; R0_w=scalar() fp-96=????mmmm fp-104=mmmmmmmm fp-112=mmmmmmmm
3月 26 10:53:23 dae[621]: ; if ((ret = @ tproxy.c:516
3月 26 10:53:23 dae[621]: 77: (67) r0 <<= 32                    ; R0_w=scalar(smax=0x7fffffff00000000,umax=0xffffffff00000000,smin32=0,smax32=umax32=0,var_off=(0x0; 0xffffffff00000000))
3月 26 10:53:23 dae[621]: 78: (77) r0 >>= 32                    ; R0=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
3月 26 10:53:23 dae[621]: 79: (15) if r0 == 0x0 goto pc+55 135: R0=0 R6=ctx() R7=0 R8=6 R9=scalar(id=3,smin=umin=smin32=umin32=14,smax=umax=smax32=umax32=74,var_off=(0x2; 0x7c)) R10=fp0 fp-8=??mmmmmm fp-16=mmmmmmmm fp-24=????mmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=0 fp-56=0 fp-64=0 fp-72=0 fp-80=0 fp-88=0 fp-96=????mmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=0 fp-128=mmmm???? fp-176=????mmmm
3月 26 10:53:23 dae[621]: ; if ((ret = @ tproxy.c:523
3月 26 10:53:23 dae[621]: 135: (b7) r1 = 0                      ; R1_w=0
3月 26 10:53:23 dae[621]: 136: (05) goto pc+67
3月 26 10:53:23 dae[621]: ; if (ret) { @ tproxy.c:869
3月 26 10:53:23 dae[621]: 204: (67) r1 <<= 32                   ; R1_w=0
3月 26 10:53:23 dae[621]: 205: (77) r1 >>= 32                   ; R1_w=0
3月 26 10:53:23 dae[621]: 206: (55) if r1 != 0x0 goto pc-182    ; R1_w=0
3月 26 10:53:23 dae[621]: 207: (15) if r8 == 0x3a goto pc-183   ; R8=6
3月 26 10:53:23 dae[621]: 208: (b7) r1 = 0                      ; R1_w=0
3月 26 10:53:23 dae[621]: ; __builtin_memset(tuples, 0, sizeof(*tuples)); @ tproxy.c:354
3月 26 10:53:23 dae[621]: 209: (7b) *(u64 *)(r10 -144) = r1     ; R1_w=0 R10=fp0 fp-144_w=0
3月 26 10:53:23 dae[621]: 210: (7b) *(u64 *)(r10 -136) = r1     ; R1_w=0 R10=fp0 fp-136_w=0
3月 26 10:53:23 dae[621]: 211: (7b) *(u64 *)(r10 -152) = r1     ; R1_w=0 R10=fp0 fp-152_w=0
3月 26 10:53:23 dae[621]: 212: (7b) *(u64 *)(r10 -160) = r1     ; R1_w=0 R10=fp0 fp-160_w=0
3月 26 10:53:23 dae[621]: 213: (7b) *(u64 *)(r10 -168) = r1     ; R1_w=0 R10=fp0 fp-168_w=0
3月 26 10:53:23 dae[621]: 214: (7b) *(u64 *)(r10 -176) = r1     ; R1_w=0 R10=fp0 fp-176_w=0
3月 26 10:53:23 dae[621]: ; tuples->five.l4proto = l4proto; @ tproxy.c:355
3月 26 10:53:23 dae[621]: 215: (73) *(u8 *)(r10 -140) = r8      ; R8=6 R10=fp0 fp-144_w=mmmmmmmm
3月 26 10:53:23 dae[621]: ; if (skb->protocol == bpf_htons(ETH_P_IP)) { @ tproxy.c:357
3月 26 10:53:23 dae[621]: 216: (61) r1 = *(u32 *)(r6 +16)       ; R1_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R6=ctx()
3月 26 10:53:23 dae[621]: 217: (55) if r1 != 0x8 goto pc+13     ; R1_w=8
3月 26 10:53:23 dae[621]: 218: (18) r1 = 0xffff0000             ; R1_w=0xffff0000
3月 26 10:53:23 dae[621]: ; tuples->five.sip.u6_addr32[2] = bpf_htonl(0x0000ffff); @ tproxy.c:358
3月 26 10:53:23 dae[621]: 220: (63) *(u32 *)(r10 -168) = r1     ; R1_w=0xffff0000 R10=fp0 fp-168_w=mmmm0xffff0000
3月 26 10:53:23 dae[621]: ; tuples->five.dip.u6_addr32[2] = bpf_htonl(0x0000ffff); @ tproxy.c:361
3月 26 10:53:23 dae[621]: 221: (63) *(u32 *)(r10 -152) = r1     ; R1_w=0xffff0000 R10=fp0 fp-152_w=mmmm0xffff0000
3月 26 10:53:23 dae[621]: 222: (bf) r1 = r10                    ; R1_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; tuples->five.sip.u6_addr32[2] = bpf_htonl(0x0000ffff); @ tproxy.c:358
3月 26 10:53:23 dae[621]: 223: (07) r1 += -40                   ; R1_w=fp-40
3月 26 10:53:23 dae[621]: ; tuples->five.sip.u6_addr32[3] = iph->saddr; @ tproxy.c:359
3月 26 10:53:23 dae[621]: 224: (61) r2 = *(u32 *)(r1 +12)       ; R1_w=fp-40 R2_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) fp-32=mmmmmmmm
3月 26 10:53:23 dae[621]: 225: (63) *(u32 *)(r10 -164) = r2     ; R2_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R10=fp0 fp-168_w=mmmmmmmm
3月 26 10:53:23 dae[621]: ; tuples->five.dip.u6_addr32[3] = iph->daddr; @ tproxy.c:362
3月 26 10:53:23 dae[621]: 226: (61) r2 = *(u32 *)(r1 +16)       ; R1_w=fp-40 R2_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) fp-24=????mmmm
3月 26 10:53:23 dae[621]: 227: (63) *(u32 *)(r10 -148) = r2     ; R2_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R10=fp0 fp-152_w=mmmmmmmm
3月 26 10:53:23 dae[621]: ; return (iph->tos & 0xfc) >> 2; @ tproxy.c:345
3月 26 10:53:23 dae[621]: 228: (71) r2 = *(u8 *)(r1 +1)         ; R1_w=fp-40 R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) fp-40=mmmmmmmm
3月 26 10:53:23 dae[621]: 229: (77) r2 >>= 2                    ; R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=63,var_off=(0x0; 0x3f))
3月 26 10:53:23 dae[621]: 230: (05) goto pc+34
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 265: (73) *(u8 *)(r10 -136) = r2      ; R2=scalar(id=4,smin=smin32=0,smax=umax=smax32=umax32=63,var_off=(0x0; 0x3f)) R10=fp0 fp-136=mmmmmmmscalar(id=4,smin=smin32=0,smax=umax=smax32=umax32=63,var_off=(0x0; 0x3f))
3月 26 10:53:23 dae[621]: ; if (l4proto == IPPROTO_TCP) { @ tproxy.c:372
3月 26 10:53:23 dae[621]: 266: (55) if r8 != 0x6 goto pc+7      ; R8=6
3月 26 10:53:23 dae[621]: 267: (b7) r3 = 2                      ; R3_w=2
3月 26 10:53:23 dae[621]: 268: (bf) r1 = r10                    ; R1_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; } else { @ tproxy.c:375
3月 26 10:53:23 dae[621]: 269: (07) r1 += -112                  ; R1_w=fp-112
3月 26 10:53:23 dae[621]: 270: (bf) r2 = r1                     ; R1_w=fp-112 R2_w=fp-112
3月 26 10:53:23 dae[621]: 271: (0f) r2 += r3                    ; R2_w=fp-110 R3_w=2
3月 26 10:53:23 dae[621]: 272: (b7) r3 = 0                      ; R3_w=0
3月 26 10:53:23 dae[621]: 273: (05) goto pc+6
3月 26 10:53:23 dae[621]: 280: (0f) r1 += r3                    ; R1_w=fp-112 R3=0
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 281: (69) r1 = *(u16 *)(r1 +0)        ; R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) fp-112=mmmmmmmm
3月 26 10:53:23 dae[621]: 282: (6b) *(u16 *)(r10 -144) = r1     ; R1_w=scalar(id=5,smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) R10=fp0 fp-144=mmmmmmscalar(id=5,smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff))
3月 26 10:53:23 dae[621]: 283: (69) r1 = *(u16 *)(r2 +0)        ; R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) R2=fp-110 fp-112=mmmmmmmm
3月 26 10:53:23 dae[621]: 284: (6b) *(u16 *)(r10 -142) = r1     ; R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) R10=fp0 fp-144=mmmmmmmm
3月 26 10:53:23 dae[621]: 285: (b7) r1 = 0                      ; R1_w=0
3月 26 10:53:23 dae[621]: ; struct bpf_sock_tuple tuple = {0}; @ tproxy.c:893
3月 26 10:53:23 dae[621]: 286: (63) *(u32 *)(r10 -184) = r1     ; R1_w=0 R10=fp0 fp-184=????0
3月 26 10:53:23 dae[621]: 287: (7b) *(u64 *)(r10 -192) = r1     ; R1_w=0 R10=fp0 fp-192_w=0
3月 26 10:53:23 dae[621]: 288: (7b) *(u64 *)(r10 -200) = r1     ; R1_w=0 R10=fp0 fp-200_w=0
3月 26 10:53:23 dae[621]: 289: (7b) *(u64 *)(r10 -208) = r1     ; R1_w=0 R10=fp0 fp-208_w=0
3月 26 10:53:23 dae[621]: 290: (7b) *(u64 *)(r10 -216) = r1     ; R1_w=0 R10=fp0 fp-216_w=0
3月 26 10:53:23 dae[621]: ; if (skb->protocol == bpf_htons(ETH_P_IP)) { @ tproxy.c:900
3月 26 10:53:23 dae[621]: 291: (61) r1 = *(u32 *)(r6 +16)       ; R1_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R6=ctx()
3月 26 10:53:23 dae[621]: 292: (7b) *(u64 *)(r10 -320) = r8     ; R8=6 R10=fp0 fp-320_w=6
3月 26 10:53:23 dae[621]: 293: (55) if r1 != 0x8 goto pc+24     ; R1_w=8
3月 26 10:53:23 dae[621]: 294: (bf) r1 = r10                    ; R1_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; tuple.ipv4.daddr = tuples.five.dip.u6_addr32[3]; @ tproxy.c:901
3月 26 10:53:23 dae[621]: 295: (07) r1 += -216                  ; R1_w=fp-216
3月 26 10:53:23 dae[621]: 296: (61) r2 = *(u32 *)(r10 -148)     ; R2_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R10=fp0 fp-152=mmmmmmmm
3月 26 10:53:23 dae[621]: 297: (63) *(u32 *)(r1 +4) = r2        ; R1_w=fp-216 R2_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) fp-216_w=mmmmmmmm
3月 26 10:53:23 dae[621]: ; tuple.ipv4.saddr = tuples.five.sip.u6_addr32[3]; @ tproxy.c:902
3月 26 10:53:23 dae[621]: 298: (61) r2 = *(u32 *)(r10 -164)     ; R2_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R10=fp0 fp-168=mmmmmmmm
3月 26 10:53:23 dae[621]: 299: (63) *(u32 *)(r1 +0) = r2        ; R1_w=fp-216 R2_w=scalar(id=6,smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) fp-216_w=mmmmscalar(id=6,smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
3月 26 10:53:23 dae[621]: ; tuple.ipv4.dport = tuples.five.dport; @ tproxy.c:903
3月 26 10:53:23 dae[621]: 300: (69) r2 = *(u16 *)(r10 -142)     ; R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) R10=fp0 fp-144=mmmmmmmm
3月 26 10:53:23 dae[621]: 301: (6b) *(u16 *)(r1 +10) = r2       ; R1_w=fp-216 R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) fp-208_w=mmmmmmmm
3月 26 10:53:23 dae[621]: ; tuple.ipv4.sport = tuples.five.sport; @ tproxy.c:904
3月 26 10:53:23 dae[621]: 302: (69) r2 = *(u16 *)(r10 -144)     ; R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) R10=fp0 fp-144=mmmmmmmm
3月 26 10:53:23 dae[621]: 303: (6b) *(u16 *)(r1 +8) = r2        ; R1_w=fp-216 R2_w=scalar(id=7,smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) fp-208_w=mmmmmmscalar(id=7,smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff))
3月 26 10:53:23 dae[621]: 304: (b7) r3 = 12                     ; R3_w=12
3月 26 10:53:23 dae[621]: ; if (l4proto == IPPROTO_TCP) { @ tproxy.c:914
3月 26 10:53:23 dae[621]: 305: (55) if r8 != 0x6 goto pc+1      ; R8=6
3月 26 10:53:23 dae[621]: 306: (05) goto pc+43
3月 26 10:53:23 dae[621]: 350: (bf) r1 = r10                    ; R1_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; if (tcph.syn && !tcph.ack) { @ tproxy.c:916
3月 26 10:53:23 dae[621]: 351: (07) r1 += -112                  ; R1_w=fp-112
3月 26 10:53:23 dae[621]: 352: (69) r1 = *(u16 *)(r1 +12)       ; R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) fp-104=mmmmmmmm
3月 26 10:53:23 dae[621]: 353: (57) r1 &= 512                   ; R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=512,var_off=(0x0; 0x200))
3月 26 10:53:23 dae[621]: 354: (15) if r1 == 0x0 goto pc+5      ; R1_w=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=512,var_off=(0x0; 0x200))
3月 26 10:53:23 dae[621]: 355: (bf) r1 = r10                    ; R1_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: 356: (07) r1 += -112                  ; R1_w=fp-112
3月 26 10:53:23 dae[621]: 357: (69) r1 = *(u16 *)(r1 +12)       ; R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) fp-104=mmmmmmmm
3月 26 10:53:23 dae[621]: 358: (57) r1 &= 4096                  ; R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=4096,var_off=(0x0; 0x1000))
3月 26 10:53:23 dae[621]: 359: (15) if r1 == 0x0 goto pc+13     ; R1_w=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=4096,var_off=(0x0; 0x1000))
3月 26 10:53:23 dae[621]: 360: (bf) r2 = r10                    ; R2_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; sk = bpf_skc_lookup_tcp(skb, &tuple, tuple_size, BPF_F_CURRENT_NETNS, 0); @ tproxy.c:920
3月 26 10:53:23 dae[621]: 361: (07) r2 += -216                  ; R2_w=fp-216
3月 26 10:53:23 dae[621]: 362: (bf) r1 = r6                     ; R1_w=ctx() R6=ctx()
3月 26 10:53:23 dae[621]: 363: (b7) r4 = -1                     ; R4_w=-1
3月 26 10:53:23 dae[621]: 364: (b7) r5 = 0                      ; R5_w=0
3月 26 10:53:23 dae[621]: 365: (85) call bpf_skc_lookup_tcp#99          ; R0_w=sock_common_or_null(id=9,ref_obj_id=9) refs=9
3月 26 10:53:23 dae[621]: 366: (bf) r8 = r0                     ; R0_w=sock_common_or_null(id=9,ref_obj_id=9) R8_w=sock_common_or_null(id=9,ref_obj_id=9) refs=9
3月 26 10:53:23 dae[621]: ; if (sk) { @ tproxy.c:921
3月 26 10:53:23 dae[621]: 367: (15) if r8 == 0x0 goto pc+5      ; R8_w=sock_common(ref_obj_id=9) refs=9
3月 26 10:53:23 dae[621]: 368: (b7) r9 = 1                      ; R9_w=1 refs=9
3月 26 10:53:23 dae[621]: ; if (sk->state != BPF_TCP_LISTEN) { @ tproxy.c:922
3月 26 10:53:23 dae[621]: 369: (61) r1 = *(u32 *)(r8 +72)       ; R1=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R8=sock_common(ref_obj_id=9) refs=9
3月 26 10:53:23 dae[621]: 370: (55) if r1 != 0xa goto pc+136    ; R1=10 refs=9
3月 26 10:53:23 dae[621]: ; bpf_sk_release(sk); @ tproxy.c:926
3月 26 10:53:23 dae[621]: 371: (bf) r1 = r8                     ; R1_w=sock_common(ref_obj_id=9) R8=sock_common(ref_obj_id=9) refs=9
3月 26 10:53:23 systemd[1]: dae.service: Main process exited, code=exited, status=1/FAILURE
3月 26 10:53:23 dae[621]: 372: (85) call bpf_sk_release#86      ; R0_w=scalar()
3月 26 10:53:23 dae[621]: 373: (bf) r1 = r10                    ; R1_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 374: (07) r1 += -112                  ; R1_w=fp-112
3月 26 10:53:23 dae[621]: ; __builtin_memset(flag, 0, sizeof(flag)); @ tproxy.c:932
3月 26 10:53:23 dae[621]: 375: (63) *(u32 *)(r10 -220) = r7     ; R7=0 R10=fp0 fp-224=0000????
3月 26 10:53:23 dae[621]: 376: (63) *(u32 *)(r10 -224) = r7     ; R7=0 R10=fp0 fp-224=00000
3月 26 10:53:23 dae[621]: 377: (63) *(u32 *)(r10 -228) = r7     ; R7=0 R10=fp0 fp-232=0000????
3月 26 10:53:23 dae[621]: 378: (63) *(u32 *)(r10 -232) = r7     ; R7=0 R10=fp0 fp-232=00000
3月 26 10:53:23 dae[621]: 379: (63) *(u32 *)(r10 -236) = r7     ; R7=0 R10=fp0 fp-240=0000????
3月 26 10:53:23 dae[621]: 380: (63) *(u32 *)(r10 -240) = r7     ; R7=0 R10=fp0 fp-240=00000
3月 26 10:53:23 dae[621]: ; if (!(tcph.syn && !tcph.ack)) { @ tproxy.c:934
3月 26 10:53:23 dae[621]: 381: (69) r1 = *(u16 *)(r1 +12)       ; R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) fp-104=mmmmmmmm
3月 26 10:53:23 dae[621]: 382: (57) r1 &= 512                   ; R1=scalar(smin=smin32=0,smax=umax=smax32=umax32=512,var_off=(0x0; 0x200))
3月 26 10:53:23 dae[621]: 383: (15) if r1 == 0x0 goto pc-359    ; R1=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=512,var_off=(0x0; 0x200))
3月 26 10:53:23 dae[621]: 384: (bf) r2 = r10                    ; R2_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: 385: (07) r2 += -112                  ; R2_w=fp-112
3月 26 10:53:23 dae[621]: 386: (b7) r1 = 1                      ; R1_w=1
3月 26 10:53:23 dae[621]: 387: (69) r3 = *(u16 *)(r2 +12)       ; R2_w=fp-112 R3_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) fp-104=mmmmmmmm
3月 26 10:53:23 dae[621]: 388: (57) r3 &= 4096                  ; R3_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=4096,var_off=(0x0; 0x1000))
3月 26 10:53:23 dae[621]: 389: (55) if r3 != 0x0 goto pc-365    ; R3_w=0
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 390: (63) *(u32 *)(r10 -248) = r1     ; R1_w=1 R10=fp0 fp-248=????1
3月 26 10:53:23 dae[621]: 391: (bf) r1 = r10                    ; R1_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: 392: (07) r1 += -16                   ; R1_w=fp-16
3月 26 10:53:23 dae[621]: ; bpf_htonl((ethh.h_source[0] << 8) + (ethh.h_source[1])), @ tproxy.c:954
3月 26 10:53:23 dae[621]: 393: (71) r4 = *(u8 *)(r1 +7)         ; R1_w=fp-16 R4_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) fp-16=mmmmmmmm
3月 26 10:53:23 systemd[1]: dae.service: Failed with result 'exit-code'.
3月 26 10:53:23 dae[621]: 394: (71) r3 = *(u8 *)(r1 +6)         ; R1_w=fp-16 R3_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) fp-16=mmmmmmmm
3月 26 10:53:23 dae[621]: 395: (67) r3 <<= 8                    ; R3_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xff00,var_off=(0x0; 0xff00))
3月 26 10:53:23 dae[621]: 396: (4f) r3 |= r4                    ; R3_w=scalar() R4_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff))
3月 26 10:53:23 dae[621]: ; if (skb->protocol == bpf_htons(ETH_P_IP)) { @ tproxy.c:945
3月 26 10:53:23 dae[621]: 397: (61) r4 = *(u32 *)(r6 +16)       ; R4_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R6=ctx()
3月 26 10:53:23 dae[621]: 398: (b7) r7 = 2                      ; R7_w=2
3月 26 10:53:23 dae[621]: 399: (b7) r5 = 1                      ; R5=1
3月 26 10:53:23 dae[621]: 400: (15) if r4 == 0x8 goto pc+1      ; R4=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
3月 26 10:53:23 dae[621]: 401: (b7) r5 = 2                      ; R5_w=2
3月 26 10:53:23 dae[621]: ; bpf_htonl((ethh.h_source[2] << 24) + (ethh.h_source[3] << 16) + @ tproxy.c:955
3月 26 10:53:23 dae[621]: 402: (71) r0 = *(u8 *)(r1 +8)         ; R0_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) R1=fp-16 fp-8=??mmmmmm
3月 26 10:53:23 dae[621]: 403: (67) r0 <<= 24                   ; R0_w=scalar(smin=0,smax=umax=umax32=0xff000000,smax32=0x7f000000,var_off=(0x0; 0xff000000))
3月 26 10:53:23 dae[621]: 404: (71) r4 = *(u8 *)(r1 +9)         ; R1=fp-16 R4_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) fp-8=??mmmmmm
3月 26 10:53:23 dae[621]: 405: (67) r4 <<= 16                   ; R4_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xff0000,var_off=(0x0; 0xff0000))
3月 26 10:53:23 dae[621]: 406: (4f) r4 |= r0                    ; R0_w=scalar(smin=0,smax=umax=umax32=0xff000000,smax32=0x7f000000,var_off=(0x0; 0xff000000)) R4_w=scalar()
3月 26 10:53:23 dae[621]: ; flag[6] = tuples.dscp; @ tproxy.c:950
3月 26 10:53:23 dae[621]: 407: (71) r0 = *(u8 *)(r10 -136)      ; R0_w=scalar(id=4,smin=smin32=0,smax=umax=smax32=umax32=63,var_off=(0x0; 0x3f)) R10=fp0 fp-136=mmmmmmmscalar(id=4,smin=smin32=0,smax=umax=smax32=umax32=63,var_off=(0x0; 0x3f))
3月 26 10:53:23 dae[621]: 408: (63) *(u32 *)(r10 -224) = r0     ; R0_w=scalar(id=4,smin=smin32=0,smax=umax=smax32=umax32=63,var_off=(0x0; 0x3f)) R10=fp0 fp-224=0000scalar(id=4,smin=smin32=0,smax=umax=smax32=umax32=63,var_off=(0x0; 0x3f))
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 systemd[1]: Failed to start dae Service.
3月 26 10:53:23 dae[621]: 409: (63) *(u32 *)(r10 -244) = r5     ; R5_w=2 R10=fp0 fp-248=mmmmmmmm
3月 26 10:53:23 dae[621]: ; bpf_htonl((ethh.h_source[0] << 8) + (ethh.h_source[1])), @ tproxy.c:954
3月 26 10:53:23 dae[621]: 410: (dc) r3 = be32 r3                ; R3_w=scalar()
3月 26 10:53:23 dae[621]: ; __be32 mac[4] = { @ tproxy.c:951
3月 26 10:53:23 dae[621]: 411: (63) *(u32 *)(r10 -256) = r3     ; R3_w=scalar() R10=fp0 fp-256=????scalar()
3月 26 10:53:23 dae[621]: ; bpf_htonl((ethh.h_source[2] << 24) + (ethh.h_source[3] << 16) + @ tproxy.c:955
3月 26 10:53:23 dae[621]: 412: (71) r3 = *(u8 *)(r1 +10)        ; R1=fp-16 R3_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) fp-8=??mmmmmm
3月 26 10:53:23 dae[621]: 413: (67) r3 <<= 8                    ; R3_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xff00,var_off=(0x0; 0xff00))
3月 26 10:53:23 dae[621]: 414: (4f) r4 |= r3                    ; R3_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xff00,var_off=(0x0; 0xff00)) R4_w=scalar()
3月 26 10:53:23 dae[621]: 415: (71) r1 = *(u8 *)(r1 +11)        ; R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) fp-8=??mmmmmm
3月 26 10:53:23 dae[621]: 416: (4f) r4 |= r1                    ; R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) R4_w=scalar()
3月 26 10:53:23 dae[621]: 417: (b7) r8 = 0                      ; R8_w=0
3月 26 10:53:23 dae[621]: ; __be32 mac[4] = { @ tproxy.c:951
3月 26 10:53:23 dae[621]: 418: (63) *(u32 *)(r10 -260) = r8     ; R8_w=0 R10=fp0 fp-264=0000????
3月 26 10:53:23 dae[621]: 419: (63) *(u32 *)(r10 -264) = r8     ; R8_w=0 R10=fp0 fp-264=00000
3月 26 10:53:23 dae[621]: ; bpf_htonl((ethh.h_source[2] << 24) + (ethh.h_source[3] << 16) + @ tproxy.c:955
3月 26 10:53:23 dae[621]: 420: (dc) r4 = be32 r4                ; R4_w=scalar()
3月 26 10:53:23 dae[621]: ; __be32 mac[4] = { @ tproxy.c:951
3月 26 10:53:23 dae[621]: 421: (63) *(u32 *)(r10 -252) = r4     ; R4_w=scalar() R10=fp0 fp-256=mmmmmmmm
3月 26 10:53:23 dae[621]: 422: (bf) r1 = r10                    ; R1_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; bpf_htonl((ethh.h_source[2] << 24) + (ethh.h_source[3] << 16) + @ tproxy.c:955
3月 26 10:53:23 dae[621]: 423: (07) r1 += -248                  ; R1_w=fp-248
3月 26 10:53:23 dae[621]: 424: (bf) r3 = r10                    ; R3_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: 425: (07) r3 += -176                  ; R3_w=fp-176
3月 26 10:53:23 dae[621]: ; tuples.five.dip.u6_addr32, mac)) < 0) { @ tproxy.c:960
3月 26 10:53:23 dae[621]: 426: (bf) r4 = r10                    ; R4_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: 427: (07) r4 += -160                  ; R4_w=fp-160
3月 26 10:53:23 dae[621]: 428: (bf) r5 = r10                    ; R5_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; bpf_htonl((ethh.h_source[2] << 24) + (ethh.h_source[3] << 16) + @ tproxy.c:955
3月 26 10:53:23 dae[621]: 429: (07) r5 += -264                  ; R5_w=fp-264
3月 26 10:53:23 dae[621]: ; if ((s64_ret = route(flag, l4hdr, tuples.five.sip.u6_addr32, @ tproxy.c:959
3月 26 10:53:23 dae[621]: 430: (85) call pc+252
3月 26 10:53:23 dae[621]: caller:
3月 26 10:53:23 dae[621]:  R6=ctx() R7=2 R8_w=0 R9=1 R10=fp0 fp-8=??mmmmmm fp-16=mmmmmmmm fp-24=????mmmm fp-32=mmmmmmmm fp-40=mmmmmmmm fp-48=0 fp-56=0 fp-64=0 fp-72=0 fp-80=0 fp-88=0 fp-96=????mmmm fp-104=mmmmmmmm fp-112=mmmmmmmm fp-120=0 fp-128=mmmm???? fp-136=mmmmmmmscalar(id=4,smin=smin32=0,smax=umax=smax32=umax32=63,var_off=(0x0; 0x3f)) fp-144=mmmmmmmm fp-152=mmmmmmmm fp-160=0 fp-168=mmmmmmmm fp-176=0 fp-184=????0 fp-192=0 fp-200=0 fp-208=mmmmmmmm fp-216=mmmmmmmm fp-224=0000scalar(id=4,smin=smin32=0,smax=umax=smax32=umax32=63,var_off=(0x0; 0x3f)) fp-232=00000 fp-240=00000 fp-248=mmmmmmmm fp-256=mmmmmmmm fp-264=00000 fp-320=6
3月 26 10:53:23 dae[621]: callee:
3月 26 10:53:23 dae[621]:  frame1: R1_w=fp[0]-248 R2=fp[0]-112 R3_w=fp[0]-176 R4_w=fp[0]-160 R5_w=fp[0]-264 R10=fp0
3月 26 10:53:23 dae[621]: 683: frame1: R1=fp[0]-248 R2=fp[0]-112 R3=fp[0]-176 R4=fp[0]-160 R5=fp[0]-264 R10=fp0
3月 26 10:53:23 dae[621]: ; route(const __u32 flag[8], const void *l4hdr, const __be32 saddr[4], @ tproxy.c:558
3月 26 10:53:23 dae[621]: 683: (7b) *(u64 *)(r10 -56) = r5      ; frame1: R5=fp[0]-264 R10=fp0 fp-56_w=fp[0]-264
3月 26 10:53:23 dae[621]: 684: (7b) *(u64 *)(r10 -48) = r4      ; frame1: R4=fp[0]-160 R10=fp0 fp-48_w=fp[0]-160
3月 26 10:53:23 dae[621]: 685: (bf) r7 = r3                     ; frame1: R3=fp[0]-176 R7_w=fp[0]-176
3月 26 10:53:23 dae[621]: 686: (bf) r6 = r2                     ; frame1: R2=fp[0]-112 R6_w=fp[0]-112
3月 26 10:53:23 dae[621]: 687: (bf) r8 = r1                     ; frame1: R1=fp[0]-248 R8_w=fp[0]-248
3月 26 10:53:23 dae[621]: 688: (b7) r1 = 5                      ; frame1: R1_w=5
3月 26 10:53:23 dae[621]: ; __u32 key = MatchType_L4Proto; @ tproxy.c:568
3月 26 10:53:23 dae[621]: 689: (63) *(u32 *)(r10 -28) = r1      ; frame1: R1_w=5 R10=fp0 fp-32=mmmm????
3月 26 10:53:23 dae[621]: 690: (bf) r2 = r10                    ; frame1: R2_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 691: (07) r2 += -28                   ; frame1: R2_w=fp-28
3月 26 10:53:23 dae[621]: ; if (unlikely((ret = bpf_map_update_elem(&l4proto_ipversion_map, &key, @ tproxy.c:573
3月 26 10:53:23 dae[621]: 692: (18) r1 = 0xffff8ec8103e8c00     ; frame1: R1_w=map_ptr(map=l4proto_ipversi,ks=4,vs=4)
3月 26 10:53:23 dae[621]: 694: (bf) r3 = r8                     ; frame1: R3_w=fp[0]-248 R8_w=fp[0]-248
3月 26 10:53:23 dae[621]: 695: (b7) r4 = 0                      ; frame1: R4_w=0
3月 26 10:53:23 dae[621]: 696: (85) call bpf_map_update_elem#2          ; frame1: R0_w=scalar()
3月 26 10:53:23 dae[621]: 697: (bf) r9 = r0                     ; frame1: R0_w=scalar(id=10) R9_w=scalar(id=10)
3月 26 10:53:23 dae[621]: 698: (67) r9 <<= 32                   ; frame1: R9_w=scalar(smax=0x7fffffff00000000,umax=0xffffffff00000000,smin32=0,smax32=umax32=0,var_off=(0x0; 0xffffffff00000000))
3月 26 10:53:23 dae[621]: 699: (15) if r9 == 0x0 goto pc+2 702: frame1: R0=scalar(id=10) R6=fp[0]-112 R7=fp[0]-176 R8=fp[0]-248 R9=0 R10=fp0 fp-32=mmmm???? fp-48=fp[0]-160 fp-56=fp[0]-264
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 702: (b7) r1 = 6                      ; frame1: R1_w=6
3月 26 10:53:23 dae[621]: ; key = MatchType_IpVersion; @ tproxy.c:577
3月 26 10:53:23 dae[621]: 703: (63) *(u32 *)(r10 -28) = r1      ; frame1: R1_w=6 R10=fp0 fp-32=mmmm????
3月 26 10:53:23 dae[621]: ; if (unlikely((ret = bpf_map_update_elem(&l4proto_ipversion_map, &key, @ tproxy.c:578
3月 26 10:53:23 dae[621]: 704: (bf) r3 = r8                     ; frame1: R3_w=fp[0]-248 R8=fp[0]-248
3月 26 10:53:23 dae[621]: 705: (07) r3 += 4                     ; frame1: R3_w=fp[0]-244
3月 26 10:53:23 dae[621]: 706: (bf) r2 = r10                    ; frame1: R2_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 707: (07) r2 += -28                   ; frame1: R2_w=fp-28
3月 26 10:53:23 dae[621]: ; if (unlikely((ret = bpf_map_update_elem(&l4proto_ipversion_map, &key, @ tproxy.c:578
3月 26 10:53:23 dae[621]: 708: (18) r1 = 0xffff8ec8103e8c00     ; frame1: R1_w=map_ptr(map=l4proto_ipversi,ks=4,vs=4)
3月 26 10:53:23 dae[621]: 710: (b7) r4 = 0                      ; frame1: R4_w=0
3月 26 10:53:23 dae[621]: 711: (85) call bpf_map_update_elem#2          ; frame1: R0_w=scalar()
3月 26 10:53:23 dae[621]: 712: (bf) r9 = r0                     ; frame1: R0_w=scalar(id=22) R9_w=scalar(id=22)
3月 26 10:53:23 dae[621]: 713: (67) r9 <<= 32                   ; frame1: R9_w=scalar(smax=0x7fffffff00000000,umax=0xffffffff00000000,smin32=0,smax32=umax32=0,var_off=(0x0; 0xffffffff00000000))
3月 26 10:53:23 dae[621]: 714: (15) if r9 == 0x0 goto pc+1 716: frame1: R0=scalar(id=22) R6=fp[0]-112 R7=fp[0]-176 R8=fp[0]-248 R9=0 R10=fp0 fp-32=mmmm???? fp-48=fp[0]-160 fp-56=fp[0]-264
3月 26 10:53:23 dae[621]: ; if (_l4proto_type == L4ProtoType_TCP) { @ tproxy.c:584
3月 26 10:53:23 dae[621]: 716: (61) r1 = *(u32 *)(r8 +0)        ; frame1: R1_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R8=fp[0]-248
3月 26 10:53:23 dae[621]: 717: (7b) *(u64 *)(r10 -64) = r8      ; frame1: R8=fp[0]-248 R10=fp0 fp-64_w=fp[0]-248
3月 26 10:53:23 dae[621]: 718: (55) if r1 != 0x1 goto pc+5      ; frame1: R1_w=1
3月 26 10:53:23 dae[621]: ; h_dport = bpf_ntohs(((struct tcphdr *)l4hdr)->dest); @ tproxy.c:585
3月 26 10:53:23 dae[621]: 719: (69) r1 = *(u16 *)(r6 +2)        ; frame1: R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) R6=fp[0]-112
3月 26 10:53:23 dae[621]: 720: (dc) r1 = be16 r1                ; frame1: R1_w=scalar()
3月 26 10:53:23 dae[621]: 721: (6b) *(u16 *)(r10 -30) = r1      ; frame1: R1_w=scalar() R10=fp0 fp-32=mmmmmm??
3月 26 10:53:23 dae[621]: 722: (b7) r1 = 0                      ; frame1: R1_w=0
3月 26 10:53:23 dae[621]: 723: (05) goto pc+4
3月 26 10:53:23 dae[621]: ; h_dport = bpf_ntohs(((struct udphdr *)l4hdr)->dest); @ tproxy.c:588
3月 26 10:53:23 dae[621]: 728: (0f) r6 += r1                    ; frame1: R1=0 R6_w=fp[0]-112
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 729: (69) r1 = *(u16 *)(r6 +0)        ; frame1: R1_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff)) R6_w=fp[0]-112
3月 26 10:53:23 dae[621]: 730: (b7) r2 = 4                      ; frame1: R2_w=4
3月 26 10:53:23 dae[621]: ; key = MatchType_SourcePort; @ tproxy.c:592
3月 26 10:53:23 dae[621]: 731: (63) *(u32 *)(r10 -28) = r2      ; frame1: R2_w=4 R10=fp0 fp-32=mmmmmm??
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 732: (dc) r1 = be16 r1                ; frame1: R1_w=scalar()
3月 26 10:53:23 dae[621]: 733: (6b) *(u16 *)(r10 -32) = r1      ; frame1: R1_w=scalar() R10=fp0 fp-32=mmmmmmscalar()
3月 26 10:53:23 dae[621]: 734: (bf) r2 = r10                    ; frame1: R2_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: 735: (07) r2 += -28                   ; frame1: R2_w=fp-28
3月 26 10:53:23 dae[621]: 736: (bf) r3 = r10                    ; frame1: R3_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: 737: (07) r3 += -32                   ; frame1: R3_w=fp-32
3月 26 10:53:23 dae[621]: ; if (unlikely( @ tproxy.c:593
3月 26 10:53:23 dae[621]: 738: (18) r1 = 0xffff8ec8103ec400     ; frame1: R1_w=map_ptr(map=h_port_map,ks=4,vs=2)
3月 26 10:53:23 dae[621]: 740: (b7) r4 = 0                      ; frame1: R4_w=0
3月 26 10:53:23 dae[621]: 741: (85) call bpf_map_update_elem#2          ; frame1: R0_w=scalar()
3月 26 10:53:23 dae[621]: 742: (bf) r9 = r0                     ; frame1: R0_w=scalar(id=23) R9_w=scalar(id=23)
3月 26 10:53:23 dae[621]: 743: (67) r9 <<= 32                   ; frame1: R9_w=scalar(smax=0x7fffffff00000000,umax=0xffffffff00000000,smin32=0,smax32=umax32=0,var_off=(0x0; 0xffffffff00000000))
3月 26 10:53:23 dae[621]: 744: (15) if r9 == 0x0 goto pc+1 746: frame1: R0=scalar(id=23) R6=fp[0]-112 R7=fp[0]-176 R8=fp[0]-248 R9=0 R10=fp0 fp-32=mmmmmmmm fp-48=fp[0]-160 fp-56=fp[0]-264 fp-64=fp[0]-248
3月 26 10:53:23 dae[621]: 746: (b7) r1 = 3                      ; frame1: R1_w=3
3月 26 10:53:23 dae[621]: ; key = MatchType_Port; @ tproxy.c:597
3月 26 10:53:23 dae[621]: 747: (63) *(u32 *)(r10 -28) = r1      ; frame1: R1_w=3 R10=fp0 fp-32=mmmmmmmm
3月 26 10:53:23 dae[621]: 748: (bf) r2 = r10                    ; frame1: R2_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ;  @ tproxy.c:0
3月 26 10:53:23 dae[621]: 749: (07) r2 += -28                   ; frame1: R2_w=fp-28
3月 26 10:53:23 dae[621]: 750: (bf) r3 = r10                    ; frame1: R3_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: 751: (07) r3 += -30                   ; frame1: R3_w=fp-30
3月 26 10:53:23 dae[621]: ; if (unlikely( @ tproxy.c:598
3月 26 10:53:23 dae[621]: 752: (18) r1 = 0xffff8ec8103ec400     ; frame1: R1_w=map_ptr(map=h_port_map,ks=4,vs=2)
3月 26 10:53:23 dae[621]: 754: (b7) r4 = 0                      ; frame1: R4_w=0
3月 26 10:53:23 dae[621]: 755: (85) call bpf_map_update_elem#2          ; frame1: R0_w=scalar()
3月 26 10:53:23 dae[621]: 756: (bf) r9 = r0                     ; frame1: R0_w=scalar(id=24) R9_w=scalar(id=24)
3月 26 10:53:23 dae[621]: 757: (67) r9 <<= 32                   ; frame1: R9_w=scalar(smax=0x7fffffff00000000,umax=0xffffffff00000000,smin32=0,smax32=umax32=0,var_off=(0x0; 0xffffffff00000000))
3月 26 10:53:23 dae[621]: 758: (15) if r9 == 0x0 goto pc+1 760: frame1: R0=scalar(id=24) R6=fp[0]-112 R7=fp[0]-176 R8=fp[0]-248 R9=0 R10=fp0 fp-32=mmmmmmmm fp-48=fp[0]-160 fp-56=fp[0]-264 fp-64=fp[0]-248
3月 26 10:53:23 dae[621]: 760: (bf) r3 = r10                    ; frame1: R3_w=fp0 R10=fp0
3月 26 10:53:23 dae[621]: ; lpm_key_instance.trie_key.prefixlen = IPV6_BYTE_LENGTH * 8; @ tproxy.c:603
3月 26 10:53:23 dae[621]: 761: (07) r3 += -24                   ; frame1: R3_w=fp-24
3月 26 10:53:23 dae[621]: 762: (b7) r1 = 128                    ; frame1: R1_w=128
3月 26 10:53:23 dae[621]: 763: (85) call unknown#195896080
3月 26 10:53:23 dae[621]: invalid func unknown#195896080
3月 26 10:53:23 dae[621]: processed 423 insns (limit 1000000) max_states_per_insn 1 total_states 29 peak_states 29 mark_read 11"
jschwinger233 commented 6 months ago

可能是 CO-RE 的问题，你能导出一份 vmlinux 吗？

bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

或者安装 pahole 运行 pahole -C bpf_lpm_trie_key 会更简单

sakarie9 commented 6 months ago

vmlinux.h.tar.gz

但是 pahole: type 'bpf_lpm_trie_key' not found

jschwinger233 commented 6 months ago

~~ 那就对了，~~ 没有 bpf_lpm_trie_key 这个类型，无法做 CO-RE，所以挂了。

jschwinger233 commented 6 months ago

看到你的 vmlinux 里定义的是 bpf_lpm_trie_key_u8 而不是 bpf_lpm_trie_key

sakarie9 commented 6 months ago

https://lore.kernel.org/bpf/202402221046.020C94D@keescook/T/

大概是这里改动的

jschwinger233 commented 6 months ago

在 vmlinux 里做兼容吧，大概是这样的：

if (LINUX_KERNEL_VERSION > KERNEL_VERSION(6, 9, 0)) {
    /* bpf_lpm_trie_key_u8 */
} else {
    /* bpf_lpm_trie_key */ 
}

我看看有没有更好的办法

jschwinger233 commented 6 months ago

bpf 和内核社区还完全不知道这个问题，恭喜成为全球第一个报告此 bug 的人类。

请试试 https://github.com/daeuniverse/dae/pull/483 这个 PR，可以直接从 PR build 里下载二进制。

sakarie9 commented 6 months ago

正常了！

dae version unstable-20240326.pr-483.r1.9aafbb

3月 26 18:19:42 systemd[1]: Starting dae Service...
3月 26 18:19:42 dae[17759]: level=info msg="Include config files: [/etc/dae/config.dae]"
3月 26 18:19:42 dae[17759]: level=info msg="Loading eBPF programs and maps into the kernel..."
3月 26 18:19:42 dae[17759]: level=info msg="The loading process takes about 120MB free memory, which will be released after loading. Insufficient memory will cause loading failure."
3月 26 18:19:43 dae[17759]: level=info msg="Loaded eBPF programs and maps"
3月 26 18:19:43 dae[17759]: level=info msg="Bind to WAN: enp34s0"
3月 26 18:19:43 dae[17759]: level=info msg="Group "clash" node list:"
3月 26 18:19:43 dae[17759]: level=info msg="        clash"
3月 26 18:19:43 dae[17759]: level=info msg="Routing match set len: 14/64"
3月 26 18:19:43 systemd[1]: Started dae Service.