search

kmesh-net / kmesh

High Performance ServiceMesh Data Plane Based on Programmable Kernel
https://kmesh.net
Apache License 2.0
464 stars 70 forks source link

Encounter crash using the latest version of Kmesh #1053

Closed delavet closed 19 hours ago

delavet commented 22 hours ago

What happened: The Kmesh daemon is crashing on started when I user the latest Kmesh Image.

image

The logs are as belows:

none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
time="2024-11-19T05:43:27Z" level=info msg="FLAG: --bpf-fs-path=\"/sys/fs/bpf\"" subsys=manager
time="2024-11-19T05:43:27Z" level=info msg="FLAG: --cgroup2-path=\"/mnt/kmesh_cgroup2\"" subsys=manager
time="2024-11-19T05:43:27Z" level=info msg="FLAG: --cni-etc-path=\"/etc/cni/net.d\"" subsys=manager
time="2024-11-19T05:43:27Z" level=info msg="FLAG: --conflist-name=\"\"" subsys=manager
time="2024-11-19T05:43:27Z" level=info msg="FLAG: --enable-accesslog=\"false\"" subsys=manager
time="2024-11-19T05:43:27Z" level=info msg="FLAG: --enable-bpf-log=\"true\"" subsys=manager
time="2024-11-19T05:43:27Z" level=info msg="FLAG: --enable-bypass=\"false\"" subsys=manager
time="2024-11-19T05:43:27Z" level=info msg="FLAG: --enable-mda=\"false\"" subsys=manager
time="2024-11-19T05:43:27Z" level=info msg="FLAG: --enable-secret-manager=\"false\"" subsys=manager
time="2024-11-19T05:43:27Z" level=info msg="FLAG: --help=\"false\"" subsys=manager
time="2024-11-19T05:43:27Z" level=info msg="FLAG: --mode=\"dual-engine\"" subsys=manager
time="2024-11-19T05:43:27Z" level=info msg="FLAG: --plugin-cni-chained=\"true\"" subsys=manager
time="2024-11-19T05:43:27Z" level=info msg="FLAG: --profiling=\"false\"" subsys=manager
time="2024-11-19T05:43:27Z" level=info msg="kmesh version map loadfailed: no such file or directory, start normally" subsys=bpf
time="2024-11-19T05:43:27Z" level=info msg="kmesh start with Normal" subsys=bpf
time="2024-11-19T05:43:27Z" level=info msg="cleaned kmesh_version map" subsys=bpf
time="2024-11-19T05:43:27Z" level=info msg="cleanup bpf map success" subsys=bpf
Error: bpf Load failed: load program: permission denied:
    ; int xdp_authz(struct xdp_md *ctx)
    0: (bf) r6 = r1
    1: (b7) r8 = 0
    ; int kmesh_config_key = 0;
    2: (63) *(u32 *)(r10 -72) = r8
    last_idx 2 first_idx 0
    regs=100 stack=0 before 1: (b7) r8 = 0
    3: (bf) r2 = r10
    ;
    4: (07) r2 += -72
    ; return bpf_map_lookup_elem(map, key);
    5: (18) r1 = 0xffff9bad87e03a00
    7: (85) call bpf_map_lookup_elem#1
    8: (b7) r7 = 0
    ; if (!value)
    9: (15) if r0 == 0x0 goto pc+4
     R0_w=map_value(id=0,off=0,ks=4,vs=40,imm=0) R6_w=ctx(id=0,off=0,imm=0) R7_w=inv0 R8_w=invP0 R10=fp0 fp-72=????mmmm
    ; return ((*value).authz_offload == 1);
    10: (61) r1 = *(u32 *)(r0 +36)
     R0_w=map_value(id=0,off=0,ks=4,vs=40,imm=0) R6_w=ctx(id=0,off=0,imm=0) R7_w=inv0 R8_w=invP0 R10=fp0 fp-72=????mmmm
    11: (b7) r7 = 1
    ; return ((*value).authz_offload == 1);
    12: (15) if r1 == 0x1 goto pc+1

    from 12 to 14: R0=map_value(id=0,off=0,ks=4,vs=40,imm=0) R1=inv1 R6=ctx(id=0,off=0,imm=0) R7=inv1 R8=invP0 R10=fp0 fp-72=????mmmm
    ; int zero = 0;
    14: (63) *(u32 *)(r10 -72) = r8
    15: (bf) r2 = r10
    ;
    16: (07) r2 += -72
    ; return bpf_map_lookup_elem(map, key);
    17: (18) r1 = 0xffff9bad87e03a00
    19: (85) call bpf_map_lookup_elem#1
    20: (b7) r1 = 2
    ; if (!value)
    21: (15) if r0 == 0x0 goto pc+1
     R0=map_value(id=0,off=0,ks=4,vs=40,imm=0) R1_w=inv2 R6=ctx(id=0,off=0,imm=0) R7=inv1 R8=invP0 R10=fp0 fp-72=????mmmm
    ; return value->bpf_log_level;
    22: (61) r1 = *(u32 *)(r0 +0)
     R0=map_value(id=0,off=0,ks=4,vs=40,imm=0) R1_w=inv2 R6=ctx(id=0,off=0,imm=0) R7=inv1 R8=invP0 R10=fp0 fp-72=????mmmm
    ;
    23: (67) r1 <<= 32
    24: (c7) r1 s>>= 32
    25: (b7) r2 = 3
    26: (6d) if r2 s> r1 goto pc+1
     R0=map_value(id=0,off=0,ks=4,vs=40,imm=0) R1_w=inv(id=0,umin_value=3,umax_value=2147483647,var_off=(0x0; 0x7fffffff)) R2_w=inv3 R6=ctx(id=0,off=0,imm=0) R7=inv1 R8=invP0 R10=fp0 fp-72=????mmmm
    27: (b7) r1 = 3
    ; if (!is_authz_offload_enabled()) {
    28: (57) r7 &= 1
    29: (55) if r7 != 0x0 goto pc+15
    last_idx 29 first_idx 28
    regs=80 stack=0 before 28: (57) r7 &= 1
     R0=map_value(id=0,off=0,ks=4,vs=40,imm=0) R1_w=inv3 R2_w=inv3 R6=ctx(id=0,off=0,imm=0) R7_r=invP1 R8=invP0 R10=fp0 fp-72=????mmmm
    parent didn't have regs=80 stack=0 marks
    last_idx 27 first_idx 20
    regs=80 stack=0 before 27: (b7) r1 = 3
    regs=80 stack=0 before 26: (6d) if r2 s> r1 goto pc+1
    regs=80 stack=0 before 25: (b7) r2 = 3
    regs=80 stack=0 before 24: (c7) r1 s>>= 32
    regs=80 stack=0 before 23: (67) r1 <<= 32
    regs=80 stack=0 before 22: (61) r1 = *(u32 *)(r0 +0)
    regs=80 stack=0 before 21: (15) if r0 == 0x0 goto pc+1
    regs=80 stack=0 before 20: (b7) r1 = 2
     R0_rw=map_value_or_null(id=4,off=0,ks=4,vs=40,imm=0) R6=ctx(id=0,off=0,imm=0) R7_r=invP1 R8=invP0 R10=fp0 fp-72=????mmmm
    parent didn't have regs=80 stack=0 marks
    last_idx 19 first_idx 12
    regs=80 stack=0 before 19: (85) call bpf_map_lookup_elem#1
    regs=80 stack=0 before 17: (18) r1 = 0xffff9bad87e03a00
    regs=80 stack=0 before 16: (07) r2 += -72
    regs=80 stack=0 before 15: (bf) r2 = r10
    regs=80 stack=0 before 14: (63) *(u32 *)(r10 -72) = r8
    regs=80 stack=0 before 12: (15) if r1 == 0x1 goto pc+1
     R0_w=map_value(id=0,off=0,ks=4,vs=40,imm=0) R1_rw=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6_rw=ctx(id=0,off=0,imm=0) R7_rw=invP1 R8_rw=invP0 R10=fp0 fp-72_r=????mmmm
    parent didn't have regs=80 stack=0 marks
    last_idx 11 first_idx 0
    regs=80 stack=0 before 11: (b7) r7 = 1
    ;
    45: (67) r1 <<= 32
    46: (c7) r1 s>>= 32
    47: (b7) r7 = 2
    ; BPF_LOG(INFO, AUTH, "authz is enabled, processing");
    48: (6d) if r7 s> r1 goto pc+4
    last_idx 48 first_idx 28
    regs=80 stack=0 before 47: (b7) r7 = 2
    last_idx 48 first_idx 28
    regs=2 stack=0 before 47: (b7) r7 = 2
    regs=2 stack=0 before 46: (c7) r1 s>>= 32
    regs=2 stack=0 before 45: (67) r1 <<= 32
    regs=2 stack=0 before 29: (55) if r7 != 0x0 goto pc+15
    regs=2 stack=0 before 28: (57) r7 &= 1
     R0=map_value(id=0,off=0,ks=4,vs=40,imm=0) R1_rw=invP3 R2_w=inv3 R6=ctx(id=0,off=0,imm=0) R7_r=invP1 R8=invP0 R10=fp0 fp-72=????mmmm
    parent didn't have regs=2 stack=0 marks
    last_idx 27 first_idx 20
    regs=2 stack=0 before 27: (b7) r1 = 3
    ; BPF_LOG(INFO, AUTH, "authz is enabled, processing");
    49: (18) r1 = 0xffff9bae7e1374db
    51: (b7) r2 = 42
    52: (85) call bpf_trace_printk#6
     R0=map_value(id=0,off=0,ks=4,vs=40,imm=0) R1_w=map_value(id=0,off=907,ks=4,vs=1768,imm=0) R2_w=inv42 R6=ctx(id=0,off=0,imm=0) R7_w=invP2 R8=invP0 R10=fp0 fp-72=????mmmm
    last_idx 52 first_idx 28
    regs=4 stack=0 before 51: (b7) r2 = 42
    53: (b7) r1 = 0
    ; struct bpf_sock_tuple tuple_key = {0};
    54: (63) *(u32 *)(r10 -40) = r1
    last_idx 54 first_idx 53
    regs=2 stack=0 before 53: (b7) r1 = 0
    55: (7b) *(u64 *)(r10 -48) = r1
    56: (7b) *(u64 *)(r10 -56) = r1
    57: (7b) *(u64 *)(r10 -64) = r1
    58: (7b) *(u64 *)(r10 -72) = r1
    ; void *begin = (void *)(long)(ctx->data);
    59: (61) r1 = *(u32 *)(r6 +0)
    ; void *end = (void *)(long)(ctx->data_end);
    60: (61) r3 = *(u32 *)(r6 +4)
    ; if ((void *)(info->ethh + 1) > end)
    61: (bf) r8 = r1
    62: (07) r8 += 14
    ; if ((void *)(info->ethh + 1) > end)
    63: (2d) if r8 > r3 goto pc-21
     R0=inv(id=0) R1_w=pkt(id=0,off=0,r=14,imm=0) R3_w=pkt_end(id=0,off=0,imm=0) R6=ctx(id=0,off=0,imm=0) R7=invP2 R8_w=pkt(id=0,off=14,r=14,imm=0) R10=fp0 fp-40=????0000 fp-48_w=00000000 fp-56_w=00000000 fp-64_w=00000000 fp-72_w=00000000
    64: (bf) r2 = r1
    65: (07) r2 += 15
    66: (2d) if r2 > r3 goto pc-24
     R0=inv(id=0) R1_w=pkt(id=0,off=0,r=15,imm=0) R2_w=pkt(id=0,off=15,r=15,imm=0) R3_w=pkt_end(id=0,off=0,imm=0) R6=ctx(id=0,off=0,imm=0) R7=invP2 R8_w=pkt(id=0,off=14,r=15,imm=0) R10=fp0 fp-40=????0000 fp-48_w=00000000 fp-56_w=00000000 fp-64_w=00000000 fp-72_w=00000000
    ; if (((struct iphdr *)begin)->version == 4) {
    67: (71) r4 = *(u8 *)(r8 +0)
    68: (bf) r5 = r4
    69: (77) r5 >>= 4
    ; if (((struct iphdr *)begin)->version == 4) {
    70: (15) if r5 == 0x6 goto pc+7
     R0=inv(id=0) R1=pkt(id=0,off=0,r=15,imm=0) R2=pkt(id=0,off=15,r=15,imm=0) R3=pkt_end(id=0,off=0,imm=0) R4=inv(id=5,umax_value=255,var_off=(0x0; 0xff)) R5=inv(id=0,umax_value=15,var_off=(0x0; 0xf)) R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=15,imm=0) R10=fp0 fp-40=????0000 fp-48=00000000 fp-56=00000000 fp-64=00000000 fp-72=00000000
    71: (55) if r5 != 0x4 goto pc-29
     R0=inv(id=0) R1=pkt(id=0,off=0,r=15,imm=0) R2=pkt(id=0,off=15,r=15,imm=0) R3=pkt_end(id=0,off=0,imm=0) R4=inv(id=5,umax_value=255,var_off=(0x0; 0xff)) R5=inv4 R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=15,imm=0) R10=fp0 fp-40=????0000 fp-48=00000000 fp-56=00000000 fp-64=00000000 fp-72=00000000
    ; if ((void *)(info->iph + 1) > end || (info->iph->protocol != IPPROTO_TCP))
    72: (bf) r2 = r1
    73: (07) r2 += 34
    ; if ((void *)(info->iph + 1) > end || (info->iph->protocol != IPPROTO_TCP))
    74: (2d) if r2 > r3 goto pc-32
     R0=inv(id=0) R1=pkt(id=0,off=0,r=34,imm=0) R2_w=pkt(id=0,off=34,r=34,imm=0) R3=pkt_end(id=0,off=0,imm=0) R4=inv(id=5,umax_value=255,var_off=(0x0; 0xff)) R5=inv4 R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=34,imm=0) R10=fp0 fp-40=????0000 fp-48=00000000 fp-56=00000000 fp-64=00000000 fp-72=00000000
    ; if ((void *)(info->iph + 1) > end || (info->iph->protocol != IPPROTO_TCP))
    75: (71) r0 = *(u8 *)(r1 +23)
    ; if ((void *)(info->iph + 1) > end || (info->iph->protocol != IPPROTO_TCP))
    76: (15) if r0 == 0x6 goto pc+6

    from 76 to 83: R0_w=inv6 R1=pkt(id=0,off=0,r=34,imm=0) R2_w=pkt(id=0,off=34,r=34,imm=0) R3=pkt_end(id=0,off=0,imm=0) R4=inv(id=5,umax_value=255,var_off=(0x0; 0xff)) R5=inv4 R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=34,imm=0) R10=fp0 fp-40=????0000 fp-48=00000000 fp-56=00000000 fp-64=00000000 fp-72=00000000
    ; if ((void *)(info->tcph + 1) > end)
    83: (bf) r0 = r2
    84: (07) r0 += 20
    ; if (parser_xdp_info(ctx, &info) == PARSER_FAILED)
    85: (2d) if r0 > r3 goto pc-43
     R0_w=pkt(id=0,off=54,r=54,imm=0) R1=pkt(id=0,off=0,r=54,imm=0) R2_w=pkt(id=0,off=34,r=54,imm=0) R3=pkt_end(id=0,off=0,imm=0) R4=inv(id=5,umax_value=255,var_off=(0x0; 0xff)) R5=inv4 R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=54,imm=0) R10=fp0 fp-40=????0000 fp-48=00000000 fp-56=00000000 fp-64=00000000 fp-72=00000000
    ; if (info.iph->version != 4 && info.iph->version != 6)
    86: (47) r5 |= 2
    87: (15) if r5 == 0x6 goto pc+1
    last_idx 87 first_idx 70
    regs=20 stack=0 before 86: (47) r5 |= 2
    regs=20 stack=0 before 85: (2d) if r0 > r3 goto pc-43
    regs=20 stack=0 before 84: (07) r0 += 20
    regs=20 stack=0 before 83: (bf) r0 = r2
    regs=20 stack=0 before 76: (15) if r0 == 0x6 goto pc+6
    regs=20 stack=0 before 75: (71) r0 = *(u8 *)(r1 +23)
    regs=20 stack=0 before 74: (2d) if r2 > r3 goto pc-32
    regs=20 stack=0 before 73: (07) r2 += 34
    regs=20 stack=0 before 72: (bf) r2 = r1
    regs=20 stack=0 before 71: (55) if r5 != 0x4 goto pc-29
    regs=20 stack=0 before 70: (15) if r5 == 0x6 goto pc+7
     R0=inv(id=0) R1_rw=pkt(id=0,off=0,r=15,imm=0) R2_w=pkt(id=0,off=15,r=15,imm=0) R3_rw=pkt_end(id=0,off=0,imm=0) R4_w=inv(id=5,umax_value=255,var_off=(0x0; 0xff)) R5_rw=invP(id=0,umax_value=15,var_off=(0x0; 0xf)) R6=ctx(id=0,off=0,imm=0) R7_r=invP2 R8_w=pkt(id=0,off=14,r=15,imm=0) R10=fp0 fp-40=????0000 fp-48_w=00000000 fp-56_w=00000000 fp-64_w=00000000 fp-72_w=00000000
    parent didn't have regs=20 stack=0 marks
    last_idx 69 first_idx 53
    regs=20 stack=0 before 69: (77) r5 >>= 4
    regs=20 stack=0 before 68: (bf) r5 = r4
    regs=10 stack=0 before 67: (71) r4 = *(u8 *)(r8 +0)
    ; if (info->iph->version == 4) {
    89: (57) r4 &= 240
    ; if (info->iph->version == 4) {
    90: (55) if r4 != 0x40 goto pc+9
     R0=pkt(id=0,off=54,r=54,imm=0) R1=pkt(id=0,off=0,r=54,imm=0) R2=pkt(id=0,off=34,r=54,imm=0) R3=pkt_end(id=0,off=0,imm=0) R4=inv64 R5=invP6 R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=54,imm=0) R10=fp0 fp-40=????0000 fp-48=00000000 fp-56=00000000 fp-64=00000000 fp-72=00000000
    ; tuple_info->ipv4.saddr = info->iph->saddr;
    91: (61) r3 = *(u32 *)(r1 +26)
    ; tuple_info->ipv4.saddr = info->iph->saddr;
    92: (63) *(u32 *)(r10 -72) = r3
    ; tuple_info->ipv4.daddr = info->iph->daddr;
    93: (61) r1 = *(u32 *)(r1 +30)
    ; tuple_info->ipv4.daddr = info->iph->daddr;
    94: (63) *(u32 *)(r10 -68) = r1
    ; tuple_info->ipv4.sport = info->tcph->source;
    95: (69) r1 = *(u16 *)(r2 +0)
    ; tuple_info->ipv4.sport = info->tcph->source;
    96: (6b) *(u16 *)(r10 -64) = r1
    ; tuple_info->ipv4.dport = info->tcph->dest;
    97: (69) r1 = *(u16 *)(r2 +2)
    ; tuple_info->ipv4.dport = info->tcph->dest;
    98: (6b) *(u16 *)(r10 -62) = r1
    99: (05) goto pc+24
    ; tuple_info->ipv6.dport = info->tcph->dest;
    124: (bf) r2 = r10
    ; int *value = bpf_map_lookup_elem(&map_of_auth, &tuple_key);
    125: (07) r2 += -72
    126: (18) r1 = 0xffff9bad82972c00
    128: (85) call bpf_map_lookup_elem#1
    ; if (!value) {
    129: (55) if r0 != 0x0 goto pc+9
     R0_w=inv0 R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=54,imm=0) R10=fp0 fp-40=????mmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm
    130: (b7) r1 = 0
    ; frontend_key frontend_k = {};
    131: (7b) *(u64 *)(r10 -24) = r1
    last_idx 131 first_idx 124
    regs=2 stack=0 before 130: (b7) r1 = 0
    132: (7b) *(u64 *)(r10 -32) = r1
    ; if (info->iph->version == 4) {
    133: (71) r1 = *(u8 *)(r8 +0)
    ; if (info->iph->version == 4) {
    134: (57) r1 &= 240
    ; if (info->iph->version == 4) {
    135: (55) if r1 != 0x40 goto pc+8
     R0=inv0 R1=inv64 R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=54,imm=0) R10=fp0 fp-24=00000000 fp-32=00000000 fp-40=????mmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm
    ; frontend_k.addr.ip4 = tuple_info->ipv4.daddr;
    136: (61) r1 = *(u32 *)(r10 -68)
    ; frontend_k.addr.ip4 = tuple_info->ipv4.daddr;
    137: (63) *(u32 *)(r10 -32) = r1
    138: (05) goto pc+28
    ; bpf_memcpy(frontend_k.addr.ip6, tuple_info->ipv6.daddr, IPV6_ADDR_LEN);
    167: (bf) r2 = r10
    ;
    168: (07) r2 += -32
    ; return bpf_map_lookup_elem(map, key);
    169: (18) r1 = 0xffff9bad809ba000
    171: (85) call bpf_map_lookup_elem#1
    ; if (!frontend_v) {
    172: (55) if r0 != 0x0 goto pc+22

    from 172 to 195: R0=map_value(id=0,off=0,ks=16,vs=4,imm=0) R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=54,imm=0) R10=fp0 fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=????mmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm
    ; workload_uid = frontend_v->upstream_id;
    195: (71) r1 = *(u8 *)(r0 +1)
     R0=map_value(id=0,off=0,ks=16,vs=4,imm=0) R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=54,imm=0) R10=fp0 fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=????mmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm
    196: (67) r1 <<= 8
    197: (71) r2 = *(u8 *)(r0 +0)
     R0=map_value(id=0,off=0,ks=16,vs=4,imm=0) R1_w=inv(id=0,umax_value=65280,var_off=(0x0; 0xff00)) R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=54,imm=0) R10=fp0 fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=????mmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm
    198: (4f) r1 |= r2
    199: (71) r2 = *(u8 *)(r0 +2)
     R0=map_value(id=0,off=0,ks=16,vs=4,imm=0) R1_w=inv(id=0) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=54,imm=0) R10=fp0 fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=????mmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm
    200: (71) r3 = *(u8 *)(r0 +3)
     R0=map_value(id=0,off=0,ks=16,vs=4,imm=0) R1_w=inv(id=0) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=54,imm=0) R10=fp0 fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=????mmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm
    201: (67) r3 <<= 8
    202: (4f) r3 |= r2
    203: (67) r3 <<= 16
    204: (4f) r3 |= r1
    205: (63) *(u32 *)(r10 -4) = r3
    206: (bf) r2 = r10
    207: (07) r2 += -4
    ; return bpf_map_lookup_elem(map, key);
    208: (18) r1 = 0xffff9bad83042c00
    210: (85) call bpf_map_lookup_elem#1
    ; if (!policies) {
    211: (15) if r0 == 0x0 goto pc-169
     R0=map_value(id=0,off=0,ks=4,vs=16,imm=0) R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=54,imm=0) R10=fp0 fp-8=mmmm???? fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=????mmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm
    ; match_ctx.policies = policies;
    212: (7b) *(u64 *)(r10 -24) = r0
    213: (b7) r7 = 0
    ; match_ctx.policy_index = 0;
    214: (73) *(u8 *)(r10 -28) = r7
    last_idx 214 first_idx 211
    regs=80 stack=0 before 213: (b7) r7 = 0
    215: (bf) r2 = r10
    ; match_ctx.policies = policies;
    216: (07) r2 += -72
    217: (bf) r3 = r10
    218: (07) r3 += -32
    ; ret = bpf_map_update_elem(&kmesh_tc_args, &tuple_key, &match_ctx, BPF_ANY);
    219: (18) r1 = 0xffff9baed8312000
    221: (b7) r4 = 0
    222: (85) call bpf_map_update_elem#2
    invalid indirect read from stack R3 off -32+16 size 24
    processed 211 insns (limit 1000000) max_states_per_insn 1 total_states 20 peak_states 20 mark_read 9
time="2024-11-19T05:43:27Z" level=error msg="bpf Load failed: load program: permission denied:\n\t; int xdp_authz(struct xdp_md *ctx)\n\t0: (bf) r6 = r1\n\t1: (b7) r8 = 0\n\t; int kmesh_config_key = 0;\n\t2: (63) *(u32 *)(r10 -72) = r8\n\tlast_idx 2 first_idx 0\n\tregs=100 stack=0 before 1: (b7) r8 = 0\n\t3: (bf) r2 = r10\n\t;\n\t4: (07) r2 += -72\n\t; return bpf_map_lookup_elem(map, key);\n\t5: (18) r1 = 0xffff9bad87e03a00\n\t7: (85) call bpf_map_lookup_elem#1\n\t8: (b7) r7 = 0\n\t; if (!value)\n\t9: (15) if r0 == 0x0 goto pc+4\n\t R0_w=map_value(id=0,off=0,ks=4,vs=40,imm=0) R6_w=ctx(id=0,off=0,imm=0) R7_w=inv0 R8_w=invP0 R10=fp0 fp-72=????mmmm\n\t; return ((*value).authz_offload == 1);\n\t10: (61) r1 = *(u32 *)(r0 +36)\n\t R0_w=map_value(id=0,off=0,ks=4,vs=40,imm=0) R6_w=ctx(id=0,off=0,imm=0) R7_w=inv0 R8_w=invP0 R10=fp0 fp-72=????mmmm\n\t11: (b7) r7 = 1\n\t; return ((*value).authz_offload == 1);\n\t12: (15) if r1 == 0x1 goto pc+1\n\t\n\tfrom 12 to 14: R0=map_value(id=0,off=0,ks=4,vs=40,imm=0) R1=inv1 R6=ctx(id=0,off=0,imm=0) R7=inv1 R8=invP0 R10=fp0 fp-72=????mmmm\n\t; int zero = 0;\n\t14: (63) *(u32 *)(r10 -72) = r8\n\t15: (bf) r2 = r10\n\t;\n\t16: (07) r2 += -72\n\t; return bpf_map_lookup_elem(map, key);\n\t17: (18) r1 = 0xffff9bad87e03a00\n\t19: (85) call bpf_map_lookup_elem#1\n\t20: (b7) r1 = 2\n\t; if (!value)\n\t21: (15) if r0 == 0x0 goto pc+1\n\t R0=map_value(id=0,off=0,ks=4,vs=40,imm=0) R1_w=inv2 R6=ctx(id=0,off=0,imm=0) R7=inv1 R8=invP0 R10=fp0 fp-72=????mmmm\n\t; return value->bpf_log_level;\n\t22: (61) r1 = *(u32 *)(r0 +0)\n\t R0=map_value(id=0,off=0,ks=4,vs=40,imm=0) R1_w=inv2 R6=ctx(id=0,off=0,imm=0) R7=inv1 R8=invP0 R10=fp0 fp-72=????mmmm\n\t;\n\t23: (67) r1 <<= 32\n\t24: (c7) r1 s>>= 32\n\t25: (b7) r2 = 3\n\t26: (6d) if r2 s> r1 goto pc+1\n\t R0=map_value(id=0,off=0,ks=4,vs=40,imm=0) R1_w=inv(id=0,umin_value=3,umax_value=2147483647,var_off=(0x0; 0x7fffffff)) R2_w=inv3 R6=ctx(id=0,off=0,imm=0) R7=inv1 R8=invP0 R10=fp0 fp-72=????mmmm\n\t27: (b7) r1 = 3\n\t; if (!is_authz_offload_enabled()) {\n\t28: (57) r7 &= 1\n\t29: (55) if r7 != 0x0 goto pc+15\n\tlast_idx 29 first_idx 28\n\tregs=80 stack=0 before 28: (57) r7 &= 1\n\t R0=map_value(id=0,off=0,ks=4,vs=40,imm=0) R1_w=inv3 R2_w=inv3 R6=ctx(id=0,off=0,imm=0) R7_r=invP1 R8=invP0 R10=fp0 fp-72=????mmmm\n\tparent didn't have regs=80 stack=0 marks\n\tlast_idx 27 first_idx 20\n\tregs=80 stack=0 before 27: (b7) r1 = 3\n\tregs=80 stack=0 before 26: (6d) if r2 s> r1 goto pc+1\n\tregs=80 stack=0 before 25: (b7) r2 = 3\n\tregs=80 stack=0 before 24: (c7) r1 s>>= 32\n\tregs=80 stack=0 before 23: (67) r1 <<= 32\n\tregs=80 stack=0 before 22: (61) r1 = *(u32 *)(r0 +0)\n\tregs=80 stack=0 before 21: (15) if r0 == 0x0 goto pc+1\n\tregs=80 stack=0 before 20: (b7) r1 = 2\n\t R0_rw=map_value_or_null(id=4,off=0,ks=4,vs=40,imm=0) R6=ctx(id=0,off=0,imm=0) R7_r=invP1 R8=invP0 R10=fp0 fp-72=????mmmm\n\tparent didn't have regs=80 stack=0 marks\n\tlast_idx 19 first_idx 12\n\tregs=80 stack=0 before 19: (85) call bpf_map_lookup_elem#1\n\tregs=80 stack=0 before 17: (18) r1 = 0xffff9bad87e03a00\n\tregs=80 stack=0 before 16: (07) r2 += -72\n\tregs=80 stack=0 before 15: (bf) r2 = r10\n\tregs=80 stack=0 before 14: (63) *(u32 *)(r10 -72) = r8\n\tregs=80 stack=0 before 12: (15) if r1 == 0x1 goto pc+1\n\t R0_w=map_value(id=0,off=0,ks=4,vs=40,imm=0) R1_rw=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R6_rw=ctx(id=0,off=0,imm=0) R7_rw=invP1 R8_rw=invP0 R10=fp0 fp-72_r=????mmmm\n\tparent didn't have regs=80 stack=0 marks\n\tlast_idx 11 first_idx 0\n\tregs=80 stack=0 before 11: (b7) r7 = 1\n\t;\n\t45: (67) r1 <<= 32\n\t46: (c7) r1 s>>= 32\n\t47: (b7) r7 = 2\n\t; BPF_LOG(INFO, AUTH, \"authz is enabled, processing\");\n\t48: (6d) if r7 s> r1 goto pc+4\n\tlast_idx 48 first_idx 28\n\tregs=80 stack=0 before 47: (b7) r7 = 2\n\tlast_idx 48 first_idx 28\n\tregs=2 stack=0 before 47: (b7) r7 = 2\n\tregs=2 stack=0 before 46: (c7) r1 s>>= 32\n\tregs=2 stack=0 before 45: (67) r1 <<= 32\n\tregs=2 stack=0 before 29: (55) if r7 != 0x0 goto pc+15\n\tregs=2 stack=0 before 28: (57) r7 &= 1\n\t R0=map_value(id=0,off=0,ks=4,vs=40,imm=0) R1_rw=invP3 R2_w=inv3 R6=ctx(id=0,off=0,imm=0) R7_r=invP1 R8=invP0 R10=fp0 fp-72=????mmmm\n\tparent didn't have regs=2 stack=0 marks\n\tlast_idx 27 first_idx 20\n\tregs=2 stack=0 before 27: (b7) r1 = 3\n\t; BPF_LOG(INFO, AUTH, \"authz is enabled, processing\");\n\t49: (18) r1 = 0xffff9bae7e1374db\n\t51: (b7) r2 = 42\n\t52: (85) call bpf_trace_printk#6\n\t R0=map_value(id=0,off=0,ks=4,vs=40,imm=0) R1_w=map_value(id=0,off=907,ks=4,vs=1768,imm=0) R2_w=inv42 R6=ctx(id=0,off=0,imm=0) R7_w=invP2 R8=invP0 R10=fp0 fp-72=????mmmm\n\tlast_idx 52 first_idx 28\n\tregs=4 stack=0 before 51: (b7) r2 = 42\n\t53: (b7) r1 = 0\n\t; struct bpf_sock_tuple tuple_key = {0};\n\t54: (63) *(u32 *)(r10 -40) = r1\n\tlast_idx 54 first_idx 53\n\tregs=2 stack=0 before 53: (b7) r1 = 0\n\t55: (7b) *(u64 *)(r10 -48) = r1\n\t56: (7b) *(u64 *)(r10 -56) = r1\n\t57: (7b) *(u64 *)(r10 -64) = r1\n\t58: (7b) *(u64 *)(r10 -72) = r1\n\t; void *begin = (void *)(long)(ctx->data);\n\t59: (61) r1 = *(u32 *)(r6 +0)\n\t; void *end = (void *)(long)(ctx->data_end);\n\t60: (61) r3 = *(u32 *)(r6 +4)\n\t; if ((void *)(info->ethh + 1) > end)\n\t61: (bf) r8 = r1\n\t62: (07) r8 += 14\n\t; if ((void *)(info->ethh + 1) > end)\n\t63: (2d) if r8 > r3 goto pc-21\n\t R0=inv(id=0) R1_w=pkt(id=0,off=0,r=14,imm=0) R3_w=pkt_end(id=0,off=0,imm=0) R6=ctx(id=0,off=0,imm=0) R7=invP2 R8_w=pkt(id=0,off=14,r=14,imm=0) R10=fp0 fp-40=????0000 fp-48_w=00000000 fp-56_w=00000000 fp-64_w=00000000 fp-72_w=00000000\n\t64: (bf) r2 = r1\n\t65: (07) r2 += 15\n\t66: (2d) if r2 > r3 goto pc-24\n\t R0=inv(id=0) R1_w=pkt(id=0,off=0,r=15,imm=0) R2_w=pkt(id=0,off=15,r=15,imm=0) R3_w=pkt_end(id=0,off=0,imm=0) R6=ctx(id=0,off=0,imm=0) R7=invP2 R8_w=pkt(id=0,off=14,r=15,imm=0) R10=fp0 fp-40=????0000 fp-48_w=00000000 fp-56_w=00000000 fp-64_w=00000000 fp-72_w=00000000\n\t; if (((struct iphdr *)begin)->version == 4) {\n\t67: (71) r4 = *(u8 *)(r8 +0)\n\t68: (bf) r5 = r4\n\t69: (77) r5 >>= 4\n\t; if (((struct iphdr *)begin)->version == 4) {\n\t70: (15) if r5 == 0x6 goto pc+7\n\t R0=inv(id=0) R1=pkt(id=0,off=0,r=15,imm=0) R2=pkt(id=0,off=15,r=15,imm=0) R3=pkt_end(id=0,off=0,imm=0) R4=inv(id=5,umax_value=255,var_off=(0x0; 0xff)) R5=inv(id=0,umax_value=15,var_off=(0x0; 0xf)) R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=15,imm=0) R10=fp0 fp-40=????0000 fp-48=00000000 fp-56=00000000 fp-64=00000000 fp-72=00000000\n\t71: (55) if r5 != 0x4 goto pc-29\n\t R0=inv(id=0) R1=pkt(id=0,off=0,r=15,imm=0) R2=pkt(id=0,off=15,r=15,imm=0) R3=pkt_end(id=0,off=0,imm=0) R4=inv(id=5,umax_value=255,var_off=(0x0; 0xff)) R5=inv4 R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=15,imm=0) R10=fp0 fp-40=????0000 fp-48=00000000 fp-56=00000000 fp-64=00000000 fp-72=00000000\n\t; if ((void *)(info->iph + 1) > end || (info->iph->protocol != IPPROTO_TCP))\n\t72: (bf) r2 = r1\n\t73: (07) r2 += 34\n\t; if ((void *)(info->iph + 1) > end || (info->iph->protocol != IPPROTO_TCP))\n\t74: (2d) if r2 > r3 goto pc-32\n\t R0=inv(id=0) R1=pkt(id=0,off=0,r=34,imm=0) R2_w=pkt(id=0,off=34,r=34,imm=0) R3=pkt_end(id=0,off=0,imm=0) R4=inv(id=5,umax_value=255,var_off=(0x0; 0xff)) R5=inv4 R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=34,imm=0) R10=fp0 fp-40=????0000 fp-48=00000000 fp-56=00000000 fp-64=00000000 fp-72=00000000\n\t; if ((void *)(info->iph + 1) > end || (info->iph->protocol != IPPROTO_TCP))\n\t75: (71) r0 = *(u8 *)(r1 +23)\n\t; if ((void *)(info->iph + 1) > end || (info->iph->protocol != IPPROTO_TCP))\n\t76: (15) if r0 == 0x6 goto pc+6\n\t\n\tfrom 76 to 83: R0_w=inv6 R1=pkt(id=0,off=0,r=34,imm=0) R2_w=pkt(id=0,off=34,r=34,imm=0) R3=pkt_end(id=0,off=0,imm=0) R4=inv(id=5,umax_value=255,var_off=(0x0; 0xff)) R5=inv4 R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=34,imm=0) R10=fp0 fp-40=????0000 fp-48=00000000 fp-56=00000000 fp-64=00000000 fp-72=00000000\n\t; if ((void *)(info->tcph + 1) > end)\n\t83: (bf) r0 = r2\n\t84: (07) r0 += 20\n\t; if (parser_xdp_info(ctx, &info) == PARSER_FAILED)\n\t85: (2d) if r0 > r3 goto pc-43\n\t R0_w=pkt(id=0,off=54,r=54,imm=0) R1=pkt(id=0,off=0,r=54,imm=0) R2_w=pkt(id=0,off=34,r=54,imm=0) R3=pkt_end(id=0,off=0,imm=0) R4=inv(id=5,umax_value=255,var_off=(0x0; 0xff)) R5=inv4 R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=54,imm=0) R10=fp0 fp-40=????0000 fp-48=00000000 fp-56=00000000 fp-64=00000000 fp-72=00000000\n\t; if (info.iph->version != 4 && info.iph->version != 6)\n\t86: (47) r5 |= 2\n\t87: (15) if r5 == 0x6 goto pc+1\n\tlast_idx 87 first_idx 70\n\tregs=20 stack=0 before 86: (47) r5 |= 2\n\tregs=20 stack=0 before 85: (2d) if r0 > r3 goto pc-43\n\tregs=20 stack=0 before 84: (07) r0 += 20\n\tregs=20 stack=0 before 83: (bf) r0 = r2\n\tregs=20 stack=0 before 76: (15) if r0 == 0x6 goto pc+6\n\tregs=20 stack=0 before 75: (71) r0 = *(u8 *)(r1 +23)\n\tregs=20 stack=0 before 74: (2d) if r2 > r3 goto pc-32\n\tregs=20 stack=0 before 73: (07) r2 += 34\n\tregs=20 stack=0 before 72: (bf) r2 = r1\n\tregs=20 stack=0 before 71: (55) if r5 != 0x4 goto pc-29\n\tregs=20 stack=0 before 70: (15) if r5 == 0x6 goto pc+7\n\t R0=inv(id=0) R1_rw=pkt(id=0,off=0,r=15,imm=0) R2_w=pkt(id=0,off=15,r=15,imm=0) R3_rw=pkt_end(id=0,off=0,imm=0) R4_w=inv(id=5,umax_value=255,var_off=(0x0; 0xff)) R5_rw=invP(id=0,umax_value=15,var_off=(0x0; 0xf)) R6=ctx(id=0,off=0,imm=0) R7_r=invP2 R8_w=pkt(id=0,off=14,r=15,imm=0) R10=fp0 fp-40=????0000 fp-48_w=00000000 fp-56_w=00000000 fp-64_w=00000000 fp-72_w=00000000\n\tparent didn't have regs=20 stack=0 marks\n\tlast_idx 69 first_idx 53\n\tregs=20 stack=0 before 69: (77) r5 >>= 4\n\tregs=20 stack=0 before 68: (bf) r5 = r4\n\tregs=10 stack=0 before 67: (71) r4 = *(u8 *)(r8 +0)\n\t; if (info->iph->version == 4) {\n\t89: (57) r4 &= 240\n\t; if (info->iph->version == 4) {\n\t90: (55) if r4 != 0x40 goto pc+9\n\t R0=pkt(id=0,off=54,r=54,imm=0) R1=pkt(id=0,off=0,r=54,imm=0) R2=pkt(id=0,off=34,r=54,imm=0) R3=pkt_end(id=0,off=0,imm=0) R4=inv64 R5=invP6 R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=54,imm=0) R10=fp0 fp-40=????0000 fp-48=00000000 fp-56=00000000 fp-64=00000000 fp-72=00000000\n\t; tuple_info->ipv4.saddr = info->iph->saddr;\n\t91: (61) r3 = *(u32 *)(r1 +26)\n\t; tuple_info->ipv4.saddr = info->iph->saddr;\n\t92: (63) *(u32 *)(r10 -72) = r3\n\t; tuple_info->ipv4.daddr = info->iph->daddr;\n\t93: (61) r1 = *(u32 *)(r1 +30)\n\t; tuple_info->ipv4.daddr = info->iph->daddr;\n\t94: (63) *(u32 *)(r10 -68) = r1\n\t; tuple_info->ipv4.sport = info->tcph->source;\n\t95: (69) r1 = *(u16 *)(r2 +0)\n\t; tuple_info->ipv4.sport = info->tcph->source;\n\t96: (6b) *(u16 *)(r10 -64) = r1\n\t; tuple_info->ipv4.dport = info->tcph->dest;\n\t97: (69) r1 = *(u16 *)(r2 +2)\n\t; tuple_info->ipv4.dport = info->tcph->dest;\n\t98: (6b) *(u16 *)(r10 -62) = r1\n\t99: (05) goto pc+24\n\t; tuple_info->ipv6.dport = info->tcph->dest;\n\t124: (bf) r2 = r10\n\t; int *value = bpf_map_lookup_elem(&map_of_auth, &tuple_key);\n\t125: (07) r2 += -72\n\t126: (18) r1 = 0xffff9bad82972c00\n\t128: (85) call bpf_map_lookup_elem#1\n\t; if (!value) {\n\t129: (55) if r0 != 0x0 goto pc+9\n\t R0_w=inv0 R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=54,imm=0) R10=fp0 fp-40=????mmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm\n\t130: (b7) r1 = 0\n\t; frontend_key frontend_k = {};\n\t131: (7b) *(u64 *)(r10 -24) = r1\n\tlast_idx 131 first_idx 124\n\tregs=2 stack=0 before 130: (b7) r1 = 0\n\t132: (7b) *(u64 *)(r10 -32) = r1\n\t; if (info->iph->version == 4) {\n\t133: (71) r1 = *(u8 *)(r8 +0)\n\t; if (info->iph->version == 4) {\n\t134: (57) r1 &= 240\n\t; if (info->iph->version == 4) {\n\t135: (55) if r1 != 0x40 goto pc+8\n\t R0=inv0 R1=inv64 R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=54,imm=0) R10=fp0 fp-24=00000000 fp-32=00000000 fp-40=????mmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm\n\t; frontend_k.addr.ip4 = tuple_info->ipv4.daddr;\n\t136: (61) r1 = *(u32 *)(r10 -68)\n\t; frontend_k.addr.ip4 = tuple_info->ipv4.daddr;\n\t137: (63) *(u32 *)(r10 -32) = r1\n\t138: (05) goto pc+28\n\t; bpf_memcpy(frontend_k.addr.ip6, tuple_info->ipv6.daddr, IPV6_ADDR_LEN);\n\t167: (bf) r2 = r10\n\t;\n\t168: (07) r2 += -32\n\t; return bpf_map_lookup_elem(map, key);\n\t169: (18) r1 = 0xffff9bad809ba000\n\t171: (85) call bpf_map_lookup_elem#1\n\t; if (!frontend_v) {\n\t172: (55) if r0 != 0x0 goto pc+22\n\t\n\tfrom 172 to 195: R0=map_value(id=0,off=0,ks=16,vs=4,imm=0) R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=54,imm=0) R10=fp0 fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=????mmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm\n\t; workload_uid = frontend_v->upstream_id;\n\t195: (71) r1 = *(u8 *)(r0 +1)\n\t R0=map_value(id=0,off=0,ks=16,vs=4,imm=0) R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=54,imm=0) R10=fp0 fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=????mmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm\n\t196: (67) r1 <<= 8\n\t197: (71) r2 = *(u8 *)(r0 +0)\n\t R0=map_value(id=0,off=0,ks=16,vs=4,imm=0) R1_w=inv(id=0,umax_value=65280,var_off=(0x0; 0xff00)) R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=54,imm=0) R10=fp0 fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=????mmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm\n\t198: (4f) r1 |= r2\n\t199: (71) r2 = *(u8 *)(r0 +2)\n\t R0=map_value(id=0,off=0,ks=16,vs=4,imm=0) R1_w=inv(id=0) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=54,imm=0) R10=fp0 fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=????mmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm\n\t200: (71) r3 = *(u8 *)(r0 +3)\n\t R0=map_value(id=0,off=0,ks=16,vs=4,imm=0) R1_w=inv(id=0) R2_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=54,imm=0) R10=fp0 fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=????mmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm\n\t201: (67) r3 <<= 8\n\t202: (4f) r3 |= r2\n\t203: (67) r3 <<= 16\n\t204: (4f) r3 |= r1\n\t205: (63) *(u32 *)(r10 -4) = r3\n\t206: (bf) r2 = r10\n\t207: (07) r2 += -4\n\t; return bpf_map_lookup_elem(map, key);\n\t208: (18) r1 = 0xffff9bad83042c00\n\t210: (85) call bpf_map_lookup_elem#1\n\t; if (!policies) {\n\t211: (15) if r0 == 0x0 goto pc-169\n\t R0=map_value(id=0,off=0,ks=4,vs=16,imm=0) R6=ctx(id=0,off=0,imm=0) R7=invP2 R8=pkt(id=0,off=14,r=54,imm=0) R10=fp0 fp-8=mmmm???? fp-24=mmmmmmmm fp-32=mmmmmmmm fp-40=????mmmm fp-48=mmmmmmmm fp-56=mmmmmmmm fp-64=mmmmmmmm fp-72=mmmmmmmm\n\t; match_ctx.policies = policies;\n\t212: (7b) *(u64 *)(r10 -24) = r0\n\t213: (b7) r7 = 0\n\t; match_ctx.policy_index = 0;\n\t214: (73) *(u8 *)(r10 -28) = r7\n\tlast_idx 214 first_idx 211\n\tregs=80 stack=0 before 213: (b7) r7 = 0\n\t215: (bf) r2 = r10\n\t; match_ctx.policies = policies;\n\t216: (07) r2 += -72\n\t217: (bf) r3 = r10\n\t218: (07) r3 += -32\n\t; ret = bpf_map_update_elem(&kmesh_tc_args, &tuple_key, &match_ctx, BPF_ANY);\n\t219: (18) r1 = 0xffff9baed8312000\n\t221: (b7) r4 = 0\n\t222: (85) call bpf_map_update_elem#2\n\tinvalid indirect read from stack R3 off -32+16 size 24\n\tprocessed 211 insns (limit 1000000) max_states_per_insn 1 total_states 20 peak_states 20 mark_read 9" subsys=main
Error: parse BpfConfig failed, stat /mnt/kmesh_cgroup2: no such file or directory
time="2024-11-19T05:43:28Z" level=error msg="parse BpfConfig failed, stat /mnt/kmesh_cgroup2: no such file or directory" subsys=main
kmesh exit

I'm using ghcr.io/kmesh-net/kmesh:latest when I reproduce this. However, everything seems to work fine when I use ghcr.io/kmesh-net/kmesh:v0.5.0

According to Error: bpf Load failed: load program: permission denied:, I suspect that this is a problem related to pod privilege. But the daemon is actually running as privileged. Here is the deployed Daemonset:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  annotations:
    deprecated.daemonset.template.generation: '2'
    meta.helm.sh/release-name: kmesh
    meta.helm.sh/release-namespace: kmesh-system
  creationTimestamp: '2024-11-19T05:35:13Z'
  generation: 2
  labels:
    app: kmesh
    app.kubernetes.io/instance: kmesh
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kmesh-helm
    app.kubernetes.io/version: 0.1.0
    helm.sh/chart: kmesh-helm-0.1.0
  name: kmesh
  namespace: kmesh-system
  resourceVersion: '74799'
  uid: 5d1c1af3-f9c1-46e2-9aad-bab14cf4415c
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: kmesh
      app.kubernetes.io/instance: kmesh
      app.kubernetes.io/name: kmesh-helm
  template:
    metadata:
      annotations:
        prometheus.io/path: status/metric
        prometheus.io/port: '15020'
        prometheus.io/scrape: 'true'
      labels:
        app: kmesh
        app.kubernetes.io/instance: kmesh
        app.kubernetes.io/name: kmesh-helm
    spec:
      containers:
        - args:
            - >-
              ./start_kmesh.sh --mode=dual-engine --enable-bypass=false
              --enable-bpf-log=true
          command:
            - /bin/sh
            - '-c'
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: INSTANCE_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
            - name: XDS_ADDRESS
              value: 'istiod.istio-system.svc:15012'
            - name: KUBERNETES_CLUSTER_DOMAIN
              value: cluster.local
            - name: SERVICE_ACCOUNT
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.serviceAccountName
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
          image: 'ghcr.io/kmesh-net/kmesh:latest'
          imagePullPolicy: Always
          name: kmesh
          resources:
            limits:
              cpu: '1'
              memory: 800Mi
          securityContext:
            capabilities:
              add:
                - all
            privileged: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /mnt
              name: mnt
            - mountPath: /sys/fs/bpf
              name: sys-fs-bpf
            - mountPath: /lib/modules
              name: lib-modules
            - mountPath: /etc/cni/net.d
              name: cni
            - mountPath: /opt/cni/bin
              name: kmesh-cni-install-path
            - mountPath: /host/proc
              name: host-procfs
              readOnly: true
            - mountPath: /var/run/secrets/istio
              name: istiod-ca-cert
            - mountPath: /var/run/secrets/tokens
              name: istio-token
      dnsPolicy: ClusterFirst
      priorityClassName: system-node-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: kmesh
      serviceAccountName: kmesh
      terminationGracePeriodSeconds: 30
      tolerations:
        - effect: NoSchedule
          operator: Exists
        - key: CriticalAddonsOnly
          operator: Exists
        - effect: NoExecute
          operator: Exists
      volumes:
        - hostPath:
            path: /mnt
            type: ''
          name: mnt
        - hostPath:
            path: /sys/fs/bpf
            type: ''
          name: sys-fs-bpf
        - hostPath:
            path: /lib/modules
            type: ''
          name: lib-modules
        - hostPath:
            path: /etc/cni/net.d
            type: ''
          name: cni
        - hostPath:
            path: /opt/cni/bin
            type: ''
          name: kmesh-cni-install-path
        - hostPath:
            path: /proc
            type: Directory
          name: host-procfs
        - configMap:
            defaultMode: 420
            name: istio-ca-root-cert
          name: istiod-ca-cert
        - name: istio-token
          projected:
            defaultMode: 420
            sources:
              - serviceAccountToken:
                  audience: istio-ca
                  expirationSeconds: 43200
                  path: istio-token
  updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
status:
  currentNumberScheduled: 3
  desiredNumberScheduled: 3
  numberMisscheduled: 0
  numberReady: 0
  numberUnavailable: 3
  observedGeneration: 2
  updatedNumberScheduled: 3

What you expected to happen:

I expect that the Kmesh daemon is in running state.

How to reproduce it (as minimally and precisely as possible):

Just follow the quick start (https://kmesh.net/en/docs/setup/quickstart/). I encouter this issue after install Kmesh using kubectl create namespace kmesh-system & kubectl apply -f ./deploy/yaml/

Anything else we need to know?:

I'm running Kmesh on Alibaba Cloud Container Service (known as Alibaba Cloud ACK). The mode is dual-engine mode. The node OS is AlibabaCloud Linux 3. However, It's hard to imagine how this issue is related to the OS. Besides, Kmesh turns to work on ACK using dual-engine mode several weeks before.

Environment:

nlgwcy commented 22 hours ago

Thank you for your report. This problem is caused by the failure to load the BPF PROG file, which may be related to recent modifications. @weli-l Please help to check.

hzxuzhonghu commented 22 hours ago

seems related to xdp offload @weli-l

hzxuzhonghu commented 22 hours ago

I can reproduce on my ubuntu 22.04 with kernel 5.15

weli-l commented 21 hours ago

@delavet What is the kernel version you are using?

delavet commented 21 hours ago

@delavet What is the kernel version you are using?

It's 5.10.134-17.2.al8.x86_64

hzxuzhonghu commented 19 hours ago

@delavet can you retest?

delavet commented 18 hours ago

@delavet can you retest?

I have retested it and everything seems to work fine. Really appreciated!