failed to run ./nettrace -p icmp in loongarch 4.19 kernel

[georgejguo]

nettrace version： v1.2.6. cpu：loongarch kernel：4.19 version

./nettrace -p icmp --debug

DEBUG: command: mount | grep debugfs, status:0 DEBUG: command: cat /sys/kernel/debug/tracing/events/skb/kfree_skb/format 2>/dev/null | grep NOT_SPECIFIED, status:0 DEBUG: trace name=tcp_ack_probe, prog=trace_tcp_ack_probe is made invalid for: sock or sk mode DEBUG: trace name=tcp_probe_timer, prog=__trace_tcp_probe_timer is made invalid for: sock or sk mode DEBUG: trace name=tcp_send_probe0, prog=trace_tcp_send_probe0 is made invalid for: sock or sk mode DEBUG: trace name=tcp_transmit_skb, prog=trace_tcp_transmit_skb is made invalid for: sock or sk mode DEBUG: trace name=tcp_sendmsg_locked, prog=__trace_tcp_sendmsg_locked is made invalid for: sock or sk mode DEBUG: trace name=tcp_push_pending_frames, prog=trace_tcp_push_pending_frames is made invalid for: sock or sk mode DEBUG: trace name=inet_listen, prog=trace_inet_listen is made invalid for: sock or sk mode DEBUG: trace name=tcp_v4_destroy_sock, prog=trace_tcp_v4_destroy_sock is made invalid for: sock or sk mode DEBUG: trace name=tcp_close, prog=trace_tcp_close is made invalid for: sock or sk mode DEBUG: trace name=tcp_write_timer_handler, prog=trace_tcp_write_timer_handler is made invalid for: sock or sk mode DEBUG: trace name=tcp_retransmit_timer, prog=trace_tcp_retransmit_timer is made invalid for: sock or sk mode DEBUG: trace name=tcp_enter_recovery, prog=trace_tcp_enter_recovery is made invalid for: sock or sk mode DEBUG: trace name=tcp_enter_loss, prog=trace_tcp_enter_loss is made invalid for: sock or sk mode DEBUG: trace name=tcp_try_keep_open, prog=trace_tcp_try_keep_open is made invalid for: sock or sk mode DEBUG: trace name=tcp_enter_cwr, prog=trace_tcp_enter_cwr is made invalid for: sock or sk mode DEBUG: trace name=tcp_fastretrans_alert, prog=__trace_tcp_fastretrans_alert is made invalid for: sock or sk mode DEBUG: trace name=tcp_rearm_rto, prog=trace_tcp_rearm_rto is made invalid for: sock or sk mode DEBUG: trace name=tcp_event_new_data_sent, prog=trace_tcp_event_new_data_sent is made invalid for: sock or sk mode DEBUG: trace name=tcp_schedule_loss_probe, prog=trace_tcp_schedule_loss_probe is made invalid for: sock or sk mode DEBUG: command: verlte() { [ "$1" = "$2" ] && echo 0 && return; [ "$1" = "$(/bin/echo -e "$1\n$2" | sort -V | head -n1)" ] && echo -1 && return; echo 1; }; [ ! $(verlte "$(uname -r)" "5.4") -eq -1 ], status:1 DEBUG: trace name=netif_receive_skb_core, prog=_tracenetif_receive_skb_core_pskb is made invalid for: cond DEBUG: command: verlte() { [ "$1" = "$2" ] && echo 0 && return; [ "$1" = "$(/bin/echo -e "$1\n$2" | sort -V | head -n1)" ] && echo -1 && return; echo 1; }; [ $(verlte "$(uname -r)" "5.16") -eq -1 ], status:0 DEBUG: begin to resolve kernel symbol... kernel function netif_receive_generic_xdp not founded, skipped DEBUG: trace name=netif_receive_generic_xdp, prog=trace_netif_receive_generic_xdp is made invalid for: not found kernel function cls_bpf_classify not founded, skipped DEBUG: trace name=cls_bpf_classify, prog=trace_cls_bpf_classify is made invalid for: not found kernel function tcf_bpf_act not founded, skipped DEBUG: trace name=tcf_bpf_act, prog=trace_tcf_bpf_act is made invalid for: not found kernel function bond_dev_queue_xmit not founded, skipped DEBUG: trace name=bond_dev_queue_xmit, prog=trace_bond_dev_queue_xmit is made invalid for: not found kernel function nft_do_chain not founded, skipped DEBUG: trace name=nft_do_chain, prog=trace_nft_do_chain is made invalid for: not found DEBUG: ip_rcv_core.isra.2 is made manual attach DEBUG: ip6_rcv_core.isra.2 is made manual attach kernel function ip_finish_output_gso not founded, skipped DEBUG: trace name=ip_finish_output_gso, prog=trace_ip_finish_output_gso is made invalid for: not found DEBUG: xfrm_output_gso.isra.1 is made manual attach kernel function xfrm4_transport_output not founded, skipped DEBUG: trace name=xfrm4_transport_output, prog=trace_xfrm4_transport_output is made invalid for: not found kernel function xfrm4_policy_check not founded, skipped DEBUG: trace name=xfrm4_policy_check, prog=__trace_xfrm4_policy_check is made invalid for: not found kernel function xfrm4_transport_input not founded, skipped DEBUG: trace name=xfrm4_transport_input, prog=trace_xfrm4_transport_input is made invalid for: not found kernel function ah_output not founded, skipped DEBUG: trace name=ah_output, prog=trace_ah_output is made invalid for: not found kernel function esp_output not founded, skipped DEBUG: trace name=esp_output, prog=trace_esp_output is made invalid for: not found kernel function esp_output_tail not founded, skipped DEBUG: trace name=esp_output_tail, prog=trace_esp_output_tail is made invalid for: not found kernel function ah_input not founded, skipped DEBUG: trace name=ah_input, prog=trace_ah_input is made invalid for: not found kernel function esp_input not founded, skipped DEBUG: trace name=esp_input, prog=trace_esp_input is made invalid for: not found DEBUG: udp_unicast_rcv_skb.isra.14 is made manual attach kernel function udp_queue_rcv_skb not founded, skipped DEBUG: trace name=udp_queue_rcv_skb, prog=_traceudp_queue_rcv_skb is made invalid for: not found DEBUG: icmp_reply.constprop.12 is made manual attach kernel function ping_queue_rcv_skb not founded, skipped DEBUG: trace name=__ping_queue_rcv_skb, prog=trace_ping_queue_rcv_skb is made invalid for: not found DEBUG: ping_lookup.isra.3 is made manual attach DEBUG: finished to resolve kernel symbol DEBUG: kernel feature probe begin DEBUG: kernel feature probe end DEBUG: backup: valid prog for __netif_receive_skb_core is trace_netif_receive_skb_core DEBUG: backup: valid prog for ipt_do_table is trace_ipt_do_table_legacy DEBUG: trace name=ipt_do_table, prog=trace_ipt_do_table is made invalid for: backup following traces are enabled and valid: tracepoint: napi_gro_receive_entry, prog: trace_napi_gro_receive_entry kprobe: dev_gro_receive, prog: trace_dev_gro_receive kprobe: enqueue_to_backlog, prog: trace_enqueue_to_backlog kprobe: xdp_do_generic_redirect, prog: trace_xdp_do_generic_redirect kprobe: __netif_receive_skb_core, prog: trace_netif_receive_skb_core kprobe: dev_queue_xmit, prog: trace___dev_queue_xmit kprobe: dev_hard_start_xmit, prog: trace_dev_hard_start_xmit kprobe: pfifo_enqueue, prog: trace_pfifo_enqueue kprobe: sch_direct_xmit, prog: trace_sch_direct_xmit kprobe: tcf_classify, prog: trace_tcf_classify kprobe: arp_rcv, prog: __trace_arp_rcv kprobe: arp_process, prog: trace_arp_process kprobe: packet_rcv, prog: trace_packet_rcv kprobe: tpacket_rcv, prog: trace_tpacket_rcv kprobe: packet_direct_xmit, prog: trace_packet_direct_xmit kprobe: nf_nat_manip_pkt, prog: trace_nf_nat_manip_pkt kprobe: nf_hook_slow, prog: trace_nf_hook_slow kprobe: ipt_do_table, prog: trace_ipt_do_table_legacy kprobe: tcp_in_window, prog: trace_tcp_in_window kprobe: ip_rcv, prog: trace_ip_rcv kprobe: ip_rcv_core.isra.2, prog: trace_ip_rcv_core kprobe: ip_rcv_finish, prog: __trace_ip_rcv_finish kprobe: ip_local_deliver, prog: trace_ip_local_deliver kprobe: ip_local_deliver_finish, prog: trace_ip_local_deliver_finish kprobe: ip_forward, prog: trace_ip_forward kprobe: ip_forward_finish, prog: trace_ip_forward_finish kprobe: ip6_forward, prog: trace_ip6_forward kprobe: ip6_rcv_finish, prog: trace_ip6_rcv_finish kprobe: ip6_rcv_core.isra.2, prog: trace_ip6_rcv_core kprobe: ipv6_rcv, prog: trace_ipv6_rcv kprobe: ip_output, prog: trace_ip_output kprobe: ip_finish_output, prog: trace_ip_finish_output kprobe: ip_finish_output2, prog: trace_ip_finish_output2 kprobe: ip6_output, prog: trace_ip6_output kprobe: ip6_finish_output, prog: trace_ip6_finish_output kprobe: ip6_finish_output2, prog: trace_ip6_finish_output2 kprobe: ip6_send_skb, prog: __trace_ip6_send_skb kprobe: ip6_local_out, prog: trace_ip6_local_out kprobe: xfrm4_output, prog: trace_xfrm4_output kprobe: xfrm_output, prog: __trace_xfrm_output kprobe: xfrm_output2, prog: trace_xfrm_output2 kprobe: xfrm_output_gso.isra.1, prog: trace_xfrm_output_gso kprobe: xfrm_output_resume, prog: __trace_xfrm_output_resume kprobe: xfrm4_prepare_output, prog: trace_xfrm4_prepare_output kprobe: xfrm4_rcv, prog: trace_xfrm4_rcv kprobe: xfrm_input, prog: trace_xfrm_input kprobe: fib_validate_source, prog: trace_fib_validate_source kprobe: ip_route_input_slow, prog: trace_ip_route_input_slow kprobe: tcp_v4_rcv, prog: trace_tcp_v4_rcv kprobe: tcp_v6_rcv, prog: trace_tcp_v6_rcv kprobe: tcp_filter, prog: trace_tcp_filter kprobe: tcp_child_process, prog: trace_tcp_child_process kprobe: tcp_v4_send_reset, prog: trace_tcp_v4_send_reset kprobe: tcp_v6_send_reset, prog: __trace_tcp_v6_send_reset kprobe: tcp_v4_do_rcv, prog: trace_tcp_v4_do_rcv kprobe: tcp_v6_do_rcv, prog: trace_tcp_v6_do_rcv kprobe: tcp_rcv_established, prog: trace_tcp_rcv_established kprobe: tcp_rcv_state_process, prog: trace_tcp_rcv_state_process kprobe: tcp_queue_rcv, prog: trace_tcp_queue_rcv kprobe: tcp_ack, prog: trace_tcp_ack kprobe: inet_lookup_listener, prog: trace_inet_lookup_listener kprobe: inet6_lookup_listener, prog: trace_inet6_lookup_listener kprobe: udp_rcv, prog: __trace_udp_rcv kprobe: udp_unicast_rcv_skb.isra.14, prog: trace_udp_unicast_rcv_skb kprobe: udp_queue_rcv_skb, prog: trace_udp_queue_rcv_skb kprobe: xfrm4_udp_encap_rcv, prog: trace_xfrm4_udp_encap_rcv kprobe: xfrm4_rcv_encap, prog: trace_xfrm4_rcv_encap kprobe: udp_enqueue_schedule_skb, prog: trace_udp_enqueue_schedule_skb kprobe: icmp_rcv, prog: trace_icmp_rcv kprobe: icmp_echo, prog: trace_icmp_echo kprobe: icmp_reply.constprop.12, prog: trace_icmp_reply kprobe: icmpv6_rcv, prog: trace_icmpv6_rcv kprobe: icmpv6_echo_reply, prog: trace_icmpv6_echo_reply kprobe: ping_rcv, prog: __trace_ping_rcv kprobe: ping_queue_rcv_skb, prog: trace_ping_queue_rcv_skb kprobe: ping_lookup.isra.3, prog: trace_ping_lookup kprobe: kfree_skb, prog: trace_kfree_skb kprobe/kretprobe: skb_clone, prog: trace_skb_clone tracepoint: consume_skb, prog: trace_consume_skb tracepoint: kfree_skb, prog: trace_kfree_skb libbpf: failed to find skeleton program 'rettrace___netif_receive_skb_core_pskb' ERROR: failed to open kprobe-based eBPF ERROR: failed to load kprobe-based bpf

想知道这里为何libbpf: failed to find skeleton program 'ret__trace___netif_receive_skb_core_pskb'没有找到这个程序呢？

[menglongdong]

你这里用的是自己采用compat模式编译的nettrace吗？从报错信息上来看，我猜测是编译环境上所采用的libbpf和bpftool的版本的问题。

目前还没有loongarch架构的docker编译环境，这里你可以尝试使用较高版本的libbpf以及bpftool，看看能不能解决问题。

[georgejguo]

libbpf version： v1.1.0 bpftool：7.1.0 nettrace：1.2.6

$ make COMPAT=1 all 现在编译没有问题，

$ ./src/nettrace -p icmp --debug ERROR: failed to load kprobe-based eBPF ERROR: failed to load kprobe-based bpf 但是执行的时候报错：log如下：

DEBUG: prog: trace_ah_output is madlibbpf: prog 'trace_dev_gro_receive': BPF program load failed: Permission denied libbpf: prog '__trace_dev_gro_receive': -- BEGIN PROG LOAD LOG -- Unrecognized arg#0 type PTR ; DEFINE_ALL_PROBES(KPROBE_DEFAULT, TP_DEFAULT, FNC) 0: (bf) r8 = r1 ; DEFINE_ALL_PROBES(KPROBE_DEFAULT, TP_DEFAULT, FNC) 1: (7b) (u64 )(r10 -200) = r8

... 805: (79) r1 = (u64 )(r6 +55) misaligned value access off (0x0; 0x0)+0+55 size 8 processed 1461 insns (limit 1000000) max_states_per_insn 2 total_states 117 peak_states 117 mark_read 28 -- END PROG LOAD LOG -- libbpf: prog '__trace_dev_gro_receive': failed to load: -13 libbpf: failed to load object 'kprobe' libbpf: failed to load BPF skeleton 'kprobe': -13

[menglongdong]

这个我先分析一下，感觉像是loongarch对BPF的支持问题

[georgejguo]

这个是完整的日志： fail-log.txt 我主要是没有思路。不知道怎么debug了，应该和内核相关。