search

orangepi-xunlong / linux-orangepi

Other
200 stars 174 forks source link

SSH卡死问题 #5

Closed Longhao-Chen closed 2 years ago

Longhao-Chen commented 3 years ago

今早ssh无法登录，在输入密码后就没有了响应，使用 ssh -v 得到的信息如下：

root@192.168.3.50's password: 
debug3: send packet: type 50
debug2: we sent a password packet, wait for reply
debug3: receive packet: type 52
debug1: Authentication succeeded (password).
Authenticated to 192.168.3.50 ([192.168.3.50]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
debug1: Sending environment.
debug1: Sending env LANG = zh_CN.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug3: send packet: type 100
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug3: send packet: type 100
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug3: send packet: type 100
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug3: send packet: type 100
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug3: send packet: type 100
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug3: send packet: type 100
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug3: send packet: type 100
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug3: send packet: type 100
debug3: send packet: type 1

然后使用串口直接连接到板子上，也什么反应都没有。 在此期间，运行的其他的网络服务均可正常使用。

拔电源重启后在 /var/log/kern.log 得到的信息如下：

May 29 06:59:41 orangepizero kernel: [19546.888810]     (detected by 2, t=162485108 jiffies, g=172501, q=805)
May 29 06:59:41 orangepizero kernel: [19546.888823] Sending NMI from CPU 2 to CPUs 0:
May 29 06:59:41 orangepizero kernel: [19546.889396] NMI backtrace for cpu 0
May 29 06:59:41 orangepizero kernel: [19546.889401] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C        5.3.5 #1
May 29 06:59:41 orangepizero kernel: [19546.889404] Hardware name: Allwinner sun8i Family
May 29 06:59:41 orangepizero kernel: [19546.889406] PC is at __skb_gso_segment+0xc/0x140
May 29 06:59:41 orangepizero kernel: [19546.889408] LR is at validate_xmit_skb+0xe3/0x1f8
May 29 06:59:41 orangepizero kernel: [19546.889411] pc : [<c08171c0>]    lr : [<c08175f3>]    psr: 20010133
May 29 06:59:41 orangepizero kernel: [19546.889414] sp : c0f01988  ip : 00000001  fp : 00000022
May 29 06:59:41 orangepizero kernel: [19546.889416] r10: 00000b84  r9 : d5614000  r8 : c0f01a0b
May 29 06:59:41 orangepizero kernel: [19546.889419] r7 : d5e2b740  r6 : 00002080  r5 : 00004933  r4 : c8663438
May 29 06:59:41 orangepizero kernel: [19546.889422] r3 : 00002080  r2 : 00004933  r1 : 00000000  r0 : c8663438
May 29 06:59:41 orangepizero kernel: [19546.889425] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment none
May 29 06:59:41 orangepizero kernel: [19546.889428] Control: 50c5387d  Table: 55fc406a  DAC: 00000051
May 29 06:59:41 orangepizero kernel: [19546.889431] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C        5.3.5 #1
May 29 06:59:41 orangepizero kernel: [19546.889434] Hardware name: Allwinner sun8i Family
May 29 06:59:41 orangepizero kernel: [19546.889437] [<c010cfb9>] (unwind_backtrace) from [<c0109cdd>] (show_stack+0x11/0x14)
May 29 06:59:41 orangepizero kernel: [19546.889440] [<c0109cdd>] (show_stack) from [<c0961c4d>] (dump_stack+0x69/0x78)
May 29 06:59:41 orangepizero kernel: [19546.889443] [<c0961c4d>] (dump_stack) from [<c0965d09>] (nmi_cpu_backtrace+0x59/0x90)
May 29 06:59:41 orangepizero kernel: [19546.889447] [<c0965d09>] (nmi_cpu_backtrace) from [<c010bf21>] (handle_IPI+0x85/0x2c0)
May 29 06:59:41 orangepizero kernel: [19546.889450] [<c010bf21>] (handle_IPI) from [<c05c8613>] (gic_handle_irq+0x67/0x68)
May 29 06:59:41 orangepizero kernel: [19546.889454] [<c05c8613>] (gic_handle_irq) from [<c0101ae5>] (__irq_svc+0x65/0x94)
May 29 06:59:41 orangepizero kernel: [19546.889456] Exception stack(0xc0f01938 to 0xc0f01980)
May 29 06:59:41 orangepizero kernel: [19546.889459] 1920:                                                       c8663438 00000000
May 29 06:59:41 orangepizero kernel: [19546.889463] 1940: 00004933 00002080 c8663438 00004933 00002080 d5e2b740 c0f01a0b d5614000
May 29 06:59:41 orangepizero kernel: [19546.889466] 1960: 00000b84 00000022 00000001 c0f01988 c08175f3 c08171c0 20010133 ffffffff
May 29 06:59:41 orangepizero kernel: [19546.889470] [<c0101ae5>] (__irq_svc) from [<c08171c0>] (__skb_gso_segment+0xc/0x140)
May 29 06:59:41 orangepizero kernel: [19546.889473] [<c08171c0>] (__skb_gso_segment) from [<c08175f3>] (validate_xmit_skb+0xe3/0x1f8)
May 29 06:59:41 orangepizero kernel: [19546.889477] [<c08175f3>] (validate_xmit_skb) from [<c081773d>] (validate_xmit_skb_list+0x35/0x48)
May 29 06:59:41 orangepizero kernel: [19546.889481] [<c081773d>] (validate_xmit_skb_list) from [<c0850ba5>] (sch_direct_xmit+0x85/0x2a8)
May 29 06:59:41 orangepizero kernel: [19546.889484] [<c0850ba5>] (sch_direct_xmit) from [<c0818127>] (__dev_queue_xmit+0x693/0x7d0)
May 29 06:59:41 orangepizero kernel: [19546.889488] [<c0818127>] (__dev_queue_xmit) from [<c086dbb1>] (ip_finish_output2+0x1bd/0x474)
May 29 06:59:41 orangepizero kernel: [19546.889491] [<c086dbb1>] (ip_finish_output2) from [<c086faeb>] (ip_output+0x9b/0xe0)
May 29 06:59:41 orangepizero kernel: [19546.889494] [<c086faeb>] (ip_output) from [<c086f659>] (__ip_queue_xmit+0xe9/0x2ec)
May 29 06:59:41 orangepizero kernel: [19546.889498] [<c086f659>] (__ip_queue_xmit) from [<c0883a51>] (__tcp_transmit_skb+0x3e5/0x7c4)
May 29 06:59:41 orangepizero kernel: [19546.889501] [<c0883a51>] (__tcp_transmit_skb) from [<c0884797>] (tcp_write_xmit+0x17f/0xcb8)
May 29 06:59:41 orangepizero kernel: [19546.889505] [<c0884797>] (tcp_write_xmit) from [<c08852f9>] (__tcp_push_pending_frames+0x29/0xa8)
May 29 06:59:41 orangepizero kernel: [19546.889509] [<c08852f9>] (__tcp_push_pending_frames) from [<c0881087>] (tcp_rcv_established+0x49f/0x528)
May 29 06:59:41 orangepizero kernel: [19546.889513] [<c0881087>] (tcp_rcv_established) from [<c0888a7b>] (tcp_v4_do_rcv+0xe7/0x144)
May 29 06:59:41 orangepizero kernel: [19546.889516] [<c0888a7b>] (tcp_v4_do_rcv) from [<c088ad4b>] (tcp_v4_rcv+0x8bb/0xa78)
May 29 06:59:41 orangepizero kernel: [19546.889520] [<c088ad4b>] (tcp_v4_rcv) from [<c086b6b1>] (ip_protocol_deliver_rcu+0x25/0x240)
May 29 06:59:41 orangepizero kernel: [19546.889523] [<c086b6b1>] (ip_protocol_deliver_rcu) from [<c086b907>] (ip_local_deliver_finish+0x3b/0x40)
May 29 06:59:41 orangepizero kernel: [19546.889527] [<c086b907>] (ip_local_deliver_finish) from [<c086b943>] (ip_local_deliver+0x37/0x88)
May 29 06:59:41 orangepizero kernel: [19546.889531] [<c086b943>] (ip_local_deliver) from [<c086b9bf>] (ip_rcv+0x2b/0x64)
May 29 06:59:41 orangepizero kernel: [19546.889534] [<c086b9bf>] (ip_rcv) from [<c0818f15>] (__netif_receive_skb_one_core+0x29/0x2c)
May 29 06:59:41 orangepizero kernel: [19546.889539] [<c0818f15>] (__netif_receive_skb_one_core) from [<c0818f9b>] (netif_receive_skb_internal+0x23/0x70)
May 29 06:59:41 orangepizero kernel: [19546.889542] [<c0818f9b>] (netif_receive_skb_internal) from [<c0819671>] (napi_gro_receive+0xf9/0x114)
May 29 06:59:41 orangepizero kernel: [19546.889546] [<c0819671>] (napi_gro_receive) from [<c07169bd>] (stmmac_napi_poll_rx+0x2c5/0x5e4)
May 29 06:59:41 orangepizero kernel: [19546.889550] [<c07169bd>] (stmmac_napi_poll_rx) from [<c0819b91>] (net_rx_action+0x105/0x300)
May 29 06:59:41 orangepizero kernel: [19546.889553] [<c0819b91>] (net_rx_action) from [<c01022f9>] (__do_softirq+0xe1/0x2a4)
May 29 06:59:41 orangepizero kernel: [19546.889556] [<c01022f9>] (__do_softirq) from [<c011efa3>] (irq_exit+0x87/0x98)
May 29 06:59:41 orangepizero kernel: [19546.889560] [<c011efa3>] (irq_exit) from [<c015dcd1>] (__handle_domain_irq+0x49/0x84)
May 29 06:59:41 orangepizero kernel: [19546.889563] [<c015dcd1>] (__handle_domain_irq) from [<c05c85e5>] (gic_handle_irq+0x39/0x68)
May 29 06:59:41 orangepizero kernel: [19546.889566] [<c05c85e5>] (gic_handle_irq) from [<c0101ae5>] (__irq_svc+0x65/0x94)
May 29 06:59:41 orangepizero kernel: [19546.889569] Exception stack(0xc0f01f20 to 0xc0f01f68)
May 29 06:59:41 orangepizero kernel: [19546.889572] 1f20: 00000000 01da487c dff5dfb0 c0115841 ffffe000 c0f04f64 c0f04fa8 00000001
May 29 06:59:41 orangepizero kernel: [19546.889576] 1f40: 00000000 c0ec08b0 00000000 c0ff6863 000729a4 c0f01f70 c01075f7 c01075f8
May 29 06:59:41 orangepizero kernel: [19546.889578] 1f60: 40010033 ffffffff
May 29 06:59:41 orangepizero kernel: [19546.889581] [<c0101ae5>] (__irq_svc) from [<c01075f8>] (arch_cpu_idle+0x28/0x2c)
May 29 06:59:41 orangepizero kernel: [19546.889584] [<c01075f8>] (arch_cpu_idle) from [<c013cb7d>] (do_idle+0x12d/0x1b4)
May 29 06:59:41 orangepizero kernel: [19546.889588] [<c013cb7d>] (do_idle) from [<c013cdd9>] (cpu_startup_entry+0x19/0x20)
May 29 06:59:41 orangepizero kernel: [19546.889591] [<c013cdd9>] (cpu_startup_entry) from [<c0e00c17>] (start_kernel+0x3cf/0x3e4)
May 29 06:59:41 orangepizero kernel: [19546.889862] rcu_sched       I    0    10      2 0x00000000
May 29 06:59:41 orangepizero kernel: [19546.889888] [<c0971dd5>] (__schedule) from [<c097216d>] (schedule+0x35/0x84)
May 29 06:59:41 orangepizero kernel: [19546.889899] [<c097216d>] (schedule) from [<c0974595>] (schedule_timeout+0x69/0x264)
May 29 06:59:41 orangepizero kernel: [19546.889917] [<c0974595>] (schedule_timeout) from [<c0169e9f>] (rcu_gp_kthread+0x583/0x79c)
May 29 06:59:41 orangepizero kernel: [19546.889934] [<c0169e9f>] (rcu_gp_kthread) from [<c0131819>] (kthread+0xf9/0x100)
May 29 06:59:41 orangepizero kernel: [19546.889946] [<c0131819>] (kthread) from [<c01010f9>] (ret_from_fork+0x11/0x38)
May 29 06:59:41 orangepizero kernel: [19546.889951] Exception stack(0xd712ffb0 to 0xd712fff8)
May 29 06:59:41 orangepizero kernel: [19546.889959] ffa0:                                     00000000 00000000 00000000 00000000
May 29 06:59:41 orangepizero kernel: [19546.889969] ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
May 29 06:59:41 orangepizero kernel: [19546.889976] ffe0: 00000000 00000000 00000000 00000000 00000013 00000000
May 29 07:24:47 orangepizero kernel: [21053.048265] printk: systemd-journal: 141 output lines suppressed due to ratelimiting

请问此问题有什么引起以及该如何修复。

orangepi-xunlong commented 2 years ago

你好，光从log是猜不出原因的，你用的哪个镜像，安装了什么软件，做了什么操作都是需要说明白的，另外很重要的一点是此现象能不能稳定复现，如果没法复现是找不出原因以及解决办法来验证的

Longhao-Chen commented 2 years ago

这个可能是 Linux内核 的问题，我后来把CPU频率调成 performance 模式或 powersave 模式就没有出现此过问题。出现问题的内核版本应该是 5.3.5。

这个问题会不定时的出现，没有找到规律。

怀疑可能是CPU频率调节的时候某个核心没有响应，触发了 NMI watchdog ，因为我对Linux内核不熟悉，所以没有继续探究下去。

下面的这一份日志可能能够提供一些有用的信息： 在这份日志中后面的时间出现了错乱，是否可能是 hrtimer 的问题。

Oct 18 05:09:50 orangepizero kernel: [24430.223182] rcu: INFO: rcu_sched self-detected stall on CPU
Oct 18 05:09:50 orangepizero kernel: [24430.223205] rcu:        1-...!: (1 GPs behind) idle=41a/0/0x1 softirq=90558/90559 fqs=0 
Oct 18 05:09:50 orangepizero kernel: [24430.223212]     (t=519689 jiffies g=211057 q=2)
Oct 18 05:09:50 orangepizero kernel: [24430.223220] rcu: rcu_sched kthread starved for 519689 jiffies! g211057 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
Oct 18 05:09:50 orangepizero kernel: [24430.223223] rcu: RCU grace-period kthread stack dump:
Oct 18 05:09:50 orangepizero kernel: [24430.223228] rcu_sched       I    0    10      2 0x00000000
Oct 18 05:09:50 orangepizero kernel: [24430.223265] [<c0971dd5>] (__schedule) from [<c097216d>] (schedule+0x35/0x84)
Oct 18 05:09:50 orangepizero kernel: [24430.223278] [<c097216d>] (schedule) from [<c0974595>] (schedule_timeout+0x69/0x264)
Oct 18 05:09:50 orangepizero kernel: [24430.223296] [<c0974595>] (schedule_timeout) from [<c0169e9f>] (rcu_gp_kthread+0x583/0x79c)
Oct 18 05:09:50 orangepizero kernel: [24430.223312] [<c0169e9f>] (rcu_gp_kthread) from [<c0131819>] (kthread+0xf9/0x100)
Oct 18 05:09:50 orangepizero kernel: [24430.223326] [<c0131819>] (kthread) from [<c01010f9>] (ret_from_fork+0x11/0x38)
Oct 18 05:09:50 orangepizero kernel: [24430.223331] Exception stack(0xd712ffb0 to 0xd712fff8)
Oct 18 05:09:50 orangepizero kernel: [24430.223339] ffa0:                                     00000000 00000000 00000000 00000000
Oct 18 05:09:50 orangepizero kernel: [24430.223347] ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Oct 18 05:09:50 orangepizero kernel: [24430.223354] ffe0: 00000000 00000000 00000000 00000000 00000013 00000000
Oct 18 05:09:50 orangepizero kernel: [24430.223388] Sending NMI from CPU 1 to CPUs 0:
Oct 18 05:09:50 orangepizero kernel: [24440.223801] NMI backtrace for cpu 1
Oct 18 05:09:50 orangepizero kernel: [24440.223812] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G         C        5.3.5 #1
Oct 18 05:09:50 orangepizero kernel: [24440.223815] Hardware name: Allwinner sun8i Family
Oct 18 05:09:50 orangepizero kernel: [24440.223834] [<c010cfb9>] (unwind_backtrace) from [<c0109cdd>] (show_stack+0x11/0x14)
Oct 18 05:09:50 orangepizero kernel: [24440.223848] [<c0109cdd>] (show_stack) from [<c0961c4d>] (dump_stack+0x69/0x78)
Oct 18 05:09:50 orangepizero kernel: [24440.223861] [<c0961c4d>] (dump_stack) from [<c0965d3f>] (nmi_cpu_backtrace+0x8f/0x90)
Oct 18 05:09:50 orangepizero kernel: [24440.223871] [<c0965d3f>] (nmi_cpu_backtrace) from [<c0965def>] (nmi_trigger_cpumask_backtrace+0xaf/0xe0)
Oct 18 05:09:50 orangepizero kernel: [24440.223882] [<c0965def>] (nmi_trigger_cpumask_backtrace) from [<c016b2bd>] (rcu_dump_cpu_stacks+0x83/0xa2)
Oct 18 05:09:50 orangepizero kernel: [24440.223892] [<c016b2bd>] (rcu_dump_cpu_stacks) from [<c016a957>] (rcu_sched_clock_irq+0x58b/0x6b8)
Oct 18 05:09:50 orangepizero kernel: [24440.223905] [<c016a957>] (rcu_sched_clock_irq) from [<c016ffab>] (update_process_times+0x2b/0x48)
Oct 18 05:09:50 orangepizero kernel: [24440.223919] [<c016ffab>] (update_process_times) from [<c017dab3>] (tick_sched_timer+0x37/0x74)
Oct 18 05:09:50 orangepizero kernel: [24440.223932] [<c017dab3>] (tick_sched_timer) from [<c01707c1>] (__hrtimer_run_queues+0xf1/0x234)
Oct 18 05:09:50 orangepizero kernel: [24440.223942] [<c01707c1>] (__hrtimer_run_queues) from [<c017125d>] (hrtimer_interrupt+0xb5/0x200)
Oct 18 05:09:50 orangepizero kernel: [24440.223954] [<c017125d>] (hrtimer_interrupt) from [<c07dc831>] (arch_timer_handler_phys+0x25/0x28)
Oct 18 05:09:50 orangepizero kernel: [24440.223967] [<c07dc831>] (arch_timer_handler_phys) from [<c016182f>] (handle_percpu_devid_irq+0x57/0x19c)
Oct 18 05:09:50 orangepizero kernel: [24440.223980] [<c016182f>] (handle_percpu_devid_irq) from [<c015d81d>] (generic_handle_irq+0x1d/0x28)
Oct 18 05:09:50 orangepizero kernel: [24440.223991] [<c015d81d>] (generic_handle_irq) from [<c015dccd>] (__handle_domain_irq+0x45/0x84)
Oct 18 05:09:50 orangepizero kernel: [24440.224004] [<c015dccd>] (__handle_domain_irq) from [<c05c85e5>] (gic_handle_irq+0x39/0x68)
Oct 18 05:09:50 orangepizero kernel: [24440.224016] [<c05c85e5>] (gic_handle_irq) from [<c0101ae5>] (__irq_svc+0x65/0x94)
Oct 18 05:09:50 orangepizero kernel: [24440.224020] Exception stack(0xd7133f60 to 0xd7133fa8)
Oct 18 05:09:50 orangepizero kernel: [24440.224029] 3f60: 00000000 00631418 dff71fb0 c0115841 ffffe000 c0f04f64 c0f04fa8 00000002
Oct 18 05:09:50 orangepizero kernel: [24440.224039] 3f80: 00000000 c0ec08b0 00000000 c0ff6863 c0f03d00 d7133fb0 c01075f7 c01075f8
Oct 18 05:09:50 orangepizero kernel: [24440.224043] 3fa0: 40000033 ffffffff
Oct 18 05:09:50 orangepizero kernel: [24440.224056] [<c0101ae5>] (__irq_svc) from [<c01075f8>] (arch_cpu_idle+0x28/0x2c)
Oct 18 05:09:50 orangepizero kernel: [24440.224069] [<c01075f8>] (arch_cpu_idle) from [<c013cb7d>] (do_idle+0x12d/0x1b4)
Oct 18 05:09:50 orangepizero kernel: [24440.224080] [<c013cb7d>] (do_idle) from [<c013cdd9>] (cpu_startup_entry+0x19/0x20)
Oct 18 05:09:50 orangepizero kernel: [24440.224090] [<c013cdd9>] (cpu_startup_entry) from [<40102551>] (0x40102551)
Oct 18 05:09:50 orangepizero kernel: [24440.224100] Sending NMI from CPU 1 to CPUs 2:
Oct 18 05:09:50 orangepizero kernel: [24440.224267] NMI backtrace for cpu 2
Oct 18 05:09:50 orangepizero kernel: [24440.224272] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G         C        5.3.5 #1
Oct 18 05:09:50 orangepizero kernel: [24440.224274] Hardware name: Allwinner sun8i Family
Oct 18 05:09:50 orangepizero kernel: [24440.224276] PC is at arch_cpu_idle+0x28/0x2c
Oct 18 05:09:50 orangepizero kernel: [24440.224279] LR is at arch_cpu_idle+0x27/0x2c
Oct 18 05:09:50 orangepizero kernel: [24440.224281] pc : [<c01075f8>]    lr : [<c01075f7>]    psr: 40000033
Oct 18 05:09:50 orangepizero kernel: [24440.224284] sp : d7135fb0  ip : c0f03d00  fp : c0ff6863
Oct 18 05:09:50 orangepizero kernel: [24440.224286] r10: 00000000  r9 : c0ec08b0  r8 : 00000000
Oct 18 05:09:50 orangepizero kernel: [24440.224289] r7 : 00000004  r6 : c0f04fa8  r5 : c0f04f64  r4 : ffffe000
Oct 18 05:09:50 orangepizero kernel: [24440.224292] r3 : c0115841  r2 : dff85fb0  r1 : 001e3574  r0 : 00000000
Oct 18 05:09:50 orangepizero kernel: [24440.224295] Flags: nZcv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment none
Oct 18 05:09:50 orangepizero kernel: [24440.224298] Control: 50c5387d  Table: 564b806a  DAC: 00000051
Oct 18 05:09:50 orangepizero kernel: [24440.224301] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G         C        5.3.5 #1
Oct 18 05:09:50 orangepizero kernel: [24440.224303] Hardware name: Allwinner sun8i Family
Oct 18 05:09:50 orangepizero kernel: [24440.224306] [<c010cfb9>] (unwind_backtrace) from [<c0109cdd>] (show_stack+0x11/0x14)
Oct 18 05:09:50 orangepizero kernel: [24440.224310] [<c0109cdd>] (show_stack) from [<c0961c4d>] (dump_stack+0x69/0x78)
Oct 18 05:09:50 orangepizero kernel: [24440.224313] [<c0961c4d>] (dump_stack) from [<c0965d09>] (nmi_cpu_backtrace+0x59/0x90)
Oct 18 05:09:50 orangepizero kernel: [24440.224316] [<c0965d09>] (nmi_cpu_backtrace) from [<c010bf21>] (handle_IPI+0x85/0x2c0)
Oct 18 05:09:50 orangepizero kernel: [24440.224319] [<c010bf21>] (handle_IPI) from [<c05c8613>] (gic_handle_irq+0x67/0x68)
Oct 18 05:09:50 orangepizero kernel: [24440.224322] [<c05c8613>] (gic_handle_irq) from [<c0101ae5>] (__irq_svc+0x65/0x94)
Oct 18 05:09:50 orangepizero kernel: [24440.224325] Exception stack(0xd7135f60 to 0xd7135fa8)
Oct 18 05:09:50 orangepizero kernel: [24440.224328] 5f60: 00000000 001e3574 dff85fb0 c0115841 ffffe000 c0f04f64 c0f04fa8 00000004
Oct 18 05:09:50 orangepizero kernel: [24440.224331] 5f80: 00000000 c0ec08b0 00000000 c0ff6863 c0f03d00 d7135fb0 c01075f7 c01075f8
Oct 18 05:09:50 orangepizero kernel: [24440.224333] 5fa0: 40000033 ffffffff
Oct 18 05:09:50 orangepizero kernel: [24440.224337] [<c0101ae5>] (__irq_svc) from [<c01075f8>] (arch_cpu_idle+0x28/0x2c)
Oct 18 05:09:50 orangepizero kernel: [24440.224340] [<c01075f8>] (arch_cpu_idle) from [<c013cb7d>] (do_idle+0x12d/0x1b4)
Oct 18 05:09:50 orangepizero kernel: [24440.224343] [<c013cb7d>] (do_idle) from [<c013cdd9>] (cpu_startup_entry+0x19/0x20)
Oct 18 05:09:50 orangepizero kernel: [24440.224346] [<c013cdd9>] (cpu_startup_entry) from [<40102551>] (0x40102551)
Oct 18 05:09:50 orangepizero kernel: [24440.225107] Sending NMI from CPU 1 to CPUs 3:
Oct 18 05:10:00 orangepizero kernel: [24450.225770] NMI backtrace for cpu 3
Oct 18 05:10:00 orangepizero kernel: [24450.225774] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G         C        5.3.5 #1
Oct 18 05:10:00 orangepizero kernel: [24450.225777] Hardware name: Allwinner sun8i Family
Oct 18 05:10:00 orangepizero kernel: [24450.225779] PC is at _raw_spin_unlock_irqrestore+0x1c/0x20
Oct 18 05:10:00 orangepizero kernel: [24450.225781] LR is at rcu_core+0x2ed/0x3a0
Oct 18 05:10:00 orangepizero kernel: [24450.225784] pc : [<c0975110>]    lr : [<c0169645>]    psr: 00010133
Oct 18 05:10:00 orangepizero kernel: [24450.225787] sp : d7137e98  ip : 00000000  fp : c0f04f64
Oct 18 05:10:00 orangepizero kernel: [24450.225789] r10: 60010113  r9 : c0f11140  r8 : c0f11140
Oct 18 05:10:00 orangepizero kernel: [24450.225792] r7 : 1f0d8000  r6 : c0f04f64  r5 : c0ec1f40  r4 : dff99f40
Oct 18 05:10:00 orangepizero kernel: [24450.225795] r3 : 00002d91  r2 : 00033859  r1 : 60010113  r0 : c0f11140
Oct 18 05:10:00 orangepizero kernel: [24450.225798] Flags: nzcv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment none
Oct 18 05:10:00 orangepizero kernel: [24450.225801] Control: 50c5387d  Table: 4413806a  DAC: 00000051
Oct 18 05:10:00 orangepizero kernel: [24450.225804] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G         C        5.3.5 #1
Oct 18 05:10:00 orangepizero kernel: [24450.225806] Hardware name: Allwinner sun8i Family
Oct 18 05:10:00 orangepizero kernel: [24450.225810] [<c010cfb9>] (unwind_backtrace) from [<c0109cdd>] (show_stack+0x11/0x14)
Oct 18 05:10:00 orangepizero kernel: [24450.225813] [<c0109cdd>] (show_stack) from [<c0961c4d>] (dump_stack+0x69/0x78)
Oct 18 05:10:00 orangepizero kernel: [24450.225816] [<c0961c4d>] (dump_stack) from [<c0965d09>] (nmi_cpu_backtrace+0x59/0x90)
Oct 18 05:10:00 orangepizero kernel: [24450.225820] [<c0965d09>] (nmi_cpu_backtrace) from [<c010bf21>] (handle_IPI+0x85/0x2c0)
Oct 18 05:10:00 orangepizero kernel: [24450.225823] [<c010bf21>] (handle_IPI) from [<c05c8613>] (gic_handle_irq+0x67/0x68)
Oct 18 05:10:00 orangepizero kernel: [24450.225826] [<c05c8613>] (gic_handle_irq) from [<c0101ae5>] (__irq_svc+0x65/0x94)
Oct 18 05:10:00 orangepizero kernel: [24450.225828] Exception stack(0xd7137e48 to 0xd7137e90)
Oct 18 05:10:00 orangepizero kernel: [24450.225832] 7e40:                   c0f11140 60010113 00033859 00002d91 dff99f40 c0ec1f40
Oct 18 05:10:00 orangepizero kernel: [24450.225835] 7e60: c0f04f64 1f0d8000 c0f11140 c0f11140 60010113 c0f04f64 00000000 d7137e98
Oct 18 05:10:00 orangepizero kernel: [24450.225837] 7e80: c0169645 c0975110 00010133 ffffffff
Oct 18 05:10:00 orangepizero kernel: [24450.225841] [<c0101ae5>] (__irq_svc) from [<c0975110>] (_raw_spin_unlock_irqrestore+0x1c/0x20)
Oct 18 05:10:00 orangepizero kernel: [24450.225844] [<c0975110>] (_raw_spin_unlock_irqrestore) from [<c0169645>] (rcu_core+0x2ed/0x3a0)
Oct 18 05:10:00 orangepizero kernel: [24450.225847] [<c0169645>] (rcu_core) from [<c01022f9>] (__do_softirq+0xe1/0x2a4)
Oct 18 05:10:00 orangepizero kernel: [24450.225850] [<c01022f9>] (__do_softirq) from [<c011efa3>] (irq_exit+0x87/0x98)
Oct 18 05:10:00 orangepizero kernel: [24450.225854] [<c011efa3>] (irq_exit) from [<c015dcd1>] (__handle_domain_irq+0x49/0x84)
Oct 18 05:10:00 orangepizero kernel: [24450.225857] [<c015dcd1>] (__handle_domain_irq) from [<c05c85e5>] (gic_handle_irq+0x39/0x68)
Oct 18 05:10:00 orangepizero kernel: [24450.225860] [<c05c85e5>] (gic_handle_irq) from [<c0101ae5>] (__irq_svc+0x65/0x94)
Oct 18 05:10:00 orangepizero kernel: [24450.225863] Exception stack(0xd7137f60 to 0xd7137fa8)
Oct 18 05:10:00 orangepizero kernel: [24450.225866] 7f60: 00000000 000d8e64 dff99fb0 c0115841 ffffe000 c0f04f64 c0f04fa8 00000008
Oct 18 05:10:00 orangepizero kernel: [24450.225870] 7f80: 00000000 c0ec08b0 00000000 c0ff6863 ffff5e89 d7137fb0 c01075f7 c01075f8
Oct 18 05:10:00 orangepizero kernel: [24450.225872] 7fa0: 40010033 ffffffff
Oct 18 05:10:00 orangepizero kernel: [24450.225875] [<c0101ae5>] (__irq_svc) from [<c01075f8>] (arch_cpu_idle+0x28/0x2c)
Oct 18 05:10:00 orangepizero kernel: [24450.225878] [<c01075f8>] (arch_cpu_idle) from [<c013cb7d>] (do_idle+0x12d/0x1b4)
Oct 18 05:10:00 orangepizero kernel: [24450.225881] [<c013cb7d>] (do_idle) from [<c013cdd9>] (cpu_startup_entry+0x19/0x20)
Oct 18 05:10:00 orangepizero kernel: [24450.225884] [<c013cdd9>] (cpu_startup_entry) from [<40102551>] (0x40102551)
Nov  2 06:31:25 orangepizero kernel: [24967.219361] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
Nov  2 06:31:25 orangepizero kernel: [24967.219385] rcu:        0-...!: (1 ticks this GP) idle=546/0/0x1 softirq=63811/63811 fqs=1 
Nov  2 06:31:25 orangepizero kernel: [24967.219394]     (detected by 1, t=159661235 jiffies, g=211057, q=607)
Nov  2 06:31:25 orangepizero kernel: [24967.219408] Sending NMI from CPU 1 to CPUs 0:
Nov  2 06:31:25 orangepizero kernel: [24967.219783] NMI backtrace for cpu 0
Nov  2 06:31:25 orangepizero kernel: [24967.219788] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C        5.3.5 #1
Nov  2 06:31:25 orangepizero kernel: [24967.219791] Hardware name: Allwinner sun8i Family
Nov  2 06:31:25 orangepizero kernel: [24967.219793] PC is at ___bpf_prog_run+0xd36/0x18b8
Nov  2 06:31:25 orangepizero kernel: [24967.219795] LR is at __bpf_prog_run32+0x1f/0x24
Nov  2 06:31:25 orangepizero kernel: [24967.219798] pc : [<c01bed5a>]    lr : [<c01bff47>]    psr: 80000133
Nov  2 06:31:25 orangepizero kernel: [24967.219801] sp : c0f01c88  ip : c40e9824  fp : c0a11ea4
Nov  2 06:31:25 orangepizero kernel: [24967.219803] r10: c0f01d18  r9 : 00000000  r8 : c01bdff9
Nov  2 06:31:25 orangepizero kernel: [24967.219806] r7 : d7c54ba0  r6 : 00000000  r5 : 0000002b  r4 : e08110f0
Nov  2 06:31:25 orangepizero kernel: [24967.219809] r3 : c01bed5b  r2 : 0000002b  r1 : 00000071  r0 : 00000001
Nov  2 06:31:25 orangepizero kernel: [24967.219812] Flags: Nzcv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment none
Nov  2 06:31:25 orangepizero kernel: [24967.219815] Control: 50c5387d  Table: 4ee9006a  DAC: 00000051
Nov  2 06:31:25 orangepizero kernel: [24967.219818] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C        5.3.5 #1
Nov  2 06:31:25 orangepizero kernel: [24967.219820] Hardware name: Allwinner sun8i Family
Nov  2 06:31:25 orangepizero kernel: [24967.219824] [<c010cfb9>] (unwind_backtrace) from [<c0109cdd>] (show_stack+0x11/0x14)
Nov  2 06:31:25 orangepizero kernel: [24967.219827] [<c0109cdd>] (show_stack) from [<c0961c4d>] (dump_stack+0x69/0x78)
Nov  2 06:31:25 orangepizero kernel: [24967.219830] [<c0961c4d>] (dump_stack) from [<c0965d09>] (nmi_cpu_backtrace+0x59/0x90)
Nov  2 06:31:25 orangepizero kernel: [24967.219833] [<c0965d09>] (nmi_cpu_backtrace) from [<c010bf21>] (handle_IPI+0x85/0x2c0)
Nov  2 06:31:25 orangepizero kernel: [24967.219836] [<c010bf21>] (handle_IPI) from [<c05c8613>] (gic_handle_irq+0x67/0x68)
Nov  2 06:31:25 orangepizero kernel: [24967.219840] [<c05c8613>] (gic_handle_irq) from [<c0101ae5>] (__irq_svc+0x65/0x94)
Nov  2 06:31:25 orangepizero kernel: [24967.219842] Exception stack(0xc0f01c38 to 0xc0f01c80)
Nov  2 06:31:25 orangepizero kernel: [24967.219845] 1c20:                                                       00000001 00000071
Nov  2 06:31:25 orangepizero kernel: [24967.219849] 1c40: 0000002b c01bed5b e08110f0 0000002b 00000000 d7c54ba0 c01bdff9 00000000
Nov  2 06:31:25 orangepizero kernel: [24967.219852] 1c60: c0f01d18 c0a11ea4 c40e9824 c0f01c88 c01bff47 c01bed5a 80000133 ffffffff
Nov  2 06:31:25 orangepizero kernel: [24967.219855] [<c0101ae5>] (__irq_svc) from [<c01bed5a>] (___bpf_prog_run+0xd36/0x18b8)
Nov  2 06:31:25 orangepizero kernel: [24967.219859] [<c01bed5a>] (___bpf_prog_run) from [<c01bff47>] (__bpf_prog_run32+0x1f/0x24)
Nov  2 06:31:25 orangepizero kernel: [24967.219862] [<c01bff47>] (__bpf_prog_run32) from [<c084418d>] (skb_defer_rx_timestamp+0x39/0x74)
Nov  2 06:31:25 orangepizero kernel: [24967.219866] [<c084418d>] (skb_defer_rx_timestamp) from [<c0818f8f>] (netif_receive_skb_internal+0x17/0x70)
Nov  2 06:31:25 orangepizero kernel: [24967.219869] [<c0818f8f>] (netif_receive_skb_internal) from [<c0819671>] (napi_gro_receive+0xf9/0x114)
Nov  2 06:31:25 orangepizero kernel: [24967.219873] [<c0819671>] (napi_gro_receive) from [<c07169bd>] (stmmac_napi_poll_rx+0x2c5/0x5e4)
Nov  2 06:31:25 orangepizero kernel: [24967.219876] [<c07169bd>] (stmmac_napi_poll_rx) from [<c0819b91>] (net_rx_action+0x105/0x300)
Nov  2 06:31:25 orangepizero kernel: [24967.219879] [<c0819b91>] (net_rx_action) from [<c01022f9>] (__do_softirq+0xe1/0x2a4)
Nov  2 06:31:25 orangepizero kernel: [24967.219883] [<c01022f9>] (__do_softirq) from [<c011efa3>] (irq_exit+0x87/0x98)
Nov  2 06:31:25 orangepizero kernel: [24967.219886] [<c011efa3>] (irq_exit) from [<c015dcd1>] (__handle_domain_irq+0x49/0x84)
Nov  2 06:31:25 orangepizero kernel: [24967.219889] [<c015dcd1>] (__handle_domain_irq) from [<c05c85e5>] (gic_handle_irq+0x39/0x68)
Nov  2 06:31:25 orangepizero kernel: [24967.219892] [<c05c85e5>] (gic_handle_irq) from [<c0101ae5>] (__irq_svc+0x65/0x94)
Nov  2 06:31:25 orangepizero kernel: [24967.219895] Exception stack(0xc0f01f20 to 0xc0f01f68)
Nov  2 06:31:25 orangepizero kernel: [24967.219898] 1f20: 00000000 00ac6544 dff5dfb0 c0115841 ffffe000 c0f04f64 c0f04fa8 00000001
Nov  2 06:31:25 orangepizero kernel: [24967.219901] 1f40: 00000000 c0ec08b0 00000000 c0ff6863 00012782 c0f01f70 c01075f7 c01075f8
Nov  2 06:31:25 orangepizero kernel: [24967.219903] 1f60: 40000033 ffffffff
Nov  2 06:31:25 orangepizero kernel: [24967.219906] [<c0101ae5>] (__irq_svc) from [<c01075f8>] (arch_cpu_idle+0x28/0x2c)
Nov  2 06:31:25 orangepizero kernel: [24967.219909] [<c01075f8>] (arch_cpu_idle) from [<c013cb7d>] (do_idle+0x12d/0x1b4)
Nov  2 06:31:25 orangepizero kernel: [24967.219912] [<c013cb7d>] (do_idle) from [<c013cdd9>] (cpu_startup_entry+0x19/0x20)
Nov  2 06:31:25 orangepizero kernel: [24967.219916] [<c013cdd9>] (cpu_startup_entry) from [<c0e00c17>] (start_kernel+0x3cf/0x3e4)
Nov  2 06:31:25 orangepizero kernel: [24967.220426] rcu: rcu_sched kthread starved for 159141546 jiffies! g211057 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=2
Nov  2 06:31:25 orangepizero kernel: [24967.220429] rcu: RCU grace-period kthread stack dump:
Nov  2 06:31:25 orangepizero kernel: [24967.220434] rcu_sched       R  running task        0    10      2 0x00000000
Nov  2 06:31:25 orangepizero kernel: [24967.220464] [<c0971dd5>] (__schedule) from [<c097216d>] (schedule+0x35/0x84)
Nov  2 06:31:25 orangepizero kernel: [24967.220476] [<c097216d>] (schedule) from [<c0974595>] (schedule_timeout+0x69/0x264)
Nov  2 06:31:25 orangepizero kernel: [24967.220492] [<c0974595>] (schedule_timeout) from [<c0169e9f>] (rcu_gp_kthread+0x583/0x79c)
Nov  2 06:31:25 orangepizero kernel: [24967.220510] [<c0169e9f>] (rcu_gp_kthread) from [<c0131819>] (kthread+0xf9/0x100)
Nov  2 06:31:25 orangepizero kernel: [24967.220523] [<c0131819>] (kthread) from [<c01010f9>] (ret_from_fork+0x11/0x38)
Nov  2 06:31:25 orangepizero kernel: [24967.220528] Exception stack(0xd712ffb0 to 0xd712fff8)
Nov  2 06:31:25 orangepizero kernel: [24967.220536] ffa0:                                     00000000 00000000 00000000 00000000
Nov  2 06:31:25 orangepizero kernel: [24967.220545] ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Nov  2 06:31:25 orangepizero kernel: [24967.220553] ffe0: 00000000 00000000 00000000 00000000 00000013 00000000
orangepi-xunlong commented 2 years ago

你好，Linux5.3.5还是很老的镜像了，目前H3的开发板都已经更新到Linux5.4的系统了，可以试下最新的系统有没有这个问题哈。