Unable to handle kernel NULL pointer dereference

[AlBundy33]

I've updated my raspberry pi 3 with rpi-update to Linux rpi 4.14.30-v7+ #1102 SMP Mon Mar 26 16:45:49 BST 2018 armv7l GNU/Linux

But now I have some NULL pointer dereferences in my syslog.

Mar 27 20:40:42 rpi tvheadend[13605]: mpegts: 442MHz in FritzBox - tuning on SAT>IP DVB-C Tuner #1 (192.168.1.1)
Mar 27 20:40:42 rpi tvheadend[13605]: subscription: 0012: "epggrab" subscribing to mux "442MHz", weight: 4, adapter: "SAT>IP DVB-C Tuner #1 (192.168.1.1)", network: "FritzBox", service: "Raw PID Subscription" 
Mar 27 20:40:43 rpi kernel: [173275.768022] Unable to handle kernel NULL pointer dereference at virtual address 00000028
Mar 27 20:40:43 rpi kernel: [173275.771557] pgd = 89e28000
Mar 27 20:40:43 rpi kernel: [173275.773239] [00000028] *pgd=00000000
Mar 27 20:40:43 rpi kernel: [173275.774904] Internal error: Oops: 5 [#3] SMP ARM
Mar 27 20:40:43 rpi kernel: [173275.776543] Modules linked in: rpcsec_gss_krb5 fuse sg brcmfmac brcmutil cfg80211 rfkill snd_bcm2835(C) snd_pcm snd_timer snd fixed uio_pdrv_genirq uio ip_tables x_tables ipv6
Mar 27 20:40:43 rpi kernel: [173275.781496] CPU: 2 PID: 13919 Comm: tvh:tcp-start Tainted: G      D  C      4.14.29-v7+ #1101
Mar 27 20:40:43 rpi kernel: [173275.784697] Hardware name: BCM2835
Mar 27 20:40:43 rpi kernel: [173275.786242] task: b2429e00 task.stack: b361c000
Mar 27 20:40:43 rpi kernel: [173275.787777] PC is at tcp_push+0x44/0x148
Mar 27 20:40:43 rpi kernel: [173275.789243] LR is at tcp_sendmsg_locked+0x308/0xdfc
Mar 27 20:40:43 rpi kernel: [173275.790672] pc : [<806ce28c>]    lr : [<806d1858>]    psr: 40000013
Mar 27 20:40:43 rpi kernel: [173275.792099] sp : b361dd78  ip : b9b845bc  fp : b361dd9c
Mar 27 20:40:43 rpi kernel: [173275.793491] r10: 825c8300  r9 : ffffffe0  r8 : b9b845bc
Mar 27 20:40:43 rpi kernel: [173275.794837] r7 : 00000000  r6 : 000005a8  r5 : 00000000  r4 : b9b844c0
Mar 27 20:40:43 rpi kernel: [173275.796170] r3 : 00000001  r2 : 000005a8  r1 : 00000000  r0 : 841a6f00
Mar 27 20:40:43 rpi kernel: [173275.797439] Flags: nZcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Mar 27 20:40:43 rpi kernel: [173275.798712] Control: 10c5383d  Table: 09e2806a  DAC: 00000055
Mar 27 20:40:43 rpi kernel: [173275.799946] Process tvh:tcp-start (pid: 13919, stack limit = 0xb361c210)
Mar 27 20:40:43 rpi kernel: [173275.801180] Stack: (0xb361dd78 to 0xb361e000)
Mar 27 20:40:43 rpi kernel: [173275.802419] dd60:                                                       000005a8 806d1838
Mar 27 20:40:43 rpi kernel: [173275.804897] dd80: 0000dc98 b361de88 b9b844c0 0000dc98 b361de34 b361dda0 806d1858 806ce254
Mar 27 20:40:43 rpi kernel: [173275.807407] dda0: 000005a8 00000000 baf57b98 014200ca 80c069c4 80d0a08c 80c049c0 b361c020
Mar 27 20:40:43 rpi kernel: [173275.810039] ddc0: 00000000 00000000 00000000 00000000 00000001 00000000 b361de04 00000200
Mar 27 20:40:43 rpi kernel: [173275.812625] dde0: b361de90 b9b84610 000005a8 00000000 00000000 000005a8 00000bb8 b361de08
Mar 27 20:40:43 rpi kernel: [173275.815369] de00: 801235c4 801e0000 b9b844c0 b9b844c0 00015b72 b361de88 00000000 b361df08
Mar 27 20:40:43 rpi kernel: [173275.818271] de20: b74b2540 00000000 b361de54 b361de38 806d2384 806d155c 806d234c b9b844c0
Mar 27 20:40:43 rpi kernel: [173275.821277] de40: 00000000 00000000 b361de74 b361de58 806fe478 806d2358 89e29a18 00000000
Mar 27 20:40:43 rpi kernel: [173275.824450] de60: b361def0 00000000 b361de84 b361de78 80657b8c 806fe43c b361dedc b361de88
Mar 27 20:40:43 rpi kernel: [173275.827723] de80: 80657c34 80657b74 00000000 00000000 00000001 0000dc98 00007eda b361dee8
Mar 27 20:40:43 rpi kernel: [173275.831060] dea0: 00000001 00400040 00000000 00000000 00000000 b361df08 b74b2540 b361df80
Mar 27 20:40:43 rpi kernel: [173275.834532] dec0: 00000000 00000000 00015b72 b361c000 b361df4c b361dee0 8028a930 80657ba8
Mar 27 20:40:43 rpi kernel: [173275.838071] dee0: 00015b72 b361dfb0 6863abe8 00015b72 00000001 00000000 00015b72 b361dee8
Mar 27 20:40:43 rpi kernel: [173275.841587] df00: 00000001 00400040 b74b2540 00000000 00000000 00000000 00000000 00000000
Mar 27 20:40:43 rpi kernel: [173275.845112] df20: 00000000 00000000 b74b2540 00015b72 b74b2540 6863abe8 b361df80 80108224
Mar 27 20:40:43 rpi kernel: [173275.848636] df40: b361df7c b361df50 8028ab50 8028a85c 802aa234 802a9a10 b74b2540 b74b2541
Mar 27 20:40:43 rpi kernel: [173275.852161] df60: 00015b72 6863abe8 80108224 b361c000 b361dfa4 b361df80 8028ada0 8028aaa8
Mar 27 20:40:43 rpi kernel: [173275.855677] df80: 00000000 00000000 00015b72 6863abe8 597bdf18 00000004 00000000 b361dfa8
Mar 27 20:40:43 rpi kernel: [173275.859194] dfa0: 80108080 8028ad58 00015b72 6863abe8 00000021 6863abe8 00015b72 00000000
Mar 27 20:40:43 rpi kernel: [173275.862717] dfc0: 00015b72 6863abe8 597bdf18 00000004 00000000 00000000 00000021 007dba38
Mar 27 20:40:43 rpi kernel: [173275.866240] dfe0: 00000000 68ffe388 00000000 753bf314 80000010 00000021 1e030374 52726567
Mar 27 20:40:43 rpi kernel: [173275.869775] [<806ce28c>] (tcp_push) from [<806d1858>] (tcp_sendmsg_locked+0x308/0xdfc)
Mar 27 20:40:43 rpi kernel: [173275.873287] [<806d1858>] (tcp_sendmsg_locked) from [<806d2384>] (tcp_sendmsg+0x38/0x4c)
Mar 27 20:40:43 rpi kernel: [173275.876819] [<806d2384>] (tcp_sendmsg) from [<806fe478>] (inet_sendmsg+0x48/0xe4)
Mar 27 20:40:43 rpi kernel: [173275.880350] [<806fe478>] (inet_sendmsg) from [<80657b8c>] (sock_sendmsg+0x24/0x34)
Mar 27 20:40:43 rpi kernel: [173275.883885] [<80657b8c>] (sock_sendmsg) from [<80657c34>] (sock_write_iter+0x98/0xcc)
Mar 27 20:40:43 rpi kernel: [173275.887434] [<80657c34>] (sock_write_iter) from [<8028a930>] (__vfs_write+0xe0/0x138)
Mar 27 20:40:43 rpi kernel: [173275.890979] [<8028a930>] (__vfs_write) from [<8028ab50>] (vfs_write+0xb4/0x1bc)
Mar 27 20:40:43 rpi kernel: [173275.894533] [<8028ab50>] (vfs_write) from [<8028ada0>] (SyS_write+0x54/0xb0)
Mar 27 20:40:43 rpi kernel: [173275.896362] [<8028ada0>] (SyS_write) from [<80108080>] (ret_fast_syscall+0x0/0x28)
Mar 27 20:40:43 rpi kernel: [173275.899836] Code: e155000c 03a05000 e2117902 1a000039 (e5d50028)
Mar 27 20:40:43 rpi kernel: [173275.901647] ---[ end trace 0e5784276b7b80de ]---
Mar 27 20:40:53 rpi tvheadend[13605]: subscription: 0010: "epggrab" unsubscribing

Al

[jsebosik]

Got similar crash on custom kernel 4.9.80-v7 a7b4dd27c1c0d6510b8066b91ef01be0928d8529 while watching logs of dhclient inside journald ( using Raspi 3 B+ )

[  136.718174] Unable to handle kernel NULL pointer dereference at virtual address 00000164 pgd = 80004000
[  136.744103] [00000164] *pgd=00000000
[  136.751852] Internal error: Oops: 17 [#1] PREEMPT SMP ARM
[  136.761115] Modules linked in: asix bcm2835_gpiomem uio_pdrv_genirq uio fixed iptable_filter ipv6
[  136.777970] CPU: 0 PID: 38 Comm: kworker/0:1 Not tainted 4.9.80-v7+ #37
[  136.788665] Hardware name: Generic DT based system
[  136.797569] Workqueue: events lan78xx_delayedwork
[  136.806392] task: b8610000 task.stack: b860c000
[  136.814992] PC is at lan78xx_delayedwork+0xc8/0x514
[  136.823905] LR is at __pm_runtime_resume+0x64/0x6c
[  136.832673] pc : [<8046c51c>]    lr : [<804362c0>]    psr: 60000113
[  136.832673] sp : b860de98  ip : b860de58  fp : b860deec
[  136.852096] r10: 80a02d00  r9 : 00000000  r8 : 00000001
[  136.861238] r7 : b8b9b000  r6 : 00000000  r5 : b8786500  r4 : b878657c
[  136.871701] r3 : 00000000  r2 : 0000001a  r1 : b8786000  r0 : 00000000
[  136.882074] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  136.893111] Control: 10c5383d  Table: 37f3c06a  DAC: 00000055
[  136.902706] Process kworker/0:1 (pid: 38, stack limit = 0xb860c210)
[  136.912791] Stack: (0xb860de98 to 0xb860e000)
[  136.920883] de80:                                                       00000002 80a32088
[  136.936400] dea0: 00000000 00000000 b860dedc b860deb8 801388ec 80640b6c 00000000 8093fe00
[  136.951872] dec0: b8b97e00 b8610000 b87ad000 b85eb500 b878657c b8b97a00 b8b9b000 00000000
[  136.967592] dee0: b860df24 b860def0 8012d740 8046c460 b8b97a00 00000001 b85eb518 b85eb500
[  136.983464] df00: b8b97a00 b8b97a18 00000001 b85eb518 00000008 80a02d00 b860df5c b860df28
[  136.999476] df20: 8012d970 8012d62c b85eb500 8012d938 00000000 b85e9bc0 00000000 b85eb500
[  137.015918] df40: 8012d938 00000000 00000000 00000000 b860dfac b860df60 80133448 8012d944
[  137.032460] df60: b860df94 00000000 801388ec b85eb500 00000000 00000000 b860df78 b860df78
[  137.049110] df80: 00000000 00000000 b860df88 b860df88 b85e9bc0 80133348 00000000 00000000
[  137.066074] dfa0: 00000000 b860dfb0 80107a20 80133354 00000000 00000000 00000000 00000000
[  137.083420] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  137.101066] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[  137.119067] Backtrace:
[  137.126451] [<8046c454>] (lan78xx_delayedwork) from [<8012d740>] (process_one_work+0x120/0x318)
[  137.145158]  r8:00000000 r7:b8b9b000 r6:b8b97a00 r5:b878657c r4:b85eb500
[  137.156972] [<8012d620>] (process_one_work) from [<8012d970>] (worker_thread+0x38/0x524)
[  137.175303]  r10:80a02d00 r9:00000008 r8:b85eb518 r7:00000001 r6:b8b97a18 r5:b8b97a00
[  137.188483]  r4:b85eb500
[  137.196213] [<8012d938>] (worker_thread) from [<80133448>] (kthread+0x100/0x11c)
[  137.208894]  r10:00000000 r9:00000000 r8:00000000 r7:8012d938 r6:b85eb500 r5:00000000
[  137.222002]  r4:b85e9bc0
[  137.229664] [<80133348>] (kthread) from [<80107a20>] (ret_from_fork+0x14/0x34)
[  137.242093]  r7:00000000 r6:00000000 r5:80133348 r4:b85e9bc0
[  137.252887] Code: e514107c e3a0201a e59164c0 e50b304c (e5960164)
Mar 27 20:39:48 raspi-test dhclient[148]: bound [  137.271589] ---[ end trace 5a106ba7c8ab9141 ]---
to 192.168.1.252 -- renewal in -5027907 [  137.284859] Unable to handle kernel paging request at virtual address ffffffec
[  137.298390] pgd = 80004000
[  137.305962] [ffffffec] *pgd=393fa861, *pte=00000000, *ppte=00000000
[  137.317180] Internal error: Oops: 37 [#2] PREEMPT SMP ARM
[  137.327402] Modules linked in: asix bcm2835_gpiomem uio_pdrv_genirq uio fixed iptable_filter ipv6
[  137.345972] CPU: 0 PID: 38 Comm: kworker/0:1 Tainted: G      D         4.9.80-v7+ #37
[  137.358644] Hardware name: Generic DT based system
[  137.368162] task: b8610000 task.stack: b860c000
[  137.377326] PC is at kthread_data+0x10/0x18
[  137.386031] LR is at wq_worker_sleeping+0x10/0xd0
[  137.395170] pc : [<80133c28>]    lr : [<8012e6a8>]    psr: 20000193
[  137.395170] sp : b860dc80  ip : b860dc90  fp : b860dc8c
[  137.415363] r10: b8610000  r9 : 8013c290  r8 : 80a041ec
[  137.424791] r7 : b8610340  r6 : 8093fe00  r5 : b8610000  r4 : b8b97e00
[  137.435587] r3 : 00000000  r2 : 00000000  r1 : b8610000  r0 : b8610000
[  137.446285] Flags: nzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  137.457657] Control: 10c5383d  Table: 37f3c06a  DAC: 00000055
[  137.467507] Process kworker/0:1 (pid: 38, stack limit = 0xb860c210)
[  137.477847] Stack: (0xb860dc80 to 0xb860e000)
[  137.486173] dc80: b860dc9c b860dc90 8012e6a8 80133c24 b860dccc b860dca0 8063cc70 8012e6a4
[  137.502268] dca0: b860da14 b8460000 b860da14 b8460000 b86102b8 b860dce8 60000113 00000020
[  137.518468] dcc0: b860dcdc b860dcd0 8013c290 8063ca1c b860dd04 b860dce0 8011bac4 8013c20c
[  137.534766] dce0: 60000113 00000001 b860dce8 b860dce8 80118af0 00000017 b860dd3c b860dd08
[  137.551227] dd00: 8010adb4 8011b468 b860c210 0000000b 807e5918 00000164 00000017 b860de48
[  137.567909] dd20: 00000000 b860de48 b860c000 b8610000 b860dd54 b860dd40 8011227c 8010aba4
[  137.584749] dd40: b860de48 00000000 b860dd94 b860dd58 80641da0 8011222c b8b97e00 b8b97e00
[  137.601765] dd60: 00000000 b8b97e00 00000003 00000017 80641b34 00000164 80a09904 b860de48
[  137.618822] dd80: b860c000 80a02d00 b860de44 b860dd98 80101298 80641b40 80152510 80160d9c
[  137.636091] dda0: 00000fde 00000000 b8610080 b8b97e30 b860dde4 b860ddc0 80140a0c 8015249c
[  137.653740] ddc0: b8610000 b8b97e30 00000001 b8b97e30 b8b97e30 0042cca4 b860debc b860dde8
[  137.671687] dde0: 80144ae8 80144170 b860de04 b860ddf8 8013b29c b7c45a88 b8610080 b7c45a80
[  137.690011] de00: b860de74 b860de10 80147040 b7d9788c b860de4c b860de20 8014fefc 80150344
[  137.708694] de20: 00000022 8046c51c 60000113 ffffffff b860de7c 00000001 b860deec b860de48
[  137.727545] de40: 80641330 80101268 00000000 b8786000 0000001a 00000000 b878657c b8786500
[  137.746589] de60: 00000000 b8b9b000 00000001 00000000 80a02d00 b860deec b860de58 b860de98
[  137.765652] de80: 804362c0 8046c51c 60000113 ffffffff 8046c4fc 00000000 00000002 80a32088
[  137.784710] dea0: 00000000 00000000 b860dedc b860deb8 801388ec 80640b6c 00000000 8093fe00
[  137.803761] dec0: b8b97e00 b8610000 b87ad000 b85eb500 b878657c b8b97a00 b8b9b000 00000000
[  137.822822] dee0: b860df24 b860def0 8012d740 8046c460 b8b97a00 00000001 b85eb518 b85eb500
[  137.841883] df00: b8b97a00 b8b97a18 00000001 b85eb518 00000008 80a02d00 b860df5c b860df28
[  137.860935] df20: 8012d970 8012d62c b85eb500 8012d938 00000000 b85e9bc0 00000000 b85eb500
[  137.879989] df40: 8012d938 00000000 00000000 00000000 b860dfac b860df60 80133448 8012d944
[  137.899040] df60: b860df94 00000000 801388ec b85eb500 00000000 00000000 b860df78 b860df78
[  137.918089] df80: 00000001 00010001 b860df88 b860df88 b85e9bc0 80133348 00000000 00000000
[  137.937140] dfa0: 00000000 b860dfb0 80107a20 80133354 00000000 00000000 00000000 00000000
[  137.956192] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  137.975248] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[  137.994297] Backtrace:
[  138.002036] [<80133c18>] (kthread_data) from [<8012e6a8>] (wq_worker_sleeping+0x10/0xd0)
[  138.020739] [<8012e698>] (wq_worker_sleeping) from [<8063cc70>] (__schedule+0x260/0x44c)
[  138.039451] [<8063ca10>] (__schedule) from [<8013c290>] (do_task_dead+0x90/0x94)
[  138.052232]  r9:00000020 r8:60000113 r7:b860dce8 r6:b86102b8 r5:b8460000 r4:b860da14
[  138.065354] [<8013c200>] (do_task_dead) from [<8011bac4>] (do_exit+0x668/0x9dc)
[  138.078008] [<8011b45c>] (do_exit) from [<8010adb4>] (die+0x21c/0x328)
[  138.089798]  r7:00000017
[  138.097453] [<8010ab98>] (die) from [<8011227c>] (__do_kernel_fault.part.0+0x5c/0x1ec)
[  138.110606]  r10:b8610000 r9:b860c000 r8:b860de48 r7:00000000 r6:b860de48 r5:00000017
[  138.123649]  r4:00000164
[  138.131245] [<80112220>] (__do_kernel_fault.part.0) from [<80641da0>] (do_page_fault+0x26c/0x32c)
[  138.150299]  r7:00000000 r3:b860de48
[  138.158853] [<80641b34>] (do_page_fault) from [<80101298>] (do_DataAbort+0x3c/0xbc)
[  138.171565]  r10:80a02d00 r9:b860c000 r8:b860de48 r7:80a09904 r6:00000164 r5:80641b34
[  138.184432]  r4:00000017
[  138.191855] [<8010125c>] (do_DataAbort) from [<80641330>] (__dabt_svc+0x50/0x80)
[  138.204224] Exception stack(0xb860de48 to 0xb860de90)
[  138.214169] de40:                   00000000 b8786000 0000001a 00000000 b878657c b8786500
[  138.232004] de60: 00000000 b8b9b000 00000001 00000000 80a02d00 b860deec b860de58 b860de98
[  138.249801] de80: 804362c0 8046c51c 60000113 ffffffff
[  138.259628]  r8:00000001 r7:b860de7c r6:ffffffff r5:60000113 r4:8046c51c
[  138.271101] [<8046c454>] (lan78xx_delayedwork) from [<8012d740>] (process_one_work+0x120/0x318)
[  138.289179]  r8:00000000 r7:b8b9b000 r6:b8b97a00 r5:b878657c r4:b85eb500
[  138.300736] [<8012d620>] (process_one_work) from [<8012d970>] (worker_thread+0x38/0x524)
[  138.318384]  r10:80a02d00 r9:00000008 r8:b85eb518 r7:00000001 r6:b8b97a18 r5:b8b97a00
[  138.331079]  r4:b85eb500
[  138.338326] [<8012d938>] (worker_thread) from [<80133448>] (kthread+0x100/0x11c)
[  138.350539]  r10:00000000 r9:00000000 r8:00000000 r7:8012d938 r6:b85eb500 r5:00000000
[  138.363192]  r4:b85e9bc0
[  138.370422] [<80133348>] (kthread) from [<80107a20>] (ret_from_fork+0x14/0x34)
[  138.382473]  r7:00000000 r6:00000000 r5:80133348 r4:b85e9bc0
[  138.392941] Code: e1a0c00d e92dd800 e24cb004 e5903310 (e5130014)
[  138.403803] ---[ end trace 5a106ba7c8ab9142 ]---
[  138.413107] Fixing recursive fault but reboot is needed!

[dave-p]

@AlBundy33 's issue is a duplicate of issue#2446. If the published fix isn't going to be be made upstream anytime soon, could it be cherry-picked into the RPi kernel? TVheadend seems particularly affected by this bug, as TV streaming is generally stopped by the client closing the connection.

[popcornmix]

I was hoping the upstream fix would be accepted more quickly (it appears to be a simple reset of pointer to NULL). But as the fix looks pretty innocuous and has been reported to fix an issue I'll pull it in and deal with the probable merge conflicts later.

[popcornmix]

The linked patch has been pushed to linux tree and is in latest rpi-update kernel. Can you test?

[AlBundy33]

I've executed rpi-update but I'm still on 4.14.30 as before.

[pelwell]

Check the build date - things move fast downstream.

[popcornmix]

Yes, 4.14.30 is still the latest. uname -a should report: Linux version 4.14.30+ (dc4@dc4-XPS13-9333) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)) #1103 Wed Mar 28 19:15:50 BST 2018

[AlBundy33]

On my pi it looks like this

pi@rpi:~ $ uname -a
Linux rpi 4.14.30-v7+ #1103 SMP Wed Mar 28 19:38:37 BST 2018 armv7l GNU/Linux

[pelwell]

Yes, that's the one - the v7 (Pi2/3) build ran after the v6 (Pi0/1) build.

[AlBundy33]

In syslog I've found this

Mar 28 19:17:02 rpi kernel: [    0.000000] Linux version 4.14.30-v7+ (dc4@dc4-XPS13-9333) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1Mar 28 21:16:29 rpi kernel: [    0.000000] Linux version 4.14.30-v7+ (dc4@dc4-XPS13-9333) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)) #1103 SMP Wed Mar 28 19:38:37 BST 2018

So seems to be correct. I'll check syslog from time to time for the Oops.

Thanks

[dave-p]

Tested a home-built kernel on Arch, the issue seems to be resolved. Many thanks.

[AlBundy33]

Should be ok for now.