Pi3B+ : USB+ethernet file transfer problem

Hi,

I have problems while transferring files from an USB external HDD or USB flash drive connected on the new RPi 3B+ to another computer via SFTP or samba. After a random amount of data transfered, the transfer rate drop to 0 and stay like that for at least 30 min ( I canceled the transfer after that time).

I cannot see any error on the standard raspbian system logs. I am running raspbian stretch with the last updates, I also did a firmware upgrade with rpi-update.

The RPi 3B+ is powered with an official raspberry power supply (2.5A) and I tried with a spare one. I tried to give more power with an external power supply for the USB HDD without any change, but like I got the same problem with an USB flash drive, I don't expect a power issue or ?

I tried to transfer files with thunar, pcmanfm, scp or rsync. rsync is the only one working, but with a strong fluctuation of the transfer rate (from 0 to 20Mo/s). scp tell me that the file is "stalled" and stop any progress.

If I start the same SD card with a RPi 3B (not plus) the file transfer is working like a charm ( I used it for almost 3 years like that).

I followed tips from this link without successI having no problems during ssh sessions or with vnc.

Any ideas ?

Having same strange issues with Pi3B+ [1] [2]

IMO it's network driver (lan78xx) related or the usb host driver does not handle usb transfers correctly.

After switching from eth to wlan, everything is ok

[1] Was never able to make an 3GB image backup (root fs is on external usb disk) over network (kernel 4.9, 4.14 and 4.15 tested). Same configuration works on an older Pi3B without any issues

[2] When root fs is on iSCSI target, got massive kernel OOPS when using kernel 4.14.27+. After switching to 4.9.87+, ethernet connection was better (no kernel OOPS), but maybe still not stable (more testing needed). Kernel 4.15.10+ not tested

I did another test.

Now I am back with the official kernel of the raspbian stretch : 4.9.80-v7, and it is working up to now. I transfered 10Go from an USB HDD with a rate of 20 Mo/s.

I've got the same problem here... Trying to transfer large files over ethernet fails after a minute or so... Fine on Wifi... However, when I try to downgrade to 4.9.80-v7, I get four flashing green lights on the PI3B+... I can take the card out and plug it into a Pi2 and it boots up just fine... I then have to re-update the firmware and swap everything back to the Pi3B+... Frustrating...

Is this bug related to #2446? If so, then the forthcoming kernel 4.14.28 should fix it. Reverting to an old kernel may be a temporary solution, but only for advanced users.

Are other distros affected: xbian, osmc, LibreELEC?

At least we know that OSMC used pre-production units, so may be they included a fix/workaround to this problem.

I have 10 days to decide to return the RPi 3B+, so I want to start with a working Ethernet or just return the unit unopened.

Is this bug related to #2446? If so, then the forthcoming kernel 4.14.28 should fix it. Reverting to an old kernel may be a temporary solution, but only for advanced users.

IMO there is a good chance to solve some strange issues.

As already reported, I'm struggling with my Pi3B+ since I got that part. Raspbian seems to work in my testing scenario [1], but XBian was crashing always, sooner or later. The major difference between Raspbian and XBian is, that Rasbian is using ipv6, whereas XBian is using ipv4 (ipv6 is disabled per default), and those patches fixing the ipv4 stack. After enabling tcp6 on XBian, my test seems so finish successfully.

[1] Root fs on usb disk (60GB), mounting Samba share on my server to /mnt, and then run dd if=/dev/sda of=/mnt/test.img bs=1M status=process. Raspbian succeeded, XBian crashes always in different ways: kernel oops, kernel panic of just completely freezed.

If you have a Linux target machine, can you try running this one-liner (you'll need to change the login credentials for the target):

dd if=/dev/zero bs=1M status=progress | ssh user@target "cat >/dev/null"

I've had that running overnight and it's approaching 2TB transferred without issue. The channel is encrypted, so the 29.4MB/s (235Mb/s) it's achieving isn't bad.

If that's solid, switch to:

while true; do date 1>&2; sudo dd if=/dev/mmcblk0 bs=1M status=progress; done | ssh user@target "cat >/dev/null"

Then:

while true; do date 1>&2; sudo dd if=/dev/sda bs=1M status=progress; done | ssh user@target "cat >/dev/null"

@pelwell Sure, there are many many ways to run a test. Unfortunately, after my tests has been finished successfully, my real application making an image backup started from Kodi GUI still kills the Pi3B+ completely [1], in different ways (4.14.27+ mostly throws kernel Oops #5, so built kernel with netdev patch, Oops were gone, but now still getting VFS CIFS stuck for more than 15s message, and Kernel 4.14.29+ build tonight did not make any difference).

Conclusion: Pi3B+ is at the moment absolutely unusable, crappiest Pi ever had. Extremely frustrating

[1] That backup is using btrfs send/receive, sends all local btrfs subvolumes (located on external USB disk) to an image mounted on a network share (cifs of nfs).

Pi3B+ is at the moment absolutely unusable, crappiest Pi ever had. Extremely frustrating

For you, apparently, but not for me - that's why I'm trying to be methodical and work out which step is causing the problem.

@mkreisl Out of curiosity, could you possibly try using sshfs instead of CIFS or NFS? It's a FUSE module instead of a native filesystem driver, and it uses a completely different (and much more efficient) protocol on-the-wire, so it may work even though CIFS and NFS aren't. sudo apt-get install sshfs should get it for you (I'm about 99% certain Raspbian has it). If that works, it should at least give you something you can use in the short term until this gets figured out.

For you, apparently, but not for me - that's why I'm trying to be methodical and work out which step is causing the problem.

Those basic tests you are running does cause the issue. IMO many many data in and out of the USB is causing the problem. That's the main difference between Pi3B nd Pi3B+, Pi3B has one usb hub, and Pi3B+ has two:

Pi3B:

Bus 001 Device 004: ID 046d:c503 Logitech, Inc. Cordless Mouse+Keyboard Receiver
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp. SMC9514 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Pi3B+:

Bus 001 Device 004: ID 046d:c503 Logitech, Inc. Cordless Mouse+Keyboard Receiver
Bus 001 Device 005: ID 0424:7800 Standard Microsystems Corp. 
Bus 001 Device 003: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 002: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

@Ferroin Good idea, I'll test it. Takes some time, have to modify the backup script

@Ferroin sshfs doesn't make a difference, Pi3B+ dying with kernel Oops again (Pi3B works like a charm)

Mar 22 17:40:00 kmxbilr2 kernel: [  555.453472] Unable to handle kernel NULL pointer dereference at virtual address 0000000d
Mar 22 17:40:00 kmxbilr2 kernel: [  555.453480] pgd = a12d8000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.453484] [0000000d] *pgd=00000000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.453493] Internal error: Oops: 5 [#1] SMP ARM
Mar 22 17:40:00 kmxbilr2 kernel: [  555.453578] Modules linked in: dm_mod dax fuse loop hci_uart bluetooth ecdh_generic sg uio_pdrv_genirq uio lirc_rpi(C) lirc_dev fixed frandom ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipv6 snd_bcm2835(C) snd_pcm snd_timer snd brcmfmac cfg80211 rfkill brcmutil evdev rpcsec_gss_krb5 tun uinput
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454081] CPU: 1 PID: 10626 Comm: sudo Tainted: G         C      4.14.29+ #1
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454170] Hardware name: BCM2835
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454216] task: 91ab1e00 task.stack: 93ea0000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454287] PC is at locks_remove_posix+0x30/0x14c
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454364] LR is at filp_close+0x68/0x8c
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454447] pc : [<802e08b0>]    lr : [<80286ac4>]    psr: 20000013
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454550] sp : 93ea1eb0  ip : 93ea1f58  fp : 93ea1f54
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454616] r10: 00000000  r9 : 93ea0000  r8 : 80108204
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454680] r7 : 00000006  r6 : acc63900  r5 : 9d640180  r4 : ae690440
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454754] r3 : 00000001  r2 : ad1bb940  r1 : acc63900  r0 : 9d640180
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454831] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454914] Control: 10c5383d  Table: 212d806a  DAC: 00000055
Mar 22 17:40:00 kmxbilr2 kernel: [  555.454983] Process sudo (pid: 10626, stack limit = 0x93ea0210)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.455058] Stack: (0x93ea1eb0 to 0x93ea2000)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.455112] 1ea0:                                     802fe824 91ab1e00 00000044 802a8f5c
Mar 22 17:40:00 kmxbilr2 kernel: [  555.455210] 1ec0: 00000004 00000000 00000000 00000100 81f22e80 80d093bc 00000017 808ac294
Mar 22 17:40:00 kmxbilr2 kernel: [  555.455316] 1ee0: 76ebfe2c 93ea1fb0 7ed36760 808a8780 93ea1f14 93ea1f00 808a8780 806fbae8
Mar 22 17:40:00 kmxbilr2 kernel: [  555.455440] 1f00: add51c00 add51d74 93ea1f34 93ea1f18 806fbae8 808a7758 add52400 7f0220bc
Mar 22 17:40:00 kmxbilr2 kernel: [  555.455561] 1f20: add52400 add52444 93ea1f54 00000000 9d640180 acc63900 00000006 80108204
Mar 22 17:40:00 kmxbilr2 kernel: [  555.455679] 1f40: 93ea0000 00000000 93ea1f74 93ea1f58 80286ac4 802e088c 0000003c acc63900
Mar 22 17:40:00 kmxbilr2 kernel: [  555.455787] 1f60: 9d640180 00000006 93ea1f94 93ea1f78 802aad38 80286a68 76fbd218 0000000b
Mar 22 17:40:00 kmxbilr2 kernel: [  555.455892] 1f80: 00000000 00000006 93ea1fa4 93ea1f98 80286b18 802aac7c 00000000 93ea1fa8
Mar 22 17:40:00 kmxbilr2 kernel: [  555.455993] 1fa0: 80108060 80286af4 76fbd218 0000000b 0000003c 7ed36760 76fbcb5c 76fbcb5c
Mar 22 17:40:00 kmxbilr2 kernel: [  555.456102] 1fc0: 76fbd218 0000000b 00000000 00000006 01d185b8 7ed36760 76fface8 0000003c
Mar 22 17:40:00 kmxbilr2 kernel: [  555.456205] 1fe0: 7ed36738 7ed36728 76fbcb6c 76e94fc2 60000030 0000003c 00000000 00000000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.456325] [<802e08b0>] (locks_remove_posix) from [<80286ac4>] (filp_close+0x68/0x8c)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.456435] [<80286ac4>] (filp_close) from [<802aad38>] (__close_fd+0xc8/0xec)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.456529] [<802aad38>] (__close_fd) from [<80286b18>] (SyS_close+0x30/0x58)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.456626] [<80286b18>] (SyS_close) from [<80108060>] (ret_fast_syscall+0x0/0x28)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.456728] Code: e59430e8 f57ff05b e3530000 0a000025 (e5b3200c) 
Mar 22 17:40:00 kmxbilr2 kernel: [  555.456857] ---[ end trace 124dc21b94998788 ]---
Mar 22 17:40:00 kmxbilr2 kernel: [  555.477670] Unable to handle kernel NULL pointer dereference at virtual address 0000000d
Mar 22 17:40:00 kmxbilr2 kernel: [  555.477836] pgd = 80004000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.477896] [0000000d] *pgd=00000000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.477964] Internal error: Oops: 5 [#2] SMP ARM
Mar 22 17:40:00 kmxbilr2 kernel: [  555.478032] Modules linked in: dm_mod dax fuse loop hci_uart bluetooth ecdh_generic sg uio_pdrv_genirq uio lirc_rpi(C) lirc_dev fixed frandom ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipv6 snd_bcm2835(C) snd_pcm snd_timer snd brcmfmac cfg80211 rfkill brcmutil evdev rpcsec_gss_krb5 tun uinput
Mar 22 17:40:00 kmxbilr2 kernel: [  555.478580] CPU: 3 PID: 10620 Comm: sudo Tainted: G      D  C      4.14.29+ #1
Mar 22 17:40:00 kmxbilr2 kernel: [  555.478678] Hardware name: BCM2835
Mar 22 17:40:00 kmxbilr2 kernel: [  555.478725] task: 82818f00 task.stack: 86ea8000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.478800] PC is at locks_remove_posix+0x30/0x14c
Mar 22 17:40:00 kmxbilr2 kernel: [  555.478867] LR is at filp_close+0x68/0x8c
Mar 22 17:40:00 kmxbilr2 kernel: [  555.478933] pc : [<802e08b0>]    lr : [<80286ac4>]    psr: 20040013
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479016] sp : 86ea9d08  ip : 86ea9db0  fp : 86ea9dac
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479084] r10: 0000000b  r9 : ad41a980  r8 : 00000004
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479152] r7 : ade91500  r6 : ade91500  r5 : 9d640180  r4 : ae690440
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479234] r3 : 00000001  r2 : 9d640180  r1 : ade91500  r0 : 9d640180
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479316] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479407] Control: 10c5383d  Table: 109d806a  DAC: 00000055
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479486] Process sudo (pid: 10620, stack limit = 0x86ea8210)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479561] Stack: (0x86ea9d08 to 0x86eaa000)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479621] 9d00:                   80d87980 809024bc 2e486000 808a6840 af11ad40 86ea9d48
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479728] 9d20: 8c3da480 8025bc28 ae811a40 80e15480 60040013 60040013 0000008c 00033c4b
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479832] 9d40: 00000000 808a8780 86ea9d6c 86ea9d58 808a8780 806fbae8 add51c00 add51d74
Mar 22 17:40:00 kmxbilr2 kernel: [  555.479956] 9d60: 86ea9d8c 86ea9d70 806fbae8 808a7758 add52400 7f0220bc add52400 add52444
Mar 22 17:40:00 kmxbilr2 kernel: [  555.480075] 9d80: 86ea9dac 00000000 9d640180 ade91500 ade91500 00000004 ad41a980 0000000b
Mar 22 17:40:00 kmxbilr2 kernel: [  555.480184] 9da0: 86ea9dcc 86ea9db0 80286ac4 802e088c 00000009 000000f0 00000000 ade91500
Mar 22 17:40:00 kmxbilr2 kernel: [  555.480287] 9dc0: 86ea9df4 86ea9dd0 802aa864 80286a68 82818f00 82819440 ade91500 ae811a40
Mar 22 17:40:00 kmxbilr2 kernel: [  555.480389] 9de0: 00000001 ae811a78 86ea9e14 86ea9df8 802aa974 802aa7bc 82818f00 00000000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.480493] 9e00: 00000544 ae811a40 86ea9e54 86ea9e18 80121d6c 802aa928 86ea9e74 86ea9e28
Mar 22 17:40:00 kmxbilr2 kernel: [  555.480595] 9e20: 86ea9edc 86ea9fb0 00000000 0000000b ac9a02c0 86ea9edc 86ea8000 00106001
Mar 22 17:40:00 kmxbilr2 kernel: [  555.480698] 9e40: 86ea8000 0000000b 86ea9e74 86ea9e58 8012260c 801219e0 00000000 ad6aba88
Mar 22 17:40:00 kmxbilr2 kernel: [  555.480800] 9e60: 86ea9edc 86ea8000 86ea9ec4 86ea9e78 8012d9f4 801225cc 80d02040 418004fc
Mar 22 17:40:00 kmxbilr2 kernel: [  555.480903] 9e80: 80d03d68 86ea9ec8 ac9a02c0 ad6abec4 ad6ab9c0 000000a0 0000000b 76e27574
Mar 22 17:40:00 kmxbilr2 kernel: [  555.494505] 9ea0: 86ea9ec8 86ea9fb0 76e27576 00000000 86ea8000 00000000 86ea9f8c 86ea9ec8
Mar 22 17:40:00 kmxbilr2 kernel: [  555.506477] 9ec0: 8010b318 8012d6c8 86ea9ef4 86ea9ed8 8012d084 8012cf08 00000000 0000000b
Mar 22 17:40:00 kmxbilr2 kernel: [  555.518136] 9ee0: 00000000 00000000 0000297c 00000000 ad6ab9e8 00000001 82818f00 00000000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.531395] 9f00: 00000000 0000297c 00000000 ad6ab9e8 00000001 82818f00 00000001 86ea9f68
Mar 22 17:40:00 kmxbilr2 kernel: [  555.545698] 9f20: 86ea9f64 86ea9f30 8012f17c 801edc50 82819444 00000002 7ed36aa8 00000000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.558183] 9f40: 00000002 7ed36aa8 000000ae 80108204 86ea8000 00000000 86ea9fa4 86ea9f68
Mar 22 17:40:00 kmxbilr2 kernel: [  555.570528] 9f60: 8012f6f0 00000001 86ea8010 80108204 86ea9fb0 80108204 86ea8000 00000000
Mar 22 17:40:00 kmxbilr2 kernel: [  555.582517] 9f80: 86ea9fac 86ea9f90 8010b820 8010b260 7ed36aa8 0000000b 0000000b 00000025
Mar 22 17:40:00 kmxbilr2 kernel: [  555.594335] 9fa0: 00000000 86ea9fb0 80108094 8010b774 00000000 0000000b 3a4d6900 3a4d6900
Mar 22 17:40:00 kmxbilr2 kernel: [  555.607730] 9fc0: 7ed36aa8 0000000b 0000000b 00000025 01d15c58 0049016c 00000000 7ed36a18
Mar 22 17:40:00 kmxbilr2 kernel: [  555.621658] 9fe0: 004a1dd8 7ed369c4 0047dd5b 76e27576 20040030 0000297c a302000d 64690063
Mar 22 17:40:00 kmxbilr2 kernel: [  555.635304] [<802e08b0>] (locks_remove_posix) from [<80286ac4>] (filp_close+0x68/0x8c)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.647889] [<80286ac4>] (filp_close) from [<802aa864>] (put_files_struct+0xb4/0x10c)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.660265] [<802aa864>] (put_files_struct) from [<802aa974>] (exit_files+0x58/0x5c)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.672651] [<802aa974>] (exit_files) from [<80121d6c>] (do_exit+0x398/0xba0)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.684677] [<80121d6c>] (do_exit) from [<8012260c>] (do_group_exit+0x4c/0xe4)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.696657] [<8012260c>] (do_group_exit) from [<8012d9f4>] (get_signal+0x338/0x6e0)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.708835] [<8012d9f4>] (get_signal) from [<8010b318>] (do_signal+0xc4/0x3e4)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.720791] [<8010b318>] (do_signal) from [<8010b820>] (do_work_pending+0xb8/0xd0)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.732744] [<8010b820>] (do_work_pending) from [<80108094>] (slow_work_pending+0xc/0x20)
Mar 22 17:40:00 kmxbilr2 kernel: [  555.744414] Code: e59430e8 f57ff05b e3530000 0a000025 (e5b3200c) 
Mar 22 17:40:00 kmxbilr2 kernel: [  555.755952] ---[ end trace 124dc21b94998789 ]---
Mar 22 17:40:00 kmxbilr2 kernel: [  555.772247] Fixing recursive fault but reboot is needed!
Mar 22 17:40:02 kmxbilr2 kernel: [  557.572209] Unable to handle kernel paging request at virtual address 04bc7ffc
Mar 22 17:40:02 kmxbilr2 kernel: [  557.583366] pgd = a0380000
Mar 22 17:40:02 kmxbilr2 kernel: [  557.594624] [04bc7ffc] *pgd=00000000
Mar 22 17:40:02 kmxbilr2 kernel: [  557.605878] Internal error: Oops: 5 [#3] SMP ARM
Mar 22 17:40:02 kmxbilr2 kernel: [  557.616992] Modules linked in: dm_mod dax fuse loop hci_uart bluetooth ecdh_generic sg uio_pdrv_genirq uio lirc_rpi(C) lirc_dev fixed frandom ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipv6 snd_bcm2835(C) snd_pcm snd_timer snd brcmfmac cfg80211 rfkill brcmutil evdev rpcsec_gss_krb5 tun uinput
Mar 22 17:40:02 kmxbilr2 kernel: [  557.639993] CPU: 3 PID: 10496 Comm: btrfs Tainted: G      D  C      4.14.29+ #1
Mar 22 17:40:02 kmxbilr2 kernel: [  557.651518] Hardware name: BCM2835
Mar 22 17:40:02 kmxbilr2 kernel: [  557.662917] task: 9851bc00 task.stack: a01e4000
Mar 22 17:40:02 kmxbilr2 kernel: [  557.674480] PC is at __d_lookup_rcu+0x68/0x19c
Mar 22 17:40:02 kmxbilr2 kernel: [  557.685867] LR is at lookup_fast+0x4c/0x2c8
Mar 22 17:40:02 kmxbilr2 kernel: [  557.697094] pc : [<802a4da0>]    lr : [<80295cc4>]    psr: 20010013
Mar 22 17:40:02 kmxbilr2 kernel: [  557.708335] sp : a01e5d20  ip : 80d04590  fp : a01e5d5c
Mar 22 17:40:02 kmxbilr2 kernel: [  557.719619] r10: a01e5e50  r9 : 00000006  r8 : e6ad3f44
Mar 22 17:40:02 kmxbilr2 kernel: [  557.730983] r7 : a03d8550  r6 : a03d8550  r5 : 00000000  r4 : 04bc8000
Mar 22 17:40:02 kmxbilr2 kernel: [  557.741942] r3 : 0001cd5a  r2 : 00004000  r1 : 00000000  r0 : a03d8550
Mar 22 17:40:02 kmxbilr2 kernel: [  557.752961] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Mar 22 17:40:02 kmxbilr2 kernel: [  557.763554] Control: 10c5383d  Table: 2038006a  DAC: 00000055
Mar 22 17:40:02 kmxbilr2 kernel: [  557.774189] Process btrfs (pid: 10496, stack limit = 0xa01e4210)
Mar 22 17:40:02 kmxbilr2 kernel: [  557.784760] Stack: (0xa01e5d20 to 0xa01e6000)
Mar 22 17:40:02 kmxbilr2 kernel: [  557.795359] 5d20: 8c6f2000 a01e5d6c 00000006 8c6f0038 a01e5d5c a01e5e48 00000000 a01e5da8
Mar 22 17:40:02 kmxbilr2 kernel: [  557.806204] 5d40: a03d8550 a01e5da0 ad639b10 a01e5da4 a01e5d9c a01e5d60 80295cc4 802a4d44
Mar 22 17:40:02 kmxbilr2 kernel: [  557.817041] 5d60: 8c6f2000 ae4085a0 00001455 00000006 00001542 a01e5e48 00000000 00000003
Mar 22 17:40:02 kmxbilr2 kernel: [  557.828472] 5d80: 8c6f003f a01e5e48 8cff9c59 a03d8550 a01e5dd4 a01e5da0 80298140 80295c84
Mar 22 17:40:02 kmxbilr2 kernel: [  557.843155] 5da0: a01e5dc4 a01e5db0 8029683c 8042ea20 c2d0f82e 47090a62 e6ad3f44 8c6f003f
Mar 22 17:40:02 kmxbilr2 kernel: [  557.855354] 5dc0: a01e5e48 8cff9c59 a01e5e24 a01e5dd8 8029856c 80298110 80295448 61c88647
Mar 22 17:40:02 kmxbilr2 kernel: [  557.866613] 5de0: 00000000 00000006 a01e5e48 8c6f0010 a01e5e24 a01e5e00 80294f74 8c6f0010
Mar 22 17:40:02 kmxbilr2 kernel: [  557.878046] 5e00: a01e5e48 a01e5f38 a01e5f38 a01e5f40 00000000 ffffff9c a01e5e44 a01e5e28
Mar 22 17:40:02 kmxbilr2 kernel: [  557.889410] 5e20: 802988c4 802983ec 8c6f0000 a01e5e48 00000000 a01e5f38 a01e5eec a01e5e48
Mar 22 17:40:02 kmxbilr2 kernel: [  557.900746] 5e40: 8029a4b4 8029889c ad639b10 a03d8550 e6ad3f44 00000006 8c6f0038 80276ee4
Mar 22 17:40:02 kmxbilr2 kernel: [  557.912237] 5e60: ae990c10 ae7af660 883d6660 00000050 00000006 000003a4 00000000 00000000
Mar 22 17:40:02 kmxbilr2 kernel: [  557.923647] 5e80: 00000000 a01e5e88 80d0459c 8c6f0000 80d0459c 7ea7a81c 00000000 00000000
Mar 22 17:40:02 kmxbilr2 kernel: [  557.934994] 5ea0: a01e5edc a01e5eb0 8029a30c 805d1904 a01e5e88 8c6f4000 8c6f0000 00000000
Mar 22 17:40:02 kmxbilr2 kernel: [  557.946359] 5ec0: 00000026 00000002 ffffff9c ffffff9c 8c6f4000 7ea7981c a01e5f50 00000026
Mar 22 17:40:02 kmxbilr2 kernel: [  557.957715] 5ee0: a01e5f8c a01e5ef0 8029b984 8029a444 a01e5f50 a01e5f28 98785c08 a01e5ef0
Mar 22 17:40:02 kmxbilr2 kernel: [  557.969150] 5f00: 00000001 802ad51c 98785c00 00000800 00000000 00000000 7ea7a81c ffffff9c
Mar 22 17:40:02 kmxbilr2 kernel: [  557.980558] 5f20: 00000000 00000000 98785c08 00000000 ad639b10 962fd330 a01e4000 00000000
Mar 22 17:40:02 kmxbilr2 kernel: [  557.992053] 5f40: aa64a66c 0000000a 8c6f402a 80276ee4 00000000 00000000 a01e5f7c 98785c00
Mar 22 17:40:02 kmxbilr2 kernel: [  558.003698] 5f60: 98785c00 00000000 00e9a010 00e9a050 00000026 80108204 a01e4000 00000000
Mar 22 17:40:02 kmxbilr2 kernel: [  558.015188] 5f80: a01e5fa4 a01e5f90 8029bd9c 8029b8bc 00000000 00e9a050 00000000 a01e5fa8
Mar 22 17:40:02 kmxbilr2 kernel: [  558.026723] 5fa0: 80108060 8029bd74 00000000 00e9a010 7ea7981c 7ea7a81c ffffffff 00000000
Mar 22 17:40:02 kmxbilr2 kernel: [  558.038198] 5fc0: 00000000 00e9a010 00e9a050 00000026 004fd000 76f05ce8 7ea7981c 7ea7a81c
Mar 22 17:40:02 kmxbilr2 kernel: [  558.049811] 5fe0: 004fd198 7ea79814 0049cc9b 76cd7796 00010030 7ea7981c 00000000 00000000
Mar 22 17:40:02 kmxbilr2 kernel: [  558.061386] [<802a4da0>] (__d_lookup_rcu) from [<80295cc4>] (lookup_fast+0x4c/0x2c8)
Mar 22 17:40:02 kmxbilr2 kernel: [  558.072944] [<80295cc4>] (lookup_fast) from [<80298140>] (walk_component+0x3c/0x2dc)
Mar 22 17:40:02 kmxbilr2 kernel: [  558.084330] [<80298140>] (walk_component) from [<8029856c>] (link_path_walk+0x18c/0x4b0)
Mar 22 17:40:02 kmxbilr2 kernel: [  558.095582] [<8029856c>] (link_path_walk) from [<802988c4>] (path_parentat+0x34/0x6c)
Mar 22 17:40:02 kmxbilr2 kernel: [  558.106851] [<802988c4>] (path_parentat) from [<8029a4b4>] (filename_parentat+0x7c/0x104)
Mar 22 17:40:02 kmxbilr2 kernel: [  558.117923] [<8029a4b4>] (filename_parentat) from [<8029b984>] (SyS_renameat2+0xd4/0x48c)
Mar 22 17:40:02 kmxbilr2 kernel: [  558.129156] [<8029b984>] (SyS_renameat2) from [<8029bd9c>] (SyS_rename+0x34/0x3c)
Mar 22 17:40:02 kmxbilr2 kernel: [  558.140092] [<8029bd9c>] (SyS_rename) from [<80108060>] (ret_fast_syscall+0x0/0x28)
Mar 22 17:40:02 kmxbilr2 kernel: [  558.150966] Code: ea000002 e5944000 e3540000 0a000028 (e5141004) 
Mar 22 17:40:02 kmxbilr2 kernel: [  558.161756] ---[ end trace 124dc21b9499878a ]---
Mar 22 17:40:05 kmxbilr2 kernel: [  560.624738] Alignment trap: not handling instruction e1b04f9f at [<80566d48>]
Mar 22 17:40:05 kmxbilr2 kernel: [  560.635458] Unhandled fault: alignment exception (0x001) at 0x9ac47754
Mar 22 17:40:05 kmxbilr2 kernel: [  560.646194] pgd = acecc000
Mar 22 17:40:05 kmxbilr2 kernel: [  560.656802] [9ac47754] *pgd=1ac1141e(bad)
Mar 22 17:40:05 kmxbilr2 kernel: [  560.667391] Internal error: : 1 [#4] SMP ARM
Mar 22 17:40:05 kmxbilr2 kernel: [  560.677894] Modules linked in: dm_mod dax fuse loop hci_uart bluetooth ecdh_generic sg uio_pdrv_genirq uio lirc_rpi(C) lirc_dev fixed frandom ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipv6 snd_bcm2835(C) snd_pcm snd_timer snd brcmfmac cfg80211 rfkill brcmutil evdev rpcsec_gss_krb5 tun uinput
Mar 22 17:40:05 kmxbilr2 kernel: [  560.700191] CPU: 3 PID: 4194 Comm: PeripBusUSBUdev Tainted: G      D  C      4.14.29+ #1
Mar 22 17:40:05 kmxbilr2 kernel: [  560.711520] Hardware name: BCM2835
Mar 22 17:40:05 kmxbilr2 kernel: [  560.724080] task: 81945a00 task.stack: 9e712000
Mar 22 17:40:05 kmxbilr2 kernel: [  560.737866] PC is at lockref_put_return+0x3c/0x94
Mar 22 17:40:05 kmxbilr2 kernel: [  560.750567] LR is at dput+0x40/0x2d0
Mar 22 17:40:05 kmxbilr2 kernel: [  560.761780] pc : [<80566d4c>]    lr : [<802a1abc>]    psr: 20060013
Mar 22 17:40:05 kmxbilr2 kernel: [  560.773122] sp : 9e713de0  ip : 9e713e00  fp : 9e713dfc
Mar 22 17:40:05 kmxbilr2 kernel: [  560.784351] r10: 00000000  r9 : 9e712000  r8 : 9e713f60
Mar 22 17:40:05 kmxbilr2 kernel: [  560.795460] r7 : 00000001  r6 : 00010001  r5 : 00000001  r4 : 00010001
Mar 22 17:40:05 kmxbilr2 kernel: [  560.806753] r3 : 00000000  r2 : 00010001  r1 : 00000000  r0 : 9ac47754
Mar 22 17:40:05 kmxbilr2 kernel: [  560.817978] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Mar 22 17:40:05 kmxbilr2 kernel: [  560.829007] Control: 10c5383d  Table: 2cecc06a  DAC: 00000055
Mar 22 17:40:05 kmxbilr2 kernel: [  560.839867] Process PeripBusUSBUdev (pid: 4194, stack limit = 0x9e712210)
Mar 22 17:40:05 kmxbilr2 kernel: [  560.851294] Stack: (0x9e713de0 to 0x9e714000)
Mar 22 17:40:05 kmxbilr2 kernel: [  560.863891] 3de0: 9ac47704 00080040 9ac47754 00000041 9e713e24 9e713e00 802a1abc 80566d1c
Mar 22 17:40:05 kmxbilr2 kernel: [  560.876801] 3e00: 00000000 9e713e78 9e713f60 00000041 9e713f60 9e712000 9e713e44 9e713e28
Mar 22 17:40:05 kmxbilr2 kernel: [  560.887838] 3e20: 80294b9c 802a1a88 9e713e78 fffffffe 9e713f60 00000041 9e713e74 9e713e48
Mar 22 17:40:05 kmxbilr2 kernel: [  560.900010] 3e40: 802989bc 80294b5c 9e6683f8 af57c6bc 00000000 00000000 00000001 8c6f5000
Mar 22 17:40:05 kmxbilr2 kernel: [  560.915033] 3e60: 9e713e78 00000001 9e713f24 9e713e78 8029a5d8 80298908 ae990310 9ac47704
Mar 22 17:40:05 kmxbilr2 kernel: [  560.929842] 3e80: e73d77c1 00000006 8c6f5025 00000000 ae990c10 ae7af660 ae7db2b0 00000001
Mar 22 17:40:05 kmxbilr2 kernel: [  560.942629] 3ea0: 9e713dd0 000003be 00000000 00000000 00000000 9e713eb8 00000000 00000000
Mar 22 17:40:05 kmxbilr2 kernel: [  560.955273] 3ec0: 00001000 8c6f6000 00000000 00000001 80d0459c 8c6f5000 80d0459c 6b2fea18
Mar 22 17:40:05 kmxbilr2 kernel: [  560.966619] 3ee0: 00000000 00000001 8c6f5000 00000000 8029a30c 00000002 ffffff9c 00000001
Mar 22 17:40:05 kmxbilr2 kernel: [  560.978523] 3f00: ffffff9c 00000001 ffffff9c 9e713f60 ffffff9c 6b2fea18 9e713f4c 9e713f28
Mar 22 17:40:05 kmxbilr2 kernel: [  560.992103] 3f20: 8029a720 8029a548 00000000 00000000 9e713f4c 00000001 00000000 acb3e980
Mar 22 17:40:05 kmxbilr2 kernel: [  561.005850] 3f40: 9e713f94 9e713f50 80287344 8029a6d8 00000000 00000000 aaa67500 00000000
Mar 22 17:40:05 kmxbilr2 kernel: [  561.017377] 3f60: 00000000 00000000 00000005 00000000 5c21ee28 76f78ce8 00000021 80108204
Mar 22 17:40:05 kmxbilr2 kernel: [  561.029029] 3f80: 9e712000 00000000 9e713fa4 9e713f98 802874c4 802872b4 00000000 9e713fa8
Mar 22 17:40:05 kmxbilr2 kernel: [  561.040579] 3fa0: 80108060 802874ac 00000000 5c21ee28 6b2fea18 00000000 00000000 6b2fea33
Mar 22 17:40:05 kmxbilr2 kernel: [  561.052969] 3fc0: 00000000 5c21ee28 76f78ce8 00000021 6b2fea18 6b2fea48 716d7b30 ffffffea
Mar 22 17:40:05 kmxbilr2 kernel: [  561.068944] 3fe0: 75eebf10 6b2fea04 75ee4b7b 759c1b66 20060030 6b2fea18 aa98abba faaaaaea
Mar 22 17:40:05 kmxbilr2 kernel: [  561.083256] [<80566d4c>] (lockref_put_return) from [<802a1abc>] (dput+0x40/0x2d0)
Mar 22 17:40:05 kmxbilr2 kernel: [  561.094752] [<802a1abc>] (dput) from [<80294b9c>] (terminate_walk+0x4c/0xc0)
Mar 22 17:40:05 kmxbilr2 kernel: [  561.106786] [<80294b9c>] (terminate_walk) from [<802989bc>] (path_lookupat+0xc0/0x204)
Mar 22 17:40:05 kmxbilr2 kernel: [  561.120543] [<802989bc>] (path_lookupat) from [<8029a5d8>] (filename_lookup+0x9c/0xf8)
Mar 22 17:40:05 kmxbilr2 kernel: [  561.133721] [<8029a5d8>] (filename_lookup) from [<8029a720>] (user_path_at_empty+0x54/0x5c)
Mar 22 17:40:05 kmxbilr2 kernel: [  561.144836] [<8029a720>] (user_path_at_empty) from [<80287344>] (SyS_faccessat+0x9c/0x1f8)
Mar 22 17:40:05 kmxbilr2 kernel: [  561.155787] [<80287344>] (SyS_faccessat) from [<802874c4>] (SyS_access+0x24/0x28)
Mar 22 17:40:05 kmxbilr2 kernel: [  561.166755] [<802874c4>] (SyS_access) from [<80108060>] (ret_fast_syscall+0x0/0x28)
Mar 22 17:40:05 kmxbilr2 kernel: [  561.177608] Code: e1a07005 da000015 f590f000 e1b04f9f (e1340006) 
Mar 22 17:40:05 kmxbilr2 kernel: [  561.188475] ---[ end trace 124dc21b9499878b ]---

... and dmesg reports

[  555.453472] Unable to handle kernel NULL pointer dereference at virtual address 0000000d
[  555.453480] pgd = a12d8000
[  555.453484] [0000000d] *pgd=00000000
[  555.453493] Internal error: Oops: 5 [#1] SMP ARM
[  555.453578] Modules linked in: dm_mod dax fuse loop hci_uart bluetooth ecdh_generic sg uio_pdrv_genirq uio lirc_rpi(C) lirc_dev fixed frandom ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipv6 snd_bcm2835(C) snd_pcm snd_timer snd brcmfmac cfg80211 rfkill brcmutil evdev rpcsec_gss_krb5 tun uinput
[  555.454081] CPU: 1 PID: 10626 Comm: sudo Tainted: G         C      4.14.29+ #1
[  555.454170] Hardware name: BCM2835
[  555.454216] task: 91ab1e00 task.stack: 93ea0000
[  555.454287] PC is at locks_remove_posix+0x30/0x14c
[  555.454364] LR is at filp_close+0x68/0x8c
[  555.454447] pc : [<802e08b0>]    lr : [<80286ac4>]    psr: 20000013
[  555.454550] sp : 93ea1eb0  ip : 93ea1f58  fp : 93ea1f54
[  555.454616] r10: 00000000  r9 : 93ea0000  r8 : 80108204
[  555.454680] r7 : 00000006  r6 : acc63900  r5 : 9d640180  r4 : ae690440
[  555.454754] r3 : 00000001  r2 : ad1bb940  r1 : acc63900  r0 : 9d640180
[  555.454831] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  555.454914] Control: 10c5383d  Table: 212d806a  DAC: 00000055
[  555.454983] Process sudo (pid: 10626, stack limit = 0x93ea0210)
[  555.455058] Stack: (0x93ea1eb0 to 0x93ea2000)
[  555.455112] 1ea0:                                     802fe824 91ab1e00 00000044 802a8f5c
[  555.455210] 1ec0: 00000004 00000000 00000000 00000100 81f22e80 80d093bc 00000017 808ac294
[  555.455316] 1ee0: 76ebfe2c 93ea1fb0 7ed36760 808a8780 93ea1f14 93ea1f00 808a8780 806fbae8
[  555.455440] 1f00: add51c00 add51d74 93ea1f34 93ea1f18 806fbae8 808a7758 add52400 7f0220bc
[  555.455561] 1f20: add52400 add52444 93ea1f54 00000000 9d640180 acc63900 00000006 80108204
[  555.455679] 1f40: 93ea0000 00000000 93ea1f74 93ea1f58 80286ac4 802e088c 0000003c acc63900
[  555.455787] 1f60: 9d640180 00000006 93ea1f94 93ea1f78 802aad38 80286a68 76fbd218 0000000b
[  555.455892] 1f80: 00000000 00000006 93ea1fa4 93ea1f98 80286b18 802aac7c 00000000 93ea1fa8
[  555.455993] 1fa0: 80108060 80286af4 76fbd218 0000000b 0000003c 7ed36760 76fbcb5c 76fbcb5c
[  555.456102] 1fc0: 76fbd218 0000000b 00000000 00000006 01d185b8 7ed36760 76fface8 0000003c
[  555.456205] 1fe0: 7ed36738 7ed36728 76fbcb6c 76e94fc2 60000030 0000003c 00000000 00000000
[  555.456325] [<802e08b0>] (locks_remove_posix) from [<80286ac4>] (filp_close+0x68/0x8c)
[  555.456435] [<80286ac4>] (filp_close) from [<802aad38>] (__close_fd+0xc8/0xec)
[  555.456529] [<802aad38>] (__close_fd) from [<80286b18>] (SyS_close+0x30/0x58)
[  555.456626] [<80286b18>] (SyS_close) from [<80108060>] (ret_fast_syscall+0x0/0x28)
[  555.456728] Code: e59430e8 f57ff05b e3530000 0a000025 (e5b3200c) 
[  555.456857] ---[ end trace 124dc21b94998788 ]---
[  555.477670] Unable to handle kernel NULL pointer dereference at virtual address 0000000d
[  555.477836] pgd = 80004000
[  555.477896] [0000000d] *pgd=00000000
[  555.477964] Internal error: Oops: 5 [#2] SMP ARM
[  555.478032] Modules linked in: dm_mod dax fuse loop hci_uart bluetooth ecdh_generic sg uio_pdrv_genirq uio lirc_rpi(C) lirc_dev fixed frandom ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipv6 snd_bcm2835(C) snd_pcm snd_timer snd brcmfmac cfg80211 rfkill brcmutil evdev rpcsec_gss_krb5 tun uinput
[  555.478580] CPU: 3 PID: 10620 Comm: sudo Tainted: G      D  C      4.14.29+ #1
[  555.478678] Hardware name: BCM2835
[  555.478725] task: 82818f00 task.stack: 86ea8000
[  555.478800] PC is at locks_remove_posix+0x30/0x14c
[  555.478867] LR is at filp_close+0x68/0x8c
[  555.478933] pc : [<802e08b0>]    lr : [<80286ac4>]    psr: 20040013
[  555.479016] sp : 86ea9d08  ip : 86ea9db0  fp : 86ea9dac
[  555.479084] r10: 0000000b  r9 : ad41a980  r8 : 00000004
[  555.479152] r7 : ade91500  r6 : ade91500  r5 : 9d640180  r4 : ae690440
[  555.479234] r3 : 00000001  r2 : 9d640180  r1 : ade91500  r0 : 9d640180
[  555.479316] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  555.479407] Control: 10c5383d  Table: 109d806a  DAC: 00000055
[  555.479486] Process sudo (pid: 10620, stack limit = 0x86ea8210)
[  555.479561] Stack: (0x86ea9d08 to 0x86eaa000)
[  555.479621] 9d00:                   80d87980 809024bc 2e486000 808a6840 af11ad40 86ea9d48
[  555.479728] 9d20: 8c3da480 8025bc28 ae811a40 80e15480 60040013 60040013 0000008c 00033c4b
[  555.479832] 9d40: 00000000 808a8780 86ea9d6c 86ea9d58 808a8780 806fbae8 add51c00 add51d74
[  555.479956] 9d60: 86ea9d8c 86ea9d70 806fbae8 808a7758 add52400 7f0220bc add52400 add52444
[  555.480075] 9d80: 86ea9dac 00000000 9d640180 ade91500 ade91500 00000004 ad41a980 0000000b
[  555.480184] 9da0: 86ea9dcc 86ea9db0 80286ac4 802e088c 00000009 000000f0 00000000 ade91500
[  555.480287] 9dc0: 86ea9df4 86ea9dd0 802aa864 80286a68 82818f00 82819440 ade91500 ae811a40
[  555.480389] 9de0: 00000001 ae811a78 86ea9e14 86ea9df8 802aa974 802aa7bc 82818f00 00000000
[  555.480493] 9e00: 00000544 ae811a40 86ea9e54 86ea9e18 80121d6c 802aa928 86ea9e74 86ea9e28
[  555.480595] 9e20: 86ea9edc 86ea9fb0 00000000 0000000b ac9a02c0 86ea9edc 86ea8000 00106001
[  555.480698] 9e40: 86ea8000 0000000b 86ea9e74 86ea9e58 8012260c 801219e0 00000000 ad6aba88
[  555.480800] 9e60: 86ea9edc 86ea8000 86ea9ec4 86ea9e78 8012d9f4 801225cc 80d02040 418004fc
[  555.480903] 9e80: 80d03d68 86ea9ec8 ac9a02c0 ad6abec4 ad6ab9c0 000000a0 0000000b 76e27574
[  555.494505] 9ea0: 86ea9ec8 86ea9fb0 76e27576 00000000 86ea8000 00000000 86ea9f8c 86ea9ec8
[  555.506477] 9ec0: 8010b318 8012d6c8 86ea9ef4 86ea9ed8 8012d084 8012cf08 00000000 0000000b
[  555.518136] 9ee0: 00000000 00000000 0000297c 00000000 ad6ab9e8 00000001 82818f00 00000000
[  555.531395] 9f00: 00000000 0000297c 00000000 ad6ab9e8 00000001 82818f00 00000001 86ea9f68
[  555.545698] 9f20: 86ea9f64 86ea9f30 8012f17c 801edc50 82819444 00000002 7ed36aa8 00000000
[  555.558183] 9f40: 00000002 7ed36aa8 000000ae 80108204 86ea8000 00000000 86ea9fa4 86ea9f68
[  555.570528] 9f60: 8012f6f0 00000001 86ea8010 80108204 86ea9fb0 80108204 86ea8000 00000000
[  555.582517] 9f80: 86ea9fac 86ea9f90 8010b820 8010b260 7ed36aa8 0000000b 0000000b 00000025
[  555.594335] 9fa0: 00000000 86ea9fb0 80108094 8010b774 00000000 0000000b 3a4d6900 3a4d6900
[  555.607730] 9fc0: 7ed36aa8 0000000b 0000000b 00000025 01d15c58 0049016c 00000000 7ed36a18
[  555.621658] 9fe0: 004a1dd8 7ed369c4 0047dd5b 76e27576 20040030 0000297c a302000d 64690063
[  555.635304] [<802e08b0>] (locks_remove_posix) from [<80286ac4>] (filp_close+0x68/0x8c)
[  555.647889] [<80286ac4>] (filp_close) from [<802aa864>] (put_files_struct+0xb4/0x10c)
[  555.660265] [<802aa864>] (put_files_struct) from [<802aa974>] (exit_files+0x58/0x5c)
[  555.672651] [<802aa974>] (exit_files) from [<80121d6c>] (do_exit+0x398/0xba0)
[  555.684677] [<80121d6c>] (do_exit) from [<8012260c>] (do_group_exit+0x4c/0xe4)
[  555.696657] [<8012260c>] (do_group_exit) from [<8012d9f4>] (get_signal+0x338/0x6e0)
[  555.708835] [<8012d9f4>] (get_signal) from [<8010b318>] (do_signal+0xc4/0x3e4)
[  555.720791] [<8010b318>] (do_signal) from [<8010b820>] (do_work_pending+0xb8/0xd0)
[  555.732744] [<8010b820>] (do_work_pending) from [<80108094>] (slow_work_pending+0xc/0x20)
[  555.744414] Code: e59430e8 f57ff05b e3530000 0a000025 (e5b3200c) 
[  555.755952] ---[ end trace 124dc21b94998789 ]---
[  555.772247] Fixing recursive fault but reboot is needed!
[  557.572209] Unable to handle kernel paging request at virtual address 04bc7ffc
[  557.583366] pgd = a0380000
[  557.594624] [04bc7ffc] *pgd=00000000
[  557.605878] Internal error: Oops: 5 [#3] SMP ARM
[  557.616992] Modules linked in: dm_mod dax fuse loop hci_uart bluetooth ecdh_generic sg uio_pdrv_genirq uio lirc_rpi(C) lirc_dev fixed frandom ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipv6 snd_bcm2835(C) snd_pcm snd_timer snd brcmfmac cfg80211 rfkill brcmutil evdev rpcsec_gss_krb5 tun uinput
[  557.639993] CPU: 3 PID: 10496 Comm: btrfs Tainted: G      D  C      4.14.29+ #1
[  557.651518] Hardware name: BCM2835
[  557.662917] task: 9851bc00 task.stack: a01e4000
[  557.674480] PC is at __d_lookup_rcu+0x68/0x19c
[  557.685867] LR is at lookup_fast+0x4c/0x2c8
[  557.697094] pc : [<802a4da0>]    lr : [<80295cc4>]    psr: 20010013
[  557.708335] sp : a01e5d20  ip : 80d04590  fp : a01e5d5c
[  557.719619] r10: a01e5e50  r9 : 00000006  r8 : e6ad3f44
[  557.730983] r7 : a03d8550  r6 : a03d8550  r5 : 00000000  r4 : 04bc8000
[  557.741942] r3 : 0001cd5a  r2 : 00004000  r1 : 00000000  r0 : a03d8550
[  557.752961] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  557.763554] Control: 10c5383d  Table: 2038006a  DAC: 00000055
[  557.774189] Process btrfs (pid: 10496, stack limit = 0xa01e4210)
[  557.784760] Stack: (0xa01e5d20 to 0xa01e6000)
[  557.795359] 5d20: 8c6f2000 a01e5d6c 00000006 8c6f0038 a01e5d5c a01e5e48 00000000 a01e5da8
[  557.806204] 5d40: a03d8550 a01e5da0 ad639b10 a01e5da4 a01e5d9c a01e5d60 80295cc4 802a4d44
[  557.817041] 5d60: 8c6f2000 ae4085a0 00001455 00000006 00001542 a01e5e48 00000000 00000003
[  557.828472] 5d80: 8c6f003f a01e5e48 8cff9c59 a03d8550 a01e5dd4 a01e5da0 80298140 80295c84
[  557.843155] 5da0: a01e5dc4 a01e5db0 8029683c 8042ea20 c2d0f82e 47090a62 e6ad3f44 8c6f003f
[  557.855354] 5dc0: a01e5e48 8cff9c59 a01e5e24 a01e5dd8 8029856c 80298110 80295448 61c88647
[  557.866613] 5de0: 00000000 00000006 a01e5e48 8c6f0010 a01e5e24 a01e5e00 80294f74 8c6f0010
[  557.878046] 5e00: a01e5e48 a01e5f38 a01e5f38 a01e5f40 00000000 ffffff9c a01e5e44 a01e5e28
[  557.889410] 5e20: 802988c4 802983ec 8c6f0000 a01e5e48 00000000 a01e5f38 a01e5eec a01e5e48
[  557.900746] 5e40: 8029a4b4 8029889c ad639b10 a03d8550 e6ad3f44 00000006 8c6f0038 80276ee4
[  557.912237] 5e60: ae990c10 ae7af660 883d6660 00000050 00000006 000003a4 00000000 00000000
[  557.923647] 5e80: 00000000 a01e5e88 80d0459c 8c6f0000 80d0459c 7ea7a81c 00000000 00000000
[  557.934994] 5ea0: a01e5edc a01e5eb0 8029a30c 805d1904 a01e5e88 8c6f4000 8c6f0000 00000000
[  557.946359] 5ec0: 00000026 00000002 ffffff9c ffffff9c 8c6f4000 7ea7981c a01e5f50 00000026
[  557.957715] 5ee0: a01e5f8c a01e5ef0 8029b984 8029a444 a01e5f50 a01e5f28 98785c08 a01e5ef0
[  557.969150] 5f00: 00000001 802ad51c 98785c00 00000800 00000000 00000000 7ea7a81c ffffff9c
[  557.980558] 5f20: 00000000 00000000 98785c08 00000000 ad639b10 962fd330 a01e4000 00000000
[  557.992053] 5f40: aa64a66c 0000000a 8c6f402a 80276ee4 00000000 00000000 a01e5f7c 98785c00
[  558.003698] 5f60: 98785c00 00000000 00e9a010 00e9a050 00000026 80108204 a01e4000 00000000
[  558.015188] 5f80: a01e5fa4 a01e5f90 8029bd9c 8029b8bc 00000000 00e9a050 00000000 a01e5fa8
[  558.026723] 5fa0: 80108060 8029bd74 00000000 00e9a010 7ea7981c 7ea7a81c ffffffff 00000000
[  558.038198] 5fc0: 00000000 00e9a010 00e9a050 00000026 004fd000 76f05ce8 7ea7981c 7ea7a81c
[  558.049811] 5fe0: 004fd198 7ea79814 0049cc9b 76cd7796 00010030 7ea7981c 00000000 00000000
[  558.061386] [<802a4da0>] (__d_lookup_rcu) from [<80295cc4>] (lookup_fast+0x4c/0x2c8)
[  558.072944] [<80295cc4>] (lookup_fast) from [<80298140>] (walk_component+0x3c/0x2dc)
[  558.084330] [<80298140>] (walk_component) from [<8029856c>] (link_path_walk+0x18c/0x4b0)
[  558.095582] [<8029856c>] (link_path_walk) from [<802988c4>] (path_parentat+0x34/0x6c)
[  558.106851] [<802988c4>] (path_parentat) from [<8029a4b4>] (filename_parentat+0x7c/0x104)
[  558.117923] [<8029a4b4>] (filename_parentat) from [<8029b984>] (SyS_renameat2+0xd4/0x48c)
[  558.129156] [<8029b984>] (SyS_renameat2) from [<8029bd9c>] (SyS_rename+0x34/0x3c)
[  558.140092] [<8029bd9c>] (SyS_rename) from [<80108060>] (ret_fast_syscall+0x0/0x28)
[  558.150966] Code: ea000002 e5944000 e3540000 0a000028 (e5141004) 
[  558.161756] ---[ end trace 124dc21b9499878a ]---
[  560.624738] Alignment trap: not handling instruction e1b04f9f at [<80566d48>]
[  560.635458] Unhandled fault: alignment exception (0x001) at 0x9ac47754
[  560.646194] pgd = acecc000
[  560.656802] [9ac47754] *pgd=1ac1141e(bad)
[  560.667391] Internal error: : 1 [#4] SMP ARM
[  560.677894] Modules linked in: dm_mod dax fuse loop hci_uart bluetooth ecdh_generic sg uio_pdrv_genirq uio lirc_rpi(C) lirc_dev fixed frandom ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipv6 snd_bcm2835(C) snd_pcm snd_timer snd brcmfmac cfg80211 rfkill brcmutil evdev rpcsec_gss_krb5 tun uinput
[  560.700191] CPU: 3 PID: 4194 Comm: PeripBusUSBUdev Tainted: G      D  C      4.14.29+ #1
[  560.711520] Hardware name: BCM2835
[  560.724080] task: 81945a00 task.stack: 9e712000
[  560.737866] PC is at lockref_put_return+0x3c/0x94
[  560.750567] LR is at dput+0x40/0x2d0
[  560.761780] pc : [<80566d4c>]    lr : [<802a1abc>]    psr: 20060013
[  560.773122] sp : 9e713de0  ip : 9e713e00  fp : 9e713dfc
[  560.784351] r10: 00000000  r9 : 9e712000  r8 : 9e713f60
[  560.795460] r7 : 00000001  r6 : 00010001  r5 : 00000001  r4 : 00010001
[  560.806753] r3 : 00000000  r2 : 00010001  r1 : 00000000  r0 : 9ac47754
[  560.817978] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  560.829007] Control: 10c5383d  Table: 2cecc06a  DAC: 00000055
[  560.839867] Process PeripBusUSBUdev (pid: 4194, stack limit = 0x9e712210)
[  560.851294] Stack: (0x9e713de0 to 0x9e714000)
[  560.863891] 3de0: 9ac47704 00080040 9ac47754 00000041 9e713e24 9e713e00 802a1abc 80566d1c
[  560.876801] 3e00: 00000000 9e713e78 9e713f60 00000041 9e713f60 9e712000 9e713e44 9e713e28
[  560.887838] 3e20: 80294b9c 802a1a88 9e713e78 fffffffe 9e713f60 00000041 9e713e74 9e713e48
[  560.900010] 3e40: 802989bc 80294b5c 9e6683f8 af57c6bc 00000000 00000000 00000001 8c6f5000
[  560.915033] 3e60: 9e713e78 00000001 9e713f24 9e713e78 8029a5d8 80298908 ae990310 9ac47704
[  560.929842] 3e80: e73d77c1 00000006 8c6f5025 00000000 ae990c10 ae7af660 ae7db2b0 00000001
[  560.942629] 3ea0: 9e713dd0 000003be 00000000 00000000 00000000 9e713eb8 00000000 00000000
[  560.955273] 3ec0: 00001000 8c6f6000 00000000 00000001 80d0459c 8c6f5000 80d0459c 6b2fea18
[  560.966619] 3ee0: 00000000 00000001 8c6f5000 00000000 8029a30c 00000002 ffffff9c 00000001
[  560.978523] 3f00: ffffff9c 00000001 ffffff9c 9e713f60 ffffff9c 6b2fea18 9e713f4c 9e713f28
[  560.992103] 3f20: 8029a720 8029a548 00000000 00000000 9e713f4c 00000001 00000000 acb3e980
[  561.005850] 3f40: 9e713f94 9e713f50 80287344 8029a6d8 00000000 00000000 aaa67500 00000000
[  561.017377] 3f60: 00000000 00000000 00000005 00000000 5c21ee28 76f78ce8 00000021 80108204
[  561.029029] 3f80: 9e712000 00000000 9e713fa4 9e713f98 802874c4 802872b4 00000000 9e713fa8
[  561.040579] 3fa0: 80108060 802874ac 00000000 5c21ee28 6b2fea18 00000000 00000000 6b2fea33
[  561.052969] 3fc0: 00000000 5c21ee28 76f78ce8 00000021 6b2fea18 6b2fea48 716d7b30 ffffffea
[  561.068944] 3fe0: 75eebf10 6b2fea04 75ee4b7b 759c1b66 20060030 6b2fea18 aa98abba faaaaaea
[  561.083256] [<80566d4c>] (lockref_put_return) from [<802a1abc>] (dput+0x40/0x2d0)
[  561.094752] [<802a1abc>] (dput) from [<80294b9c>] (terminate_walk+0x4c/0xc0)
[  561.106786] [<80294b9c>] (terminate_walk) from [<802989bc>] (path_lookupat+0xc0/0x204)
[  561.120543] [<802989bc>] (path_lookupat) from [<8029a5d8>] (filename_lookup+0x9c/0xf8)
[  561.133721] [<8029a5d8>] (filename_lookup) from [<8029a720>] (user_path_at_empty+0x54/0x5c)
[  561.144836] [<8029a720>] (user_path_at_empty) from [<80287344>] (SyS_faccessat+0x9c/0x1f8)
[  561.155787] [<80287344>] (SyS_faccessat) from [<802874c4>] (SyS_access+0x24/0x28)
[  561.166755] [<802874c4>] (SyS_access) from [<80108060>] (ret_fast_syscall+0x0/0x28)
[  561.177608] Code: e1a07005 da000015 f590f000 e1b04f9f (e1340006) 
[  561.188475] ---[ end trace 124dc21b9499878b ]---
[  627.422503] Unable to handle kernel NULL pointer dereference at virtual address 0000000d
[  627.436018] pgd = 82af0000
[  627.449159] [0000000d] *pgd=00000000
[  627.460412] Internal error: Oops: 5 [#5] SMP ARM
[  627.470877] Modules linked in: dm_mod dax fuse loop hci_uart bluetooth ecdh_generic sg uio_pdrv_genirq uio lirc_rpi(C) lirc_dev fixed frandom ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipv6 snd_bcm2835(C) snd_pcm snd_timer snd brcmfmac cfg80211 rfkill brcmutil evdev rpcsec_gss_krb5 tun uinput
[  627.492654] CPU: 0 PID: 10887 Comm: sudo Tainted: G      D  C      4.14.29+ #1
[  627.503546] Hardware name: BCM2835
[  627.514368] task: 91ab3c00 task.stack: 8287a000
[  627.525158] PC is at locks_remove_posix+0x30/0x14c
[  627.535881] LR is at filp_close+0x68/0x8c
[  627.547086] pc : [<802e08b0>]    lr : [<80286ac4>]    psr: 20000013
[  627.557884] sp : 8287beb0  ip : 8287bf58  fp : 8287bf54
[  627.568690] r10: 00000000  r9 : 8287a000  r8 : 80108204
[  627.579439] r7 : 00000006  r6 : acc63300  r5 : 9d640180  r4 : ae690440
[  627.590265] r3 : 00000001  r2 : ace99fc0  r1 : acc63300  r0 : 9d640180
[  627.601025] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  627.611944] Control: 10c5383d  Table: 02af006a  DAC: 00000055
[  627.622831] Process sudo (pid: 10887, stack limit = 0x8287a210)
[  627.633779] Stack: (0x8287beb0 to 0x8287c000)
[  627.644595] bea0:                                     802fe824 91ab3c00 00000041 802a8f5c
[  627.655547] bec0: 00000004 00000000 00000000 00000100 81efc688 80d093bc 00000017 808ac294
[  627.666714] bee0: 76e3be2c 8287bfb0 7e94f750 808a8780 8287bf14 8287bf00 808a8780 806fbae8
[  627.677972] bf00: add51c00 add51d74 8287bf34 8287bf18 806fbae8 808a7758 add52400 7f0220bc
[  627.689092] bf20: add52400 add52444 8287bf54 00000000 9d640180 acc63300 00000006 80108204
[  627.699934] bf40: 8287a000 00000000 8287bf74 8287bf58 80286ac4 802e088c 0000003c acc63300
[  627.710825] bf60: 9d640180 00000006 8287bf94 8287bf78 802aad38 80286a68 76f39218 0000000b
[  627.721610] bf80: 00000000 00000006 8287bfa4 8287bf98 80286b18 802aac7c 00000000 8287bfa8
[  627.732532] bfa0: 80108060 80286af4 76f39218 0000000b 0000003c 7e94f750 76f38b5c 76f38b5c
[  627.743512] bfc0: 76f39218 0000000b 00000000 00000006 022b85d0 7e94f750 76f76ce8 0000003c
[  627.754665] bfe0: 7e94f728 7e94f718 76f38b6c 76e10fc2 60000030 0000003c 00000000 00000000
[  627.765898] [<802e08b0>] (locks_remove_posix) from [<80286ac4>] (filp_close+0x68/0x8c)
[  627.777075] [<80286ac4>] (filp_close) from [<802aad38>] (__close_fd+0xc8/0xec)
[  627.788388] [<802aad38>] (__close_fd) from [<80286b18>] (SyS_close+0x30/0x58)
[  627.799587] [<80286b18>] (SyS_close) from [<80108060>] (ret_fast_syscall+0x0/0x28)
[  627.810901] Code: e59430e8 f57ff05b e3530000 0a000025 (e5b3200c) 
[  627.823021] ---[ end trace 124dc21b9499878c ]---
[  627.851770] Unable to handle kernel NULL pointer dereference at virtual address 0000000d
[  627.863130] pgd = 80004000
[  627.874478] [0000000d] *pgd=00000000
[  627.885738] Internal error: Oops: 5 [#6] SMP ARM
[  627.896909] Modules linked in: dm_mod dax fuse loop hci_uart bluetooth ecdh_generic sg uio_pdrv_genirq uio lirc_rpi(C) lirc_dev fixed frandom ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipv6 snd_bcm2835(C) snd_pcm snd_timer snd brcmfmac cfg80211 rfkill brcmutil evdev rpcsec_gss_krb5 tun uinput
[  627.920757] CPU: 3 PID: 10879 Comm: sudo Tainted: G      D  C      4.14.29+ #1
[  627.932515] Hardware name: BCM2835
[  627.944207] task: 91ab6900 task.stack: aeb86000
[  627.955689] PC is at locks_remove_posix+0x30/0x14c
[  627.967555] LR is at filp_close+0x68/0x8c
[  627.979366] pc : [<802e08b0>]    lr : [<80286ac4>]    psr: 28070013
[  627.994526] sp : aeb87d08  ip : aeb87db0  fp : aeb87dac
[  628.006388] r10: 0000000b  r9 : 8d5bba00  r8 : 00000004
[  628.017356] r7 : acc63a00  r6 : acc63a00  r5 : 9d640180  r4 : ae690440
[  628.028340] r3 : 00000001  r2 : 9d640180  r1 : acc63a00  r0 : 9d640180
[  628.039180] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  628.050082] Control: 10c5383d  Table: 2cecc06a  DAC: 00000055
[  628.060845] Process sudo (pid: 10879, stack limit = 0xaeb86210)
[  628.071621] Stack: (0xaeb87d08 to 0xaeb88000)
[  628.082270] 7d00:                   802777a0 801edc50 00000000 80d87c00 ac80efc0 af2bfff0
[  628.093150] 7d20: 8012000e 00000001 ac80efc0 00000001 48070013 000b21eb aeb87d84 8012000e
[  628.104069] 7d40: 80224d2c 808a8780 aeb87d6c aeb87d58 808a8780 806fbae8 add51c00 add51d74
[  628.115133] 7d60: aeb87d8c aeb87d70 806fbae8 808a7758 add52400 7f0220bc add52400 add52444
[  628.126354] 7d80: aeb87dac 00000000 9d640180 acc63a00 acc63a00 00000004 8d5bba00 0000000b
[  628.137591] 7da0: aeb87dcc aeb87db0 80286ac4 802e088c 00000009 000000f0 00000000 acc63a00
[  628.149003] 7dc0: aeb87df4 aeb87dd0 802aa864 80286a68 91ab6900 91ab6e40 acc63a00 ac80efc0
[  628.160356] 7de0: 00000001 ac80eff8 aeb87e14 aeb87df8 802aa974 802aa7bc 91ab6900 00000000
[  628.171934] 7e00: 00000544 ac80efc0 aeb87e54 aeb87e18 80121d6c 802aa928 aeb87e74 aeb87e28
[  628.183447] 7e20: aeb87edc aeb87fb0 00000000 0000000b 82e42100 aeb87edc aeb86000 00106001
[  628.194987] 7e40: aeb86000 0000000b aeb87e74 aeb87e58 8012260c 801219e0 00000000 8bf6da08
[  628.206554] 7e60: aeb87edc aeb86000 aeb87ec4 aeb87e78 8012d9f4 801225cc 80d02040 418004fc
[  628.218157] 7e80: 80d03d68 aeb87ec8 82e42100 8bf6de44 8bf6d940 000000a0 0000000b 76da3574
[  628.230003] 7ea0: aeb87ec8 aeb87fb0 76da3576 00000000 aeb86000 00000000 aeb87f8c aeb87ec8
[  628.241510] 7ec0: 8010b318 8012d6c8 aeb87ef4 aeb87ed8 8012d084 8012cf08 00000000 0000000b
[  628.252971] 7ee0: 00000000 00000000 00002a7f 00000000 8bf6d968 00000001 91ab6900 00000000
[  628.264251] 7f00: 00000000 00002a7f 00000000 8bf6d968 00000001 91ab6900 00000001 aeb87f68
[  628.275572] 7f20: aeb87f64 aeb87f30 8012f17c 801edc50 91ab6e44 00000002 7e94fa98 00000000
[  628.287125] 7f40: 00000002 7e94fa98 000000ae 80108204 aeb86000 00000000 aeb87fa4 aeb87f68
[  628.298685] 7f60: 8012f6f0 00000001 aeb86010 80108204 aeb87fb0 80108204 aeb86000 00000000
[  628.310431] 7f80: aeb87fac aeb87f90 8010b820 8010b260 7e94fa98 0000000b 0000000b 00000025
[  628.322157] 7fa0: 00000000 aeb87fb0 80108094 8010b774 00000000 0000000b 453b0700 453b0700
[  628.334017] 7fc0: 7e94fa98 0000000b 0000000b 00000025 022b5c58 004e516c 00000000 7e94fa08
[  628.345896] 7fe0: 004f6dd8 7e94f9b4 004d2d5b 76da3576 20040030 00002a7f 000001b8 00000000
[  628.357879] [<802e08b0>] (locks_remove_posix) from [<80286ac4>] (filp_close+0x68/0x8c)
[  628.369957] [<80286ac4>] (filp_close) from [<802aa864>] (put_files_struct+0xb4/0x10c)
[  628.381868] [<802aa864>] (put_files_struct) from [<802aa974>] (exit_files+0x58/0x5c)
[  628.393828] [<802aa974>] (exit_files) from [<80121d6c>] (do_exit+0x398/0xba0)
[  628.405659] [<80121d6c>] (do_exit) from [<8012260c>] (do_group_exit+0x4c/0xe4)
[  628.417476] [<8012260c>] (do_group_exit) from [<8012d9f4>] (get_signal+0x338/0x6e0)
[  628.429281] [<8012d9f4>] (get_signal) from [<8010b318>] (do_signal+0xc4/0x3e4)
[  628.440899] [<8010b318>] (do_signal) from [<8010b820>] (do_work_pending+0xb8/0xd0)
[  628.452491] [<8010b820>] (do_work_pending) from [<80108094>] (slow_work_pending+0xc/0x20)
[  628.463887] Code: e59430e8 f57ff05b e3530000 0a000025 (e5b3200c) 
[  628.475205] ---[ end trace 124dc21b9499878d ]---
[  628.488855] Fixing recursive fault but reboot is needed!

If you have vcgencmd, please report the output of vcgencmd get_throttled.

Hmm, that's interesting...

All of the stack traces in those kernel OOPS messages implicate absolutely nothing from the networking stack, they've just got generic core kernel stuff and VFS layer functions referenced.

The complaint about a kernel NULL pointer dereference at a very low virtual address is also somewhat suspicious. Stuff like that is usually indicative of either hardware failure, or a very poorly behaved kernel driver scribbling on memory locations it shouldn't be touching.

All of the stack traces in those kernel OOPS messages implicate absolutely nothing from the networking stack, they've just got generic core kernel stuff and VFS layer functions referenced.

I know, could it be that corrupted data read from usb disk caused this?

'Ported' the backup script to Raspbian, an run it there to backup one of my btrfs partition. Similar result, did not finished. btrfs send/receive stucks, nothing in logs. Tested with original 4.14.27-v7+ kernel and my 4.15.10+ kernel, and of course, backup was running successful on Pi3B

@pelwell

vcgencmd get_throttled
throttled=0x0

Thanks - that rules out a power supply issue.

Thanks - that rules out a power supply issue.

Already tried different PSU's, usually using brand new 2.5A PSU for Pi3B+ and an older one (2A) for Pi3B

Yes, so you said, but I'd rather hear it from the Pi.

@pelwell Are there any tests I can run to make absolutely sure that hardware is ok?

Btw, I have similar issues if root fs is on iSCSI target and no usb drive is connected and used.

@pelwell @Ferroin Oh, and before I forget: If I'm using onboard WLAN instead of onboard ethernet, issues are still there and, already changed network cable and port on the network switch and, switching root to sd-card does not make any difference, getting this on my final test for today:

[  515.046170] CIFS VFS: sends on sock a715d500 stuck for 15 seconds
[  515.046196] CIFS VFS: Error -11 sending data on socket to server
[  517.443871] systemd-logind[3010]: New session c2 of user root.
[  530.166192] CIFS VFS: sends on sock a715d500 stuck for 15 seconds
[  530.166217] CIFS VFS: Error -11 sending data on socket to server
[  540.648102] Status code returned 0xc0000008 NT_STATUS_INVALID_HANDLE
[  540.648134] CIFS VFS: Send error in read = -9
[  540.648161] print_req_error: I/O error, dev loop0, sector 231328
[  540.648195] BTRFS error (device dm-1): bdev /dev/mapper/loop0p2 errs: wr 0, rd 1, flush 0, corrupt 0, gen 0
[  735.206538] INFO: task kworker/u8:2:219 blocked for more than 120 seconds.
[  735.206553]       Tainted: G         C      4.14.29+ #1
[  735.206558] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  735.206567] kworker/u8:2    D    0   219      2 0x00000000
[  735.206596] Workqueue: writeback wb_workfn (flush-btrfs-3)
[  735.206633] [<808a5ecc>] (__schedule) from [<808a6544>] (schedule+0x50/0xa8)
[  735.206652] [<808a6544>] (schedule) from [<80476418>] (btrfs_tree_lock+0x15c/0x278)
[  735.206672] [<80476418>] (btrfs_tree_lock) from [<804578e0>] (lock_extent_buffer_for_io+0xec/0x270)
[  735.206694] [<804578e0>] (lock_extent_buffer_for_io) from [<8045b160>] (btree_write_cache_pages+0x248/0x344)
[  735.206716] [<8045b160>] (btree_write_cache_pages) from [<80422eb0>] (btree_writepages+0x84/0x8c)
[  735.206734] [<80422eb0>] (btree_writepages) from [<8022edb4>] (do_writepages+0x30/0x8c)
[  735.206750] [<8022edb4>] (do_writepages) from [<802bb208>] (__writeback_single_inode+0x44/0x434)
[  735.206766] [<802bb208>] (__writeback_single_inode) from [<802bbb08>] (writeback_sb_inodes+0x214/0x4c4)
[  735.206781] [<802bbb08>] (writeback_sb_inodes) from [<802bbe48>] (__writeback_inodes_wb+0x90/0xd0)
[  735.206797] [<802bbe48>] (__writeback_inodes_wb) from [<802bc120>] (wb_writeback+0x298/0x33c)
[  735.206811] [<802bc120>] (wb_writeback) from [<802bc9b8>] (wb_workfn+0xdc/0x4d8)
[  735.206831] [<802bc9b8>] (wb_workfn) from [<801374b8>] (process_one_work+0x158/0x454)
[  735.206850] [<801374b8>] (process_one_work) from [<80137810>] (worker_thread+0x5c/0x5b0)
[  735.206868] [<80137810>] (worker_thread) from [<8013d8a4>] (kthread+0x13c/0x16c)
[  735.206886] [<8013d8a4>] (kthread) from [<8010810c>] (ret_from_fork+0x14/0x28)
[  735.206920] INFO: task kworker/u8:3:399 blocked for more than 120 seconds.
[  735.206926]       Tainted: G         C      4.14.29+ #1
[  735.206931] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  735.206936] kworker/u8:3    D    0   399      2 0x00000000
[  735.206954] Workqueue: btrfs-endio-write btrfs_endio_write_helper
[  735.206972] [<808a5ecc>] (__schedule) from [<808a6544>] (schedule+0x50/0xa8)
[  735.206988] [<808a6544>] (schedule) from [<808aa12c>] (schedule_timeout+0x1d0/0x3e4)
[  735.207005] [<808aa12c>] (schedule_timeout) from [<808a71cc>] (wait_for_common+0xc0/0x184)
[  735.207021] [<808a71cc>] (wait_for_common) from [<808a72b0>] (wait_for_completion+0x20/0x24)
[  735.207036] [<808a72b0>] (wait_for_completion) from [<8040c478>] (btrfs_async_run_delayed_refs+0x12c/0x14c)
[  735.207053] [<8040c478>] (btrfs_async_run_delayed_refs) from [<8042d7ec>] (__btrfs_end_transaction+0x228/0x324)
[  735.207069] [<8042d7ec>] (__btrfs_end_transaction) from [<8042d904>] (btrfs_end_transaction+0x1c/0x20)
[  735.207085] [<8042d904>] (btrfs_end_transaction) from [<80438b40>] (btrfs_finish_ordered_io+0x230/0x848)
[  735.207102] [<80438b40>] (btrfs_finish_ordered_io) from [<8043955c>] (finish_ordered_fn+0x1c/0x20)
[  735.207118] [<8043955c>] (finish_ordered_fn) from [<8046a1d8>] (normal_work_helper+0xb0/0x3e0)
[  735.207135] [<8046a1d8>] (normal_work_helper) from [<8046a8f0>] (btrfs_endio_write_helper+0x1c/0x20)
[  735.207152] [<8046a8f0>] (btrfs_endio_write_helper) from [<801374b8>] (process_one_work+0x158/0x454)
[  735.207170] [<801374b8>] (process_one_work) from [<80137810>] (worker_thread+0x5c/0x5b0)
[  735.207187] [<80137810>] (worker_thread) from [<8013d8a4>] (kthread+0x13c/0x16c)
[  735.207203] [<8013d8a4>] (kthread) from [<8010810c>] (ret_from_fork+0x14/0x28)
[  735.207217] INFO: task kworker/u8:6:759 blocked for more than 120 seconds.
[  735.207224]       Tainted: G         C      4.14.29+ #1
[  735.207229] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  735.207233] kworker/u8:6    D    0   759      2 0x00000000
[  735.207250] Workqueue: btrfs-extent-refs btrfs_extent_refs_helper
[  735.207269] [<808a5ecc>] (__schedule) from [<808a6544>] (schedule+0x50/0xa8)
[  735.207284] [<808a6544>] (schedule) from [<80475e2c>] (btrfs_tree_read_lock+0x120/0x198)
[  735.207303] [<80475e2c>] (btrfs_tree_read_lock) from [<803f9660>] (btrfs_read_lock_root_node+0x38/0x50)
[  735.207321] [<803f9660>] (btrfs_read_lock_root_node) from [<803ff188>] (btrfs_search_slot+0x8b0/0xb10)
[  735.207339] [<803ff188>] (btrfs_search_slot) from [<80400f10>] (btrfs_insert_empty_items+0x7c/0xd4)
[  735.207357] [<80400f10>] (btrfs_insert_empty_items) from [<8040f01c>] (__btrfs_run_delayed_refs+0xe90/0x16fc)
[  735.207374] [<8040f01c>] (__btrfs_run_delayed_refs) from [<80412d9c>] (btrfs_run_delayed_refs+0x9c/0x310)
[  735.207390] [<80412d9c>] (btrfs_run_delayed_refs) from [<804130c4>] (delayed_ref_async_start+0xb4/0xc0)
[  735.207405] [<804130c4>] (delayed_ref_async_start) from [<8046a1d8>] (normal_work_helper+0xb0/0x3e0)
[  735.207419] [<8046a1d8>] (normal_work_helper) from [<8046a990>] (btrfs_extent_refs_helper+0x1c/0x20)
[  735.207437] [<8046a990>] (btrfs_extent_refs_helper) from [<801374b8>] (process_one_work+0x158/0x454)
[  735.207455] [<801374b8>] (process_one_work) from [<80137810>] (worker_thread+0x5c/0x5b0)
[  735.207471] [<80137810>] (worker_thread) from [<8013d8a4>] (kthread+0x13c/0x16c)
[  735.207487] [<8013d8a4>] (kthread) from [<8010810c>] (ret_from_fork+0x14/0x28)
[  735.207625] INFO: task btrfs-transacti:8152 blocked for more than 120 seconds.
[  735.207632]       Tainted: G         C      4.14.29+ #1
[  735.207637] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  735.207641] btrfs-transacti D    0  8152      2 0x00000000
[  735.207663] [<808a5ecc>] (__schedule) from [<808a6544>] (schedule+0x50/0xa8)
[  735.207678] [<808a6544>] (schedule) from [<8014b128>] (io_schedule+0x20/0x40)
[  735.207693] [<8014b128>] (io_schedule) from [<8021cd60>] (wait_on_page_bit+0x120/0x140)
[  735.207711] [<8021cd60>] (wait_on_page_bit) from [<80459a00>] (read_extent_buffer_pages+0x258/0x33c)
[  735.207731] [<80459a00>] (read_extent_buffer_pages) from [<80422350>] (btree_read_extent_buffer_pages+0xb0/0x120)
[  735.207747] [<80422350>] (btree_read_extent_buffer_pages) from [<80423818>] (read_tree_block+0x38/0x54)
[  735.207763] [<80423818>] (read_tree_block) from [<803f7914>] (read_node_slot+0xc4/0x100)
[  735.207779] [<803f7914>] (read_node_slot) from [<803fd508>] (push_leaf_right+0xb8/0x1f0)
[  735.207796] [<803fd508>] (push_leaf_right) from [<803fe6c8>] (split_leaf+0x5f4/0x804)
[  735.207813] [<803fe6c8>] (split_leaf) from [<803ff270>] (btrfs_search_slot+0x998/0xb10)
[  735.207830] [<803ff270>] (btrfs_search_slot) from [<80400f10>] (btrfs_insert_empty_items+0x7c/0xd4)
[  735.207847] [<80400f10>] (btrfs_insert_empty_items) from [<8040f01c>] (__btrfs_run_delayed_refs+0xe90/0x16fc)
[  735.207863] [<8040f01c>] (__btrfs_run_delayed_refs) from [<80412d9c>] (btrfs_run_delayed_refs+0x9c/0x310)
[  735.207879] [<80412d9c>] (btrfs_run_delayed_refs) from [<8042c378>] (btrfs_commit_transaction+0x38/0xc2c)
[  735.207893] [<8042c378>] (btrfs_commit_transaction) from [<804274bc>] (transaction_kthread+0x1b4/0x1c8)
[  735.207908] [<804274bc>] (transaction_kthread) from [<8013d8a4>] (kthread+0x13c/0x16c)
[  735.207923] [<8013d8a4>] (kthread) from [<8010810c>] (ret_from_fork+0x14/0x28)
[  735.207947] INFO: task btrfs:9851 blocked for more than 120 seconds.
[  735.207953]       Tainted: G         C      4.14.29+ #1
[  735.207958] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  735.207963] btrfs           D    0  9851   9850 0x00000000
[  735.207984] [<808a5ecc>] (__schedule) from [<808a6544>] (schedule+0x50/0xa8)
[  735.207999] [<808a6544>] (schedule) from [<8044fdec>] (btrfs_start_ordered_extent+0x128/0x158)
[  735.208016] [<8044fdec>] (btrfs_start_ordered_extent) from [<80450394>] (btrfs_wait_ordered_range+0x140/0x18c)
[  735.208032] [<80450394>] (btrfs_wait_ordered_range) from [<8043b4f0>] (btrfs_truncate+0x54/0x2c8)
[  735.208048] [<8043b4f0>] (btrfs_truncate) from [<8043c074>] (btrfs_setattr+0x2f4/0x488)
[  735.208067] [<8043c074>] (btrfs_setattr) from [<802a9318>] (notify_change+0x1cc/0x408)
[  735.208084] [<802a9318>] (notify_change) from [<80286c98>] (do_truncate+0x90/0xc0)
[  735.208099] [<80286c98>] (do_truncate) from [<80286eac>] (vfs_truncate+0x1e4/0x25c)
[  735.208113] [<80286eac>] (vfs_truncate) from [<80286fa0>] (do_sys_truncate+0x7c/0xac)
[  735.208127] [<80286fa0>] (do_sys_truncate) from [<80287200>] (SyS_truncate64+0x18/0x1c)
[  735.208143] [<80287200>] (SyS_truncate64) from [<80108060>] (ret_fast_syscall+0x0/0x28)
[  858.087442] INFO: task kworker/u8:2:219 blocked for more than 120 seconds.
[  858.087457]       Tainted: G         C      4.14.29+ #1
[  858.087462] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  858.087469] kworker/u8:2    D    0   219      2 0x00000000
[  858.087503] Workqueue: writeback wb_workfn (flush-btrfs-3)
[  858.087538] [<808a5ecc>] (__schedule) from [<808a6544>] (schedule+0x50/0xa8)
[  858.087557] [<808a6544>] (schedule) from [<80476418>] (btrfs_tree_lock+0x15c/0x278)
[  858.087579] [<80476418>] (btrfs_tree_lock) from [<804578e0>] (lock_extent_buffer_for_io+0xec/0x270)
[  858.087601] [<804578e0>] (lock_extent_buffer_for_io) from [<8045b160>] (btree_write_cache_pages+0x248/0x344)
[  858.087624] [<8045b160>] (btree_write_cache_pages) from [<80422eb0>] (btree_writepages+0x84/0x8c)
[  858.087642] [<80422eb0>] (btree_writepages) from [<8022edb4>] (do_writepages+0x30/0x8c)
[  858.087659] [<8022edb4>] (do_writepages) from [<802bb208>] (__writeback_single_inode+0x44/0x434)
[  858.087675] [<802bb208>] (__writeback_single_inode) from [<802bbb08>] (writeback_sb_inodes+0x214/0x4c4)
[  858.087690] [<802bbb08>] (writeback_sb_inodes) from [<802bbe48>] (__writeback_inodes_wb+0x90/0xd0)
[  858.087705] [<802bbe48>] (__writeback_inodes_wb) from [<802bc120>] (wb_writeback+0x298/0x33c)
[  858.087719] [<802bc120>] (wb_writeback) from [<802bc9b8>] (wb_workfn+0xdc/0x4d8)
[  858.087739] [<802bc9b8>] (wb_workfn) from [<801374b8>] (process_one_work+0x158/0x454)
[  858.087758] [<801374b8>] (process_one_work) from [<80137810>] (worker_thread+0x5c/0x5b0)
[  858.087775] [<80137810>] (worker_thread) from [<8013d8a4>] (kthread+0x13c/0x16c)
[  858.087793] [<8013d8a4>] (kthread) from [<8010810c>] (ret_from_fork+0x14/0x28)
[  858.087827] INFO: task kworker/u8:3:399 blocked for more than 120 seconds.
[  858.087833]       Tainted: G         C      4.14.29+ #1
[  858.087838] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  858.087843] kworker/u8:3    D    0   399      2 0x00000000
[  858.087861] Workqueue: btrfs-endio-write btrfs_endio_write_helper
[  858.087879] [<808a5ecc>] (__schedule) from [<808a6544>] (schedule+0x50/0xa8)
[  858.087895] [<808a6544>] (schedule) from [<808aa12c>] (schedule_timeout+0x1d0/0x3e4)
[  858.087912] [<808aa12c>] (schedule_timeout) from [<808a71cc>] (wait_for_common+0xc0/0x184)
[  858.087929] [<808a71cc>] (wait_for_common) from [<808a72b0>] (wait_for_completion+0x20/0x24)
[  858.087944] [<808a72b0>] (wait_for_completion) from [<8040c478>] (btrfs_async_run_delayed_refs+0x12c/0x14c)
[  858.087961] [<8040c478>] (btrfs_async_run_delayed_refs) from [<8042d7ec>] (__btrfs_end_transaction+0x228/0x324)
[  858.087976] [<8042d7ec>] (__btrfs_end_transaction) from [<8042d904>] (btrfs_end_transaction+0x1c/0x20)
[  858.087994] [<8042d904>] (btrfs_end_transaction) from [<80438b40>] (btrfs_finish_ordered_io+0x230/0x848)
[  858.088011] [<80438b40>] (btrfs_finish_ordered_io) from [<8043955c>] (finish_ordered_fn+0x1c/0x20)
[  858.088027] [<8043955c>] (finish_ordered_fn) from [<8046a1d8>] (normal_work_helper+0xb0/0x3e0)
[  858.088042] [<8046a1d8>] (normal_work_helper) from [<8046a8f0>] (btrfs_endio_write_helper+0x1c/0x20)
[  858.088060] [<8046a8f0>] (btrfs_endio_write_helper) from [<801374b8>] (process_one_work+0x158/0x454)
[  858.088077] [<801374b8>] (process_one_work) from [<80137810>] (worker_thread+0x5c/0x5b0)
[  858.088094] [<80137810>] (worker_thread) from [<8013d8a4>] (kthread+0x13c/0x16c)
[  858.088109] [<8013d8a4>] (kthread) from [<8010810c>] (ret_from_fork+0x14/0x28)
[  858.088124] INFO: task kworker/u8:6:759 blocked for more than 120 seconds.
[  858.088130]       Tainted: G         C      4.14.29+ #1
[  858.088135] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  858.088140] kworker/u8:6    D    0   759      2 0x00000000
[  858.088157] Workqueue: btrfs-extent-refs btrfs_extent_refs_helper
[  858.088176] [<808a5ecc>] (__schedule) from [<808a6544>] (schedule+0x50/0xa8)
[  858.088191] [<808a6544>] (schedule) from [<80475e2c>] (btrfs_tree_read_lock+0x120/0x198)
[  858.088210] [<80475e2c>] (btrfs_tree_read_lock) from [<803f9660>] (btrfs_read_lock_root_node+0x38/0x50)
[  858.088228] [<803f9660>] (btrfs_read_lock_root_node) from [<803ff188>] (btrfs_search_slot+0x8b0/0xb10)
[  858.088246] [<803ff188>] (btrfs_search_slot) from [<80400f10>] (btrfs_insert_empty_items+0x7c/0xd4)
[  858.088264] [<80400f10>] (btrfs_insert_empty_items) from [<8040f01c>] (__btrfs_run_delayed_refs+0xe90/0x16fc)
[  858.088281] [<8040f01c>] (__btrfs_run_delayed_refs) from [<80412d9c>] (btrfs_run_delayed_refs+0x9c/0x310)
[  858.088297] [<80412d9c>] (btrfs_run_delayed_refs) from [<804130c4>] (delayed_ref_async_start+0xb4/0xc0)
[  858.088312] [<804130c4>] (delayed_ref_async_start) from [<8046a1d8>] (normal_work_helper+0xb0/0x3e0)
[  858.088326] [<8046a1d8>] (normal_work_helper) from [<8046a990>] (btrfs_extent_refs_helper+0x1c/0x20)
[  858.088344] [<8046a990>] (btrfs_extent_refs_helper) from [<801374b8>] (process_one_work+0x158/0x454)
[  858.088362] [<801374b8>] (process_one_work) from [<80137810>] (worker_thread+0x5c/0x5b0)
[  858.088378] [<80137810>] (worker_thread) from [<8013d8a4>] (kthread+0x13c/0x16c)
[  858.088394] [<8013d8a4>] (kthread) from [<8010810c>] (ret_from_fork+0x14/0x28)
[  858.088532] INFO: task btrfs-transacti:8152 blocked for more than 120 seconds.
[  858.088539]       Tainted: G         C      4.14.29+ #1
[  858.088544] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  858.088548] btrfs-transacti D    0  8152      2 0x00000000
[  858.088570] [<808a5ecc>] (__schedule) from [<808a6544>] (schedule+0x50/0xa8)
[  858.088585] [<808a6544>] (schedule) from [<8014b128>] (io_schedule+0x20/0x40)
[  858.088600] [<8014b128>] (io_schedule) from [<8021cd60>] (wait_on_page_bit+0x120/0x140)
[  858.088617] [<8021cd60>] (wait_on_page_bit) from [<80459a00>] (read_extent_buffer_pages+0x258/0x33c)
[  858.088637] [<80459a00>] (read_extent_buffer_pages) from [<80422350>] (btree_read_extent_buffer_pages+0xb0/0x120)
[  858.088653] [<80422350>] (btree_read_extent_buffer_pages) from [<80423818>] (read_tree_block+0x38/0x54)
[  858.088669] [<80423818>] (read_tree_block) from [<803f7914>] (read_node_slot+0xc4/0x100)
[  858.088686] [<803f7914>] (read_node_slot) from [<803fd508>] (push_leaf_right+0xb8/0x1f0)
[  858.088703] [<803fd508>] (push_leaf_right) from [<803fe6c8>] (split_leaf+0x5f4/0x804)
[  858.088720] [<803fe6c8>] (split_leaf) from [<803ff270>] (btrfs_search_slot+0x998/0xb10)
[  858.088737] [<803ff270>] (btrfs_search_slot) from [<80400f10>] (btrfs_insert_empty_items+0x7c/0xd4)
[  858.088754] [<80400f10>] (btrfs_insert_empty_items) from [<8040f01c>] (__btrfs_run_delayed_refs+0xe90/0x16fc)
[  858.088770] [<8040f01c>] (__btrfs_run_delayed_refs) from [<80412d9c>] (btrfs_run_delayed_refs+0x9c/0x310)
[  858.088786] [<80412d9c>] (btrfs_run_delayed_refs) from [<8042c378>] (btrfs_commit_transaction+0x38/0xc2c)
[  858.088800] [<8042c378>] (btrfs_commit_transaction) from [<804274bc>] (transaction_kthread+0x1b4/0x1c8)
[  858.088814] [<804274bc>] (transaction_kthread) from [<8013d8a4>] (kthread+0x13c/0x16c)
[  858.088830] [<8013d8a4>] (kthread) from [<8010810c>] (ret_from_fork+0x14/0x28)
[  858.088854] INFO: task btrfs:9851 blocked for more than 120 seconds.
[  858.088860]       Tainted: G         C      4.14.29+ #1
[  858.088865] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  858.088869] btrfs           D    0  9851   9850 0x00000000
[  858.088891] [<808a5ecc>] (__schedule) from [<808a6544>] (schedule+0x50/0xa8)
[  858.088906] [<808a6544>] (schedule) from [<8044fdec>] (btrfs_start_ordered_extent+0x128/0x158)
[  858.088924] [<8044fdec>] (btrfs_start_ordered_extent) from [<80450394>] (btrfs_wait_ordered_range+0x140/0x18c)
[  858.088941] [<80450394>] (btrfs_wait_ordered_range) from [<8043b4f0>] (btrfs_truncate+0x54/0x2c8)
[  858.088957] [<8043b4f0>] (btrfs_truncate) from [<8043c074>] (btrfs_setattr+0x2f4/0x488)
[  858.088975] [<8043c074>] (btrfs_setattr) from [<802a9318>] (notify_change+0x1cc/0x408)
[  858.088993] [<802a9318>] (notify_change) from [<80286c98>] (do_truncate+0x90/0xc0)
[  858.089008] [<80286c98>] (do_truncate) from [<80286eac>] (vfs_truncate+0x1e4/0x25c)
[  858.089022] [<80286eac>] (vfs_truncate) from [<80286fa0>] (do_sys_truncate+0x7c/0xac)
[  858.089036] [<80286fa0>] (do_sys_truncate) from [<80287200>] (SyS_truncate64+0x18/0x1c)
[  858.089052] [<80287200>] (SyS_truncate64) from [<80108060>] (ret_fast_syscall+0x0/0x28)

Btw, system (root partition) was cloned from usb partition to sd-card using same backup procedure and ran without any issues.

Have you got another Pi3B+ you can try this on. It really is very bizarre, and bizarre always makes me think HW fault.

Have you got another Pi3B+ you can try this on. It really is very bizarre, and bizarre always makes me think HW fault.

No, unfortunately not

It doesn't feel like a hardware fault to me - if so, you would expect to see it on a stock Raspbian kernel as well, and so far I don't think we have.

if so, you would expect to see it on a stock Raspbian kernel as well

Already posted. It happens on Raspbian using standard kernel (4.14.27+) as well

Yes, reading your earlier comment again I can see how it means that.

Can you try limiting the ARM cores to 1.2GHz by adding the following to config.txt?:

arm_freq=1200

@pelwell I'll try this. Currently running another test to a different machine, with very strange result

getting periodically message like this:

[ 1416.176886] nfs: server kmxbmc not responding, still trying
[ 1417.217789] nfs: server kmxbmc not responding, still trying
[ 1417.575681] nfs: server kmxbmc OK
[ 1417.588839] nfs: server kmxbmc OK

The interval is exactly 239s. Any idea where this value comes from. NFS share is mounted as follows

kmxbmc:/srv on /mnt type nfs (rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.5,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=192.168.1.5)

@pelwell Doesn't help :disappointed:

... and it does not depend on kernel version. 4.9.xx producing same issues

There is an interesting thread on OSMC forum about a 3B+ network issue. When user connected his RPi directly to the switch, issue disappeared.

RPi 3B (without +) can reach 100 mbps max, so can this issue be related to the higher speed that requires network adjustments i.e. a bigger buffer. Did changing MTU size, or Ethernet cables, or using different 100mbps/1Gbps switches make any difference?

It would be helpful if @mkreisl explain how is his RPi connected exactly? When working what speeds RPi 3B and 3B+ achieve?

@fiery- Nothing special. Connected directly to my router (4-port GBit switch) via 5m CAT5 ethernet cable Sending data from /dev/zero to server 33MB/s Receiving data from server 29MB/s You haven't read this thread completey. Issue appears on WLAN connection also

@mkreisl Any suspicious output in dmesg? What is the MTU of your onboard ethernet interface? Does it help to reduce the MTU to 1496? What the output of ethtool -S eth0 after the issue appeared?

@lategoodbye

Any suspicious output in dmesg?

Already posted dmesg (see above). But output differs extremely. Sometimes nothing, copy process just stucks, sometimes system completely freezed, sometimes message below, sometimes kernel Oops ...

[  347.207072] CIFS VFS: sends on sock 81c33340 stuck for 15 seconds
[  347.207099] CIFS VFS: Error -11 sending data on socket to server
[  362.328036] CIFS VFS: sends on sock 81c33340 stuck for 15 seconds
[  362.328066] CIFS VFS: Error -11 sending data on socket to server
[  362.418887] CIFS VFS: Free previous auth_key.response = ad8a3540
[  368.135439] CIFS VFS: Free previous auth_key.response = 9cd2a840
[  372.970214] Status code returned 0xc0000128 STATUS_FILE_CLOSED
[  372.970259] CIFS VFS: Send error in read = -9
[  372.970297] print_req_error: I/O error, dev loop0, sector 296512
[  372.970334] BTRFS error (device dm-1): bdev /dev/mapper/loop0p2 errs: wr 0, rd 1, flush 0, corrupt 0, gen 0
[  389.918475] print_req_error: I/O error, dev loop0, sector 0
[  389.918755] BTRFS error (device dm-1): bdev /dev/mapper/loop0p2 errs: wr 0, rd 1, flush 1, corrupt 0, gen 0
[  389.918775] BTRFS warning (device dm-1): chunk 1048576 missing 1 devices, max tolerance is 0 for writeable mount
[  389.918808] BTRFS: error (device dm-1) in write_all_supers:3670: errno=-5 IO failure (errors while submitting device barriers.)
[  389.918825] BTRFS info (device dm-1): forced readonly
[  389.918841] BTRFS warning (device dm-1): Skipping commit of aborted transaction.
[  389.918851] BTRFS: error (device dm-1) in cleanup_transaction:1873: errno=-5 IO failure
[  389.918863] BTRFS info (device dm-1): delayed_refs has NO entry

What is the MTU of your onboard ethernet interface?

Default (1500)

Does it help to reduce the MTU to 1496?

What the output of ethtool -S eth0 after the issue appeared?

NIC statistics:
     RX FCS Errors: 0
     RX Alignment Errors: 0
     Rx Fragment Errors: 0
     RX Jabber Errors: 0
     RX Undersize Frame Errors: 0
     RX Oversize Frame Errors: 0
     RX Dropped Frames: 0
     RX Unicast Byte Count: 82919099
     RX Broadcast Byte Count: 32537
     RX Multicast Byte Count: 199380
     RX Unicast Frames: 460932
     RX Broadcast Frames: 394
     RX Multicast Frames: 554
     RX Pause Frames: 708426
     RX 64 Byte Frames: 329
     RX 65 - 127 Byte Frames: 428916
     RX 128 - 255 Byte Frames: 2601
     RX 256 - 511 Bytes Frames: 592
     RX 512 - 1023 Byte Frames: 307
     RX 1024 - 1518 Byte Frames: 29135
     RX Greater 1518 Byte Frames: 0
     EEE RX LPI Transitions: 0
     EEE RX LPI Time: 0
     TX FCS Errors: 0
     TX Excess Deferral Errors: 0
     TX Carrier Errors: 0
     TX Bad Byte Count: 0
     TX Single Collisions: 0
     TX Multiple Collisions: 0
     TX Excessive Collision: 0
     TX Late Collisions: 0
     TX Unicast Byte Count: 1673525473
     TX Broadcast Byte Count: 4804
     TX Multicast Byte Count: 26394
     TX Unicast Frames: 1126605
     TX Broadcast Frames: 44
     TX Multicast Frames: 118
     TX Pause Frames: 8
     TX 64 Byte Frames: 37
     TX 65 - 127 Byte Frames: 13575
     TX 128 - 255 Byte Frames: 8217
     TX 256 - 511 Bytes Frames: 1747
     TX 512 - 1023 Byte Frames: 144
     TX 1024 - 1518 Byte Frames: 1103047
     TX Greater 1518 Byte Frames: 0
     EEE TX LPI Transitions: 0
     EEE TX LPI Time: 0

Now I did final test, put 100MBit switch between, to force a 100MBit link instead of 1GBit.

Makes no difference, now sending back that crap :angry:

The same with samba and share (NFS / etx4 ...) In HDD extern or SD internal. After reboot its work perfectly for 30/40min and after kick of share and reconnect again and again. Network work perfectly (no ping out ...) I feel owerload the queue or something like that.

Same problem here. I'm using Samba and external USB HDD. It's quite random, but it's just a matter of time until it happens, no dmesg problems indicated, throttled=0x0. RPi is connected directly to 8 port GB switch via cat5e cable.

I have exactly the same problem and I'm glad to see I'm not alone in this, after rebooting I can transfer files through sftp (using filezilla on windows) without issue but after half an hour or so after booting up the transfers don't work anymore: It hangs after a second or two of high speed transfer (18MBps ), then times out after and tries to reconnect after 20 seconds only to fail again after a second or two of transfer. I also tried the same with transferring the files from the sd card (without hdd connected) and it has the same issues, when I put the sd card into a 3B it has no issues whatsoever tranferring these files even from the hard drive. I'm using Raspbian and the official Raspberry Pi charger if that helps. If it is hardware related I'd love to know so I can get it exchanged by the seller.

Seems that the Raspberry Pi team hasn't tested anything before releasing the Pi3B+ :angry:

Yes, that right, because we just knocked up some hardware, and never checked anything worked before releasing it.....

Or on the other hand perhaps we spent a year making this product and did, actually, test it, but just never saw this error?

Might also be worth considering that we have sold >200k B+ already, and the number of reports of this error are small in comparison, so its perhaps some weird interaction between hubs, drivers, firmware and hardware. Whatever it is, we are looking in to it. My suspicion is that its a driver issue that we will figure out, although sometimes these things can take a while. On the 3B, there was a bug in the smsc95xx driver, matched with a bug in the brcmfmac software, that we didn't get to the bottom off for about a year - but its was a difficult bug to reproduce. Neigher pieces of software written by us incidentally, in much the same way that the lan78xx driver or the chip firmware, which is presumably at fault here, wasn't written by us either.

On 24 March 2018 at 17:47, Manfred Kreisl notifications@github.com wrote:

Seems that the Raspberry Pi team hasn't tested anything before releasing the Pi3B+ 😠

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/raspberrypi/linux/issues/2449#issuecomment-375911662, or mute the thread https://github.com/notifications/unsubscribe-auth/ADqrHVU1fgXWVDM31GSuGtcwNmzeW65yks5thobIgaJpZM4SvXe8 .

-- James Hughes Principal Software Engineer, Raspberry Pi (Trading) Ltd

I'd just like to say that the team at Raspberry Pi are awesome! There's always niggles with new hardware and software, it's par for the course, and impossible to test every permutation out there. I have full confidence that they'll look into whatever issues there are and get a fix out when they can!

@JamesH65 You havent read this thread completely

This issue happens regardless if lan78xx driver is used or not. I was building lan78xx as module and unloaded it before starting test And issue remains

I did say I think its a driver problem (lan78xx might be a culprit, but apparently not), and having skimmed the thread that still seems the most likely explanation. Something somewhere is dumping on memory it shouldn't (which, incidentally, was the cause of the bug I referred to above, one driver was dumping on another drivers skb allocation). As always with these things, if we can replicate the issue at our end, then we have a much better chance of figuring it out.

The effort to find and fix this problem would be helped enormously if someone could write down the steps to get from a known OS image (ideally Raspbian, but it doesn't have to be) to an observed failure.

@pelwell

The effort to find and fix this problem would be helped enormously if someone could write down the steps to get from a known OS image (ideally Raspbian, but it doesn't have to be) to an observed failure.

1) Install XBian 2) mount network share 3) run backup inside of xbian-config, menu item 6 (xbian copier), use file:/mount-to network-share/test.img as destination

Maybe some more data has to be written to the fs before running test (in my case I have about 2.5GB data to copy)

That's in short words what I'm doing

ps Actually I have been successfuly written 5 images after

1) building lan78xx driver as module 2) using wlan and not ethernet, unloading lan78xx module (this makes things a bit better but issue still remains and Kodi crashes randomly) 3) powering off the usb part by running script (after this everything seems to be stable and Kodi does not crash)

#!/bin/bash

BUSPOWER="/sys/devices/platform/bcm2708_usb/buspower"
for USB in $(find /sys/devices/platform/soc/ -name *.usb 2>/dev/null); do
    [ -e $USB ] && { BUSPOWER=$USB/buspower; break; }
done

echo 0x0 > $BUSPOWER

For me it sounds like the usbhub/ethernet (that's one of the major difference to Pi3B) part does not get stable power for a reliable work (that chip has higher power consumption than the chip used on Pi3B)

Thank you - your instructions look mercifully short and simple. As you say, the lan78xx and its driver seems the most likely culprit, but I don't want to leap to any conclusions.

Thank you - your instructions look mercifully short and simple. As you say, the lan78xx and its driver seems the most likely culprit, but I don't want to leap to any conclusions.

... and for running test in non-interactive loop a script like this:

#!/bin/bash

for i in $(seq 1 10); do
    logger running test $i
    /usr/sbin/btrfs-auto-snapshot xbiancopy --helper --img /dev/root /srv/backup/test.img
done

That's what I'm running at the moment

@pelwell @JamesH65

it's very simple to reproduce !!! mount a network share paste a video and read it from the share. You will systematically logout the video after several minutes of reading... it works also with a copy of a very large file.

Here some questions for those who can reproduce the issue: Are you able to reproduce it with Raspbian? Do you have an external HDD connected via USB during the issue? Which filesystem is the base on the server? Are there any modifications to the cmdline.txt or config.txt?

@lategoodbye

Me yes I am on Raspbian with firmware basic (4.9)

Again on HDD or internal storage it's the same things.

My cmdline and config


 $ cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=8d333b2b-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait rootdelay=3
pi@MyBilly:~ $ cat /boot/config.txt
# For more options and information see
# http://rpf.io/configtxt
# Some settings may impact device functionality. See link above for details

# uncomment if you get no picture on HDMI for a default "safe" mode
#hdmi_safe=1

# uncomment this if your display has a black border of unused pixels visible
# and your display can output without overscan
#disable_overscan=1

# uncomment the following to adjust overscan. Use positive numbers if console
# goes off screen, and negative if there is too much border
#overscan_left=16
#overscan_right=16
#overscan_top=16
#overscan_bottom=16

# uncomment to force a console size. By default it will be display's size minus
# overscan.
#framebuffer_width=1280
#framebuffer_height=720

# uncomment if hdmi display is not detected and composite is being output
#hdmi_force_hotplug=1

# uncomment to force a specific HDMI mode (this will force VGA)
#hdmi_group=1
#hdmi_mode=1

# uncomment to force a HDMI mode rather than DVI. This can make audio work in
# DMT (computer monitor) modes
#hdmi_drive=2

# uncomment to increase signal to HDMI, if you have interference, blanking, or
# no display
#config_hdmi_boost=4

# uncomment for composite PAL
#sdtv_mode=2

#uncomment to overclock the arm. 700 MHz is the default.
#arm_freq=800

# Uncomment some or all of these to enable the optional hardware interfaces
#dtparam=i2c_arm=on
#dtparam=i2s=on
#dtparam=spi=on

# Uncomment this to enable the lirc-rpi module
#dtoverlay=lirc-rpi

dtoverlay=pi3-disable-bt
dtoverlay=pi3-disable-wifi

# Additional overlays and parameters are documented /boot/overlays/README

# Enable audio (loads snd_bcm2835)
dtparam=audio=off
`

Here some questions for those who can reproduce the issue: Are you able to reproduce it with Raspbian?

Yes, already posted

Do you have an external HDD connected via USB during the issue?

No/Yes, does not matter

Which filesystem is the base on the server?

Doesn't matter, cifs, nfs4 or sshfs share mounted on server

Are there any modifications to the cmdline.txt or config.txt?

raspberrypi / linux

Pi3B+ : USB+ethernet file transfer problem #2449