hc1/xu4: kernel panic in ipv6_route_dev_notify stack

[paralin]

Experiencing an intermittent kernel panic on the hc1. Full log here: https://gist.github.com/paralin/7eec5df5d208d55d4bc04e022eafffa4#file-panic-log

Relevant snip:

Nov 03 13:17:16 skiff-21fac66 kernel: Unable to handle kernel NULL pointer dereference at virtual address 0000007c
Nov 03 13:17:16 skiff-21fac66 kernel: pgd = c0003000
Nov 03 13:17:17 skiff-21fac66 kernel: [0000007c] *pgd=80000040004003, *pmd=00000000
Nov 03 13:17:17 skiff-21fac66 kernel: Internal error: Oops: 207 [#1] PREEMPT SMP ARM
Nov 03 13:17:17 skiff-21fac66 kernel: Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc ipv6 joydev ads7846 spidev sch_fq_codel spi_s3c64xx w1_gpio wire exynos_gpiomem
Nov 03 13:17:17 skiff-21fac66 kernel: CPU: 4 PID: 6 Comm: kworker/u16:0 Not tainted 4.9.47 #1
Nov 03 13:17:17 skiff-21fac66 kernel: Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
Nov 03 13:17:17 skiff-21fac66 kernel: Workqueue: netns cleanup_net
Nov 03 13:17:17 skiff-21fac66 kernel: task: ee899f40 task.stack: ee8b2000
Nov 03 13:17:17 skiff-21fac66 kernel: PC is at ip6_route_dev_notify+0x44/0x1a8 [ipv6]
Nov 03 13:17:17 skiff-21fac66 kernel: LR is at notifier_call_chain+0x44/0x84
Nov 03 13:17:17 skiff-21fac66 kernel: pc : [<bf0442c0>]    lr : [<c0252208>]    psr: 60070013
                                      sp : ee8b3e18  ip : bf077d60  fp : ffffffff
Nov 03 13:17:17 skiff-21fac66 kernel: r10: c1045180  r9 : c0bba400  r8 : 00000000
Nov 03 13:17:17 skiff-21fac66 kernel: r7 : 00000000  r6 : 00000006  r5 : c5770000  r4 : c5748800
Nov 03 13:17:17 skiff-21fac66 kernel: r3 : 0000007c  r2 : ee8b3e5c  r1 : 00000006  r0 : 00000000
Nov 03 13:17:17 skiff-21fac66 kernel: Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
Nov 03 13:17:17 skiff-21fac66 kernel: Control: 30c5387d  Table: 46bf8dc0  DAC: 55555555
Nov 03 13:17:17 skiff-21fac66 kernel: Process kworker/u16:0 (pid: 6, stack limit = 0xee8b2210)
Nov 03 13:17:17 skiff-21fac66 kernel: Stack: (0xee8b3e18 to 0xee8b4000)
Nov 03 13:17:17 skiff-21fac66 kernel: 3e00:                                                       ffffffeb ee8b3e5c
Nov 03 13:17:17 skiff-21fac66 kernel: 3e20: 00000006 00000000 00000000 c0252208 c5748800 024000c0 ee8b3e94 c0bba400
Nov 03 13:17:17 skiff-21fac66 kernel: 3e40: c0bba1a4 c025237c 00000000 024000c0 ee8b3e94 c07d9f44 00000000 c5748800
Nov 03 13:17:17 skiff-21fac66 kernel: 3e60: ee8b3e60 ee8b3e60 c028a3b8 ee8b3e94 ee8b3ee0 ee8b3e94 c57700a4 ee8b3eb8
Nov 03 13:17:17 skiff-21fac66 kernel: 3e80: ee8b3ee0 c07da084 c5770070 c07dcc78 ee8b3ee0 c5748844 c5748844 00000000
Nov 03 13:17:17 skiff-21fac66 kernel: 3ea0: ee899f40 c0273f80 00000100 00000200 c1046370 c10451c0 ee8b3eb8 c10451b4
Nov 03 13:17:17 skiff-21fac66 kernel: 3ec0: 00000000 c07d2298 ee89ddc0 ee8b3ed8 00000000 0098d500 c5770020 c5770020
Nov 03 13:17:17 skiff-21fac66 kernel: 3ee0: c5770028 c5770028 ee04b5c4 c10451a4 ee858480 ee81b800 00000000 ee9fe300
Nov 03 13:17:17 skiff-21fac66 kernel: 3f00: 00000000 ee81b800 ee858480 c024b364 c1002d00 ee81b818 ee81b800 ee858498
Nov 03 13:17:17 skiff-21fac66 kernel: 3f20: 00000088 c1002d00 ee81b818 ee8b2000 ee81b800 c024bf4c ee81b960 00000000
Nov 03 13:17:17 skiff-21fac66 kernel: 3f40: ee858480 00000000 ee84fdc0 ee8b2000 ee858480 c024bf14 00000000 00000000
Nov 03 13:17:17 skiff-21fac66 kernel: 3f60: 00000000 c0250ca0 00030000 00000000 20038082 ee858480 00000000 00000000
Nov 03 13:17:17 skiff-21fac66 kernel: 3f80: ee8b3f80 ee8b3f80 00000000 00000000 ee8b3f90 ee8b3f90 ee84fdc0 c0250ba8
Nov 03 13:17:17 skiff-21fac66 kernel: 3fa0: 00000000 00000000 00000000 c021c6e8 00000000 00000000 00000000 00000000
Nov 03 13:17:17 skiff-21fac66 kernel: 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Nov 03 13:17:17 skiff-21fac66 kernel: 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00138003 80d2d4d5
Nov 03 13:17:17 skiff-21fac66 kernel: [<bf0442c0>] (ip6_route_dev_notify [ipv6]) from [<c0252208>] (notifier_call_chain+0x44/0x84)
Nov 03 13:17:17 skiff-21fac66 kernel: [<c0252208>] (notifier_call_chain) from [<c025237c>] (raw_notifier_call_chain+0x18/0x20)
Nov 03 13:17:17 skiff-21fac66 kernel: [<c025237c>] (raw_notifier_call_chain) from [<c07d9f44>] (rollback_registered_many+0x29c/0x3c4)
Nov 03 13:17:17 skiff-21fac66 kernel: [<c07d9f44>] (rollback_registered_many) from [<c07da084>] (unregister_netdevice_many+0x18/0x94)
Nov 03 13:17:17 skiff-21fac66 kernel: [<c07da084>] (unregister_netdevice_many) from [<c07dcc78>] (default_device_exit_batch+0x140/0x150)
Nov 03 13:17:17 skiff-21fac66 kernel: [<c07dcc78>] (default_device_exit_batch) from [<c07d2298>] (cleanup_net+0x19c/0x2a0)
Nov 03 13:17:17 skiff-21fac66 kernel: [<c07d2298>] (cleanup_net) from [<c024b364>] (process_one_work+0x1d8/0x3f0)
Nov 03 13:17:17 skiff-21fac66 kernel: [<c024b364>] (process_one_work) from [<c024bf4c>] (worker_thread+0x38/0x554)
Nov 03 13:17:17 skiff-21fac66 kernel: [<c024bf4c>] (worker_thread) from [<c0250ca0>] (kthread+0xf8/0x110)
Nov 03 13:17:17 skiff-21fac66 kernel: [<c0250ca0>] (kthread) from [<c021c6e8>] (ret_from_fork+0x14/0x2c)
Nov 03 13:17:17 skiff-21fac66 kernel: Code: e59300cc f57ff05b e280307c f593f000 (e1932f9f) 
Nov 03 13:17:17 skiff-21fac66 kernel: ---[ end trace 188886ba7f395e72 ]---

Version information:

sha256 5a8d9fd875c6c2edec3df328923ccc671c7d0851e9b40df6cb998fac4b3b73ed  linux-xu4-4.9.47.tar.gz
Commit hash ec8448dd6e243e2e860dda8d514034b34875db97

[ardje]

As a short fix I would purge network-manager which is the first thing I do because it keeps messing up ipv6. It shouldn't oops though. Anyway: ipv6 works fine without network-manager, so if you can do without purge it now. As far as I know it installs handlers to prevent the system from configuring EUI64 based addresses. You can also exclude eth0 from being configured by network manager.

[ardje]

https://patchwork.ozlabs.org/patch/778449/ seems relevant though... Reading that, if I create and up a link and then remove it again it should trigger this. So networkmanager might not be the thing that triggers it, but an lxc or anything that creates a network device bound to ipv6.

[ardje]

https://github.com/hardkernel/linux/commit/8795ee7bed243c291d30830efe925966253aa2d6 is the culprit introducing the problem. So it is an upstream problem, bound to be patched in the next release I guess?

[paralin]

Any patches we can revert to fix this? A panic is pretty extreme no matter what is going on in userspace.

[mdrjr]

@ardje can you test if reverting it fixes and has no side effects?

[ardje]

I think https://patchwork.ozlabs.org/patch/778449/ just needs to be applied, stopping the recursion. Unfortunately I've looked around it, and it has nothing to do with network manager. So to test the patch I first have to probably create a virtual device that has an ipv6 route, and then remove it. I haven't hit the problem myself, but as I am 100% depending on working IPv6 I was researching this ticket before I hit my head ;-). I will try some link creation and crashing within the next 24 hours.

[paralin]

Thanks. A reproduction would be very useful.

We are operating in an environment where kernel panics is extremely bad, and I think the ipv6 issue is being caused by Docker.

[paralin]

OK guys this is a really really bad bug. We are consistently getting panics, which completely halt the system. On every boot. I can't believe the kernel is allowed to continue in this broken state.

We are rolling back now to an older revision to avoid this bug, but respectfully, I think it should be a bigger priority at hardkernel to fix this.

@mdrjr

[paralin]

(note: we are actually now testing 4.9.50 to see if it fixes the panic. will also test the linked patch. if that fixes it, then I retract my last statement and instead thank Hardkernel for keeping the 4.9 branch up to date with upstream)

[ardje]

I am unable to create the oops, I fear it needs complex network namespace whacking before it decides there is something to oops about. My LXC machines boot without problems. Christian: Is the oops still there in 4.9.51? (I've seen a bunch of ipv6 fixes going in .51) And is the oops always or racy? Some bugs can only be reproduced by a decent pounding (like creating and deleting vlans a few 1000 times revealed some severe memory leaks somewhere in 2.4.9 ;-) ).

[paralin]

This is still broken with just standard usage of Docker. Even with the patch at https://patchwork.ozlabs.org/patch/778449/

Log: http://sprunge.us/JJCB

Snip:

Oct 03 22:44:33 skiff-c4cc3f5 kernel: Unable to handle kernel NULL pointer dereference at virtual address 0000007c
Oct 03 22:44:33 skiff-c4cc3f5 kernel: pgd = c0004000
Oct 03 22:44:33 skiff-c4cc3f5 kernel: [0000007c] *pgd=00000000
Oct 03 22:44:33 skiff-c4cc3f5 kernel: Internal error: Oops: 17 [#1] PREEMPT SMP ARM
Oct 03 22:44:33 skiff-c4cc3f5 kernel: Modules linked in: ccm ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc ipv6 arc4 joydev ads7846 spidev rt2800usb rt2800lib rt2x00usb rt2x00lib mac80211 cfg80211 rfkill spi_s3c64xx exynos_gpiomem w1_gpio wire sch_fq_codel
Oct 03 22:44:33 skiff-c4cc3f5 kernel: CPU: 5 PID: 6 Comm: kworker/u16:0 Not tainted 4.9.52 #1
Oct 03 22:44:33 skiff-c4cc3f5 kernel: Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
Oct 03 22:44:33 skiff-c4cc3f5 kernel: Workqueue: netns cleanup_net
Oct 03 22:44:33 skiff-c4cc3f5 kernel: task: ee891f40 task.stack: ee8b6000
Oct 03 22:44:33 skiff-c4cc3f5 kernel: PC is at ip6_route_dev_notify+0x50/0x1b4 [ipv6]
Oct 03 22:44:33 skiff-c4cc3f5 kernel: LR is at notifier_call_chain+0x44/0x84
Oct 03 22:44:33 skiff-c4cc3f5 kernel: pc : [<bf134358>]    lr : [<c013dfb4>]    psr: 800a0013
                                      sp : ee8b7e18  ip : bf1684e0  fp : ffffffff
Oct 03 22:44:33 skiff-c4cc3f5 kernel: r10: c0c44880  r9 : c09b4e84  r8 : 00000000
Oct 03 22:44:33 skiff-c4cc3f5 kernel: r7 : 00000000  r6 : 00000006  r5 : c63d8000  r4 : c6775800
Oct 03 22:44:33 skiff-c4cc3f5 kernel: r3 : 0000007c  r2 : ee8b7e5c  r1 : 00000006  r0 : 00000000
Oct 03 22:44:33 skiff-c4cc3f5 kernel: Flags: Nzcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
Oct 03 22:44:33 skiff-c4cc3f5 kernel: Control: 10c5387d  Table: 457e006a  DAC: 00000051
Oct 03 22:44:33 skiff-c4cc3f5 kernel: Process kworker/u16:0 (pid: 6, stack limit = 0xee8b6210)
Oct 03 22:44:33 skiff-c4cc3f5 kernel: Stack: (0xee8b7e18 to 0xee8b8000)
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7e00:                                                       ffffffe9 ee8b7e5c
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7e20: 00000006 00000000 00000000 c013dfb4 c6775800 ee8b7e94 024000c0 c09b4e84
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7e40: c0c030ec c013e128 00000000 ee8b7e94 024000c0 c06c7784 00000000 c6775800
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7e60: ee8b7e60 ee8b7e60 c63d8000 ee8b7e94 ee8b7ee0 ee8b7e94 c63d80a4 ee8b7eb8
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7e80: ee8b7ee0 c06c78a4 c63d8070 c06ca498 ee8b7ee0 c6775844 c6775844 00000000
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7ea0: ee891f40 c015fd58 00000100 00000200 c0c45a70 c0c448c0 ee8b7eb8 c0c448b4
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7ec0: 00000000 c06bfa8c ee891f40 ee8b7ed8 00000000 00d548c0 c63d8020 c63d8020
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7ee0: c63d8028 c63d8028 00000000 c0c448a4 ee858480 ee81b800 00000000 eea02300
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7f00: 00000000 ee81b800 ee858480 c0137108 c0c02d00 ee81b818 ee81b800 ee858498
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7f20: 00000088 c0c02d00 ee81b818 ee8b6000 ee81b800 c0137cf0 ee81b960 00000000
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7f40: ee858480 00000000 ee84fc00 ee8b6000 ee858480 c0137cb8 00000000 00000000
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7f60: 00000000 c013ca48 00000000 00000000 00000000 ee858480 00000000 00000000
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7f80: ee8b7f80 ee8b7f80 00000000 00000000 ee8b7f90 ee8b7f90 ee84fc00 c013c950
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7fa0: 00000000 00000000 00000000 c0107838 00000000 00000000 00000000 00000000
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000 ffffffff fff7ffff
Oct 03 22:44:33 skiff-c4cc3f5 kernel: [<bf134358>] (ip6_route_dev_notify [ipv6]) from [<c013dfb4>] (notifier_call_chain+0x44/0x84)
Oct 03 22:44:33 skiff-c4cc3f5 kernel: [<c013dfb4>] (notifier_call_chain) from [<c013e128>] (raw_notifier_call_chain+0x18/0x20)
Oct 03 22:44:33 skiff-c4cc3f5 kernel: [<c013e128>] (raw_notifier_call_chain) from [<c06c7784>] (rollback_registered_many+0x288/0x390)
Oct 03 22:44:33 skiff-c4cc3f5 kernel: [<c06c7784>] (rollback_registered_many) from [<c06c78a4>] (unregister_netdevice_many+0x18/0x94)
Oct 03 22:44:33 skiff-c4cc3f5 kernel: [<c06c78a4>] (unregister_netdevice_many) from [<c06ca498>] (default_device_exit_batch+0x140/0x150)
Oct 03 22:44:33 skiff-c4cc3f5 kernel: [<c06ca498>] (default_device_exit_batch) from [<c06bfa8c>] (cleanup_net+0x19c/0x2a0)
Oct 03 22:44:33 skiff-c4cc3f5 kernel: [<c06bfa8c>] (cleanup_net) from [<c0137108>] (process_one_work+0x1d8/0x3f0)
Oct 03 22:44:33 skiff-c4cc3f5 kernel: [<c0137108>] (process_one_work) from [<c0137cf0>] (worker_thread+0x38/0x554)
Oct 03 22:44:33 skiff-c4cc3f5 kernel: [<c0137cf0>] (worker_thread) from [<c013ca48>] (kthread+0xf8/0x110)
Oct 03 22:44:33 skiff-c4cc3f5 kernel: [<c013ca48>] (kthread) from [<c0107838>] (ret_from_fork+0x14/0x3c)
Oct 03 22:44:33 skiff-c4cc3f5 kernel: Code: e59300cc f57ff05b e280307c f593f000 (e1932f9f) 
Oct 03 22:44:33 skiff-c4cc3f5 kernel: ---[ end trace 6b165a93760d7243 ]---
Oct 03 22:44:33 skiff-c4cc3f5 kernel: Unable to handle kernel paging request at virtual address ffffffec
Oct 03 22:44:33 skiff-c4cc3f5 kernel: pgd = c0004000
Oct 03 22:44:33 skiff-c4cc3f5 kernel: [ffffffec] *pgd=6fffd861, *pte=00000000, *ppte=00000000
Oct 03 22:44:33 skiff-c4cc3f5 kernel: Internal error: Oops: 37 [#2] PREEMPT SMP ARM
Oct 03 22:44:33 skiff-c4cc3f5 kernel: Modules linked in: ccm ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc ipv6 arc4 joydev ads7846 spidev rt2800usb rt2800lib rt2x00usb rt2x00lib mac80211 cfg80211 rfkill spi_s3c64xx exynos_gpiomem w1_gpio wire sch_fq_codel
Oct 03 22:44:33 skiff-c4cc3f5 kernel: CPU: 5 PID: 6 Comm: kworker/u16:0 Tainted: G      D         4.9.52 #1
Oct 03 22:44:33 skiff-c4cc3f5 kernel: Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
Oct 03 22:44:33 skiff-c4cc3f5 kernel: task: ee891f40 task.stack: ee8b6000
Oct 03 22:44:33 skiff-c4cc3f5 kernel: PC is at kthread_data+0x4/0xc
Oct 03 22:44:33 skiff-c4cc3f5 kernel: LR is at wq_worker_sleeping+0x8/0xd0
Oct 03 22:44:33 skiff-c4cc3f5 kernel: pc : [<c013d3d8>]    lr : [<c0138294>]    psr: 200a0193
                                      sp : ee8b7bf0  ip : 00000000  fp : ee8b7c34
Oct 03 22:44:33 skiff-c4cc3f5 kernel: r10: 00000000  r9 : ee8922b0  r8 : c0c0329c
Oct 03 22:44:33 skiff-c4cc3f5 kernel: r7 : c0146b2c  r6 : c0b68cc0  r5 : ee891f40  r4 : eedcacc0
Oct 03 22:44:33 skiff-c4cc3f5 kernel: r3 : 00000000  r2 : 00000000  r1 : fffffffb  r0 : ee891f40
Oct 03 22:44:33 skiff-c4cc3f5 kernel: Flags: nzCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
Oct 03 22:44:33 skiff-c4cc3f5 kernel: Control: 10c5387d  Table: 457e006a  DAC: 00000051
Oct 03 22:44:33 skiff-c4cc3f5 kernel: Process kworker/u16:0 (pid: 6, stack limit = 0xee8b6210)
Oct 03 22:44:33 skiff-c4cc3f5 kernel: Stack: (0xee8b7bf0 to 0xee8b8000)
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7be0:                                     eedcacc0 c07b2c5c ee89ce30 00000011
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7c00: 00000000 c0122e14 c0c02040 ee8a1a40 c0b63240 ee891f40 ee8b793c ee890000
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7c20: 00000000 ee8b7c40 ee892228 00000000 ee8b7c3c c0146b2c ee8b6000 c012498c
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7c40: ee8b7c40 ee8b7c40 00000001 bf13435c ee8b6000 c010b544 ee8b6210 0000000b
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7c60: 00000000 600a0113 bf000000 00000008 65000000 30333935 20636330 66373566
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7c80: 62353066 38326520 37303330 35662063 30663339 28203030 33393165 66396632
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7ca0: 00002029 c01be8c8 0000007c 0000007c ee8b7dc8 00000017 00000000 0000007c
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7cc0: ee8b6000 c0c44880 ffffffff c011a8a4 00000017 c01161cc c5b0bcc0 c0280be0
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7ce0: 00000004 c6ed4840 00000001 c63d2800 00000001 80200017 c5925b8c c63d2800
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7d00: c5b0bcc0 c0c0824c 00000017 c0115e54 0000007c ee8b7dc8 ee8b6000 c0c44880
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7d20: ffffffff c01012d8 c6ed4840 c6ed487c 80200017 c5b0bcc0 bf1685ec bf1685ec
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7d40: c6775800 bf168508 00000040 ee8b7d78 bf168618 c017a53c bf168604 ee8b7d68
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7d60: c6775800 c06d5c44 00000000 c5337e3c c6775800 c06d8050 00000200 ee8b7da8
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7d80: 00000100 c5925c04 c5925c04 c5925b8c c5925c00 bf12f10c 00000040 00000000
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7da0: 00000001 c6775800 ee8b7da8 ee8b7da8 bf134358 800a0013 ffffffff ee8b7dfc
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7dc0: 00000000 c010bd78 00000000 00000006 ee8b7e5c 0000007c c6775800 c63d8000
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7de0: 00000006 00000000 00000000 c09b4e84 c0c44880 ffffffff bf1684e0 ee8b7e18
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7e00: c013dfb4 bf134358 800a0013 ffffffff 00000051 00000000 ffffffe9 ee8b7e5c
Oct 03 22:44:33 skiff-c4cc3f5 kernel: 7e20: 00000006 00000000 00000000 c013dfb4 c6775800 ee8b7e94 024000c0 c09b4e84
Oct 03 22:44:34 skiff-c4cc3f5 kernel: 7e40: c0c030ec c013e128 00000000 ee8b7e94 024000c0 c06c7784 00000000 c6775800
Oct 03 22:44:34 skiff-c4cc3f5 kernel: 7e60: ee8b7e60 ee8b7e60 c63d8000 ee8b7e94 ee8b7ee0 ee8b7e94 c63d80a4 ee8b7eb8
Oct 03 22:44:34 skiff-c4cc3f5 kernel: 7e80: ee8b7ee0 c06c78a4 c63d8070 c06ca498 ee8b7ee0 c6775844 c6775844 00000000
Oct 03 22:44:34 skiff-c4cc3f5 kernel: 7ea0: ee891f40 c015fd58 00000100 00000200 c0c45a70 c0c448c0 ee8b7eb8 c0c448b4
Oct 03 22:44:34 skiff-c4cc3f5 kernel: 7ec0: 00000000 c06bfa8c ee891f40 ee8b7ed8 00000000 00d548c0 c63d8020 c63d8020
Oct 03 22:44:34 skiff-c4cc3f5 kernel: 7ee0: c63d8028 c63d8028 00000000 c0c448a4 ee858480 ee81b800 00000000 eea02300
Oct 03 22:44:34 skiff-c4cc3f5 kernel: 7f00: 00000000 ee81b800 ee858480 c0137108 c0c02d00 ee81b818 ee81b800 ee858498
Oct 03 22:44:34 skiff-c4cc3f5 kernel: 7f20: 00000088 c0c02d00 ee81b818 ee8b6000 ee81b800 c0137cf0 ee81b960 00000000
Oct 03 22:44:34 skiff-c4cc3f5 kernel: 7f40: ee858480 00000000 ee84fc00 ee8b6000 ee858480 c0137cb8 00000000 00000000
Oct 03 22:44:34 skiff-c4cc3f5 kernel: 7f60: 00000000 c013ca48 00000000 00000000 00000000 ee858480 00000000 00000000
Oct 03 22:44:34 skiff-c4cc3f5 kernel: 7f80: ee8b7f80 ee8b7f80 00000001 00010001 ee8b7f90 ee8b7f90 ee84fc00 c013c950
Oct 03 22:44:34 skiff-c4cc3f5 kernel: 7fa0: 00000000 00000000 00000000 c0107838 00000000 00000000 00000000 00000000
Oct 03 22:44:34 skiff-c4cc3f5 kernel: 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Oct 03 22:44:34 skiff-c4cc3f5 kernel: 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000 ffffffff fff7ffff
Oct 03 22:44:34 skiff-c4cc3f5 kernel: [<c013d3d8>] (kthread_data) from [<c0138294>] (wq_worker_sleeping+0x8/0xd0)
Oct 03 22:44:34 skiff-c4cc3f5 kernel: [<c0138294>] (wq_worker_sleeping) from [<c07b2c5c>] (__schedule+0x334/0x580)
Oct 03 22:44:34 skiff-c4cc3f5 kernel: [<c07b2c5c>] (__schedule) from [<c0146b2c>] (do_task_dead+0x8c/0x90)
Oct 03 22:44:34 skiff-c4cc3f5 kernel: [<c0146b2c>] (do_task_dead) from [<c012498c>] (do_exit+0x69c/0xa4c)
Oct 03 22:44:34 skiff-c4cc3f5 kernel: [<c012498c>] (do_exit) from [<c010b544>] (die+0x244/0x438)
Oct 03 22:44:34 skiff-c4cc3f5 kernel: [<c010b544>] (die) from [<c011a8a4>] (__do_kernel_fault.part.0+0x64/0x1e4)
Oct 03 22:44:34 skiff-c4cc3f5 kernel: [<c011a8a4>] (__do_kernel_fault.part.0) from [<c01161cc>] (do_page_fault+0x378/0x38c)
Oct 03 22:44:34 skiff-c4cc3f5 kernel: [<c01161cc>] (do_page_fault) from [<c01012d8>] (do_DataAbort+0x3c/0xbc)
Oct 03 22:44:34 skiff-c4cc3f5 kernel: [<c01012d8>] (do_DataAbort) from [<c010bd78>] (__dabt_svc+0x58/0x80)
Oct 03 22:44:34 skiff-c4cc3f5 kernel: Exception stack(0xee8b7dc8 to 0xee8b7e10)
Oct 03 22:44:34 skiff-c4cc3f5 kernel: 7dc0:                   00000000 00000006 ee8b7e5c 0000007c c6775800 c63d8000
Oct 03 22:44:34 skiff-c4cc3f5 kernel: 7de0: 00000006 00000000 00000000 c09b4e84 c0c44880 ffffffff bf1684e0 ee8b7e18
Oct 03 22:44:34 skiff-c4cc3f5 kernel: 7e00: c013dfb4 bf134358 800a0013 ffffffff
Oct 03 22:44:34 skiff-c4cc3f5 kernel: [<c010bd78>] (__dabt_svc) from [<bf134358>] (ip6_route_dev_notify+0x50/0x1b4 [ipv6])
Oct 03 22:44:34 skiff-c4cc3f5 kernel: [<bf134358>] (ip6_route_dev_notify [ipv6]) from [<c013dfb4>] (notifier_call_chain+0x44/0x84)
Oct 03 22:44:34 skiff-c4cc3f5 kernel: [<c013dfb4>] (notifier_call_chain) from [<c013e128>] (raw_notifier_call_chain+0x18/0x20)
Oct 03 22:44:34 skiff-c4cc3f5 kernel: [<c013e128>] (raw_notifier_call_chain) from [<c06c7784>] (rollback_registered_many+0x288/0x390)
Oct 03 22:44:34 skiff-c4cc3f5 kernel: [<c06c7784>] (rollback_registered_many) from [<c06c78a4>] (unregister_netdevice_many+0x18/0x94)
Oct 03 22:44:34 skiff-c4cc3f5 kernel: [<c06c78a4>] (unregister_netdevice_many) from [<c06ca498>] (default_device_exit_batch+0x140/0x150)
Oct 03 22:44:34 skiff-c4cc3f5 kernel: [<c06ca498>] (default_device_exit_batch) from [<c06bfa8c>] (cleanup_net+0x19c/0x2a0)
Oct 03 22:44:34 skiff-c4cc3f5 kernel: [<c06bfa8c>] (cleanup_net) from [<c0137108>] (process_one_work+0x1d8/0x3f0)
Oct 03 22:44:34 skiff-c4cc3f5 kernel: [<c0137108>] (process_one_work) from [<c0137cf0>] (worker_thread+0x38/0x554)
Oct 03 22:44:34 skiff-c4cc3f5 kernel: [<c0137cf0>] (worker_thread) from [<c013ca48>] (kthread+0xf8/0x110)
Oct 03 22:44:34 skiff-c4cc3f5 kernel: [<c013ca48>] (kthread) from [<c0107838>] (ret_from_fork+0x14/0x3c)
Oct 03 22:44:34 skiff-c4cc3f5 kernel: Code: e34c0092 ebff8d71 eafffff2 e5903340 (e5130014) 
Oct 03 22:44:34 skiff-c4cc3f5 kernel: ---[ end trace 6b165a93760d7244 ]---
Oct 03 22:44:34 skiff-c4cc3f5 kernel: Fixing recursive fault but reboot is needed!

[paralin]

https://github.com/moby/moby/issues/5618

[paralin]

[mdrjr]

I've found two patchs that may address this issue :) They are merged.. can you test it?

[paralin]

Hey, testing now.

[paralin]

Seems to be fixed, thanks