search

dotmesh-io / dotmesh

dotmesh (dm) is like git for your data volumes (databases, files etc) in Docker and Kubernetes
https://dotmesh.com
Apache License 2.0
538 stars 29 forks source link

kernel panics from docker causing runner reboots #540

Open lukemarsden opened 5 years ago

lukemarsden commented 5 years ago 
[12089.355792] swap_info_get: Unused swap offset entry 00000004
[12089.423218] BUG: Bad page map in process docker-runc  pte:00010000 pmd:8fb898067
[12089.511473] addr:00000000db1ad60b vm_flags:08000070 anon_vma:          (null) mapping:0000000033adc394 index:211
[12089.632825] file:libseccomp.so.2.3.1 fault:ext4_filemap_fault mmap:ext4_file_mmap readpage:ext4_readpage
[12089.745914] CPU: 5 PID: 29685 Comm: docker-runc Tainted: P           O     4.15.0-29-generic #31-Ubuntu
[12089.745916] Hardware name: System manufacturer System Product Name/STRIX Z270F GAMING, BIOS 1203 12/26/2017
[12089.745916] Call Trace:
[12089.745922]  dump_stack+0x63/0x8b
[12089.745925]  print_bad_pte+0x222/0x2e0
[12089.745926]  ? _swap_info_get+0x4a/0x50
[12089.745927]  unmap_page_range+0x908/0xcf0
[12089.745929]  unmap_single_vma+0x7d/0xf0
[12089.745930]  unmap_vmas+0x51/0xb0
[12089.745931]  exit_mmap+0x9f/0x190
[12089.745937]  mmput+0x57/0x140
[12089.745940]  do_exit+0x295/0xb40
[12089.745943]  do_group_exit+0x43/0xb0
[12089.745946]  get_signal+0x27b/0x590
[12089.745953]  do_signal+0x37/0x730
[12089.745956]  ? do_futex+0x325/0x500
[12089.745958]  ? handle_mm_fault+0xb1/0x1f0
[12089.745960]  ? SyS_futex+0x13b/0x180
[12089.745962]  exit_to_usermode_loop+0x73/0xd0
[12089.745966]  do_syscall_64+0x115/0x130
[12089.745970]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[12089.745975] RIP: 0033:0x55ea2648b3a3
[12089.745978] RSP: 002b:000000c42003d6c0 EFLAGS: 00000286 ORIG_RAX: 00000000000000ca
[12089.745979] RAX: fffffffffffffe00 RBX: 000000c420040c00 RCX: 000055ea2648b3a3
[12089.745982] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000055ea26c03fe0
[12089.745983] RBP: 000000c42003d708 R08: 0000000000000000 R09: 0000000000000000
[12089.745983] R10: 0000000000000000 R11: 0000000000000286 R12: 0000000000000000
[12089.745987] R13: ffffffffffffffff R14: 000000c4201403e8 R15: 000000000000000c
[12089.746567] BUG: Bad rss-counter state mm:00000000137ad24f idx:2 val:-1
alaric-dotmesh commented 5 years ago

I Am Not A Linux Kernel Expect, but I see this as:

1) A syscall was happening 2) It was probably a futex syscall (the thing underlying mutexes in userland, for non-fast cases) 3) something happened while doing that (I'm not sure if it's the handle_mm_fault meaning that the futex syscall was passed an invalid pointer from userland, or the do_signal meaning that some other signal hit the process - I'd say the do_signal was sending a SIGBUS or SIGSEGV due to the handle_mm_fault, except the stack trace winds back through do_futex, but I'm not sure). 4) As a result of that, the kernel decided to kill the docker-runc process (do_exit) 5) As part of that, it went unmapping the processes's memory 6) But the page map was corrupted

The problems in (6) and (3) are quite likely to have a single common cause.

Most likely problem: Something mangled the docker-runc process' page maps. This is privileged kernel-only data, so it's either a hardware fault, or something bad in the kernel.

lukemarsden commented 5 years ago

Another one:

[18346.187171] ------------[ cut here ]------------
[18346.187175] kernel BUG at /build/linux-60XibS/linux-4.15.0/fs/inode.c:513!
[18346.269926] invalid opcode: 0000 [#1] SMP PTI
[18346.322276] Modules linked in: xt_statistic ipt_REJECT nf_reject_ipv4 ip_set ip_vs xt_comment xt_mark veth xt_nat xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay aufs nls_iso8859_1 zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel snd_hda_intel kvm snd_hda_codec snd_hda_core snd_hwdep snd_pcm irqbypass intel_cstate snd_timer eeepc_wmi mei_me intel_rapl_perf snd asus_wmi soundcore sparse_keymap mei shpchp wmi_bmof mac_hid acpi_pad sch_fq_codel ib_iser
[18347.170303]  rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 multipath linear usbhid hid pata_via netxen_nic 3w_9xxx qlge ixgbe mdio sata_nv forcedeth via686a mptctl mptsas scsi_transport_sas mptspi mptscsih mptbase dm_crypt raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 raid1 dm_mirror dm_region_hash dm_log sata_via sata_sis sym53c8xx megaraid_sas megaraid aic7xxx scsi_transport_spi 3w_xxxx sky2 r8169 skge e1000e e1000 via_rhine sis900 8139too e100 mii i915 crct10dif_pclmul drm_kms_helper crc32_pclmul ghash_clmulni_intel syscopyarea pcbc sysfillrect igb sysimgblt dca aesni_intel fb_sys_fops aes_x86_64 i2c_algo_bit crypto_simd nvme glue_helper ahci ptp mxm_wmi drm
[18348.013525]  cryptd libahci nvme_core pps_core wmi video
[18348.076910] CPU: 4 PID: 2167 Comm: systemd-udevd Tainted: P           O     4.15.0-29-generic #31-Ubuntu
[18348.190818] Hardware name: System manufacturer System Product Name/STRIX Z270F GAMING, BIOS 1203 12/26/2017
[18348.307932] RIP: 0010:clear_inode+0x97/0xa0
[18348.358132] RSP: 0018:ffffa9bb0872fd68 EFLAGS: 00010287
[18348.420871] RAX: ffff9b5fcf3d0178 RBX: ffff9b5fcf3c0048 RCX: 0000000000000020
[18348.506573] RDX: ffff9b5fcf3c0178 RSI: 0000000000000055 RDI: ffff9b5fcf3c01d8
[18348.592304] RBP: ffffa9bb0872fd78 R08: 00000000000281b0 R09: ffffffffb1bc60d3
[18348.678086] R10: fffff93b6730c680 R11: ffff9b5e1d23bd10 R12: ffff9b5fcf3c01d8
[18348.763786] R13: ffffffffb2645e40 R14: ffff9b5fcf3c01a0 R15: ffff9b5df37e70c0
[18348.849488] FS:  00007f6914489680(0000) GS:ffff9b65aed00000(0000) knlGS:0000000000000000
[18348.946673] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[18349.015669] CR2: 000055baeb8b8550 CR3: 00000008b6230006 CR4: 00000000003606e0
[18349.101371] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[18349.186664] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[18349.271787] Call Trace:
[18349.300918]  proc_evict_inode+0x21/0x60
[18349.346639]  evict+0xca/0x1a0
[18349.381997]  iput+0x156/0x220
[18349.417350]  dentry_unlink_inode+0xe5/0x140
[18349.467708]  __dentry_kill+0xd4/0x170
[18349.511849]  dput.part.22+0x12b/0x1e0
[18349.556005]  dput+0x13/0x20
[18349.589896]  __fput+0x18b/0x220
[18349.627847]  ____fput+0xe/0x10
[18349.664807]  task_work_run+0x9d/0xc0
[18349.707879]  exit_to_usermode_loop+0xc0/0xd0
[18349.759323]  do_syscall_64+0x115/0x130
[18349.804551]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[18349.865301] RIP: 0033:0x7f6913f8d947
[18349.908465] RSP: 002b:00007fff2a6c2468 EFLAGS: 00000206 ORIG_RAX: 0000000000000003
[18349.999272] RAX: 0000000000000000 RBX: 000055baeb8c78d0 RCX: 00007f6913f8d947
[18350.084871] RDX: 00007f6914264760 RSI: 0000000000000000 RDI: 0000000000000007
[18350.170499] RBP: 00007f69142652a0 R08: 00007f6914268c40 R09: 0000000000000000
[18350.256694] R10: 000055baeb8a6010 R11: 0000000000000206 R12: 0000000000000000
[18350.343004] R13: 000055bae9bd69ee R14: 000055baeb8a7e10 R15: 000055bae9bcf17c
[18350.429128] Code: 83 30 01 00 00 48 8d 93 30 01 00 00 48 39 c2 75 1a 48 c7 83 a0 00 00 00 60 00 00 00 5b 41 5c 5d c3 0f 0b 0f 0b 0f 0b 0f 0b 0f 0b <0f> 0b 0f 1f 80 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 54 4c
[18350.656712] RIP: clear_inode+0x97/0xa0 RSP: ffffa9bb0872fd68
[18350.725317] ---[ end trace e80b145c1a51ead4 ]---
lukemarsden commented 5 years ago

More, on sprinter, which is now hung (pinging, but not responding on SSH):

[163732.660294] INFO: task systemd:1 blocked for more than 120 seconds.
[163732.736384]       Tainted: P           O     4.15.0-29-generic #31-Ubuntu
[163732.818806] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[163732.913640] systemd         D    0     1      0 0x00000000
[163732.913643] Call Trace:
[163732.913648]  __schedule+0x291/0x8a0
[163732.913650]  schedule+0x2c/0x80
[163732.913651]  schedule_preempt_disabled+0xe/0x10
[163732.913652]  __mutex_lock.isra.2+0x18c/0x4d0
[163732.913654]  __mutex_lock_slowpath+0x13/0x20
[163732.913654]  ? __mutex_lock_slowpath+0x13/0x20
[163732.913655]  mutex_lock+0x2f/0x40
[163732.913658]  proc_cgroup_show+0x4c/0x2a0
[163732.913660]  proc_single_show+0x56/0x80
[163732.913661]  seq_read+0xe5/0x430
[163732.913664]  __vfs_read+0x1b/0x40
[163732.913666]  vfs_read+0x8e/0x130
[163732.913667]  SyS_read+0x55/0xc0
[163732.913668]  do_syscall_64+0x73/0x130
[163732.913670]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[163732.913672] RIP: 0033:0x7ff4675c0081
[163732.913673] RSP: 002b:00007ffe29fa25c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[163732.913674] RAX: ffffffffffffffda RBX: 000055e279252670 RCX: 00007ff4675c0081
[163732.913675] RDX: 0000000000000400 RSI: 000055e27939fbd0 RDI: 0000000000000012
[163732.913676] RBP: 0000000000000d68 R08: 0000000000000001 R09: 0000000000000000
[163732.913676] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ff467897760
[163732.913677] R13: 00007ff4678982a0 R14: 000055e279252670 R15: 00000000000007ff
[163732.913690] INFO: task systemd-journal:649 blocked for more than 120 seconds.
[163733.000170]       Tainted: P           O     4.15.0-29-generic #31-Ubuntu
[163733.082528] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[163733.177374] systemd-journal D    0   649      1 0x00000100
[163733.177376] Call Trace:
[163733.177382]  __schedule+0x291/0x8a0
[163733.177384]  schedule+0x2c/0x80
[163733.177386]  schedule_preempt_disabled+0xe/0x10
[163733.177387]  __mutex_lock.isra.2+0x18c/0x4d0
[163733.177388]  __mutex_lock_slowpath+0x13/0x20
[163733.177389]  ? __mutex_lock_slowpath+0x13/0x20
[163733.177390]  mutex_lock+0x2f/0x40
[163733.177393]  proc_cgroup_show+0x4c/0x2a0
[163733.177396]  proc_single_show+0x56/0x80
[163733.177398]  seq_read+0xe5/0x430
[163733.177400]  __vfs_read+0x1b/0x40
[163733.177405]  vfs_read+0x8e/0x130
[163733.177409]  SyS_read+0x55/0xc0
[163733.177414]  do_syscall_64+0x73/0x130
[163733.177418]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[163733.177422] RIP: 0033:0x7fcd423460b4
[163733.177425] RSP: 002b:00007ffc2804c1c0 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[163733.177428] RAX: ffffffffffffffda RBX: 0000000000000014 RCX: 00007fcd423460b4
[163733.177429] RDX: 0000000000000400 RSI: 000055d40fe0ed50 RDI: 0000000000000014
[163733.177430] RBP: 000055d40fe0ed50 R08: 0000000000000000 R09: 0000000000000000
[163733.177431] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000400
[163733.177431] R13: 00007fcd4261e2a0 R14: 000055d40fe200f0 R15: 00000000000007ff
[163733.187028] INFO: task systemd-journal:27895 blocked for more than 120 seconds.
[163733.275710]       Tainted: P           O     4.15.0-29-generic #31-Ubuntu
[163733.358037] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[163733.452921] systemd-journal D    0 27895  27317 0x00000104
[163733.452923] Call Trace:
[163733.452928]  __schedule+0x291/0x8a0
[163733.452930]  schedule+0x2c/0x80
[163733.452930]  schedule_preempt_disabled+0xe/0x10
[163733.452931]  __mutex_lock.isra.2+0x18c/0x4d0
[163733.452932]  __mutex_lock_slowpath+0x13/0x20
[163733.452933]  ? __mutex_lock_slowpath+0x13/0x20
[163733.452934]  mutex_lock+0x2f/0x40
[163733.452936]  proc_cgroup_show+0x4c/0x2a0
[163733.452937]  proc_single_show+0x56/0x80
[163733.452939]  seq_read+0xe5/0x430
[163733.452940]  __vfs_read+0x1b/0x40
[163733.452941]  vfs_read+0x8e/0x130
[163733.452942]  SyS_read+0x55/0xc0
[163733.452944]  do_syscall_64+0x73/0x130
[163733.452945]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[163733.452946] RIP: 0033:0x7ff1c5dfd6ed
[163733.452947] RSP: 002b:00007fffdabf7d20 EFLAGS: 00000293 ORIG_RAX: 0000000000000000
[163733.452948] RAX: ffffffffffffffda RBX: 000055a482f358f0 RCX: 00007ff1c5dfd6ed
[163733.452948] RDX: 0000000000000400 RSI: 000055a482f33560 RDI: 0000000000000014
[163733.452949] RBP: 0000000000000d68 R08: 00007ff1c60bc088 R09: 0000000000000410
[163733.452949] R10: 0000000000000060 R11: 0000000000000293 R12: 00007ff1c60b8440
[163733.452949] R13: 00007ff1c60b7900 R14: 00000000000007ff R15: 000055a482f358f0
[163733.452960] INFO: task systemd-journal:28966 blocked for more than 120 seconds.
[163733.541596]       Tainted: P           O     4.15.0-29-generic #31-Ubuntu
[163733.623979] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[163733.718817] systemd-journal D    0 28966  28332 0x00000100
[163733.718819] Call Trace:
[163733.718825]  __schedule+0x291/0x8a0
[163733.718828]  schedule+0x2c/0x80
[163733.718829]  schedule_preempt_disabled+0xe/0x10
[163733.718832]  __mutex_lock.isra.2+0x18c/0x4d0
[163733.718837]  __mutex_lock_slowpath+0x13/0x20
[163733.718841]  ? __mutex_lock_slowpath+0x13/0x20
[163733.718842]  mutex_lock+0x2f/0x40
[163733.718845]  proc_cgroup_show+0x4c/0x2a0
[163733.718850]  proc_single_show+0x56/0x80
[163733.718855]  seq_read+0xe5/0x430
[163733.718858]  __vfs_read+0x1b/0x40
[163733.718862]  vfs_read+0x8e/0x130
[163733.718865]  SyS_read+0x55/0xc0
[163733.718869]  do_syscall_64+0x73/0x130
[163733.718871]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[163733.718873] RIP: 0033:0x7f2dd50236ed
[163733.718875] RSP: 002b:00007fff90281900 EFLAGS: 00000293 ORIG_RAX: 0000000000000000
[163733.718877] RAX: ffffffffffffffda RBX: 000055bd426a9120 RCX: 00007f2dd50236ed
[163733.718879] RDX: 0000000000000400 RSI: 000055bd426a9350 RDI: 0000000000000014
[163733.718880] RBP: 0000000000000d68 R08: 00007f2dd52e2158 R09: 0000000000000410
[163733.718881] R10: 0000000000000060 R11: 0000000000000293 R12: 00007f2dd52de440
[163733.718883] R13: 00007f2dd52dd900 R14: 00000000000007ff R15: 000055bd426a9120
[163733.718893] INFO: task systemd-journal:32726 blocked for more than 120 seconds.
[163733.807561]       Tainted: P           O     4.15.0-29-generic #31-Ubuntu
[163733.889901] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[163733.984740] systemd-journal D    0 32726  31935 0x00000104
[163733.984742] Call Trace:
[163733.984747]  __schedule+0x291/0x8a0
[163733.984750]  schedule+0x2c/0x80
[163733.984751]  schedule_preempt_disabled+0xe/0x10
[163733.984752]  __mutex_lock.isra.2+0x18c/0x4d0
[163733.984753]  __mutex_lock_slowpath+0x13/0x20
[163733.984757]  ? __mutex_lock_slowpath+0x13/0x20
[163733.984761]  mutex_lock+0x2f/0x40
[163733.984765]  proc_cgroup_show+0x4c/0x2a0
[163733.984769]  proc_single_show+0x56/0x80
[163733.984771]  seq_read+0xe5/0x430
[163733.984774]  __vfs_read+0x1b/0x40
[163733.984776]  vfs_read+0x8e/0x130
[163733.984779]  SyS_read+0x55/0xc0
[163733.984783]  do_syscall_64+0x73/0x130
[163733.984785]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[163733.984788] RIP: 0033:0x7f304db2b6ed
[163733.984789] RSP: 002b:00007ffc997d2700 EFLAGS: 00000293 ORIG_RAX: 0000000000000000
[163733.984791] RAX: ffffffffffffffda RBX: 00005556b6a055e0 RCX: 00007f304db2b6ed
[163733.984792] RDX: 0000000000000400 RSI: 00005556b6a04970 RDI: 0000000000000010
[163733.984793] RBP: 0000000000000d68 R08: 00007f304dde9f78 R09: 0000000000000410
[163733.984794] R10: 00005556b69fc7f0 R11: 0000000000000293 R12: 00007f304dde6440
[163733.984795] R13: 00007f304dde5900 R14: 00000000000007ff R15: 00005556b6a055e0
[163733.984797] INFO: task systemd-journal:607 blocked for more than 120 seconds.
[163734.071389]       Tainted: P           O     4.15.0-29-generic #31-Ubuntu
[163734.153733] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[163734.248589] systemd-journal D    0   607  32544 0x00000104
[163734.248593] Call Trace:
[163734.248599]  __schedule+0x291/0x8a0
[163734.248603]  schedule+0x2c/0x80
[163734.248604]  schedule_preempt_disabled+0xe/0x10
[163734.248605]  __mutex_lock.isra.2+0x18c/0x4d0
[163734.248606]  __mutex_lock_slowpath+0x13/0x20
[163734.248607]  ? __mutex_lock_slowpath+0x13/0x20
[163734.248608]  mutex_lock+0x2f/0x40
[163734.248610]  proc_cgroup_show+0x4c/0x2a0
[163734.248613]  proc_single_show+0x56/0x80
[163734.248615]  seq_read+0xe5/0x430
[163734.248617]  __vfs_read+0x1b/0x40
[163734.248620]  vfs_read+0x8e/0x130
[163734.248621]  SyS_read+0x55/0xc0
[163734.248624]  do_syscall_64+0x73/0x130
[163734.248626]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[163734.248628] RIP: 0033:0x7f3b1e8636ed
[163734.248630] RSP: 002b:00007ffc363e68a0 EFLAGS: 00000293 ORIG_RAX: 0000000000000000
[163734.248631] RAX: ffffffffffffffda RBX: 000055632ab9bce0 RCX: 00007f3b1e8636ed
[163734.248632] RDX: 0000000000000400 RSI: 000055632ab9bf10 RDI: 0000000000000014
[163734.248633] RBP: 0000000000000d68 R08: 00007f3b1eb22108 R09: 0000000000000410
[163734.248634] R10: 000055632ab9d490 R11: 0000000000000293 R12: 00007f3b1eb1e440
[163734.248635] R13: 00007f3b1eb1d900 R14: 00000000000007ff R15: 000055632ab9bce0
[163734.248642] INFO: task systemd-journal:3694 blocked for more than 120 seconds.
[163734.336258]       Tainted: P           O     4.15.0-29-generic #31-Ubuntu
[163734.418619] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[163734.513474] systemd-journal D    0  3694   2905 0x00000104
[163734.513476] Call Trace:
[163734.513481]  __schedule+0x291/0x8a0
[163734.513483]  schedule+0x2c/0x80
[163734.513484]  schedule_preempt_disabled+0xe/0x10
[163734.513484]  __mutex_lock.isra.2+0x18c/0x4d0
[163734.513485]  __mutex_lock_slowpath+0x13/0x20
[163734.513487]  ? __mutex_lock_slowpath+0x13/0x20
[163734.513489]  mutex_lock+0x2f/0x40
[163734.513493]  proc_cgroup_show+0x4c/0x2a0
[163734.513497]  proc_single_show+0x56/0x80
[163734.513500]  seq_read+0xe5/0x430
[163734.513503]  __vfs_read+0x1b/0x40
[163734.513506]  vfs_read+0x8e/0x130
[163734.513508]  SyS_read+0x55/0xc0
[163734.513509]  do_syscall_64+0x73/0x130
[163734.513511]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[163734.513512] RIP: 0033:0x7f65d1d656ed
[163734.513513] RSP: 002b:00007ffdba551850 EFLAGS: 00000293 ORIG_RAX: 0000000000000000
[163734.513514] RAX: ffffffffffffffda RBX: 0000555be40077b0 RCX: 00007f65d1d656ed
[163734.513515] RDX: 0000000000000400 RSI: 0000555be40079e0 RDI: 0000000000000014
[163734.513515] RBP: 0000000000000d68 R08: 00007f65d2024158 R09: 0000000000000410
[163734.513516] R10: 00000000000000d0 R11: 0000000000000293 R12: 00007f65d2020440
[163734.513517] R13: 00007f65d201f900 R14: 00000000000007ff R15: 0000555be40077b0
[163734.513519] INFO: task systemd-journal:3750 blocked for more than 120 seconds.
[163734.601029]       Tainted: P           O     4.15.0-29-generic #31-Ubuntu
[163734.683406] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[163734.683406] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[163734.778248] systemd-journal D    0  3750   3239 0x00000104
[163734.778250] Call Trace:
[163734.778256]  __schedule+0x291/0x8a0
[163734.778258]  schedule+0x2c/0x80
[163734.778260]  schedule_preempt_disabled+0xe/0x10
[163734.778261]  __mutex_lock.isra.2+0x18c/0x4d0
[163734.778262]  __mutex_lock_slowpath+0x13/0x20
[163734.778265]  ? __mutex_lock_slowpath+0x13/0x20
[163734.778266]  mutex_lock+0x2f/0x40
[163734.778268]  proc_cgroup_show+0x4c/0x2a0
[163734.778272]  proc_single_show+0x56/0x80
[163734.778274]  seq_read+0xe5/0x430
[163734.778276]  __vfs_read+0x1b/0x40
[163734.778279]  vfs_read+0x8e/0x130
[163734.778280]  SyS_read+0x55/0xc0
[163734.778283]  do_syscall_64+0x73/0x130
[163734.778286]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[163734.778287] RIP: 0033:0x7f12cc0656ed
[163734.778289] RSP: 002b:00007ffce2f14d70 EFLAGS: 00000293 ORIG_RAX: 0000000000000000
[163734.778291] RAX: ffffffffffffffda RBX: 0000562362807d40 RCX: 00007f12cc0656ed
[163734.778291] RDX: 0000000000000400 RSI: 0000562362807f70 RDI: 0000000000000014
[163734.778293] RBP: 0000000000000d68 R08: 00007f12cc323fc8 R09: 0000000000000410
[163734.778294] R10: 0000000000000060 R11: 0000000000000293 R12: 00007f12cc320440
[163734.778295] R13: 00007f12cc31f900 R14: 00000000000007ff R15: 0000562362807d40
[163734.778776] INFO: task kworker/6:4:23468 blocked for more than 120 seconds.
[163734.863254]       Tainted: P           O     4.15.0-29-generic #31-Ubuntu
[163734.945627] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[163735.040469] kworker/6:4     D    0 23468      2 0x80000000
[163735.040474] Workqueue: cgroup_destroy css_release_work_fn
[163735.040476] Call Trace:
[163735.040481]  __schedule+0x291/0x8a0
[163735.040483]  ? __switch_to_asm+0x40/0x70
[163735.040484]  schedule+0x2c/0x80
[163735.040485]  schedule_preempt_disabled+0xe/0x10
[163735.040486]  __mutex_lock.isra.2+0x18c/0x4d0
[163735.040487]  ? __switch_to_asm+0x34/0x70
[163735.040489]  ? __switch_to_asm+0x34/0x70
[163735.040490]  __mutex_lock_slowpath+0x13/0x20
[163735.040492]  ? __mutex_lock_slowpath+0x13/0x20
[163735.040493]  mutex_lock+0x2f/0x40
[163735.040494]  css_release_work_fn+0x2b/0x180
[163735.040496]  process_one_work+0x1de/0x410
[163735.040498]  worker_thread+0x32/0x410
[163735.040500]  kthread+0x121/0x140
[163735.040501]  ? process_one_work+0x410/0x410
[163735.040502]  ? kthread_create_worker_on_cpu+0x70/0x70
[163735.040504]  ? do_syscall_64+0x73/0x130
[163735.040506]  ? SyS_exit_group+0x14/0x20
[163735.040508]  ret_from_fork+0x35/0x40
[163735.040511] INFO: task kworker/5:3:10475 blocked for more than 120 seconds.
[163735.124984]       Tainted: P           O     4.15.0-29-generic #31-Ubuntu
[163735.207357] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[163735.302217] kworker/5:3     D    0 10475      2 0x80000000
[163735.302224] Workqueue: cgroup_destroy css_release_work_fn
[163735.302226] Call Trace:
[163735.302232]  __schedule+0x291/0x8a0
[163735.302234]  ? __switch_to_asm+0x40/0x70
[163735.302236]  schedule+0x2c/0x80
[163735.302237]  schedule_preempt_disabled+0xe/0x10
[163735.302239]  __mutex_lock.isra.2+0x18c/0x4d0
[163735.302240]  ? __switch_to_asm+0x34/0x70
[163735.302243]  ? __switch_to_asm+0x34/0x70
[163735.302246]  __mutex_lock_slowpath+0x13/0x20
[163735.302250]  ? __mutex_lock_slowpath+0x13/0x20
[163735.302254]  mutex_lock+0x2f/0x40
[163735.302257]  css_release_work_fn+0x2b/0x180
[163735.302260]  process_one_work+0x1de/0x410
[163735.302261]  worker_thread+0x32/0x410
[163735.302264]  kthread+0x121/0x140
[163735.302265]  ? process_one_work+0x410/0x410
[163735.302268]  ? kthread_create_worker_on_cpu+0x70/0x70
[163735.302269]  ? do_syscall_64+0x73/0x130
[163735.302272]  ? SyS_exit_group+0x14/0x20
[163735.302276]  ret_from_fork+0x35/0x40
lukemarsden commented 5 years ago 
[92942.863453] general protection fault: 0000 [#1] SMP PTI
[92942.926056] Modules linked in: xt_statistic ipt_REJECT nf_reject_ipv4 ip_set ip_vs xt_comment xt_mark ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs cpuid veth xt_nat xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay aufs nls_iso8859_1 zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp snd_hda_intel kvm_intel snd_hda_codec kvm snd_hda_core snd_hwdep irqbypass snd_pcm intel_cstate intel_rapl_perf snd_timer mei_me eeepc_wmi snd asus_wmi mei soundcore shpchp wmi_bmof
[92943.772426]  sparse_keymap mac_hid acpi_pad sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 multipath linear usbhid hid pata_via netxen_nic 3w_9xxx qlge ixgbe mdio sata_nv forcedeth via686a mptctl mptsas scsi_transport_sas mptspi mptscsih mptbase dm_crypt raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 dm_mirror dm_region_hash dm_log sata_via sata_sis sym53c8xx megaraid_sas megaraid aic7xxx scsi_transport_spi 3w_xxxx sky2 r8169 skge e1000e e1000 via_rhine sis900 8139too e100 mii raid1 crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i915 pcbc drm_kms_helper igb aesni_intel syscopyarea aes_x86_64 sysfillrect dca crypto_simd sysimgblt fb_sys_fops
[92944.619832]  glue_helper i2c_algo_bit ahci nvme ptp mxm_wmi libahci drm cryptd nvme_core pps_core video wmi
[92944.736511] CPU: 0 PID: 22942 Comm: hyperkube Tainted: P           O     4.15.0-29-generic #31-Ubuntu
[92944.846922] Hardware name: System manufacturer System Product Name/STRIX Z270F GAMING, BIOS 1203 12/26/2017
[92944.963586] RIP: 0010:prefetch_freepointer+0x15/0x30
[92945.022999] RSP: 0018:ffffa2756f19bc40 EFLAGS: 00010282
[92945.085545] RAX: 0000000000000000 RBX: b86768a40d000b98 RCX: 00000000001d059d
[92945.170989] RDX: 00000000001d059c RSI: b86768a40d000b98 RDI: ffff895f7676cf00
[92945.256431] RBP: ffffa2756f19bc40 R08: ffffc2753f02db20 R09: ffffffffd0000000
[92945.341888] R10: ffffa2756f19beb8 R11: 0000000000000000 R12: 00000000014080c0
[92945.427321] R13: ffff8962ae245380 R14: ffff895d1720fb00 R15: ffff895f7676cf00
[92945.512769] FS:  00007f57e27fc700(0000) GS:ffff8962eec00000(0000) knlGS:0000000000000000
[92945.609678] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[92945.678545] CR2: 00007fc349ef9dd0 CR3: 0000000bc673a001 CR4: 00000000003606f0
[92945.763985] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[92945.849425] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[92945.934859] Call Trace:
[92945.964100]  kmem_cache_alloc+0xa2/0x1b0
[92946.011034]  ? get_empty_filp+0x5c/0x1b0
[92946.057965]  get_empty_filp+0x5c/0x1b0
[92946.102845]  path_openat+0x3d/0x1770
[92946.145697]  ? vsnprintf+0xf0/0x4e0
[92946.187950]  ? seq_vprintf+0x35/0x50
[92946.231161]  do_filp_open+0x9b/0x110
[92946.274508]  ? _cond_resched+0x19/0x40
[92946.319828]  ? _cond_resched+0x19/0x40
[92946.365192]  ? dput.part.22+0x2d/0x1e0
[92946.410594]  ? __check_object_size+0xaf/0x1b0
[92946.463221]  ? __alloc_fd+0x46/0x170
[92946.506534]  do_sys_open+0x1bb/0x2c0
[92946.549926]  ? do_sys_open+0x1bb/0x2c0
[92946.595330]  ? _cond_resched+0x19/0x40
[92946.640646]  SyS_openat+0x14/0x20
[92946.680732]  do_syscall_64+0x73/0x130
[92946.725070]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[92946.786042] RIP: 0033:0x48719a
[92946.823055] RSP: 002b:000000c4240c9e40 EFLAGS: 00000202 ORIG_RAX: 0000000000000101
[92946.914114] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 000000000048719a
[92946.999955] RDX: 0000000000080000 RSI: 000000c4246d0f20 RDI: ffffffffffffff9c
[92947.085801] RBP: 000000c4240c9ec0 R08: 0000000000000000 R09: 0000000000000000
[92947.171643] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000000
[92947.257510] R13: 00000000000000f4 R14: 0000000000000074 R15: 0000000000000004
[92947.343522] Code: eb bb 49 8b 74 24 60 48 c7 c7 90 b4 ce ac e8 a3 ca ea ff eb 90 90 0f 1f 44 00 00 55 48 85 f6 48 89 e5 74 14 48 63 47 20 48 01 c6 <48> 33 36 48 33 b7 40 01 00 00 0f 18 0e 5d c3 66 90 66 2e 0f 1f
[92947.570447] RIP: prefetch_freepointer+0x15/0x30 RSP: ffffa2756f19bc40
[92947.648115] ---[ end trace 1e4948c7c737d5ea ]---
lukemarsden commented 5 years ago 
[235387.507463] INFO: task systemd:1 blocked for more than 120 seconds.                                                                                                                                                             [215/9091]
[235387.583094]       Tainted: P           O     4.15.0-29-generic #31-Ubuntu
[235387.664972] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[235387.759212] systemd         D    0     1      0 0x00000000
[235387.759215] Call Trace:
[235387.759221]  __schedule+0x291/0x8a0
[235387.759223]  schedule+0x2c/0x80
[235387.759224]  schedule_preempt_disabled+0xe/0x10
[235387.759225]  __mutex_lock.isra.2+0x18c/0x4d0
[235387.759226]  __mutex_lock_slowpath+0x13/0x20
[235387.759227]  ? __mutex_lock_slowpath+0x13/0x20
[235387.759228]  mutex_lock+0x2f/0x40
[235387.759230]  proc_cgroup_show+0x4c/0x2a0
[235387.759233]  proc_single_show+0x56/0x80
[235387.759234]  seq_read+0xe5/0x430
[235387.759236]  __vfs_read+0x1b/0x40
[235387.759238]  vfs_read+0x8e/0x130
[235387.759239]  SyS_read+0x55/0xc0
[235387.759242]  do_syscall_64+0x73/0x130
[235387.759243]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[235387.759245] RIP: 0033:0x7f6de6c6c081
[235387.759245] RSP: 002b:00007ffc0695b208 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[235387.759246] RAX: ffffffffffffffda RBX: 000055c1f0d34070 RCX: 00007f6de6c6c081
[235387.759247] RDX: 0000000000000400 RSI: 000055c1f0eb9e50 RDI: 0000000000000012
[235387.759248] RBP: 0000000000000d68 R08: 0000000000000001 R09: 0000000000000000
[235387.759248] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f6de6f43760
[235387.759249] R13: 00007f6de6f442a0 R14: 000055c1f0d34070 R15: 00000000000007ff
[235387.759263] INFO: task systemd-journal:662 blocked for more than 120 seconds.
[235387.845266]       Tainted: P           O     4.15.0-29-generic #31-Ubuntu
[235387.927094] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[235388.021319] systemd-journal D    0   662      1 0x00000100
[235388.021321] Call Trace:
[235388.021326]  __schedule+0x291/0x8a0
[235388.021327]  schedule+0x2c/0x80
[235388.021329]  schedule_preempt_disabled+0xe/0x10
[235388.021329]  __mutex_lock.isra.2+0x18c/0x4d0
[235388.021331]  __mutex_lock_slowpath+0x13/0x20
[235388.021331]  ? __mutex_lock_slowpath+0x13/0x20
[235388.021332]  mutex_lock+0x2f/0x40
[235388.021334]  proc_cgroup_show+0x4c/0x2a0
[235388.021336]  proc_single_show+0x56/0x80
[235388.021338]  seq_read+0xe5/0x430
[235388.021341]  __vfs_read+0x1b/0x40
[235388.021345]  vfs_read+0x8e/0x130
[235388.021347]  SyS_read+0x55/0xc0
[235388.021351]  do_syscall_64+0x73/0x130
[235388.021355]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[235388.021358] RIP: 0033:0x7f605318f0b4
[235388.021360] RSP: 002b:00007ffe923f2fc0 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[235388.021362] RAX: ffffffffffffffda RBX: 0000000000000012 RCX: 00007f605318f0b4
[235388.021363] RDX: 0000000000000400 RSI: 0000564744994f90 RDI: 0000000000000012
[235388.021363] RBP: 0000564744994f90 R08: 0000000000000000 R09: 0000000000000000
[235388.021364] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000400
[235388.021365] R13: 00007f60534672a0 R14: 00005647446a3320 R15: 00000000000007ff