Closed mvernimmen closed 8 years ago
mvernimmen
commented
9 years ago Since the machine locks up when trying to import the pool, there is no way to get stacktraces. None appear on the screen either, other than the mentioned rcu_sched message.
behlendorf
commented
9 years ago @mvernimmen I doubt this is related to your system update but you could always trying try importing the pool from a live CD running your distribution of choice. Also when you say you're pool is full, how full is it? 90%, 95%, 99%? If it's close to completely full you may start seeing significant performance problems.
Unfortunately, the only way to do debug this will be to get a backtrace from the system. If you can get anything from the console or dmesg prior to a complete lockup the would be very helpful.
mvernimmen
commented
9 years ago Only a small part of the backtrace is logged to disk when I redirect all rsyslog messages to a file. Much more is printed on screen. Here is the output that was logged to file:
Nov 5 21:13:52 server kernel: [ 395.139700] SELinux: initialized (dev zfs, type zfs), uses xattr
Nov 5 21:13:52 server kernel: [ 395.282857] SELinux: initialized (dev zfs, type zfs), uses xattr
Nov 5 21:13:52 server kernel: [ 395.386097] SELinux: initialized (dev zfs, type zfs), uses xattr
Nov 5 21:14:55 server kernel: [ 458.499841] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=60003 jiffies, g=2346, c=2345, q=0)
Nov 5 21:14:55 server kernel: [ 458.535923] sending NMI to all CPUs:
Nov 5 21:14:55 server kernel: [ 458.535935] NMI backtrace for cpu 0
Nov 5 21:14:55 server kernel: [ 458.535950] CPU: 0 PID: 1575 Comm: mount.zfs Tainted: P OE 3.16.6-203.fc20.x86_64 #1
Nov 5 21:14:55 server kernel: [ 458.535956] Hardware name: System manufacturer System Product Name/M4A88TD-M EVO, BIOS 1801 08/09/2012
Nov 5 21:14:55 server kernel: [ 458.535962] task: ffff8800ca562780 ti: ffff8800b9e54000 task.ti: ffff8800b9e54000
Nov 5 21:14:55 server kernel: [ 458.535968] RIP: 0010:[
The pool is 2TB in size, I think there is about 600GB free.
I tried to mount the pool with a livecd (gentoo based sysresccd) and had the same problem, but this time i was able to catch the full stacktraces:
[ 47.537271] SPL: Loaded module v0.6.3-r0-gentoo
[ 47.544222] device fsid fc266705-091a-4bca-9372-d635b19c5690 devid 1 transid 1434 /dev/sdm
[ 47.627230] znvpair: module license 'CDDL' taints kernel.
[ 47.627234] Disabling lock debugging due to kernel taint
[ 47.729276] ZFS: Loaded module v0.6.3-r0-gentoo, ZFS pool version 5000, ZFS filesystem version 5
[ 50.570080] [drm] VGACON disable radeon kernel modesetting.
[ 50.570084] [drm:radeon_init] ERROR No UMS support in radeon module!
[ 50.672108] device fsid fc266705-091a-4bca-9372-d635b19c5690 devid 1 transid 1434 /dev/sdm
[ 50.722167] warning: process `hwsetup' used the deprecated sysctl system call with 1.23.
[ 50.819096] device fsid fc266705-091a-4bca-9372-d635b19c5690 devid 1 transid 1434 /dev/sdm
[ 57.081834] r8169 0000:04:00.0 enp4s0: link down
[ 57.081856] r8169 0000:04:00.0 enp4s0: link down
[ 57.081916] IPv6: ADDRCONF(NETDEV_UP): enp4s0: link is not ready
[ 59.870683] r8169 0000:04:00.0 enp4s0: link up
[ 59.870696] IPv6: ADDRCONF(NETDEV_CHANGE): enp4s0: link becomes ready
[ 64.115285] md: Autodetecting RAID arrays.
[ 64.115295] md: Scanned 0 and added 0 devices.
[ 64.115299] md: autorun ...
[ 64.115304] md: ... autorun DONE.
[ 122.949842] SPL: using hostid 0x00000000
[ 209.600762] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=60002 jiffies, g=6360, c=6359, q=10)
[ 209.600780] sending NMI to all CPUs:
[ 209.600793] NMI backtrace for cpu 0
[ 209.600809] CPU: 0 PID: 4054 Comm: mount.zfs Tainted: P O 3.10.35-std420-amd64 #1
[ 209.600815] Hardware name: System manufacturer System Product Name/M4A88TD-M EVO, BIOS 1801 08/09/2012
[ 209.600823] task: ffff880115502530 ti: ffff880116a54000 task.ti: ffff880116a54000
[ 209.600829] RIP: 0010:[
[ 209.602005] [
[ 209.602173] [
[ 389.703327] [
[ 389.703494] [
mvernimmen
commented
9 years ago @behlendorf Is there anything else I can do or provide to help debug this?
behlendorf
commented
9 years ago @mvernimmen Can you check what CONFIG_PREEMPT is set to for your new kernel. We're had reports of similar rcu issues depended on preempt.
$ grep CONFIG_PREEMPT /boot/config-3.10.0-123.el7.x86_64
# CONFIG_PREEMPT_RCU is not set
CONFIG_PREEMPT_NOTIFIERS=y
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set@behlendorf sure, here's the output:
% grep CONFIG_PREEMPT /boot/config-3.16.6-203.fc20.x86_64 # CONFIG_PREEMPT_RCU is not set CONFIG_PREEMPT_NOTIFIERS=y # CONFIG_PREEMPT_NONE is not set CONFIG_PREEMPT_VOLUNTARY=y # CONFIG_PREEMPT is not set
behlendorf
commented
9 years ago OK, well that's not it. Thanks for the quick turn around.
mvernimmen
commented
9 years ago @behlendorf Anything else I can do or provide to help this along?
behlendorf
commented
9 years ago @mvernimmen could you try the latest testing version. We've fixed several issues in the area of the code where you're having problems. Since you're running fedora all you need to do is enable the zfs-testing repository, update the packages, and reboot.
mvernimmen
commented
9 years ago @behlendorf indeed it works fine with the testing release of the following composition: libzfs2-0.6.3-133_g9635861 spl-dkms-0.6.3-44_g46c9367 zfs-0.6.3-133_g9635861 spl-0.6.3-44_g46c9367 zfs-dkms-0.6.3-133_g9635861
Why didn't I think of trying that? Thanks for the help. If there is anything you need or anything I can do to make sure the fix gets improved or included in the 0.6.4 release, let me know. Otherwise I'm very happy with this solution and the issue can be closed.
behlendorf
commented
9 years ago Sounds good. Thanks for following up.
mvernimmen
commented
9 years ago @behlendorf Sorry to continue on this but it turns out one of the subvolumes remains unmountable and continues to display the symptoms of this issue. All volumes and subvolumes except one mount and work fine. I get these stacktraces when I try to mount the non-working subvolume:
Nov 24 20:05:16 server kernel: [ 93.094477] SPL: using hostid 0x00000000
Nov 24 20:06:19 server kernel: [ 135.200817] ------------[ cut here ]------------
Nov 24 20:06:19 server kernel: [ 135.200830] WARNING: CPU: 1 PID: 1166 at kernel/watchdog.c:294 watchdog_overflow_callback+0x9c/0xd0()
Nov 24 20:06:19 server kernel: [ 135.200832] Watchdog detected hard LOCKUP on cpu 1
Nov 24 20:06:19 server kernel: [ 135.200834] Modules linked in: ppdev btrfs joydev amd64_edac_mod kvm edac_core edac_mce_amd snd_hda_codec_realtek k10temp xor serio_raw raid6_pq snd_hda_codec_generic snd_hda_codec_hdmi snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd sp5100_tco tpm_infineon wmi soundcore i2c_piix4 parport_pc shpchp asus_atk0110 tpm_tis tpm parport acpi_cpufreq zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) ata_generic pata_acpi uas sata_promise usb_storage r8169 mii pata_via
Nov 24 20:06:19 server kernel: [ 135.200871] CPU: 1 PID: 1166 Comm: mount.zfs Tainted: P OE 3.17.2-200.fc20.x86_64 #1
Nov 24 20:06:19 server kernel: [ 135.200874] Hardware name: System manufacturer System Product Name/M4A88TD-M EVO, BIOS 1801 08/09/2012
Nov 24 20:06:19 server kernel: [ 135.200876] 0000000000000000 0000000099bdfb1d ffff88011fc46c10 ffffffff817271cc
Nov 24 20:06:19 server kernel: [ 135.200879] ffff88011fc46c58 ffff88011fc46c48 ffffffff81094ced ffff880119f78000
Nov 24 20:06:19 server kernel: [ 135.200882] 0000000000000000 ffff88011fc46d68 0000000000000000 ffff88011fc46ef8
Nov 24 20:06:19 server kernel: [ 135.200885] Call Trace:
Nov 24 20:06:19 server kernel: [ 135.200887]
etc.
This was with the testing repo enabled and installed on fc20 with kernel 3.17.2-200.fc20.x86_64 Do these traces provide any clue as to why this is happening and what I could do about it?
mvernimmen
commented
9 years ago @behlendorf Can this issue be re-opened or should I create a new one for this sub-volume issue?
behlendorf
commented
9 years ago @mvernimmen I've reopened it. Can you set the zil_replay_disable=1 module option and then attempt to mount the dataset. This will prevent the ZIL replay from occurring which is encountering the problem. You should then be able to mount the dataset.
mvernimmen
commented
9 years ago Hi @behlendorf , I updated to the latest testing version (gc944be5) and latest fedora kernel (3.17.4-200), then loaded zfs with the zil_replay_disable=1 option. I verified the module was loaded correctly with the setting by doing cat /sys/module/zfs/parameters/zil_replay_disable With that option the subvolume won't mount either; The machine outputs nothing but 'rcu_sched detected stalls' and the accompanying details that I can list if they're important. Unfortunately there have been no stacktraces so far. I've configured all rsyslog output to be logged to a file, in the hope of catching it, but once this particular event happens there is a whole lot of ^@ being written into it and nothing else.
Maybe I could force stacktraces through echo 'stacktrace' > .../tracing/events/somesys/someevent/trigger or with ftrace, but I would need some help in knowing what event and sys to trigger on or how to set up ftrace for this specifically. I'd love a pointer ;)
behlendorf
commented
9 years ago @mvernimmen If the mount hangs you could get a backtrace from it by look at /proc/mount-pid/stack.
mvernimmen
commented
9 years ago @behlendorf Here is the trace:
[<ffffffff81094df6>] do_wait+0x206/0x260
[<ffffffff81095ed0>] Sys_wait4+0x80/0x110
[<ffffffff8172e3e9>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffffAnd in case it helps, here is an strace:
[root@server origin]# strace zfs mount mir-2tb/fotos
execve("/sbin/zfs", ["zfs", "mount", "mir-2tb/fotos"], [/* 20 vars */]) = 0
brk(0) = 0x1444000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6cf8b9b000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=118681, ...}) = 0
mmap(NULL, 118681, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f6cf8b7e000
close(3) = 0
open("/lib64/libnvpair.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0pM\300\3436\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=89192, ...}) = 0
mmap(0x36e3c00000, 2180128, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x36e3c00000
mprotect(0x36e3c13000, 2097152, PROT_NONE) = 0
mmap(0x36e3e13000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13000) = 0x36e3e13000
close(3) = 0
open("/lib64/libuutil.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220W\340\3426\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=77624, ...}) = 0
mmap(0x36e2e00000, 2173272, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x36e2e00000
mprotect(0x36e2e11000, 2093056, PROT_NONE) = 0
mmap(0x36e3010000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x10000) = 0x36e3010000
mmap(0x36e3012000, 2392, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x36e3012000
close(3) = 0
open("/lib64/libzpool.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360g\"\3436\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1257560, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6cf8b7d000
mmap(0x36e3200000, 4284824, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x36e3200000
mprotect(0x36e332a000, 2093056, PROT_NONE) = 0
mmap(0x36e3529000, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x129000) = 0x36e3529000
mmap(0x36e3531000, 938392, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x36e3531000
close(3) = 0
open("/lib64/libzfs.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\246\0\3446\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=277256, ...}) = 0
mmap(0x36e4000000, 2367016, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x36e4000000
mprotect(0x36e4040000, 2097152, PROT_NONE) = 0
mmap(0x36e4240000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x40000) = 0x36e4240000
close(3) = 0
open("/lib64/libzfs_core.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \22\200\3436\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=18000, ...}) = 0
mmap(0x36e3800000, 2109840, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x36e3800000
mprotect(0x36e3803000, 2093056, PROT_NONE) = 0
mmap(0x36e3a02000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x36e3a02000
close(3) = 0
open("/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220X@\2779\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1163072, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6cf8b7c000
mmap(0x39bf400000, 3170648, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x39bf400000
mprotect(0x39bf505000, 2097152, PROT_NONE) = 0
mmap(0x39bf705000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x105000) = 0x39bf705000
close(3) = 0
open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\16\0\2779\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=22440, ...}) = 0
mmap(0x39bf000000, 2109744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x39bf000000
mprotect(0x39bf003000, 2093056, PROT_NONE) = 0
mmap(0x39bf202000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x39bf202000
close(3) = 0
open("/lib64/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\"\200\3009\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=47400, ...}) = 0
mmap(0x39c0800000, 2128952, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x39c0800000
mprotect(0x39c0807000, 2093056, PROT_NONE) = 0
mmap(0x39c0a06000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x39c0a06000
close(3) = 0
open("/lib64/libuuid.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\25\200\3109\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=22624, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6cf8b7b000
mmap(0x39c8800000, 2113920, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x39c8800000
mprotect(0x39c8804000, 2093056, PROT_NONE) = 0
mmap(0x39c8a03000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x39c8a03000
close(3) = 0
open("/lib64/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\"\200\2779\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=92560, ...}) = 0
mmap(0x39bf800000, 2183688, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x39bf800000
mprotect(0x39bf815000, 2093056, PROT_NONE) = 0
mmap(0x39bfa14000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14000) = 0x39bfa14000
close(3) = 0
open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220m\300\2769\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=150744, ...}) = 0
mmap(0x39bec00000, 2213104, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x39bec00000
mprotect(0x39bec18000, 2093056, PROT_NONE) = 0
mmap(0x39bee17000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x39bee17000
mmap(0x39bee19000, 13552, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x39bee19000
close(3) = 0
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\36\202\2769\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2096496, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6cf8b7a000
mmap(0x39be800000, 3920480, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x39be800000
mprotect(0x39be9b4000, 2093056, PROT_NONE) = 0
mmap(0x39bebb3000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b3000) = 0x39bebb3000
mmap(0x39bebb9000, 16992, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x39bebb9000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6cf8b79000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6cf8b77000
arch_prctl(ARCH_SET_FS, 0x7f6cf8b77bc0) = 0
mprotect(0x617000, 4096, PROT_READ) = 0
mprotect(0x36e3e13000, 4096, PROT_READ) = 0
mprotect(0x36e3010000, 4096, PROT_READ) = 0
mprotect(0x36e3529000, 8192, PROT_READ) = 0
mprotect(0x36e4240000, 4096, PROT_READ) = 0
mprotect(0x36e3a02000, 4096, PROT_READ) = 0
mprotect(0x39bf705000, 4096, PROT_READ) = 0
mprotect(0x39bf202000, 4096, PROT_READ) = 0
mprotect(0x39c0a06000, 4096, PROT_READ) = 0
mprotect(0x39c8a03000, 4096, PROT_READ) = 0
mprotect(0x39bfa14000, 4096, PROT_READ) = 0
mprotect(0x39bee17000, 4096, PROT_READ) = 0
mprotect(0x39bebb3000, 16384, PROT_READ) = 0
mprotect(0x39be61f000, 4096, PROT_READ) = 0
munmap(0x7f6cf8b7e000, 118681) = 0
set_tid_address(0x7f6cf8b77e90) = 3264
set_robust_list(0x7f6cf8b77ea0, 24) = 0
rt_sigaction(SIGRTMIN, {0x39bec06880, [], SA_RESTORER|SA_SIGINFO, 0x39bec0f6d0}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x39bec06910, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x39bec0f6d0}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
brk(0) = 0x1444000
brk(0x1465000) = 0x1465000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=106070960, ...}) = 0
mmap(NULL, 106070960, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f6cf264e000
close(3) = 0
access("/sys/module/zfs", F_OK) = 0
open("/dev/zfs", O_RDWR) = 3
open("/etc/mtab", O_RDONLY) = 4
open("/etc/dfs/sharetab", O_RDONLY) = 5
open("/dev/zfs", O_RDWR) = 6
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=2492, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6cf8b9a000
read(7, "# Locale name alias data base.\n#"..., 4096) = 2492
read(7, "", 4096) = 0
close(7) = 0
munmap(0x7f6cf8b9a000, 4096) = 0
open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/zfs-linux-user.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US.utf8/LC_MESSAGES/zfs-linux-user.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US/LC_MESSAGES/zfs-linux-user.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.UTF-8/LC_MESSAGES/zfs-linux-user.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.utf8/LC_MESSAGES/zfs-linux-user.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/zfs-linux-user.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
ioctl(3, 0x5a12, 0x7fffa0276a40) = 0
ioctl(3, 0x5a05, 0x7fffa02733f0) = 0
ioctl(3, 0x5a27, 0x7fffa0271c90) = 0
open("/etc/mtab", O_RDONLY) = 7
dup3(7, 4, 0) = 4
close(7) = 0
fstat(4, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6cf8b9a000
read(4, "rootfs / rootfs rw,seclabel 0 0\n"..., 1024) = 1024
read(4, "memory 0 0\ncgroup /sys/fs/cgroup"..., 1024) = 1024
read(4, "pc_pipefs rpc_pipefs rw,relatime"..., 1024) = 514
read(4, "", 1024) = 0
lstat("/mnt/origin/mir-2tb/fotos", {st_mode=S_IFDIR|0755, st_size=2, ...}) = 0
openat(AT_FDCWD, "/mnt/origin/mir-2tb/fotos", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 7
getdents(7, /* 2 entries */, 32768) = 48
getdents(7, /* 0 entries */, 32768) = 0
close(7) = 0
vfork() = 3265
wait4(3265,Hi @behlendorf , anything I can do to help this problem analysed and fixed?
behlendorf
commented
9 years ago @mvernimmen I may have a workaround for you. Try setting the property readonly=on and then mounting it. This may allow you to mount it and copy off the data.
zfs set readonly=on mir-2tb/fotos
zfs mount mir-2tb/fotosHi @behlendorf , Thanks for the suggestion but I knew this, I wrote "I am currently unable to mount one of my pools in write mode (readonly works fine)" in my first post. I love zfs and want to help make it better on linux, that's why I created this issue in the hope of being able to get this bug fixed. Is there something we can do to trace this down to its cause and fix it?
behlendorf
commented
9 years ago Ahh, I see. Then would it be possible for you to grab the latest code from github and then recreate the problem. But this time if at all possible dump the stacks from all the processes on your system using sysrq-t. The above stack indicates some sort of deadlock but without the others from the system it's hard to say why. If the system is still responsive, as root run the following:
echo t >/proc/sysrq-trigger
dmesg@behlendorf With the new git code (g2d9d57b and ga3c1eb7) and kernel 3.17.7-200-fc20 the problem stil occurs. The dmesg output follows, here are some timings: t=775: all process stacktrace before starting the mount (as reference) t=780 or so, started mount of problematic subvolume t=913: automatic stacktrace from hard lock t=927, 1108, 1288: automatic stacktrace from stalls t=1447: sys-request for stacktraces t=1468: automatic stacktrace from stall
[ 775.930556] SysRq : Show State
[ 775.930653] task PC stack pid father
[ 775.930706] systemd S ffff88011fc53840 0 1 0 0x00000000
[ 775.930814] ffff880119da7dc8 0000000000000086 ffff880119e40000 0000000000013840
[ 775.930989] ffff880119da7fd8 0000000000013840 ffff8801176fce80 ffff880119e40000
[ 775.931166] 0000000000000000 0000000000000000 ffff880036635120 ffff880119e40000
[ 775.931342] Call Trace:
[ 775.931399] [<ffffffff8172a999>] schedule+0x29/0x70
[ 775.931456] [<ffffffff8172ddfd>] schedule_hrtimeout_range_clock+0x14d/0x170
[ 775.931512] [<ffffffff812449d9>] ? ep_scan_ready_list.isra.9+0x1b9/0x1f0
[ 775.931565] [<ffffffff8172de33>] schedule_hrtimeout_range+0x13/0x20
[ 775.931617] [<ffffffff81244c6e>] ep_poll+0x23e/0x360
[ 775.931671] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
[ 775.931723] [<ffffffff812460f5>] SyS_epoll_wait+0xb5/0xe0
[ 775.931775] [<ffffffff8172ec69>] system_call_fastpath+0x16/0x1b
[ 775.931824] kthreadd S ffff88011fc13840 0 2 0 0x00000000
[ 775.931933] ffff880119e4fea8 0000000000000046 ffff880119e409d0 0000000000013840
[ 775.932106] ffff880119e4ffd8 0000000000013840 ffffffff81c1b460 ffff880119e409d0
[ 775.932281] ffff8800a64cc1c0 ffff880119e409d0 0000000000000000 ffff880119e409d0
[ 775.932457] Call Trace:
[ 775.932508] [<ffffffff8172a999>] schedule+0x29/0x70
[ 775.932561] [<ffffffff810b0ea5>] kthreadd+0x235/0x240
[ 775.932614] [<ffffffff810b0c70>] ? kthread_create_on_cpu+0x60/0x60
[ 775.932668] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 775.932720] [<ffffffff810b0c70>] ? kthread_create_on_cpu+0x60/0x60
[ 775.932769] ksoftirqd/0 S ffff88011fc13840 0 3 2 0x00000000
[ 775.932874] ffff880119e63e68 0000000000000046 ffff880119e413a0 0000000000013840
[ 775.933049] ffff880119e63fd8 0000000000013840 ffff8800a0fb8000 ffff880119e413a0
[ 775.933222] ffff880119e413a0 ffff88011b402490 ffffffff81c41a40 ffff880119e413a0
[ 775.933395] Call Trace:
[ 775.933446] [<ffffffff8172a999>] schedule+0x29/0x70
[ 775.933497] [<ffffffff810b3fa3>] smpboot_thread_fn+0xd3/0x1a0
[ 775.933548] [<ffffffff8172a999>] ? schedule+0x29/0x70
[ 775.933599] [<ffffffff810b3ed0>] ? SyS_setgroups+0x1a0/0x1a0
[ 775.933654] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 775.933707] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 775.933758] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 775.933809] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 775.933858] kworker/0:0H S ffff88011fc13840 0 5 2 0x00000000
[ 775.933973] ffff880119e73e48 0000000000000046 ffff880119e42740 0000000000013840
[ 775.934147] ffff880119e73fd8 0000000000013840 ffff880036abbae0 ffff880119e42740
[ 775.934317] ffff88011fc13440 ffff88011fc13440 ffff88011fc13458 ffff880119e42740
[ 775.934489] Call Trace:
[ 775.934540] [<ffffffff8172a999>] schedule+0x29/0x70
[ 775.934590] [<ffffffff810ab4bc>] worker_thread+0x18c/0x490
[ 775.934644] [<ffffffff810ab330>] ? rescuer_thread+0x2e0/0x2e0
[ 775.935106] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 775.935157] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 775.935208] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 775.935260] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 775.935309] rcu_sched S ffff88011fc13840 0 7 2 0x00000000
[ 775.935417] ffff880119e7be00 0000000000000046 ffff880119e43ae0 0000000000013840
[ 775.935589] ffff880119e7bfd8 0000000000013840 ffffffff81c1b460 ffff880119e43ae0
[ 775.935759] ffffffff81c56400 ffffffff81c5a560 ffff88011fc0d8c0 ffffffff81c56500
[ 775.935935] Call Trace:
[ 775.935985] [<ffffffff8172a999>] schedule+0x29/0x70
[ 775.936037] [<ffffffff810ef362>] rcu_gp_kthread+0xa2/0x750
[ 775.936093] [<ffffffff810d1840>] ? prepare_to_wait_event+0xf0/0xf0
[ 775.936146] [<ffffffff810ef2c0>] ? rcu_cpu_notify+0x440/0x440
[ 775.936198] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 775.936249] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 775.936302] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 775.936354] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 775.936404] rcuos/0 S ffff88011fc53840 0 8 2 0x00000000
[ 775.936509] ffff880119e7fe20 0000000000000046 ffff880119e444b0 0000000000013840
[ 775.936682] ffff880119e7ffd8 0000000000013840 ffff880119e44e80 ffff880119e444b0
[ 775.936855] ffff88011fc0da50 ffff88011fc0d8c0 0000000000000202 ffff880119e444b0
[ 775.937028] Call Trace:
[ 775.937078] [<ffffffff8172a999>] schedule+0x29/0x70
[ 775.937128] [<ffffffff810edf7a>] rcu_nocb_kthread+0x24a/0x520
[ 775.937180] [<ffffffff810d1840>] ? prepare_to_wait_event+0xf0/0xf0
[ 775.937233] [<ffffffff810edd30>] ? rcu_implicit_dynticks_qs+0x180/0x180
[ 775.937288] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 775.937340] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 775.937390] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 775.937442] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 775.937491] rcuos/1 S ffff88011fc53840 0 9 2 0x00000000
[ 775.937598] ffff880119e83e20 0000000000000046 ffff880119e44e80 0000000000013840
[ 775.937770] ffff880119e83fd8 0000000000013840 ffff880119e9d850 ffff880119e44e80
[ 775.937940] ffff88011fc4da50 ffff88011fc4d8c0 0000000000000001 ffff8800a649c780
[ 775.938112] Call Trace:
[ 775.938162] [<ffffffff8172a999>] schedule+0x29/0x70
[ 775.938214] [<ffffffff810edefa>] rcu_nocb_kthread+0x1ca/0x520
[ 775.938267] [<ffffffff810d1840>] ? prepare_to_wait_event+0xf0/0xf0
[ 775.938319] [<ffffffff810edd30>] ? rcu_implicit_dynticks_qs+0x180/0x180
[ 775.938372] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 775.938426] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 775.938478] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 775.938530] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 775.938579] rcuos/2 S ffff88011fc13840 0 10 2 0x00000000
[ 775.938686] ffff880119e87e20 0000000000000046 ffff880119e45850 0000000000013840
[ 775.938857] ffff880119e87fd8 0000000000013840 ffff880119e46bf0 ffff880119e45850
[ 775.939031] ffff88011fc8da50 ffff88011fc8d8c0 ffffffff810edd30 ffff880119e45850
[ 775.939204] Call Trace:
[ 775.939254] [<ffffffff810edd30>] ? rcu_implicit_dynticks_qs+0x180/0x180
[ 775.939306] [<ffffffff8172a999>] schedule+0x29/0x70
[ 775.939356] [<ffffffff810edf7a>] rcu_nocb_kthread+0x24a/0x520
[ 775.939411] [<ffffffff8172a48b>] ? __schedule+0x2eb/0x7d0
[ 775.939462] [<ffffffff810d1840>] ? prepare_to_wait_event+0xf0/0xf0
[ 775.939514] [<ffffffff810edd30>] ? rcu_implicit_dynticks_qs+0x180/0x180
[ 775.939566] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 775.939617] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 775.939671] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 775.939725] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 775.939775] rcuos/3 S ffff88011fc13840 0 11 2 0x00000000
[ 775.939878] ffff880119e8be20 0000000000000046 ffff880119e46220 0000000000013840
[ 775.940050] ffff880119e8bfd8 0000000000013840 ffff880119e46bf0 ffff880119e46220
[ 775.940224] ffff88011fccda50 ffff88011fccd8c0 ffffffff810edd30 0000000000000000
[ 775.940398] Call Trace:
[ 775.940447] [<ffffffff810edd30>] ? rcu_implicit_dynticks_qs+0x180/0x180
[ 775.940499] [<ffffffff8172a999>] schedule+0x29/0x70
[ 775.940550] [<ffffffff810edefa>] rcu_nocb_kthread+0x1ca/0x520
[ 775.940605] [<ffffffff8172a48b>] ? __schedule+0x2eb/0x7d0
[ 775.940657] [<ffffffff810d1840>] ? prepare_to_wait_event+0xf0/0xf0
[ 775.940709] [<ffffffff810edd30>] ? rcu_implicit_dynticks_qs+0x180/0x180
[ 775.940760] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 775.940815] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 775.940867] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 775.940919] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 775.940967] rcuos/4 S ffff88011fc13840 0 12 2 0x00000000
[ 775.941072] ffff880119e8fe20 0000000000000046 ffff880119e46bf0 0000000000013840
[ 775.941244] ffff880119e8ffd8 0000000000013840 ffff880119e989d0 ffff880119e46bf0
[ 775.941417] ffff88011fd0da50 ffff88011fd0d8c0 ffffffff810edd30 ffff880119e46bf0
[ 775.941590] Call Trace:
[ 775.941640] [<ffffffff810edd30>] ? rcu_implicit_dynticks_qs+0x180/0x180
[ 775.946612] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
ffff88011fc13840 [ 775.953555] Call Trace:
0000000000000046
[ 775.958767] ffff880119ecef00[ 775.958938] Call Trace:
[ 775.958989] [<ffffffff810ab050>] ? process_one_work+0x3d0/0x3d0
ffff880119ee4e80
[ 775.962548] Call Trace:
ffff880119f44e80 0000000000013840
[ 775.964484] ffff880119873fd8[ 775.966236] [<ffffffff810ab050>] ? process_one_work+0x3d0/0x3d0
[ 775.966291] [<ffffffff8172a999>] schedule+0x29/0x70
[ 775.966341] [<ffffffff810ab271>] rescuer_thread+0x221/0x2e0
[ 775.971843] [<ffffffff81505110>] ? hub_port_debounce+0x170/0x170
[ 775.973551] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 775.973600] kswapd0 S 0000000000013840 ffffffff81c1b460 ffffffff810ab050 ffff880119a27e80
[ 775.977499] Call Trace:
[ 775.978582] Call Trace:
[ 775.981503] ffff8800368abfd8[ 775.983353] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 775.985989] scsi_tmf_3 S[ 775.987182] ffff8800368c7e18[ 775.987954] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 775.991281] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 775.993150] Call Trace:
[ 775.993200] [<ffffffff810ab050>] ? process_one_work+0x3d0/0x3d0
[ 775.996678] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
ffff880036aba740 0000000000013840[ 776.000354] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 776.001542] ffff88011916fe18[ 776.004447] [<ffffffff814ae5e0>] ? scsi_eh_get_sense+0x300/0x300
ffffffff810ab050[ 776.008751] [<ffffffff814ae674>] scsi_error_handler+0x94/0x8b0
[ 776.011181] Call Trace:
ffff8800367109d0[ 776.012633] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 776.015946] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
ffff880036713ae0 0000000000013840
[ 776.017358] ffff880036743fd8[ 776.018106] [<ffffffffa002d84b>] usb_stor_control_thread+0x7b/0x270 [usb_storage]
[ 776.019417] [<ffffffff810b04a8>] kthread+0xd8/0xf0
0000000000000046
[ 776.025076] ffff880035c07c00 ffff880035d013a0 ffffffff810ab050 ffff8800367e89d0
[ 776.029436] ffff880035c94824 ffff880035c94be0 0 499 2 0x00000000
[ 776.030183] ffff880035c9fe28 0000000000000046[ 776.033344] [<ffffffff81214a1e>] do_select+0x5be/0x7a0
0000000000000046 ffff8800c9c0e588 ffff8800c9138000 ffff8800367e9d70 ffff8800cb4e6fa8[ 776.037741] [<ffffffffa0054b29>] taskq_thread+0x329/0x350 [spl]
0 597 2 0x00000000
ffff8800367ebae0 0000000000013840[ 776.044716] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
[ 776.047093] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 776.050536] [<ffffffff8172a999>] schedule+0x29/0x70
[ 776.050595] [<ffffffffa0054b29>] taskq_thread+0x329/0x350 [spl]
[ 776.055655] ffff8800ca61bfd8 0000000000013840[ 776.057535] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 776.057587] zvol/11 S ffff88011fc13840 0 613 2 0x00000000
0000000000013840 ffffffff81c1b460 ffff8800ca628000
[ 776.058035] ffff8800ca2d9420[ 776.060413] Call Trace:
ffff8800ca62b110
[ 776.063949] ffff8800ca2dbc00 ffff880036b465a8
[ 776.064120] Call Trace:
0000000000013840
[ 776.065980] ffff8800ca64ffd8[ 776.068966] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
ffffffff81c1b460[ 776.072326] zvol/24 S 0000000000013840
[ 776.072776] ffff8800ca2b4180 ffff8800ca6709d0 0000000000013840
[ 776.074108] ffff8800ca67bfd8 ffff88011fc13840 0 630 2 0x00000000
[ 776.077251] ffff8800ca687e10 0000000000000046 ffff8800ca672740[ 776.080248] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 776.080300] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 776.083374] [<ffffffff8172a999>] schedule+0x29/0x70
[ 776.083424] [<ffffffff810ab271>] rescuer_thread+0x221/0x2e0
ffff8800365c1d70 0000000000013840
[ 776.086601] ffff8800c8ebffd8 0000000000013840 ffffffff81c1b460 ffff8800365c1d70[ 776.089536] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
0000000000013840[ 776.090645] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 776.094624] Call Trace:
[ 776.094784] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
[ 776.097932] Call Trace:
[ 776.098095] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
[ 776.099829] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
0000000000000046 ffff8800cab4b110 ffff880035cf6ba8
[ 776.102758] Call Trace:
[ 776.102808] [<ffffffff8172a999>] schedule+0x29/0x70
[ 776.104023] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
0000000000000046 ffff8800caaef5c0 ffff8800c2e78000
[ 776.107007] ffff8800c376bc60 ffff8800c9d71888 ffff8800c2e78000 ffff8800c372f5c0 0000000000013840
[ 776.110142] ffff8800c2e8bfd8 0000000000013840 ffff880119e9d850 ffff8800c372f5c0
[ 776.110316] ffff8800c9083f60[ 776.113104] Call Trace:
[ 776.114366] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
ffff8800c37293a0 ffff880035cf6a88[ 776.116682] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 776.118444] ffff8800c9083c60[ 776.120160] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 776.123362] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 776.126750] Call Trace:
[ 776.126799] [<ffffffff8172a999>] schedule+0x29/0x70
[ 776.126858] [<ffffffffa0054b29>] taskq_thread+0x329/0x350 [spl]
ffff880119e9d850 ffff8800c2e7a740
[ 776.129882] ffff8800c376bde0 ffff8800c9d71688 ffff8800c2e7a740 ffff8800c2ea5850 0000000000013840
[ 776.133020] ffff8800c2ef3fd8[ 776.134584] [<ffffffffa0054b29>] taskq_thread+0x329/0x350 [spl]
[ 776.134638] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
[ 776.137570] z_fr_iss_1/0 S ffff88011fc53840 0 933 2 0x00000000
0000000000013840[ 776.140671] [<ffffffff810b04a8>] kthread+0xd8/0xf0
ffff88011fc53840 0 936 2 0x00000000
0000000000000046 ffff880035cf6888[ 776.142762] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
[ 776.145909] Call Trace:
ffff8800c2e7ebf0[ 776.149051] ffff8800ca2d92a0[ 776.149274] [<ffffffff8172a999>] schedule+0x29/0x70
[ 776.152269] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 776.153871] ffff8800c9083240 ffff8800c8ceaa88 ffff8800c2f13ae0 ffff8800c8ceaaa8
[ 776.154042] Call Trace:
[ 776.157012] ffff8800c2f57fd8 0000000000013840 ffff8800c2f30000 ffff8800c2f31d70
[ 776.157186] ffff8800ca2d9c00 ffff8800c9d71e88
[ 776.158460] Call Trace:
[ 776.158511] [<ffffffff8172a999>] schedule+0x29/0x70
[ 776.159994] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
0000000000013840
[ 776.161422] ffff8800c2f67fd8[ 776.162879] Call Trace:
[ 776.163340] [<ffffffff8172a999>] schedule+0x29/0x70
[ 776.164668] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 776.167353] ffff8800c2f7bfd8 ffff8800c9d715a8 0000000000013840 ffff8800c2e7e220 0 961 2 0x00000000
ffff8800c2f175c0[ 776.170286] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 776.174481] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
[ 776.174901] ffff8800c2f9fe10[ 776.178177] [<ffffffff810ab271>] rescuer_thread+0x221/0x2e0
[ 776.180153] Call Trace:
[ 776.180202] [<ffffffff810ab050>] ? process_one_work+0x3d0/0x3d0
[ 776.181983] ffff8800caafffd8 0000000000013840[ 776.184966] btrfs-fixup S ffff88011fc13840 0 992 2 0x00000000
[ 776.185076] ffff8800c8acbe28 0000000000000046[ 776.185694] [<ffffffff8172a999>] schedule+0x29/0x70
[ 776.185744] [<ffffffff810ab271>] rescuer_thread+0x221/0x2e0
[ 776.189713] btrfs-endio-rai S 0000000000013840
[ 776.189996] ffff8800c2dbbfd8 0 999 2 0x00000000
[ 776.193155] ffff8800caaa7e28 0000000000000046 ffff880036ab89d0 0000000000013840
[ 776.193327] ffff8800caaa7fd8 0 1002 2 0x00000000
[ 776.196417] ffff8800caa63e28
[ 776.196763] ffff8800c2d77180 0000000000000046 ffff880119f95850
[ 776.200043] ffff8801196f3000[ 776.201558] ffff880119f74000[ 776.201728] Call Trace:
[ 776.202118] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 776.205356] [<ffffffffa00587c5>] __cv_wait_interruptible+0x15/0x20 [spl]
[ 776.208047] ffff8800bf964780 0000000000000046 ffff8800c2f89d70 0000000000013840
[ 776.211156] ffff8800bf62bfd8 0000000000013840[ 776.214392] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 776.214444] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 776.217531] [<ffffffffa0054b29>] taskq_thread+0x329/0x350 [spl]
[ 776.221497] ffff8800be713fd8 0000000000013840 ffff880119e9d850 ffff880117dbe220 ffff880117dbd850
[ 776.224984] ffff88003657ae40[ 776.228148] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 776.229973] Call Trace:
[ 776.230023] [<ffffffff8172a999>] schedule+0x29/0x70
[ 776.230084] [<ffffffffa0054b29>] taskq_thread+0x329/0x350 [spl]
[ 776.230135] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
[ 776.234768] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 776.234822] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 776.238026] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 776.241845] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 776.246582] ffff8800bd05bfd8[ 776.248148] [<ffffffffa0054b29>] taskq_thread+0x329/0x350 [spl]
[ 776.248200] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
[ 776.251572] [<ffffffffa0054800>] ? taskq_cancel_id+0x140/0x140 [spl]
[ 776.251782] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 776.256549] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 776.256752] ffff8800be7a7e10 0000000000000046 ffff8800be752740 0000000000013840[ 776.259807] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 776.261061] z_fr_iss_1/3 S ffff880119e9d850 ffff8800bf654e80 ffff8800be7544b0 ffff8800000128a8
[ 776.265003] Call Trace:
[ 776.265111] [<ffffffffa0054b29>] taskq_thread+0x329/0x350 [spl]
[ 776.265997] ffff8800bd063e10[ 776.266792] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 776.271690] ffff8800bd077fd8 ffff88011fc13840 0 1179 2 0x00000000
[ 776.272620] ffff8800bd07be10 0000000000013840 ffff880117da6bf0
[ 776.276277] ffff8800bf0abba0[ 776.277786] [<ffffffffa0054800>] ? taskq_cancel_id+0x140/0x140 [spl]
[ 776.277838] [<ffffffff810b04a8>] kthread+0xd8/0xf0
ffff880117cf93a0
[ 776.282866] z_fr_iss_6/2 S ffff8800be751d70[ 776.284074] ffff8800be7ebe10 ffff8800365fd7a8[ 776.286802] Call Trace:
ffff880117cfbae0 0000000000000046 ffff880117cfc4b0[ 776.289334] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 776.290590] z_cl_iss/0 S[ 776.291723] [<ffffffffa0054b29>] taskq_thread+0x329/0x350 [spl]
[ 776.296140] [<ffffffffa0054b29>] taskq_thread+0x329/0x350 [spl]
[ 776.296305] [<ffffffff810b04a8>] kthread+0xd8/0xf0
0 1207 2 0x00000000
[ 776.298444] [<ffffffff812d9370>] ? commit_timeout+0x10/0x10
ffffffff810ab050 ffff8800bf0bb880
[ 776.299332] Call Trace:
ffffffff81c1b460 ffff880117d3a740
[ 776.305947] ffff8800c89cfeb8 00000000038f9b17 0000000000000000 ffff880117d3a740
[ 776.306120] Call Trace:
[ 776.307777] audispd S 0000000000013840
[ 776.311485] ffff8800bb9ebfd8 0000000000013840 ffff880119e9d850 ffff880117d3e220
[ 776.314992] Call Trace:
[ 776.315146] [<ffffffff810f5e80>] ? hrtimer_get_res+0x50/0x50
[ 776.316297] [<ffffffff81214190>] ? poll_select_copy_remaining+0x150/0x150
[ 776.319307] [<ffffffff81200198>] ? vfs_read+0x98/0x170
0000000000013840 ffffffff81c1b460[ 776.324091] [<ffffffff8172ec69>] system_call_fastpath+0x16/0x1b
[ 776.326713] 0000000000000000 0000000000000040[ 776.328019] [<ffffffff8160cd05>] ? skb_free_datagram+0x15/0x40
[ 776.332717] [<ffffffff8172ddfd>] schedule_hrtimeout_range_clock+0x14d/0x170
[ 776.335316] [<ffffffff81215682>] do_sys_poll+0x442/0x560
[ 776.337447] [<ffffffff810d12cc>] ? add_wait_queue+0x3c/0x50
0000000000000000
[ 776.339471] Call Trace:
[ 776.339521] [<ffffffff8172a999>] schedule+0x29/0x70
[ 776.339571] [<ffffffff81241412>] inotify_read+0x2c2/0x3e0
[ 776.340449] Call Trace:
[ 776.341251] ffff8800bba4fac8 0000000005f5e100 0000000000000000[ 776.341629] [<ffffffff8172dd5c>] ? schedule_hrtimeout_range_clock+0xac/0x170
[ 776.350148] [<ffffffff81214190>] ? poll_select_copy_remaining+0x150/0x150
0 1311 1 0x00000080
0 1321 1300 0x00000080
[ 776.353064] ffff8800c9033a48 0000000000000086 ffff8800bd0bc4b0[ 776.358165] [<ffffffff81094175>] ? wait_consider_task+0xa5/0xb30
[ 776.358217] [<ffffffff8105c9fc>] ? __do_page_fault+0x21c/0x540
[ 776.361641] [<ffffffff810f6497>] ? hrtimer_try_to_cancel+0x57/0x100
[ 776.364636] [<ffffffff813038db>] ? cred_has_capability+0x6b/0x120
[ 776.368058] [<ffffffff8172a48b>] ? __schedule+0x2eb/0x7d0
[ 776.368270] [<ffffffff81725929>] ? __slab_free+0x10e/0x277
[ 776.371537] [<ffffffff8172a999>] schedule+0x29/0x70
ffff880117db89d0
[ 776.374879] 0000000000000000
[ 776.375048] Call Trace:
[ 776.376703] Call Trace:
[ 776.376857] [<ffffffff810d12cc>] ? add_wait_queue+0x3c/0x50
0000000000013840
[ 776.380594] 0000000000000000[ 776.381075] [<ffffffff81215682>] do_sys_poll+0x442/0x560
[ 776.384131] 0000000000000000 0000000000000000 ffff8800b7073bdc 0000000000013840 ffff8800b7346220 ffff8800b7344e80
[ 776.387559] ffff8800b7344e80 0000000000000000 ffffc90010753980[ 776.390781] [<ffffffff81106f50>] futex_wait_queue_me+0xd0/0x130
[ 776.390994] [<ffffffff8172ec69>] system_call_fastpath+0x16/0x1b
[ 776.391043] libvirtd S[ 776.395605] Call Trace:
[ 776.395655] [<ffffffff8172a999>] schedule+0x29/0x70
[ 776.397263] [<ffffffff816c740d>] ? unix_dgram_sendmsg+0x58d/0x660
ffff880035d00000 0000000000013840
[ 776.401016] ffff8800b7c5bfd8[ 776.402711] ffff88011740fe28[ 776.403280] [<ffffffff810ab050>] ? process_one_work+0x3d0/0x3d0
[ 776.406091] [<ffffffff81214190>] ? poll_select_copy_remaining+0x150/0x150
[ 776.406306] [<ffffffff810baccd>] ? ttwu_do_activate.constprop.90+0x5d/0x70
[ 776.410037] [<ffffffff81094e06>] do_wait+0x206/0x260
0 2561 1256 0x00000080
[ 776.413203] ffff8800bb647900[ 776.413724] Call Trace:
[ 776.414765] [<ffffffff8105c9fc>] ? __do_page_fault+0x21c/0x540
[ 776.416733] [<ffffffff812158a4>] SyS_poll+0x74/0x110
[ 776.417606] [<ffffffff810f5e80>] ? hrtimer_get_res+0x50/0x50
ffff8800b71c6bf0
[ 776.421944] ffff8800b71c6bf0
[ 776.422118] Call Trace:
[ 776.422168] [<ffffffff8172a999>] schedule+0x29/0x70
ffffc90010747b80
[ 776.427078] Call Trace:
0 2689 2389 0x00000080
[ 776.429563] [<ffffffff8172ec69>] system_call_fastpath+0x16/0x1b
ffff88011fc53840 [ 776.435393] [<ffffffff81107256>] futex_wait+0x176/0x270
[ 776.438849] ffff880117d3c4b0[ 776.439762] ffff8800c8d87fd8 0000000000013840 ffff880035d06220
[ 776.442490] ffff8800bd0ba740 7fffffffffffffff ffff8800c92f7ed0 ffff8800bd0ba740[ 776.445809] [<ffffffff812460f5>] SyS_epoll_wait+0xb5/0xe0
[ 776.446585] Call Trace:
[ 776.446635] [<ffffffff8172a999>] schedule+0x29/0x70
[ 776.449271] [<ffffffff8172da29>] schedule_timeout+0x229/0x2a0
[ 776.452630] [<ffffffff8172ec69>] system_call_fastpath+0x16/0x1b
ffff8800b7261d70[ 776.455998] [<ffffffff8172ec69>] system_call_fastpath+0x16/0x1b
ffff8800c93093a0 0000000000013840
[ 776.456326] ffff880117417fd8[ 776.457813] [<ffffffff813048ac>] ? selinux_file_alloc_security+0x3c/0x60
[ 776.457868] [<ffffffff81302ff6>] ? file_has_perm+0xa6/0xb0
ffff8800ca676bf0 0000000000013840
[ 776.462948] ffff8800b73bffd8[ 776.466013] [<ffffffff813048ac>] ? selinux_file_alloc_security+0x3c/0x60
[ 776.467837] [<ffffffff810bd37b>] ? try_to_wake_up+0xeb/0x300
[ 776.468145] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
[ 776.471308] [<ffffffff810d12cc>] ? add_wait_queue+0x3c/0x50
[ 776.474530] [<ffffffff8172de33>] schedule_hrtimeout_range+0x13/0x20
[ 776.476291] [<ffffffff816065be>] ? skb_free_head+0x1e/0x80
[ 776.477677] 0000000000000000[ 776.479770] [<ffffffff810d12cc>] ? add_wait_queue+0x3c/0x50
[ 776.481894] [<ffffffff8172ddfd>] schedule_hrtimeout_range_clock+0x14d/0x170
[ 776.485377] [<ffffffff81214190>] ? poll_select_copy_remaining+0x150/0x150
[ 776.488691] [<ffffffff81213fe9>] poll_schedule_timeout+0x49/0x70
[ 776.489114] [<ffffffff811e2d66>] ? kmem_cache_alloc_trace+0x1d6/0x200
ffff8800a8ad5850 0000000000013840
[ 776.492181] ffff8800a886bfd8[ 776.493881] ffff8800a8aa3a48[ 776.494657] [<ffffffff81213fe9>] poll_schedule_timeout+0x49/0x70
[ 776.499124] [<ffffffff810d1178>] ? __wake_up_common+0x58/0x90
[ 776.499177] [<ffffffff81214190>] ? poll_select_copy_remaining+0x150/0x150
[ 776.502344] [<ffffffff81213fe9>] poll_schedule_timeout+0x49/0x70
0000000000013840
[ 776.505934] ffff8800a8b33fd8[ 776.506695] [<ffffffff81215682>] do_sys_poll+0x442/0x560
[ 776.509719] [<ffffffff810bab69>] ? ttwu_do_wakeup+0x19/0xd0
0000000000013840[ 776.510961] [<ffffffff815fd42b>] ? sock_recvmsg+0xab/0xf0
[ 776.514476] 0000000000000000
[ 776.514647] Call Trace:
[ 776.514700] [<ffffffff8172a999>] schedule+0x29/0x70
0000000000000000 ffffc9001074b180[ 776.519447] [<ffffffff81213fe9>] poll_schedule_timeout+0x49/0x70
[ 776.519501] [<ffffffff81215682>] do_sys_poll+0x442/0x560
[ 776.519553] [<ffffffff817270d8>] ? avc_compute_av+0x1a3/0x1b5
[ 776.522581] gdbus S 0000000000013840
[ 776.522860] ffff8800a656ffd8 0000000000013840 ffff88011fc13840 0 2944 1 0x00000080
[ 776.529332] [<ffffffff81214a1e>] do_select+0x5be/0x7a0
[ 776.529542] [<ffffffff81381519>] ? flex_array_get_ptr+0x9/0x20
[ 776.532564] [<ffffffff8172a999>] schedule+0x29/0x70
[ 776.532615] [<ffffffff8172ddfd>] schedule_hrtimeout_range_clock+0x14d/0x170
[ 776.535777] ffff8800a64dffd8 0000000000013840 ffff8800cab49d70[ 776.536431] [<ffffffff81213fe9>] poll_schedule_timeout+0x49/0x70
[ 776.539377] [<ffffffff810baccd>] ? ttwu_do_activate.constprop.90+0x5d/0x70
[ 776.541134] [<ffffffff810d1178>] ? __wake_up_common+0x58/0x90
[ 776.543137] 0000000000000000 ffff8800a88089d0 0000000000013840[ 776.545642] [<ffffffff81213fe9>] poll_schedule_timeout+0x49/0x70
[ 776.548281] [<ffffffff81248b17>] ? eventfd_ctx_read+0x67/0x1c0
[ 776.548334] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
[ 776.553264] [<ffffffff8172de33>] schedule_hrtimeout_range+0x13/0x20
[ 776.555194] [<ffffffff81215682>] do_sys_poll+0x442/0x560
[ 776.557069] [<ffffffff81213fe9>] poll_schedule_timeout+0x49/0x70
[ 776.557121] [<ffffffff81215682>] do_sys_poll+0x442/0x560
[ 776.560241] [<ffffffff8172de33>] schedule_hrtimeout_range+0x13/0x20
[ 776.563884] [<ffffffff81216d34>] ? dput+0x24/0x170
[ 776.565486] ffff8800a0e57ce8[ 776.567180] ffff8800a0d2fd7c[ 776.567401] [<ffffffff8172a999>] schedule+0x29/0x70
[ 776.567453] [<ffffffff8144d9ca>] __vt_event_wait.isra.0.part.1+0x4a/0x90
[ 776.567612] [<ffffffff812fd6e8>] ? security_capable+0x18/0x20
ffff8800a0d00000
[ 776.568521] ffff8800a0e5bd7c 0 3039 1 0x00000080
[ 776.573958] ffff8800a0e6bce8 0000000000000086[ 776.577211] [<ffffffff8172a999>] schedule+0x29/0x70
[ 776.577263] [<ffffffff8144d9ca>] __vt_event_wait.isra.0.part.1+0x4a/0x90
[ 776.582377] ffff8800a0e83d7c 0000000000005607[ 776.584620] [<ffffffff8109adc9>] ? ns_capable+0x29/0x50
ffff8800a0f43110 ffff8800a0e90000[ 776.589328] ffff8800a0ea3fd8 ffff8800bb61e400[ 776.591117] [<ffffffff8144d9ca>] __vt_event_wait.isra.0.part.1+0x4a/0x90
ffff8800a0e93ae0
[ 776.595291] ffff8800a0eb3d7c ffff8800a0eb3d7c 0000000000005607 ffff8800bb61e400
[ 776.596462] ffff8800a0eb7fd8 0000000000013840 ffff8800a0e93ae0
[ 776.596810] Call Trace:
[ 776.596859] [<ffffffff8172a999>] schedule+0x29/0x70
0 3060 1 0x00000080
0000000000013840 ffff8800a0e96220 ffff8800a6772740[ 776.606298] [<ffffffff8144dbf3>] vt_waitactive+0x73/0xd0
[ 776.607731] Call Trace:
[ 776.607780] [<ffffffff8172a999>] schedule+0x29/0x70
[ 776.608197] [<ffffffff81213500>] do_vfs_ioctl+0x2e0/0x4a0
[ 776.611445] Call Trace:
[ 776.611495] [<ffffffff8172a999>] schedule+0x29/0x70
[ 776.611547] [<ffffffff8144d9ca>] __vt_event_wait.isra.0.part.1+0x4a/0x90
[ 776.618210] [<ffffffff8172ec69>] system_call_fastpath+0x16/0x1b
[ 776.618258] console-kit-dae S ffff88011fc13840 [ 776.622996] Call Trace:
0000000000013840
[ 776.626368] ffff8800a0f4ffd8 0000000000013840[ 776.628310] [<ffffffff8144ea82>] vt_ioctl+0xcc2/0x1190
[ 776.628361] [<ffffffff81442939>] tty_ioctl+0x289/0xb70
[ 776.628412] [<ffffffff81213500>] do_vfs_ioctl+0x2e0/0x4a0
0 3083 1 0x00000080
[ 776.631552] ffff8800a0f5fce8[ 776.634755] [<ffffffff810d1840>] ? prepare_to_wait_event+0xf0/0xf0
[ 776.636781] [<ffffffff8112f426>] ? __audit_syscall_exit+0x1f6/0x2a0
ffff8800a0f45850 0000000000013840
[ 776.637162] ffff880117623fd8 0000000000013840 ffff8800a0e91d70 0000000000005607 ffff8800bb61e400
[ 776.637506] Call Trace:
[ 776.637559] [<ffffffff8172a999>] schedule+0x29/0x70
[ 776.643543] [<ffffffff8172a999>] schedule+0x29/0x70
[ 776.645146] [<ffffffff812fd6e8>] ? security_capable+0x18/0x20
[ 776.645351] [<ffffffff81442939>] tty_ioctl+0x289/0xb70
[ 776.652134] [<ffffffff8172a999>] schedule+0x29/0x70
[ 776.655878] ffff88011fc13040 ffff88011fc13040 ffff88011fc13058[ 776.658892] ffff88009e37ffd8[ 776.660297] [<ffffffff811bd102>] ? unmap_region+0xe2/0x130
[ 776.662319] [<ffffffff81248b17>] ? eventfd_ctx_read+0x67/0x1c0
[ 776.663250] Call Trace:
0000000000013840[ 776.665037] [<ffffffff8172a999>] schedule+0x29/0x70
[ 776.669562]
[ 776.669562] runnable tasks:
[ 776.669562] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 776.669562] ----------------------------------------------------------------------------------------------------------
0 /
[ 776.675491] scsi_tmf_2 63 411.710124 2 100
[ 776.677951] [ 776.679691]
[ 776.679917] vdev_file_taskq 596 1560.935306 2 100 [ 776.681481] zvol/5 607 1599.043716 2 139 [ 776.681703] zvol/6 608 1605.240400 2 139
[ 776.684825] zvol/20 622 1692.019678 2 139
[ 776.686385] zvol/27 629 1735.376433 2 139 kdmflush 698 2656.114059 2 100 0 0 0.000000 0.000000 0.000000 0 0 0.000000 0.000000 0.000000 0 /
[ 776.696387]
[ 776.697054] btrfs-endio-met 995 5566.952293 2 100 [ 776.700175] z_rd_iss/2 1128 9535.798513 3 139 /
[ 776.703741] 0 0 0.000000 0.000000 0.000000
[ 776.709715] z_fr_iss_6/2 1188 33617.957885 3 139 z_fr_iss_7/2 1192 33617.935220 4 139 0 /autogroup-68
[ 776.713288] 0 /autogroup-71[ 776.715081] rpcbind 1295 34.401264 57 120 [ 776.716878] polkitd 1344 149.172767 568 120 0 /autogroup-98
[ 776.720396] httpd 1362 186.579264 1102 120 mysqld 2688 497.443934 1 120 0 0 0.000000 0.000000 0.000000
[ 776.726918] gmain 2872 67.466552 1 120
/autogroup-127
[ 776.731855] 0 /autogroup-127
[ 776.733650] gdbus 3019 106.568720 40 120 0 0 0.000000 0.000000 0.000000console-kit-dae 3051 33.314126 11 120 0 0 0.000000 0.000000 0.000000 0 0 0.000000 0.000000 0.000000
[ 776.745255] console-kit-dae 3083 33.314366 10 120 0 /autogroup-133
[ 776.745925] console-kit-dae 3086 33.314124 11 120 [ 776.748781] zil_clean/0 3295 33616.169566 2 100 0 0 0.000000 0.000000 0.000000 0 0 0.000000 0.000000 0.000000 0 /
[ 776.752241] 0 0 0.000000 0.000000 0.000000 /
[ 776.758490] scsi_tmf_7 293 1517.669573 2 100
[ 776.761352] scsi_eh_13 306 1573.645688 2 120 0 /
[ 776.762916] spl_system_task 594 2539.939780 2 100 arc_adapt 595 41610.717912 760 100 0 0 0.000000 0.000000 0.000000
[ 776.768728] z_wr_int/11 919 15668.897193 35 139 /
[ 776.772072]
[ 776.774040] z_fr_iss_5/3 952 10662.344863 3 139 0 0 0.000000 0.000000 0.000000 0 0 0.000000 0.000000 0.000000
[ 776.782310] z_fr_iss_7/3 1193 39584.180464 4 139 0 0 0.000000 0.000000 0.000000 0 0 0.000000 0.000000 0.000000 0 0 0.000000 0.000000 0.000000 /autogroup-119
[ 776.795954] gnome-settings- 2880 9626.794542 914 120 /autogroup-118
[ 776.796179] 0 0 0.000000 0.000000 0.000000
[ 776.800899]
[ 913.811006] ------------[ cut here ]------------
[ 913.811065] WARNING: CPU: 1 PID: 3390 at kernel/watchdog.c:294 watchdog_overflow_callback+0x9c/0xd0()
[ 913.811120] Watchdog detected hard LOCKUP on cpu 1
[ 913.811147] Modules linked in: netconsole ip6table_filter ip6_tables ebtable_nat ebtables 8021q libfcoe garp stp llc mrp libfc scsi_transport_fc cfg80211 rfkill snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer btrfs snd soundcore xor joydev kvm amd64_edac_mod microcode edac_core edac_mce_amd sp5100_tco serio_raw raid6_pq k10temp shpchp asus_atk0110 tpm_infineon tpm nfsd acpi_cpufreq auth_rpcgss nfs_acl lockd sunrpc zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) ata_generic pata_acpi sata_promise usb_storage r8169 mii pata_via [last unloaded: wmi]
[ 913.812972] CPU: 1 PID: 3390 Comm: mount.zfs Tainted: P OE 3.17.7-200.fc20.x86_64 #1
[ 913.813025] Hardware name: System manufacturer System Product Name/M4A88TD-M EVO, BIOS 1801 08/09/2012
[ 913.813077] 0000000000000000 000000005c55134a ffff88011fc45c10 ffffffff8172758c
[ 913.813224] ffff88011fc45c58 ffff88011fc45c48 ffffffff81092e7d ffff880119f78000
[ 913.813370] 0000000000000000 ffff88011fc45d68 0000000000000000 ffff88011fc45ef8
[ 913.813518] Call Trace:
[ 913.813557] <NMI> [<ffffffff8172758c>] dump_stack+0x45/0x56
[ 913.813650] [<ffffffff81092e7d>] warn_slowpath_common+0x7d/0xa0
[ 913.813691] [<ffffffff81092efc>] warn_slowpath_fmt+0x5c/0x80
[ 913.813732] [<ffffffff811418e0>] ? restart_watchdog_hrtimer+0x50/0x50
[ 913.813774] [<ffffffff8114197c>] watchdog_overflow_callback+0x9c/0xd0
[ 913.813816] [<ffffffff8117f82d>] __perf_event_overflow+0x9d/0x250
[ 913.813922] [<ffffffff81180344>] perf_event_overflow+0x14/0x20
[ 913.813965] [<ffffffff8102bc24>] x86_pmu_handle_irq+0x154/0x1b0
[ 913.814007] [<ffffffff8102a2db>] perf_event_nmi_handler+0x2b/0x50
[ 913.814049] [<ffffffff81017e90>] nmi_handle+0x90/0x130
[ 913.814090] [<ffffffff8101849e>] default_do_nmi+0xce/0x130
[ 913.814131] [<ffffffff81018588>] do_nmi+0x88/0xc0
[ 913.814173] [<ffffffff81730f21>] end_repeat_nmi+0x1e/0x2e
[ 913.814216] [<ffffffff8172e481>] ? _raw_spin_lock_irq+0x31/0x40
[ 913.814257] [<ffffffff8172e481>] ? _raw_spin_lock_irq+0x31/0x40
[ 913.814299] [<ffffffff8172e481>] ? _raw_spin_lock_irq+0x31/0x40
[ 913.814340] <<EOE>> [<ffffffff8172d6f8>] rwsem_down_read_failed+0x58/0x140
[ 913.814467] [<ffffffffa01152e0>] ? dnode_rele+0x50/0x90 [zfs]
[ 913.814510] [<ffffffff8137d504>] call_rwsem_down_read_failed+0x14/0x30
[ 913.814552] [<ffffffff8172ce90>] ? down_read+0x20/0x30
[ 913.814629] [<ffffffffa01618e8>] zap_get_leaf_byblk+0xe8/0x2e0 [zfs]
[ 913.814692] [<ffffffffa00f8ff7>] ? dmu_buf_update_user+0x57/0xb0 [zfs]
[ 913.814768] [<ffffffffa0165ee0>] ? zap_match.part.5+0x80/0x80 [zfs]
[ 913.814844] [<ffffffffa0161b53>] zap_deref_leaf+0x73/0xa0 [zfs]
[ 913.814920] [<ffffffffa0163854>] fzap_cursor_retrieve+0x114/0x2b0 [zfs]
[ 913.814995] [<ffffffffa0166fa8>] zap_cursor_retrieve+0x68/0x310 [zfs]
[ 913.815045] [<ffffffffa0053299>] ? spl_kmem_cache_alloc+0x109/0x740 [spl]
[ 913.815087] [<ffffffff811e2b62>] ? kmem_cache_alloc+0x1c2/0x1f0
[ 913.815129] [<ffffffff8130237a>] ? selinux_inode_alloc_security+0x3a/0xa0
[ 913.815206] [<ffffffffa0171053>] zfs_purgedir+0x93/0x220 [zfs]
[ 913.815284] [<ffffffffa0190479>] ? zfs_inode_update+0x149/0x1b0 [zfs]
[ 913.815328] [<ffffffff810d1415>] ? wake_up_bit+0x25/0x30
[ 913.815369] [<ffffffff8121a513>] ? unlock_new_inode+0x43/0x70
[ 913.815446] [<ffffffffa01908e3>] ? zfs_znode_alloc+0x403/0x590 [zfs]
[ 913.815522] [<ffffffffa0171432>] zfs_rmnode+0x252/0x330 [zfs]
[ 913.815599] [<ffffffffa019377a>] zfs_zinactive+0x9a/0x150 [zfs]
[ 913.815676] [<ffffffffa018cf64>] zfs_inactive+0x64/0x210 [zfs]
[ 913.815754] [<ffffffffa01a3dc4>] zpl_evict_inode+0x24/0x30 [zfs]
[ 913.815795] [<ffffffff8121b353>] evict+0xa3/0x170
[ 913.815835] [<ffffffff8121bb85>] iput+0xf5/0x180
[ 913.815912] [<ffffffffa01705b5>] zfs_unlinked_drain+0xb5/0xf0 [zfs]
[ 913.815954] [<ffffffff810d1686>] ? finish_wait+0x56/0x70
[ 913.816001] [<ffffffffa0055441>] ? taskq_create+0x311/0x410 [spl]
[ 913.816042] [<ffffffff810d1840>] ? prepare_to_wait_event+0xf0/0xf0
[ 913.816119] [<ffffffffa018e7f0>] ? zfs_get_done+0x70/0x70 [zfs]
[ 913.816195] [<ffffffffa0195892>] ? zil_open+0x42/0x60 [zfs]
[ 913.816271] [<ffffffffa01849a8>] zfs_sb_setup+0x108/0x140 [zfs]
[ 913.816347] [<ffffffffa0186103>] zfs_domount+0x293/0x300 [zfs]
[ 913.816389] [<ffffffff81202850>] ? get_anon_bdev+0x100/0x100
[ 913.816464] [<ffffffffa01a3ef0>] ? zpl_mount+0x50/0x50 [zfs]
[ 913.816540] [<ffffffffa01a3efe>] zpl_fill_super+0xe/0x20 [zfs]
[ 913.816582] [<ffffffff8120377d>] mount_nodev+0x4d/0xb0
[ 913.816721] [<ffffffffa01a3ed4>] zpl_mount+0x34/0x50 [zfs]
[ 913.816763] [<ffffffff812042a9>] mount_fs+0x39/0x1b0
[ 913.816804] [<ffffffff8121eec3>] vfs_kern_mount+0x63/0x100
[ 913.816846] [<ffffffff812218bc>] do_mount+0x25c/0xb50
[ 913.816888] [<ffffffff8118e284>] ? __get_free_pages+0x14/0x50
[ 913.816929] [<ffffffff812224e6>] SyS_mount+0x96/0xf0
[ 913.816970] [<ffffffff8172ec69>] system_call_fastpath+0x16/0x1b
[ 913.817010] ---[ end trace 8f069cf5895d3dd0 ]---
[ 927.990519] INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 0, t=60004 jiffies, g=7469, c=7468, q=0)
[ 927.990724] Task dump for CPU 1:
[ 927.990775] mount.zfs R running task 0 3390 3389 0x00000088
[ 927.990885] ffffffffa00f8ff7 ffff8800a649c1d8 ffffffffa0165ee0 ffff88009e8bca68
[ 927.991060] 000000005c55134a ffff8800a649c180 0000000000000000 0000000000000002
[ 927.991233] ffff88011771b880 ffff88009e30f738 ffff88011771b748 ffffffffa0161b53
[ 927.991406] Call Trace:
[ 927.991538] [<ffffffffa00f8ff7>] ? dmu_buf_update_user+0x57/0xb0 [zfs]
[ 927.991676] [<ffffffffa0165ee0>] ? zap_match.part.5+0x80/0x80 [zfs]
[ 927.991808] [<ffffffffa0161b53>] ? zap_deref_leaf+0x73/0xa0 [zfs]
[ 927.991938] [<ffffffffa0163854>] ? fzap_cursor_retrieve+0x114/0x2b0 [zfs]
[ 927.992070] [<ffffffffa0166fa8>] ? zap_cursor_retrieve+0x68/0x310 [zfs]
[ 927.992146] [<ffffffffa0053299>] ? spl_kmem_cache_alloc+0x109/0x740 [spl]
[ 927.992202] [<ffffffff811e2b62>] ? kmem_cache_alloc+0x1c2/0x1f0
[ 927.992257] [<ffffffff8130237a>] ? selinux_inode_alloc_security+0x3a/0xa0
[ 927.992387] [<ffffffffa0171053>] ? zfs_purgedir+0x93/0x220 [zfs]
[ 927.992521] [<ffffffffa0190479>] ? zfs_inode_update+0x149/0x1b0 [zfs]
[ 927.992577] [<ffffffff810d1415>] ? wake_up_bit+0x25/0x30
[ 927.992629] [<ffffffff8121a513>] ? unlock_new_inode+0x43/0x70
[ 927.992760] [<ffffffffa01908e3>] ? zfs_znode_alloc+0x403/0x590 [zfs]
[ 927.992891] [<ffffffffa0171432>] ? zfs_rmnode+0x252/0x330 [zfs]
[ 927.993109] [<ffffffffa019377a>] ? zfs_zinactive+0x9a/0x150 [zfs]
[ 927.993242] [<ffffffffa018cf64>] ? zfs_inactive+0x64/0x210 [zfs]
[ 927.993377] [<ffffffffa01a3dc4>] ? zpl_evict_inode+0x24/0x30 [zfs]
[ 927.993430] [<ffffffff8121b353>] ? evict+0xa3/0x170
[ 927.993481] [<ffffffff8121bb85>] ? iput+0xf5/0x180
[ 927.993610] [<ffffffffa01705b5>] ? zfs_unlinked_drain+0xb5/0xf0 [zfs]
[ 927.993665] [<ffffffff810d1686>] ? finish_wait+0x56/0x70
[ 927.993727] [<ffffffffa0055441>] ? taskq_create+0x311/0x410 [spl]
[ 927.993781] [<ffffffff810d1840>] ? prepare_to_wait_event+0xf0/0xf0
[ 927.993911] [<ffffffffa018e7f0>] ? zfs_get_done+0x70/0x70 [zfs]
[ 927.994042] [<ffffffffa0195892>] ? zil_open+0x42/0x60 [zfs]
[ 927.994173] [<ffffffffa01849a8>] ? zfs_sb_setup+0x108/0x140 [zfs]
[ 927.994304] [<ffffffffa0186103>] ? zfs_domount+0x293/0x300 [zfs]
[ 927.994358] [<ffffffff81202850>] ? get_anon_bdev+0x100/0x100
[ 927.994489] [<ffffffffa01a3ef0>] ? zpl_mount+0x50/0x50 [zfs]
[ 927.994621] [<ffffffffa01a3efe>] ? zpl_fill_super+0xe/0x20 [zfs]
[ 927.994675] [<ffffffff8120377d>] ? mount_nodev+0x4d/0xb0
[ 927.994806] [<ffffffffa01a3ed4>] ? zpl_mount+0x34/0x50 [zfs]
[ 927.994859] [<ffffffff812042a9>] ? mount_fs+0x39/0x1b0
[ 927.994914] [<ffffffff8121eec3>] ? vfs_kern_mount+0x63/0x100
[ 927.994967] [<ffffffff812218bc>] ? do_mount+0x25c/0xb50
[ 927.995021] [<ffffffff8118e284>] ? __get_free_pages+0x14/0x50
[ 927.995076] [<ffffffff812224e6>] ? SyS_mount+0x96/0xf0
[ 927.995129] [<ffffffff8172ec69>] ? system_call_fastpath+0x16/0x1b
[ 1108.095586] INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 0, t=240011 jiffies, g=7469, c=7468, q=0)
[ 1108.095791] Task dump for CPU 1:
[ 1108.095841] mount.zfs R running task 0 3390 3389 0x00000088
[ 1108.095947] ffffffffa00f8ff7 ffff8800a649c1d8 ffffffffa0165ee0 ffff88009e8bca68
[ 1108.096120] 000000005c55134a ffff8800a649c180 0000000000000000 0000000000000002
[ 1108.096299] ffff88011771b880 ffff88009e30f738 ffff88011771b748 ffffffffa0161b53
[ 1108.096559] Call Trace:
[ 1108.096681] [<ffffffffa00f8ff7>] ? dmu_buf_update_user+0x57/0xb0 [zfs]
[ 1108.096818] [<ffffffffa0165ee0>] ? zap_match.part.5+0x80/0x80 [zfs]
[ 1108.096948] [<ffffffffa0161b53>] ? zap_deref_leaf+0x73/0xa0 [zfs]
[ 1108.097077] [<ffffffffa0163854>] ? fzap_cursor_retrieve+0x114/0x2b0 [zfs]
[ 1108.097208] [<ffffffffa0166fa8>] ? zap_cursor_retrieve+0x68/0x310 [zfs]
[ 1108.097272] [<ffffffffa0053299>] ? spl_kmem_cache_alloc+0x109/0x740 [spl]
[ 1108.097326] [<ffffffff811e2b62>] ? kmem_cache_alloc+0x1c2/0x1f0
[ 1108.097380] [<ffffffff8130237a>] ? selinux_inode_alloc_security+0x3a/0xa0
[ 1108.097510] [<ffffffffa0171053>] ? zfs_purgedir+0x93/0x220 [zfs]
[ 1108.097643] [<ffffffffa0190479>] ? zfs_inode_update+0x149/0x1b0 [zfs]
[ 1108.097698] [<ffffffff810d1415>] ? wake_up_bit+0x25/0x30
[ 1108.097749] [<ffffffff8121a513>] ? unlock_new_inode+0x43/0x70
[ 1108.097879] [<ffffffffa01908e3>] ? zfs_znode_alloc+0x403/0x590 [zfs]
[ 1108.098010] [<ffffffffa0171432>] ? zfs_rmnode+0x252/0x330 [zfs]
[ 1108.098143] [<ffffffffa019377a>] ? zfs_zinactive+0x9a/0x150 [zfs]
[ 1108.098276] [<ffffffffa018cf64>] ? zfs_inactive+0x64/0x210 [zfs]
[ 1108.098411] [<ffffffffa01a3dc4>] ? zpl_evict_inode+0x24/0x30 [zfs]
[ 1108.098463] [<ffffffff8121b353>] ? evict+0xa3/0x170
[ 1108.098513] [<ffffffff8121bb85>] ? iput+0xf5/0x180
[ 1108.098641] [<ffffffffa01705b5>] ? zfs_unlinked_drain+0xb5/0xf0 [zfs]
[ 1108.098695] [<ffffffff810d1686>] ? finish_wait+0x56/0x70
[ 1108.098758] [<ffffffffa0055441>] ? taskq_create+0x311/0x410 [spl]
[ 1108.098811] [<ffffffff810d1840>] ? prepare_to_wait_event+0xf0/0xf0
[ 1108.098942] [<ffffffffa018e7f0>] ? zfs_get_done+0x70/0x70 [zfs]
[ 1108.099073] [<ffffffffa0195892>] ? zil_open+0x42/0x60 [zfs]
[ 1108.099204] [<ffffffffa01849a8>] ? zfs_sb_setup+0x108/0x140 [zfs]
[ 1108.099335] [<ffffffffa0186103>] ? zfs_domount+0x293/0x300 [zfs]
[ 1108.099388] [<ffffffff81202850>] ? get_anon_bdev+0x100/0x100
[ 1108.099518] [<ffffffffa01a3ef0>] ? zpl_mount+0x50/0x50 [zfs]
[ 1108.099648] [<ffffffffa01a3efe>] ? zpl_fill_super+0xe/0x20 [zfs]
[ 1108.099701] [<ffffffff8120377d>] ? mount_nodev+0x4d/0xb0
[ 1108.099832] [<ffffffffa01a3ed4>] ? zpl_mount+0x34/0x50 [zfs]
[ 1108.099885] [<ffffffff812042a9>] ? mount_fs+0x39/0x1b0
[ 1108.099939] [<ffffffff8121eec3>] ? vfs_kern_mount+0x63/0x100
[ 1108.099992] [<ffffffff812218bc>] ? do_mount+0x25c/0xb50
[ 1108.100455] [<ffffffff8118e284>] ? __get_free_pages+0x14/0x50
[ 1108.100509] [<ffffffff812224e6>] ? SyS_mount+0x96/0xf0
[ 1108.100562] [<ffffffff8172ec69>] ? system_call_fastpath+0x16/0x1b
[ 1288.199652] INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 0, t=420017 jiffies, g=7469, c=7468, q=0)
[ 1288.199856] Task dump for CPU 1:
[ 1288.199905] mount.zfs R running task 0 3390 3389 0x00000088
[ 1288.200128] ffffffffa00f8ff7 ffff8800a649c1d8 ffffffffa0165ee0 ffff88009e8bca68
[ 1288.200299] 000000005c55134a ffff8800a649c180 0000000000000000 0000000000000002
[ 1288.200470] ffff88011771b880 ffff88009e30f738 ffff88011771b748 ffffffffa0161b53
[ 1288.200642] Call Trace:
[ 1288.200763] [<ffffffffa00f8ff7>] ? dmu_buf_update_user+0x57/0xb0 [zfs]
[ 1288.200899] [<ffffffffa0165ee0>] ? zap_match.part.5+0x80/0x80 [zfs]
[ 1288.201029] [<ffffffffa0161b53>] ? zap_deref_leaf+0x73/0xa0 [zfs]
[ 1288.201158] [<ffffffffa0163854>] ? fzap_cursor_retrieve+0x114/0x2b0 [zfs]
[ 1288.201289] [<ffffffffa0166fa8>] ? zap_cursor_retrieve+0x68/0x310 [zfs]
[ 1288.201354] [<ffffffffa0053299>] ? spl_kmem_cache_alloc+0x109/0x740 [spl]
[ 1288.201409] [<ffffffff811e2b62>] ? kmem_cache_alloc+0x1c2/0x1f0
[ 1288.201461] [<ffffffff8130237a>] ? selinux_inode_alloc_security+0x3a/0xa0
[ 1288.201590] [<ffffffffa0171053>] ? zfs_purgedir+0x93/0x220 [zfs]
[ 1288.201723] [<ffffffffa0190479>] ? zfs_inode_update+0x149/0x1b0 [zfs]
[ 1288.201777] [<ffffffff810d1415>] ? wake_up_bit+0x25/0x30
[ 1288.201827] [<ffffffff8121a513>] ? unlock_new_inode+0x43/0x70
[ 1288.201958] [<ffffffffa01908e3>] ? zfs_znode_alloc+0x403/0x590 [zfs]
[ 1288.202089] [<ffffffffa0171432>] ? zfs_rmnode+0x252/0x330 [zfs]
[ 1288.202221] [<ffffffffa019377a>] ? zfs_zinactive+0x9a/0x150 [zfs]
[ 1288.202354] [<ffffffffa018cf64>] ? zfs_inactive+0x64/0x210 [zfs]
[ 1288.202488] [<ffffffffa01a3dc4>] ? zpl_evict_inode+0x24/0x30 [zfs]
[ 1288.202540] [<ffffffff8121b353>] ? evict+0xa3/0x170
[ 1288.202590] [<ffffffff8121bb85>] ? iput+0xf5/0x180
[ 1288.202718] [<ffffffffa01705b5>] ? zfs_unlinked_drain+0xb5/0xf0 [zfs]
[ 1288.202772] [<ffffffff810d1686>] ? finish_wait+0x56/0x70
[ 1288.202835] [<ffffffffa0055441>] ? taskq_create+0x311/0x410 [spl]
[ 1288.202889] [<ffffffff810d1840>] ? prepare_to_wait_event+0xf0/0xf0
[ 1288.203019] [<ffffffffa018e7f0>] ? zfs_get_done+0x70/0x70 [zfs]
[ 1288.203150] [<ffffffffa0195892>] ? zil_open+0x42/0x60 [zfs]
[ 1288.203281] [<ffffffffa01849a8>] ? zfs_sb_setup+0x108/0x140 [zfs]
[ 1288.203413] [<ffffffffa0186103>] ? zfs_domount+0x293/0x300 [zfs]
[ 1288.203467] [<ffffffff81202850>] ? get_anon_bdev+0x100/0x100
[ 1288.203597] [<ffffffffa01a3ef0>] ? zpl_mount+0x50/0x50 [zfs]
[ 1288.203804] [<ffffffffa01a3efe>] ? zpl_fill_super+0xe/0x20 [zfs]
[ 1288.203857] [<ffffffff8120377d>] ? mount_nodev+0x4d/0xb0
[ 1288.203988] [<ffffffffa01a3ed4>] ? zpl_mount+0x34/0x50 [zfs]
[ 1288.204041] [<ffffffff812042a9>] ? mount_fs+0x39/0x1b0
[ 1288.204096] [<ffffffff8121eec3>] ? vfs_kern_mount+0x63/0x100
[ 1288.204148] [<ffffffff812218bc>] ? do_mount+0x25c/0xb50
[ 1288.204202] [<ffffffff8118e284>] ? __get_free_pages+0x14/0x50
[ 1288.204256] [<ffffffff812224e6>] ? SyS_mount+0x96/0xf0
[ 1288.204307] [<ffffffff8172ec69>] ? system_call_fastpath+0x16/0x1b
[ 1447.036643] SysRq : Show State
[ 1447.036741] task PC stack pid father
[ 1447.036802] systemd S ffff88011fc13840 0 1 0 0x00000000
[ 1447.036909] ffff880119da7dc8 0000000000000086 ffff880119e40000 0000000000013840
[ 1447.037082] ffff880119da7fd8 0000000000013840 ffff8800cab49d70 ffff880119e40000
[ 1447.037253] 0000000000000000 0000000000000000 ffff880036635120 ffff880119e40000
[ 1447.037425] Call Trace:
[ 1447.037483] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.037536] [<ffffffff8172ddfd>] schedule_hrtimeout_range_clock+0x14d/0x170
[ 1447.037592] [<ffffffff812449d9>] ? ep_scan_ready_list.isra.9+0x1b9/0x1f0
[ 1447.037644] [<ffffffff8172de33>] schedule_hrtimeout_range+0x13/0x20
[ 1447.037695] [<ffffffff81244c6e>] ep_poll+0x23e/0x360
[ 1447.037746] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
[ 1447.037799] [<ffffffff812460f5>] SyS_epoll_wait+0xb5/0xe0
[ 1447.037851] [<ffffffff8172ec69>] system_call_fastpath+0x16/0x1b
[ 1447.037899] kthreadd S ffff88011fc13840 0 2 0 0x00000000
[ 1447.038003] ffff880119e4fea8 0000000000000046 ffff880119e409d0 0000000000013840
[ 1447.038274] ffff880119e4ffd8 0000000000013840 ffff8800c8bc3ae0 ffff880119e409d0
[ 1447.038444] ffff8800a0cd6940 ffff880119e409d0 0000000000000000 ffff880119e409d0
[ 1447.038615] Call Trace:
[ 1447.038665] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.038717] [<ffffffff810b0ea5>] kthreadd+0x235/0x240
[ 1447.038769] [<ffffffff810b0c70>] ? kthread_create_on_cpu+0x60/0x60
[ 1447.038820] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 1447.038872] [<ffffffff810b0c70>] ? kthread_create_on_cpu+0x60/0x60
[ 1447.038922] ksoftirqd/0 S ffff88011fc13840 0 3 2 0x00000000
[ 1447.039026] ffff880119e63e68 0000000000000046 ffff880119e413a0 0000000000013840
[ 1447.039197] ffff880119e63fd8 0000000000013840 ffffffff81c1b460 ffff880119e413a0
[ 1447.039368] ffff880119e413a0 ffff88011b402490 ffffffff81c41a40 ffff880119e413a0
[ 1447.039537] Call Trace:
[ 1447.039587] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.039638] [<ffffffff810b3fa3>] smpboot_thread_fn+0xd3/0x1a0
[ 1447.039689] [<ffffffff8172a999>] ? schedule+0x29/0x70
[ 1447.039740] [<ffffffff810b3ed0>] ? SyS_setgroups+0x1a0/0x1a0
[ 1447.039791] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 1447.039843] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.039895] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 1447.039947] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.039996] kworker/0:0H S ffff88011fc13840 0 5 2 0x00000000
[ 1447.040109] ffff880119e73e48 0000000000000046 ffff880119e42740 0000000000013840
[ 1447.040355] ffff880119e73fd8 0000000000013840 ffff880036abbae0 ffff880119e42740
[ 1447.040524] ffff88011fc13440 ffff88011fc13440 ffff88011fc13458 ffff880119e42740
[ 1447.040694] Call Trace:
[ 1447.040744] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.040794] [<ffffffff810ab4bc>] worker_thread+0x18c/0x490
[ 1447.040845] [<ffffffff810ab330>] ? rescuer_thread+0x2e0/0x2e0
[ 1447.040896] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 1447.040947] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.040998] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 1447.041049] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.041098] rcu_sched S ffff88011fc13840 0 7 2 0x00000000
[ 1447.041202] ffff880119e7bd50 0000000000000046 ffff880119e43ae0 0000000000013840
[ 1447.041371] ffff880119e7bfd8 0000000000013840 ffffffff81c1b460 ffff880119e43ae0
[ 1447.041542] ffffffff81f38fc0 ffff880119e7bd80 0000000100117875 ffffffff81f38fc0
[ 1447.041712] Call Trace:
[ 1447.041762] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.041813] [<ffffffff8172d962>] schedule_timeout+0x162/0x2a0
[ 1447.041866] [<ffffffff810f3420>] ? internal_add_timer+0xb0/0xb0
[ 1447.041919] [<ffffffff810d17da>] ? prepare_to_wait_event+0x8a/0xf0
[ 1447.041973] [<ffffffff810ef58b>] rcu_gp_kthread+0x2cb/0x750
[ 1447.042025] [<ffffffff810d1840>] ? prepare_to_wait_event+0xf0/0xf0
[ 1447.042151] [<ffffffff810ef2c0>] ? rcu_cpu_notify+0x440/0x440
[ 1447.042203] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 1447.042255] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.042307] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 1447.042359] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.042409] rcuos/0 S ffff88011fc13840 0 8 2 0x00000000
[ 1447.042513] ffff880119e7fe20 0000000000000046 ffff880119e444b0 0000000000013840
[ 1447.042683] ffff880119e7ffd8 0000000000013840 ffff8800cab49d70 ffff880119e444b0
[ 1447.042854] ffffffff81c56400 ffff88011fc0d8c0 0000000000000202 ffff880119e444b0
[ 1447.043024] Call Trace:
[ 1447.043074] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.043125] [<ffffffff810ee08e>] rcu_nocb_kthread+0x35e/0x520
[ 1447.043176] [<ffffffff810d1840>] ? prepare_to_wait_event+0xf0/0xf0
[ 1447.043228] [<ffffffff810edd30>] ? rcu_implicit_dynticks_qs+0x180/0x180
[ 1447.043279] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 1447.043331] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.043383] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 1447.043435] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.043484] rcuos/1 S ffff88011fc53840 0 9 2 0x00000000
[ 1447.043587] ffff880119e83e20 0000000000000046 ffff880119e44e80 0000000000013840
[ 1447.043756] ffff880119e83fd8 0000000000013840 ffff880119e9d850 ffff880119e44e80
[ 1447.043925] ffff88011fc4da50 ffff88011fc4d8c0 000000000000000d ffff8800a6032230
[ 1447.044165] Call Trace:
[ 1447.044215] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.044266] [<ffffffff810edefa>] rcu_nocb_kthread+0x1ca/0x520
[ 1447.044317] [<ffffffff810d1840>] ? prepare_to_wait_event+0xf0/0xf0
[ 1447.044369] [<ffffffff810edd30>] ? rcu_implicit_dynticks_qs+0x180/0x180
[ 1447.044421] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 1447.044472] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.044524] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 1447.044575] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.044623] rcuos/2 S ffff88011fc13840 0 10 2 0x00000000
[ 1447.044727] ffff880119e87e20 0000000000000046 ffff880119e45850 0000000000013840
[ 1447.044897] ffff880119e87fd8 0000000000013840 ffff880119e46bf0 ffff880119e45850
[ 1447.045067] ffff88011fc8da50 ffff88011fc8d8c0 ffffffff810edd30 ffff880119e45850
[ 1447.045236] Call Trace:
[ 1447.045286] [<ffffffff810edd30>] ? rcu_implicit_dynticks_qs+0x180/0x180
[ 1447.045339] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.045389] [<ffffffff810edf7a>] rcu_nocb_kthread+0x24a/0x520
[ 1447.045440] [<ffffffff8172a48b>] ? __schedule+0x2eb/0x7d0
[ 1447.045492] [<ffffffff810d1840>] ? prepare_to_wait_event+0xf0/0xf0
[ 1447.045544] [<ffffffff810edd30>] ? rcu_implicit_dynticks_qs+0x180/0x180
[ 1447.045596] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 1447.046055] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.046106] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 1447.046157] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.046205] rcuos/3 S ffff88011fc13840 0 11 2 0x00000000
[ 1447.046308] ffff880119e8be20 0000000000000046 ffff880119e46220 0000000000013840
[ 1447.046549] ffff880119e8bfd8 0000000000013840 ffff880119e46bf0 ffff880119e46220
[ 1447.046717] ffff88011fccda50 ffff88011fccd8c0 ffffffff810edd30 0000000000000000
[ 1447.046887] Call Trace:
[ 1447.046936] [<ffffffff810edd30>] ? rcu_implicit_dynticks_qs+0x180/0x180
[ 1447.046988] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.047038] [<ffffffff810edefa>] rcu_nocb_kthread+0x1ca/0x520
[ 1447.047091] [<ffffffff8172a48b>] ? __schedule+0x2eb/0x7d0
[ 1447.047142] [<ffffffff810d1840>] ? prepare_to_wait_event+0xf0/0xf0
[ 1447.047193] [<ffffffff810edd30>] ? rcu_implicit_dynticks_qs+0x180/0x180
[ 1447.047244] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 1447.047295] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.047347] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 1447.047399] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.047447] rcuos/4 S ffff88011fc13840 0 12 2 0x00000000
[ 1447.047549] ffff880119e8fe20 0000000000000046[ 1447.048108] [<ffffffff810edd30>] ? rcu_implicit_dynticks_qs+0x180/0x180
[ 1447.049815] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.059015] ffff880119e9c4b0[ 1447.060732] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.064332] ffff880119f73fd8 0000000000013840 0 28 2 0x00000000
[ 1447.065256] ffff880119f7fe28 0000000000000046 0000000000013840
[ 1447.077272] ffff8801199f7fd8 0000000000013840 ffff880119e9d850 0000000000013840 ffff880119e9d850 ffff880119f91d70
[ 1447.078577] 0000000000000000 0 40 2 0x00000000
[ 1447.079315] ffff880119a2fe28
[ 1447.079653] ffff88011980ecc0 ffff880119f92740 ffff880119f93ae0 0000000000013840
[ 1447.080621] ffff8800ca7effd8 0000000000000046 ffff8800c8bc13a0 0000000000013840 0000000000000046 ffff880119a06220 0000000000000246
[ 1447.091000] ffff8800368bbfd8[ 1447.091794] scsi_eh_3 S ffffffff81c1b460 ffff8800c8b9ec80
[ 1447.095752] Call Trace:
0000000000013840
0 77 2 0x00000000
[ 1447.100079] ffff8801190c3e28
[ 1447.102878] Call Trace:
ffff8800c8bc75c0
[ 1447.104869] ffff88011b409000[ 1447.105240] [<ffffffff810b04a8>] kthread+0xd8/0xf0
0000000000013840 ffff8800368d2740[ 1447.109710] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
0000000000013840 ffff8800368d5850 ffff8800365c4e80
[ 1447.110687] ffff880036ae56c0 ffff8800365c4e80 ffffffff810ab050 0 289 2 0x00000000
[ 1447.111416] ffff880119193e18[ 1447.117324] ffff8800365f7f00[ 1447.117595] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.129894] [<ffffffffa002d7d0>] ? usb_stor_disconnect+0xd0/0xd0 [usb_storage]
[ 1447.134677] ffff88011925be28[ 1447.136837] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.139264] ffff880036a99f00[ 1447.139483] [<ffffffff810ab050>] ? process_one_work+0x3d0/0x3d0
0000000000000046 ffff8800c9138000 0000000000013840
[ 1447.144417] ffff8800cb6fbfd8 0000000000013840
[ 1447.148236] ffff8800cb7c3fd8
[ 1447.150161] ffff8800367d5d20 ffff8800ca3bbb88 0000000000000046 ffff8800367ef5c0 0000000000013840
[ 1447.152234] ffff8800cb7e3fd8 ffff880119e9d850 ffff8800367ebae0
[ 1447.154350] ffff8800367d5a80 ffff880036b46d88 ffff8800367ebae0 ffff880036b46da8[ 1447.158085] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
ffff880036b46588 ffff8800c913c4b0 ffff880036b465a8
[ 1447.160171] Call Trace:
[ 1447.160280] [<ffffffffa0054b29>] taskq_thread+0x329/0x350 [spl]
[ 1447.164404] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.166965] ffff8800ca61ffd8 ffff880036b465a8[ 1447.170867] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
[ 1447.171285] ffff8800ca63fe10 ffff8800ca62bae0
[ 1447.175440] ffff8800ca2dbf60[ 1447.179333] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
0 623 2 0x00000000
[ 1447.179587] ffff8800ca65be10[ 1447.183325] ffff8800ca2b4180 ffff880036b46588[ 1447.183544] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.187528] [<ffffffffa0054800>] ? taskq_cancel_id+0x140/0x140 [spl]
[ 1447.187581] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 1447.187633] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
0000000000013840 ffff880036b46588 ffff8800ca6744b0[ 1447.195221] Call Trace:
[ 1447.195423] [<ffffffff810ab050>] ? process_one_work+0x3d0/0x3d0
ffff880036401a80
[ 1447.197440] Call Trace:
[ 1447.197591] [<ffffffff810ab271>] rescuer_thread+0x221/0x2e0
0 892 2 0x00000000
[ 1447.201773] ffff8800c2e3fe10 ffff8800cab4f5c0
[ 1447.205511] ffff8800c376ba80[ 1447.209452] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 1447.209826] ffff8800c2e5bfd8 0000000000013840 ffff8800cab4bae0[ 1447.213200] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.213251] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 1447.217028] ffff8800c2e73fd8 0000000000013840 ffff880119e9d850 ffff8800caaef5c0
[ 1447.217198] ffff8800c376bc00 ffff8800c9d71888[ 1447.220526] ffff8800c376bd20 ffff880035cf6a88 ffff8800c372ebf0 0000000000013840
[ 1447.222674] ffff8800c2e8ffd8 0000000000013840 ffff880119e9d850 ffff8800c3729d70
[ 1447.222844] ffff8800c9083ea0[ 1447.226242] ffff8800c2e9be10 0000000000000046 ffff8800c37293a0[ 1447.228226] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.228277] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
0000000000000046 ffff8800c2ea1d70 0000000000013840
[ 1447.232054] ffff8800c2eb7fd8
[ 1447.233551] Call Trace:
[ 1447.236063] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 1447.238084] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.240515] ffff8800c376bd80[ 1447.242262] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.244414] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.248074] ffff8800c2effe10 0000000000000046 ffff8800c2ea75c0 0000000000013840 ffff8800c2e7d850
[ 1447.251808] ffff8800c376b840 ffff8800c9d71788 ffff8800c2e7d850[ 1447.253651] [<ffffffffa0054b29>] taskq_thread+0x329/0x350 [spl]
[ 1447.253703] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
ffff88011fc13840 0 940 2 0x00000000
[ 1447.257694] ffff8800c2f27fd8 0000000000013840 ffff8800c2f15850 ffff8800c2f11d70 ffff8800c2f30000
[ 1447.261268] ffff8800ca2d92a0[ 1447.261599] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
[ 1447.265333] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
[ 1447.265670] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
ffff8800c2f33110 0000000000013840[ 1447.272355] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 1447.277621] Call Trace:
[ 1447.277671] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.277731] [<ffffffffa0054b29>] taskq_thread+0x329/0x350 [spl]
ffff8800c9d715a8
[ 1447.281358] Call Trace:
[ 1447.281407] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.281538] [<ffffffffa0054b29>] taskq_thread+0x329/0x350 [spl]
[ 1447.285125] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 1447.285177] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.288740] ffff880118a07e28[ 1447.290694] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.290745] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 1447.293524] ffff8800caafbe28[ 1447.296725] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.300797] [<ffffffff810ab050>] ? process_one_work+0x3d0/0x3d0
[ 1447.301312] ffff8800c2db7e28[ 1447.304718] [<ffffffff810ab271>] rescuer_thread+0x221/0x2e0
[ 1447.306880] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.306931] [<ffffffff810ab271>] rescuer_thread+0x221/0x2e0
[ 1447.310545] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.310594] btrfs-extent-re S[ 1447.311257] [<ffffffff810ab050>] ? process_one_work+0x3d0/0x3d0
[ 1447.312175] ffff8800c890e6c0 ffff8800ca2d8f88
[ 1447.318317] ffff8801196f4b08 ffff8801196f4a10 ffff8801196f4b10 0000000000000000
[ 1447.318486] Call Trace:
[ 1447.318536] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.322018] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.322066] bioset S 0000000000013840
[ 1447.325727] ffff8800bf5f3fd8 0000000000013840
[ 1447.325896] ffff8800ca2d9f60[ 1447.329629] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
[ 1447.329844] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 1447.333860] ffff8800bf9ffe10 0000000000000046[ 1447.334714] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 1447.338039] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 1447.340058] [<ffffffff8172a999>] schedule+0x29/0x70
ffff8800bd0409d0 ffff8800000124a8
[ 1447.342318] Call Trace:
ffff8800bf686bf0
[ 1447.345884] ffff880117c85f60 ffff8800365fd188 ffff8800bf686bf0 ffff8800365fd1a8
[ 1447.346053] Call Trace:
[ 1447.349452] Call Trace:
[ 1447.349502] [<ffffffff8172a999>] schedule+0x29/0x70
0000000000013840 ffff8800367175c0 ffff8800bd043110
[ 1447.351522] ffff88003657a780 ffff8800365fd188 ffff8800bd043110
[ 1447.355093] Call Trace:
[ 1447.355143] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.356970] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.357021] [<ffffffff8172ebbc>] ret_from_fork+0x7c/0xb0
[ 1447.358658] ffff88003657a300 0000000000013840
[ 1447.360804] ffff8800bd05bfd8 ffff8800365fdf88 ffff8800bf6513a0 ffff8800365fdfa8
[ 1447.364473] Call Trace:
ffff8800bf652740
[ 1447.366618] ffff880117c85ba0 ffff8800365fdf88 ffff8800bf652740 ffff8800365fdfa8
[ 1447.366788] Call Trace:
[ 1447.366837] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.371953] [<ffffffff810b04a8>] kthread+0xd8/0xf0
[ 1447.372006] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.372953] [<ffffffffa0054b29>] taskq_thread+0x329/0x350 [spl]
[ 1447.374200] [<ffffffff810b04a8>] kthread+0xd8/0xf0
0000000000000046 ffff8800bf656bf0 ffff8800be754e80 ffff8800000128a8 0 1175 2 0x00000000
0000000000013840
[ 1447.382973] ffff8800bd06bfd8[ 1447.383809] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.386714] Call Trace:
0 1180 2 0x00000000
ffff88011fc13840 ffff880117cf8000[ 1447.391434] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
[ 1447.394760] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
0 1187 2 0x00000000
[ 1447.396913] ffff8800be7dbfd8 ffff880117da3ae0 0000000000013840
[ 1447.399152] ffff8800be7ebfd8[ 1447.402729] ffff880117c86900 0000000000000046[ 1447.405540] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
[ 1447.412356] Call Trace:
[ 1447.412405] [<ffffffff8172a999>] schedule+0x29/0x70
ffff8800bf0c73e0 ffff8800bf0c70a0 ffff8800bf677e80
[ 1447.413512] Call Trace:
[ 1447.414246] ffff8800bd093fd8[ 1447.420046] Call Trace:
[ 1447.420198] [<ffffffff810ab271>] rescuer_thread+0x221/0x2e0
ffffc90010757280
[ 1447.422711] Call Trace:
[ 1447.426332] [<ffffffff810c865b>] ? put_prev_entity+0x5b/0x3b0
[ 1447.426383] [<ffffffff810c2b29>] ? pick_next_entity+0xa9/0x190
[ 1447.426539] [<ffffffff81012615>] ? __switch_to+0x165/0x590
[ 1447.426590] [<ffffffff8101cf09>] ? read_tsc+0x9/0x10
[ 1447.433656] [<ffffffff8172ddfd>] schedule_hrtimeout_range_clock+0x14d/0x170
[ 1447.433710] [<ffffffff810d12cc>] ? add_wait_queue+0x3c/0x50
[ 1447.441358] [<ffffffff815fd12c>] ? sock_sendmsg+0x9c/0xe0
[ 1447.443519] [<ffffffff8160cd05>] ? skb_free_datagram+0x15/0x40
[ 1447.447479] [<ffffffff811ffb2e>] new_sync_write+0x8e/0xd0
ffff8800cab493a0[ 1447.448710] [<ffffffff8172ec69>] system_call_fastpath+0x16/0x1b
[ 1447.451895] [<ffffffff81213f2f>] ? __pollwait+0x7f/0xf0
0000000000000000 ffff8800bb57fbbc ffff8800bb57fbbc
[ 1447.454162] Call Trace:
[ 1447.455948] [<ffffffff8172a999>] schedule+0x29/0x70
0000000000013840 ffff880119e9d850
[ 1447.458355] Call Trace:
[ 1447.458405] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.461863] [<ffffffff812158a4>] SyS_poll+0x74/0x110
[ 1447.463853] [<ffffffff810f5e80>] ? hrtimer_get_res+0x50/0x50
[ 1447.467569] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
[ 1447.467771] [<ffffffff8112f426>] ? __audit_syscall_exit+0x1f6/0x2a0
0000000000013840 ffff8800cab49d70 ffff8800bd0bc4b0
[ 1447.470487] 0000000000000000 0000000000013840
[ 1447.477323] ffff8800bb697fd8 0000000000013840[ 1447.478141] [<ffffffff810bd37b>] ? try_to_wake_up+0xeb/0x300
ffff8800b71c5850
[ 1447.479512] ffff8800b71c5850 ffff8800b71c5850 0000000000000000 ffff8800bd0b8000 ffff8800b71c6220
[ 1447.481264] ffff8800b71c6220[ 1447.481533] [<ffffffff81106f50>] futex_wait_queue_me+0xd0/0x130
[ 1447.486992] [<ffffffff8172a999>] schedule+0x29/0x70
ffff880036abebf0[ 1447.489011] [<ffffffff810ab271>] rescuer_thread+0x221/0x2e0
[ 1447.489421] ffff8800bb48ba48[ 1447.493239] [<ffffffff8105c9fc>] ? __do_page_fault+0x21c/0x540
[ 1447.493291] [<ffffffff81389aad>] ? list_del+0xd/0x30
ffff880117db93a0[ 1447.500670] [<ffffffff810d1600>] ? __wake_up_sync_key+0x50/0x60
0 1360 1 0x00000080
[ 1447.501342] ffff8800b7073a48
[ 1447.501755] 0000000000000000[ 1447.503165] [<ffffffff8112f426>] ? __audit_syscall_exit+0x1f6/0x2a0
[ 1447.503949] Call Trace:
[ 1447.505156] ffff8800b7344e80 ffff8800b7344e80[ 1447.510534] [<ffffffff81107256>] futex_wait+0x176/0x270
ffffc900107596c0 ffff880117d393a0[ 1447.514696] [<ffffffff81213fe9>] poll_schedule_timeout+0x49/0x70
0000000000000000 0000000000013840
[ 1447.518918] 0000000000000000 0000000000000040
[ 1447.520832] ffff880118609a80[ 1447.521001] Call Trace:
[ 1447.521408] [<ffffffff810b03d0>] ? kthread_create_on_node+0x190/0x190
ffff8800b7266bf0
[ 1447.525165] ffff8800b732bac8 0000000000989623 0000000000000000 0000000000000007[ 1447.526006] [<ffffffff8160157b>] ? sock_wfree+0x5b/0x70
[ 1447.530569] Call Trace:
[ 1447.532528] [<ffffffff811b6f02>] ? do_wp_page+0x3b2/0x7a0
[ 1447.533552] ffff8800b725bb88[ 1447.533976] [<ffffffff8172de33>] schedule_hrtimeout_range+0x13/0x20
[ 1447.538389] [<ffffffff81214190>] ? poll_select_copy_remaining+0x150/0x150
ffff8800b71c13a0 0000000000013840
[ 1447.542322] ffff8800b4433fd8 0000000000013840 ffff880119e9d850 ffff8800b71c13a0
[ 1447.542491] ffff8800b71c13a0 ffff8800b71c13a0[ 1447.547820] [<ffffffff8172ec69>] system_call_fastpath+0x16/0x1b
ffff880117d39d70[ 1447.551929] Call Trace:
[ 1447.552183] [<ffffffff810c70fa>] ? dequeue_entity+0x11a/0x550
[ 1447.552235] [<ffffffff810c8631>] ? put_prev_entity+0x31/0x3b0
[ 1447.552288] [<ffffffff811094ae>] do_futex+0x10e/0xa70
0 2743 2389 0x00000080
[ 1447.561450] [<ffffffff8112f426>] ? __audit_syscall_exit+0x1f6/0x2a0
[ 1447.563720] [<ffffffff81244c6e>] ep_poll+0x23e/0x360
[ 1447.565651] 7fffffffffffffff ffff8800b7262740 7fffffffffffffff ffff8800365bfe6c
[ 1447.566228] Call Trace:
[ 1447.569247] [<ffffffff813048ac>] ? selinux_file_alloc_security+0x3c/0x60
ffff880035d06220
[ 1447.573253] Call Trace:
[ 1447.573304] [<ffffffff8172a999>] schedule+0x29/0x70
ffff8800c9309d70 ffff8800c92efed0 ffff8800c930c4b0
[ 1447.580787] Call Trace:
[ 1447.581142] [<ffffffff810eebcd>] ? call_rcu_sched+0x1d/0x20
[ 1447.584376] [<ffffffff810c6ba7>] ? update_curr+0x67/0x160
[ 1447.588146] [<ffffffff81215682>] do_sys_poll+0x442/0x560
[ 1447.588198] [<ffffffff81192772>] ? __alloc_pages_nodemask+0x172/0xa40
ffff8800bf96bba0 ffff8800caae89d0[ 1447.592143] [<ffffffff812449d9>] ? ep_scan_ready_list.isra.9+0x1b9/0x1f0
[ 1447.595831] [<ffffffff8105c9fc>] ? __do_page_fault+0x21c/0x540
[ 1447.595883] [<ffffffff81248b17>] ? eventfd_ctx_read+0x67/0x1c0
0000000000000000[ 1447.601450] [<ffffffff811b9955>] ? handle_mm_fault+0xa15/0x10b0
[ 1447.603149] dbus-daemon S ffff88011fc53840 0000000000013840
[ 1447.603421] ffff8800a891bfd8 0000000000013840 ffff880119e9d850 ffff8800c93089d0 0000000000000086 ffff8800a8b13110 0000000000013840[ 1447.609037] ffff8800a8b07fd8 0000000000013840 ffff880119e9d850[ 1447.609375] Call Trace:
[ 1447.609425] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.609883] [<ffffffff81606984>] ? consume_skb+0x34/0x80
[ 1447.616438] [<ffffffff81725929>] ? __slab_free+0x10e/0x277
[ 1447.616488] [<ffffffff811988bc>] ? lru_cache_add+0x1c/0x50
[ 1447.623986] [<ffffffff810d12cc>] ? add_wait_queue+0x3c/0x50
[ 1447.625896] [<ffffffff810d12cc>] ? add_wait_queue+0x3c/0x50
[ 1447.629850] 0000000000000000 0000000000000000 0000000000013840
[ 1447.631450] ffff8800a650ffd8 0000000000013840 ffffffff81c1b460[ 1447.632197] [<ffffffff810c6ba7>] ? update_curr+0x67/0x160
[ 1447.636114] Call Trace:
[ 1447.637944] [<ffffffff81106d87>] ? get_futex_key+0x1e7/0x2e0
[ 1447.640005] [<ffffffff8172ec69>] system_call_fastpath+0x16/0x1b
[ 1447.640053] colord S[ 1447.643754] [<ffffffff815fe1c5>] ? __sys_sendmsg+0x85/0x90
[ 1447.644738] [<ffffffff8172ddfd>] schedule_hrtimeout_range_clock+0x14d/0x170
ffff88011fc13840 [ 1447.646461] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.647893] [<ffffffff810f7896>] ? do_setitimer+0xd6/0x230
[ 1447.656456] [<ffffffff815fd12c>] ? sock_sendmsg+0x9c/0xe0
0000000000000000 0000000000000000[ 1447.662416] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.666177] [<ffffffff81248b17>] ? eventfd_ctx_read+0x67/0x1c0
[ 1447.670034] [<ffffffff81214190>] ? poll_select_copy_remaining+0x150/0x150
0000000000000086[ 1447.671478] [<ffffffff81215682>] do_sys_poll+0x442/0x560
[ 1447.674316] [<ffffffff8172ec69>] system_call_fastpath+0x16/0x1b
[ 1447.674365] goa-daemon S[ 1447.678100] [<ffffffff8172ec69>] system_call_fastpath+0x16/0x1b
[ 1447.678148] pool S[ 1447.681768] ffff8800a67a7fd8[ 1447.682107] Call Trace:
[ 1447.683053] 0000000000000000[ 1447.683497] [<ffffffff810d12cc>] ? add_wait_queue+0x3c/0x50
[ 1447.688925] ffff8800a0e5bd7c
[ 1447.691879] Call Trace:
[ 1447.692031] [<ffffffff810d1840>] ? prepare_to_wait_event+0xf0/0xf0
[ 1447.693919] [<ffffffff8109adc9>] ? ns_capable+0x29/0x50
[ 1447.697720] Call Trace:
[ 1447.697770] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.697821] [<ffffffff8144d9ca>] __vt_event_wait.isra.0.part.1+0x4a/0x90
[ 1447.703437] [<ffffffff8144dbf3>] vt_waitactive+0x73/0xd0
[ 1447.703488] [<ffffffff812fd6e8>] ? security_capable+0x18/0x20
0000000000013840
[ 1447.707410] ffff8800a0e9bfd8[ 1447.711149] [<ffffffff81213741>] SyS_ioctl+0x81/0xa0
[ 1447.711520] ffff8800a0ea7fd8 0000000000013840 0000000000013840
[ 1447.717026] ffff8800a0eb7fd8[ 1447.720644] [<ffffffff810d1840>] ? prepare_to_wait_event+0xf0/0xf0
0000000000013840 ffff8800a6772740[ 1447.722446] [<ffffffff81213500>] do_vfs_ioctl+0x2e0/0x4a0
[ 1447.726354] [<ffffffff8144ea82>] vt_ioctl+0xcc2/0x1190
[ 1447.729898] [<ffffffff8144d9ca>] __vt_event_wait.isra.0.part.1+0x4a/0x90
ffff88011fc53840 [ 1447.731646] [<ffffffff810d1840>] ? prepare_to_wait_event+0xf0/0xf0
ffff8800a0f013a0 0000000000005607
[ 1447.736350] ffff8800a0f1fd7c ffff8800a0f1fd7c 0000000000005607 ffff8800a0f04e80
[ 1447.740154] ffff8800a0f2bd7c[ 1447.740476] [<ffffffff810d1840>] ? prepare_to_wait_event+0xf0/0xf0
0000000000013840
[ 1447.748169] ffff8800a0f4fd7c ffff8800a0f4fd7c 0000000000005607 ffff8800bb61e400[ 1447.751367] [<ffffffff81213741>] SyS_ioctl+0x81/0xa0
[ 1447.752148] Call Trace:
[ 1447.752198] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.752658] [<ffffffff81213741>] SyS_ioctl+0x81/0xa0
[ 1447.753366] Call Trace:
[ 1447.759899] [<ffffffff81213500>] do_vfs_ioctl+0x2e0/0x4a0
[ 1447.759950] [<ffffffff81213741>] SyS_ioctl+0x81/0xa0
[ 1447.760000] [<ffffffff8112f426>] ? __audit_syscall_exit+0x1f6/0x2a0
0000000000013840[ 1447.765760] [<ffffffff8172ec69>] system_call_fastpath+0x16/0x1b
[ 1447.765808] console-kit-dae S ffff88011fc53840 0 3092 1 0x00000080
[ 1447.769544] 0000000000000000 0000000000000000[ 1447.770432] [<ffffffff810bd602>] ? default_wake_function+0x12/0x20
0000000000013840 ffff8800cab489d0 ffff8801176fd850
[ 1447.771502] ffff88009e1bbd7c 0000000000013840
[ 1447.779164] ffff8800a0decd80 ffff88009e256288[ 1447.781065] [<ffffffff810bd5f0>] ? wake_up_state+0x20/0x20
0000000000013840 ffff8800a0fbf5c0
[ 1447.785187] Call Trace:
[ 1447.785237] [<ffffffff8172a999>] schedule+0x29/0x70
[ 1447.788762] [<ffffffffa018cf64>] ? zfs_inactive+0x64/0x210 [zfs]
[ 1447.788897] [<ffffffffa01a3dc4>] ? zpl_evict_inode+0x24/0x30 [zfs]
ffff8800a6688aa8
[ 1447.792506] Call Trace:
[ 1447.792615] [<ffffffffa0054b29>] taskq_thread+0x329/0x350 [spl]
0 0 0.000000 0.000000 0.000000
[ 1447.800518] 0 /
[ 1447.802633] scsi_tmf_2 63 411.710124 2 100 0 0 0.000000 0.000000 0.000000 0 /
[ 1447.808520] 0 0 0.000000 0.000000 0.000000[ 1447.812383] zvol/19 621 1685.819081 2 139 zvol/20 622 1692.019678 2 139 0 /
[ 1447.818288] z_wr_iss/0 902 11333.726855 67 138 0 /
[ 1447.821915] z_fr_iss_6/3 956 10835.095341 3 139 0 0 0.000000 0.000000 0.000000
[ 1447.827602] btrfs-transacti 1011 40189.350981 50 120 0 0 0.000000 0.000000 0.000000 /
[ 1447.834429] z_wr_int/12 1153 33756.991823 34 139 z_fr_iss_7/1 1191 11552.309121 4 139 0 0 0.000000 0.000000 0.000000
[ 1447.844645] /autogroup-74
[ 1447.844939] 0 0 0.000000 0.000000 0.000000
[ 1447.850853] libvirtd 2573 35.816011 1 120 0 0 0.000000 0.000000 0.000000 0 0 0.000000 0.000000 0.000000
[ 1447.862714] null-sink 2966 3.668760 13 120 0 0 0.000000 0.000000 0.000000 0 0 0.000000 0.000000 0.000000[ 1447.872088] .MIN_vruntime : 0.000001
[ 1447.873175] .throttled : 0
[ 1447.874358] .tg->runnable_avg : 5
0 /
[ 1447.878365] ksoftirqd/1 25 41941.725019 2082 120 kworker/1:0H 27 1593.342092 7 100
[ 1447.879094] kdevtmpfs 29 25700.348602 226 120 0 /
[ 1447.879534] /
[ 1447.885851] scsi_tmf_10 300 1549.284024 2 100 0 0 0.000000 0.000000 0.000000[ 1447.891335] z_wr_int/0 908 15670.102714 37 139 0 /
[ 1447.893668]
[ 1447.894326] z_wr_int_h/0 924 6486.017517 2 139 z_fr_iss_3/1 942 10662.699372 3 139 0 0 0.000000 0.000000 0.000000[ 1447.905247] z_fr_iss_3/1 1175 15877.618199 3 139 chronyd 1287 28.140328 156 120 /autogroup-93
[ 1447.911174] 0 0 0.000000 0.000000 0.000000[ 1447.914824] mysqld 2744 700.201540 2 120 0 0 0.000000 0.000000 0.000000 0 /autogroup-118
[ 1447.919378]
[ 1447.920110] gnome-shell 2939 9895.887007 4855 120 0 /autogroup-119
[ 1447.922606] systemd 3026 13.422008 40 120 /autogroup-133
[ 1447.928299] console-kit-dae 3055 73.318209 23 120 0 0 0.000000 0.000000 0.000000 0 /autogroup-133
[ 1447.934759] 0 /autogroup-133
[ 1447.935638] console-kit-dae 3091 73.316717 23 120 zfs 3388 657.493506 7 120 mount.zfs 3390 665.736062 36 120 [ 1468.303717] INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 0, t=600023 jiffies, g=7469, c=7468, q=0)
[ 1468.304036] Task dump for CPU 1:
[ 1468.304086] mount.zfs R running task 0 3390 1 0x0000008c
[ 1468.304193] ffffffffa00f8ff7 ffff8800a649c1d8 ffffffffa0165ee0 ffff88009e8bca68
[ 1468.304365] 000000005c55134a ffff8800a649c180 0000000000000000 0000000000000002
[ 1468.304537] ffff88011771b880 ffff88009e30f738 ffff88011771b748 ffffffffa0161b53
[ 1468.304710] Call Trace:
[ 1468.304832] [<ffffffffa00f8ff7>] ? dmu_buf_update_user+0x57/0xb0 [zfs]
[ 1468.304969] [<ffffffffa0165ee0>] ? zap_match.part.5+0x80/0x80 [zfs]
[ 1468.305098] [<ffffffffa0161b53>] ? zap_deref_leaf+0x73/0xa0 [zfs]
[ 1468.305228] [<ffffffffa0163854>] ? fzap_cursor_retrieve+0x114/0x2b0 [zfs]
[ 1468.305359] [<ffffffffa0166fa8>] ? zap_cursor_retrieve+0x68/0x310 [zfs]
[ 1468.305425] [<ffffffffa0053299>] ? spl_kmem_cache_alloc+0x109/0x740 [spl]
[ 1468.305479] [<ffffffff811e2b62>] ? kmem_cache_alloc+0x1c2/0x1f0
[ 1468.305531] [<ffffffff8130237a>] ? selinux_inode_alloc_security+0x3a/0xa0
[ 1468.305660] [<ffffffffa0171053>] ? zfs_purgedir+0x93/0x220 [zfs]
[ 1468.305793] [<ffffffffa0190479>] ? zfs_inode_update+0x149/0x1b0 [zfs]
[ 1468.305848] [<ffffffff810d1415>] ? wake_up_bit+0x25/0x30
[ 1468.305899] [<ffffffff8121a513>] ? unlock_new_inode+0x43/0x70
[ 1468.306030] [<ffffffffa01908e3>] ? zfs_znode_alloc+0x403/0x590 [zfs]
[ 1468.306162] [<ffffffffa0171432>] ? zfs_rmnode+0x252/0x330 [zfs]
[ 1468.306294] [<ffffffffa019377a>] ? zfs_zinactive+0x9a/0x150 [zfs]
[ 1468.306427] [<ffffffffa018cf64>] ? zfs_inactive+0x64/0x210 [zfs]
[ 1468.306562] [<ffffffffa01a3dc4>] ? zpl_evict_inode+0x24/0x30 [zfs]
[ 1468.306613] [<ffffffff8121b353>] ? evict+0xa3/0x170
[ 1468.306663] [<ffffffff8121bb85>] ? iput+0xf5/0x180
[ 1468.306792] [<ffffffffa01705b5>] ? zfs_unlinked_drain+0xb5/0xf0 [zfs]
[ 1468.306846] [<ffffffff810d1686>] ? finish_wait+0x56/0x70
[ 1468.306908] [<ffffffffa0055441>] ? taskq_create+0x311/0x410 [spl]
[ 1468.306962] [<ffffffff810d1840>] ? prepare_to_wait_event+0xf0/0xf0
[ 1468.307167] [<ffffffffa018e7f0>] ? zfs_get_done+0x70/0x70 [zfs]
[ 1468.307299] [<ffffffffa0195892>] ? zil_open+0x42/0x60 [zfs]
[ 1468.307430] [<ffffffffa01849a8>] ? zfs_sb_setup+0x108/0x140 [zfs]
[ 1468.307562] [<ffffffffa0186103>] ? zfs_domount+0x293/0x300 [zfs]
[ 1468.307616] [<ffffffff81202850>] ? get_anon_bdev+0x100/0x100
[ 1468.307746] [<ffffffffa01a3ef0>] ? zpl_mount+0x50/0x50 [zfs]
[ 1468.307878] [<ffffffffa01a3efe>] ? zpl_fill_super+0xe/0x20 [zfs]
[ 1468.307931] [<ffffffff8120377d>] ? mount_nodev+0x4d/0xb0
[ 1468.308062] [<ffffffffa01a3ed4>] ? zpl_mount+0x34/0x50 [zfs]
[ 1468.308116] [<ffffffff812042a9>] ? mount_fs+0x39/0x1b0
[ 1468.308170] [<ffffffff8121eec3>] ? vfs_kern_mount+0x63/0x100
[ 1468.308222] [<ffffffff812218bc>] ? do_mount+0x25c/0xb50
[ 1468.308275] [<ffffffff8118e284>] ? __get_free_pages+0x14/0x50
[ 1468.308328] [<ffffffff812224e6>] ? SyS_mount+0x96/0xf0
[ 1468.308381] [<ffffffff8172ec69>] ? system_call_fastpath+0x16/0x1b@behlendorf could it maybe have something to do with this? https://github.com/openzfsonosx/zfs/issues/255
behlendorf
commented
9 years ago @mvernimmen Thanks pointing this out, openzfsonosx/zfs#255 does look very similar. If somehow there are damaged entries it could manifest itself like this.
mvernimmen
commented
9 years ago @behlendorf Anything that I can do to help at this point? Anything we can try as a workaround or possible fix ?
I'm not sure when this started, whether it was when I upgraded from fedora 19 to 20, when I upgraded zfsonlinux to 0.6.3 or when the volume became almost full but I am currently unable to mount one of my pools in write mode (readonly works fine). When trying to do so the system seems to hang and periodically prints out a message that rcu_sched found hanging cpu/tasks. The pool contains my family photo albums and some other things. Although I have backups I would prefer to get the pool back in normal working order. Here follows information as listed at http://zfsonlinux.org/faq.html#HowDoIReportProblems
machine: 1 socket, amd athlon II X2 250 4GB ECC ram fedora 20 runs directly on the hardware, no hypervisors.
dmesg: [ 16.272825] SPL: Loaded module v0.6.3-1.1 [ 16.660578] ZFS: Loaded module v0.6.3-1.1, ZFS pool version 5000, ZFS filesystem version 5 [ 23.501920] SPL: using hostid 0x00000000
uname: Linux server.local 3.16.6-203.fc20.x86_64 #1 SMP Sat Oct 25 12:44:32 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
rpm versions: zfs-0.6.3-1.1.fc20.x86_64 libzfs2-0.6.3-1.1.fc20.x86_64 spl-0.6.3-1.1.fc20.x86_64 zfs-dkms-0.6.3-1.1.fc20.noarch spl-dkms-0.6.3-1.1.fc20.noarch zfs-release-1-3.fc20.noarch
zdb: mir-2tb: version: 23 name: 'mir-2tb' state: 0 txg: 2535963 pool_guid: 12859994781307727703 errata: 0 hostname: 'server.local' vdev_children: 1 vdev_tree: type: 'root' id: 0 guid: 12859994781307727703 children[0]: type: 'mirror' id: 0 guid: 5092114349905149760 whole_disk: 0 metaslab_array: 23 metaslab_shift: 31 ashift: 9 asize: 2000394125312 is_log: 0 create_txg: 4 children[0]: type: 'disk' id: 0 guid: 7233940709623163225 path: '/dev/disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F30SHWWD' whole_disk: 0 DTL: 282 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 12314288060364456657 path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2H7J9BZC05894' whole_disk: 0 DTL: 1017 create_txg: 4 children[2]: type: 'disk' id: 2 guid: 18210654718933789590 path: '/dev/disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F30U4RKD' whole_disk: 0 DTL: 1023 create_txg: 4 features_for_read: mir-2tb2: version: 23 name: 'mir-2tb2' state: 0 txg: 6386512 pool_guid: 107074157304257492 errata: 0 hostname: 'server.local' vdev_children: 1 vdev_tree: type: 'root' id: 0 guid: 107074157304257492 children[0]: type: 'mirror' id: 0 guid: 5600193728825606093 whole_disk: 0 metaslab_array: 23 metaslab_shift: 31 ashift: 9 asize: 2000394125312 is_log: 0 create_txg: 4 children[0]: type: 'disk' id: 0 guid: 8074817977783627925 path: '/dev/disk/by-id/ata-Hitachi_HDS5C3020ALA632_ML0220F30TG22D' whole_disk: 0 DTL: 184 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 1126741303624888001 path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2H7J9BZC05884' whole_disk: 0 DTL: 780 create_txg: 4 features_for_read:
params: spl_debug_mask 188 spl_debug_mb -1 spl_debug_panic_on_bug 0 spl_debug_printk 60 spl_debug_subsys 18446744073709551615 spl_hostid 0 spl_hostid_path /etc/hostid spl_kmem_cache_expire 2 spl_kmem_cache_kmem_limit 1024 spl_kmem_cache_max_size 32 spl_kmem_cache_obj_per_slab 16 spl_kmem_cache_obj_per_slab_min 8 spl_kmem_cache_reclaim 1 spl_kmem_cache_slab_limit 0 spl_taskq_thread_bind 0 l2arc_feed_again 1 l2arc_feed_min_ms 200 l2arc_feed_secs 1 l2arc_headroom 2 l2arc_headroom_boost 200 l2arc_nocompress 0 l2arc_noprefetch 1 l2arc_norw 0 l2arc_write_boost 8388608 l2arc_write_max 8388608 metaslab_debug_load 0 metaslab_debug_unload 0 spa_asize_inflation 24 spa_config_path /etc/zfs/zpool.cache zfetch_array_rd_sz 1048576 zfetch_block_cap 256 zfetch_max_streams 8 zfetch_min_sec_reap 2 zfs_arc_grow_retry 5 zfs_arc_max 0 zfs_arc_memory_throttle_disable 1 zfs_arc_meta_limit 0 zfs_arc_meta_prune 1048576 zfs_arc_min 0 zfs_arc_min_prefetch_lifespan 1000 zfs_arc_p_aggressive_disable 1 zfs_arc_p_dampener_disable 1 zfs_arc_shrink_shift 5 zfs_autoimport_disable 0 zfs_dbuf_state_index 0 zfs_deadman_enabled 1 zfs_deadman_synctime_ms 1000000 zfs_dedup_prefetch 1 zfs_delay_min_dirty_percent 60 zfs_delay_scale 500000 zfs_dirty_data_max 388050534 zfs_dirty_data_max_max 970126336 zfs_dirty_data_max_max_percent 25 zfs_dirty_data_max_percent 10 zfs_dirty_data_sync 67108864 zfs_disable_dup_eviction 0 zfs_expire_snapshot 300 zfs_flags 1 zfs_free_min_time_ms 1000 zfs_immediate_write_sz 32768 zfs_mdcomp_disable 0 zfs_nocacheflush 0 zfs_nopwrite_enabled 1 zfs_no_scrub_io 0 zfs_no_scrub_prefetch 0 zfs_pd_blks_max 100 zfs_prefetch_disable 0 zfs_read_chunk_size 1048576 zfs_read_history 0 zfs_read_history_hits 0 zfs_recover 0 zfs_resilver_delay 2 zfs_resilver_min_time_ms 3000 zfs_scan_idle 50 zfs_scan_min_time_ms 1000 zfs_scrub_delay 4 zfs_send_corrupt_data 0 zfs_sync_pass_deferred_free 2 zfs_sync_pass_dont_compress 5 zfs_sync_pass_rewrite 2 zfs_top_maxinflight 32 zfs_txg_history 0 zfs_txg_timeout 5 zfs_vdev_aggregation_limit 131072 zfs_vdev_async_read_max_active 3 zfs_vdev_async_read_min_active 1 zfs_vdev_async_write_active_max_dirty_percent 60 zfs_vdev_async_write_active_min_dirty_percent 30 zfs_vdev_async_write_max_active 10 zfs_vdev_async_write_min_active 1 zfs_vdev_cache_bshift 16 zfs_vdev_cache_max 16384 zfs_vdev_cache_size 0 zfs_vdev_max_active 1000 zfs_vdev_mirror_switch_us 10000 zfs_vdev_read_gap_limit 32768 zfs_vdev_scheduler noop zfs_vdev_scrub_max_active 2 zfs_vdev_scrub_min_active 1 zfs_vdev_sync_read_max_active 10 zfs_vdev_sync_read_min_active 10 zfs_vdev_sync_write_max_active 10 zfs_vdev_sync_write_min_active 10 zfs_vdev_write_gap_limit 4096 zfs_zevent_cols 80 zfs_zevent_console 0 zfs_zevent_len_max 96 zil_replay_disable 0 zil_slog_limit 1048576 zio_bulk_flags 0 zio_delay_max 30000 zio_injection_enabled 0 zio_requeue_io_start_cut_in_line 1 zvol_inhibit_dev 0 zvol_major 230 zvol_max_discard_blocks 16384 zvol_threads 32
The /proc/spl folder does not exist. I'll try and get the stacktraces.