search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.68k stars 1.76k forks source link

zfs hungs during file removal (Ubuntu 14.04 + 0.6.2-2~trusty) #2280

Closed Hudu closed 8 years ago

Hudu commented 10 years ago

During a huge remove (with big files) I get the following:

Apr 28 09:51:21 nas kernel: [ 1081.527707] INFO: task rm:4038 blocked for more than 120 seconds.
Apr 28 09:51:21 nas kernel: [ 1081.527773]       Tainted: PF          O 3.13.0-24-generic #46-Ubuntu
Apr 28 09:51:21 nas kernel: [ 1081.527829] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 09:51:21 nas kernel: [ 1081.527896] rm              D ffff88047fc14440     0  4038   4012 0x00000000
Apr 28 09:51:21 nas kernel: [ 1081.527902]  ffff880278911bb0 0000000000000002 ffff880282fec7d0 ffff880278911fd8
Apr 28 09:51:21 nas kernel: [ 1081.527908]  0000000000014440 0000000000014440 ffff880282fec7d0 ffff88003691b350
Apr 28 09:51:21 nas kernel: [ 1081.527912]  ffff88003691b220 ffff88003691b358 0000000000000000 0000000000000002
Apr 28 09:51:21 nas kernel: [ 1081.527917] Call Trace:
Apr 28 09:51:21 nas kernel: [ 1081.527927]  [<ffffffff81719e89>] schedule+0x29/0x70
Apr 28 09:51:21 nas kernel: [ 1081.527948]  [<ffffffffa00bbf45>] cv_wait_common+0x105/0x1a0 [spl]
Apr 28 09:51:21 nas kernel: [ 1081.527955]  [<ffffffff810aae90>] ? prepare_to_wait_event+0x100/0x100
Apr 28 09:51:21 nas kernel: [ 1081.527966]  [<ffffffffa00bbff5>] __cv_wait+0x15/0x20 [spl]
Apr 28 09:51:21 nas kernel: [ 1081.528008]  [<ffffffffa02d6f7b>] txg_wait_open+0x8b/0x110 [zfs]
Apr 28 09:51:21 nas kernel: [ 1081.528035]  [<ffffffffa0297f0d>] dmu_tx_wait+0xdd/0xe0 [zfs]
Apr 28 09:51:21 nas kernel: [ 1081.528060]  [<ffffffffa0297fa3>] dmu_tx_assign+0x93/0x4a0 [zfs]
Apr 28 09:51:21 nas kernel: [ 1081.528096]  [<ffffffffa02f42a1>] zfs_rmnode+0x141/0x340 [zfs]
Apr 28 09:51:21 nas kernel: [ 1081.528101]  [<ffffffff8171c2a2>] ? mutex_lock+0x12/0x2f
Apr 28 09:51:21 nas kernel: [ 1081.528137]  [<ffffffffa0314bf8>] zfs_zinactive+0x78/0xd0 [zfs]
Apr 28 09:51:21 nas kernel: [ 1081.528174]  [<ffffffffa030ec34>] zfs_inactive+0x64/0x200 [zfs]
Apr 28 09:51:21 nas kernel: [ 1081.528180]  [<ffffffff8115d739>] ? truncate_pagecache+0x59/0x60
Apr 28 09:51:21 nas kernel: [ 1081.528216]  [<ffffffffa0323114>] zpl_evict_inode+0x24/0x30 [zfs]
Apr 28 09:51:21 nas kernel: [ 1081.528221]  [<ffffffff811d4770>] evict+0xb0/0x1b0
Apr 28 09:51:21 nas kernel: [ 1081.528225]  [<ffffffff811d4f85>] iput+0xf5/0x180
Apr 28 09:51:21 nas kernel: [ 1081.528230]  [<ffffffff811c995e>] do_unlinkat+0x18e/0x2b0
Apr 28 09:51:21 nas kernel: [ 1081.528236]  [<ffffffff81020d35>] ? syscall_trace_enter+0x145/0x250
Apr 28 09:51:21 nas kernel: [ 1081.528241]  [<ffffffff811ca87b>] SyS_unlinkat+0x1b/0x40
Apr 28 09:51:21 nas kernel: [ 1081.528246]  [<ffffffff8172663f>] tracesys+0xe1/0xe6
Apr 28 09:51:21 nas kernel: [ 1081.528251] INFO: task chown:6980 blocked for more than 120 seconds.
Apr 28 09:51:21 nas kernel: [ 1081.528307]       Tainted: PF          O 3.13.0-24-generic #46-Ubuntu
Apr 28 09:51:21 nas kernel: [ 1081.528362] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 09:51:21 nas kernel: [ 1081.528435] chown           D ffff88047fd14440     0  6980   3942 0x00000000
Apr 28 09:51:21 nas kernel: [ 1081.528439]  ffff880261925b70 0000000000000002 ffff8802871b17f0 ffff880261925fd8
Apr 28 09:51:21 nas kernel: [ 1081.528443]  0000000000014440 0000000000014440 ffff8802871b17f0 ffff88003691b350
Apr 28 09:51:21 nas kernel: [ 1081.528447]  ffff88003691b220 ffff88003691b358 0000000000000000 0000000000000002
Apr 28 09:51:21 nas kernel: [ 1081.528451] Call Trace:
Apr 28 09:51:21 nas kernel: [ 1081.528455]  [<ffffffff81719e89>] schedule+0x29/0x70
Apr 28 09:51:21 nas kernel: [ 1081.528466]  [<ffffffffa00bbf45>] cv_wait_common+0x105/0x1a0 [spl]
Apr 28 09:51:21 nas kernel: [ 1081.528471]  [<ffffffff810aae90>] ? prepare_to_wait_event+0x100/0x100
Apr 28 09:51:21 nas kernel: [ 1081.528480]  [<ffffffffa00bbff5>] __cv_wait+0x15/0x20 [spl]
Apr 28 09:51:21 nas kernel: [ 1081.528511]  [<ffffffffa02d6f7b>] txg_wait_open+0x8b/0x110 [zfs]
Apr 28 09:51:21 nas kernel: [ 1081.528535]  [<ffffffffa0297f0d>] dmu_tx_wait+0xdd/0xe0 [zfs]
Apr 28 09:51:21 nas kernel: [ 1081.528569]  [<ffffffffa030c434>] zfs_setattr+0x1414/0x1910 [zfs]
Apr 28 09:51:21 nas kernel: [ 1081.528579]  [<ffffffffa0092a38>] ? nv_alloc_sleep_spl+0x28/0x30 [znvpair]
Apr 28 09:51:21 nas kernel: [ 1081.528584]  [<ffffffff811a001f>] ? __kmalloc+0x15f/0x190
Apr 28 09:51:21 nas kernel: [ 1081.528593]  [<ffffffffa00b31b6>] ? kmem_alloc_debug+0x96/0x3c0 [spl]
Apr 28 09:51:21 nas kernel: [ 1081.528602]  [<ffffffffa00b31b6>] ? kmem_alloc_debug+0x96/0x3c0 [spl]
Apr 28 09:51:21 nas kernel: [ 1081.528607]  [<ffffffff812d0118>] ? security_capable+0x18/0x20
Apr 28 09:51:21 nas kernel: [ 1081.528611]  [<ffffffff810f379e>] ? from_kuid+0xe/0x10
Apr 28 09:51:21 nas kernel: [ 1081.528616]  [<ffffffff81071041>] ? inode_capable+0x41/0x50
Apr 28 09:51:21 nas kernel: [ 1081.528649]  [<ffffffffa032273d>] zpl_setattr+0xdd/0x110 [zfs]
Apr 28 09:51:21 nas kernel: [ 1081.528653]  [<ffffffff811d5bd1>] notify_change+0x231/0x390
Apr 28 09:51:21 nas kernel: [ 1081.528658]  [<ffffffff811b67aa>] chown_common+0x15a/0x180
Apr 28 09:51:21 nas kernel: [ 1081.528661]  [<ffffffff811b826f>] SyS_fchownat+0xaf/0x110
Apr 28 09:51:21 nas kernel: [ 1081.528666]  [<ffffffff8172663f>] tracesys+0xe1/0xe6

The zpool-configuration is:

share:
    version: 5000
    name: 'share'
    state: 0
    txg: 11188
    pool_guid: 6604686797426360714
    hostname: 'nas'
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 6604686797426360714
        create_txg: 4
        children[0]:
            type: 'raidz'
            id: 0
            guid: 2754157955488200239
            nparity: 2
            metaslab_array: 34
            metaslab_shift: 37
            ashift: 12
            asize: 15002891714560
            is_log: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 16619676944599742585
                path: '/dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-0-part1'
                whole_disk: 1
                DTL: 81
                create_txg: 4
            children[1]:
                type: 'disk'
                id: 1
                guid: 17792812393439067462
                path: '/dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-1-part1'
                whole_disk: 1
                DTL: 80
                create_txg: 4
            children[2]:
                type: 'disk'
                id: 2
                guid: 4271262038771301126
                path: '/dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-2-part1'
                whole_disk: 1
                DTL: 79
                create_txg: 4
            children[3]:
                type: 'disk'
                id: 3
                guid: 15080917129105367318
                path: '/dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-3-part1'
                whole_disk: 1
                DTL: 78
                create_txg: 4
            children[4]:
                type: 'disk'
                id: 4
                guid: 2429349315307654443
                path: '/dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-4-part1'
                whole_disk: 1
                DTL: 77
                create_txg: 4
    features_for_read:

The arc-stats are

4 1 0x01 84 4032 15328816146 3778385897506
name                            type data
hits                            4    8515003
misses                          4    3143861
demand_data_hits                4    22
demand_data_misses              4    11
demand_metadata_hits            4    7539453
demand_metadata_misses          4    1395902
prefetch_data_hits              4    0
prefetch_data_misses            4    0
prefetch_metadata_hits          4    975529
prefetch_metadata_misses        4    1747948
mru_hits                        4    378418
mru_ghost_hits                  4    1468950
mfu_hits                        4    7161157
mfu_ghost_hits                  4    792712
deleted                         4    10
recycle_miss                    4    265546
mutex_miss                      4    2155
evict_skip                      4    163206618
evict_l2_cached                 4    4265023488
evict_l2_eligible               4    845031424
evict_l2_ineligible             4    6194059264
hash_elements                   4    882695
hash_elements_max               4    885935
hash_collisions                 4    1210827
hash_chains                     4    231567
hash_chain_max                  4    12
p                               4    1120050944
c                               4    8409665536
c_min                           4    4294967296
c_max                           4    16106127360
size                            4    2102880656
hdr_size                        4    284418096
data_size                       4    1673152000
other_size                      4    132300544
anon_size                       4    279552
anon_evict_data                 4    0
anon_evict_metadata             4    0
mru_size                        4    965329408
mru_evict_data                  4    0
mru_evict_metadata              4    885606912
mru_ghost_size                  4    1675704320
mru_ghost_evict_data            4    131072
mru_ghost_evict_metadata        4    1675573248
mfu_size                        4    707543040
mfu_evict_data                  4    0
mfu_evict_metadata              4    634978304
mfu_ghost_size                  4    412261888
mfu_ghost_evict_data            4    0
mfu_ghost_evict_metadata        4    412261888
l2_hits                         4    853981
l2_misses                       4    2289859
l2_feeds                        4    3604
l2_rw_clash                     4    158
l2_read_bytes                   4    1795242496
l2_write_bytes                  4    1552194048
l2_writes_sent                  4    923
l2_writes_done                  4    923
l2_writes_error                 4    0
l2_writes_hdr_miss              4    267
l2_evict_lock_retry             4    0
l2_evict_reading                4    0
l2_free_on_write                4    21245
l2_abort_lowmem                 4    0
l2_cksum_bad                    4    0
l2_io_error                     4    0
l2_size                         4    1715082240
l2_asize                        4    820861952
l2_hdr_size                     4    13010016
l2_compress_successes           4    736367
l2_compress_zeros               4    0
l2_compress_failures            4    17
memory_throttle_count           4    0
duplicate_buffers               4    0
duplicate_buffers_size          4    0
duplicate_reads                 4    0
memory_direct_count             4    0
memory_indirect_count           4    0
arc_no_grow                     4    0
arc_tempreserve                 4    0
arc_loaned_bytes                4    0
arc_prune                       4    223281
arc_meta_used                   4    2102880944
arc_meta_limit                  4    2102416384
arc_meta_max                    4    2755050368

Information about the computer:

Kernel: 3.13.0-24-generic
CPU: Intel(R) Atom(TM) CPU  C2550  @ 2.40GHz
MemTotal:       16425128 kB (ECC)

ii  dkms                                2.2.0.3-1.1ubuntu5+zfs9~trusty all          Dynamic Kernel Module Support Framework
ii  libzfs1                             0.6.2-2~trusty                 amd64        Native ZFS filesystem library for Linux
ii  mountall                            2.53-zfs1                      amd64        filesystem mounting tool
ii  ubuntu-zfs                          7~trusty                       amd64        Native ZFS filesystem metapackage for Ubuntu.
ii  zfs-dkms                            0.6.2-2~trusty                 amd64        Native ZFS filesystem kernel modules for Linux
ii  zfsutils                            0.6.2-2~trusty                 amd64        Native ZFS management utilities for Linux
Hudu commented 10 years ago

I have to add that I see another minor issue. The name of the processes/threads of zfs have no names.

Example from the command "ps -elf":

1 S root       335     2  0  99  19 -     0 taskq_ 09:33 ?        00:00:00 [?/0]
1 S root       336     2  0  99  19 -     0 taskq_ 09:33 ?        00:00:00 [?/0]
1 S root       337     2  0  99  19 -     0 taskq_ 09:33 ?        00:00:00 [?/-2144731094]
1 S root       338     2  0  99  19 -     0 taskq_ 09:33 ?        00:00:00 [?/0]
1 S root       339     2  0  99  19 -     0 taskq_ 09:33 ?        00:00:00 [?/0]
1 S root       340     2  0  99  19 -     0 taskq_ 09:33 ?        00:00:00 [?/0]
1 S root       341     2  0  99  19 -     0 taskq_ 09:33 ?        00:00:00 [?/0]
1 S root       342     2  0  99  19 -     0 taskq_ 09:33 ?        00:00:00 [?/0]
1 S root       343     2  0  99  19 -     0 taskq_ 09:33 ?        00:00:00 [?/0]
1 S root       344     2  0  99  19 -     0 taskq_ 09:33 ?        00:00:00 [?/0]
1 S root       345     2  0  99  19 -     0 taskq_ 09:33 ?        00:00:00 [?/0]
1 S root       346     2  0  99  19 -     0 taskq_ 09:33 ?        00:00:00 [?/0]
1 S root       347     2  0  99  19 -     0 taskq_ 09:33 ?        00:00:00 [?/0]
1 S root       348     2  0  99  19 -     0 taskq_ 09:33 ?        00:00:00 [?/0]
1 S root       349     2  0  99  19 -     0 taskq_ 09:33 ?        00:00:00 [?/0]
1 S root       350     2  0  99  19 -     0 taskq_ 09:33 ?        00:00:00 [?/0]
Hudu commented 10 years ago

Again during a rsync:

Apr 28 20:27:29 nas kernel: [ 5885.334024] INFO: task rsync:25942 blocked for more than 120 seconds.
Apr 28 20:27:29 nas kernel: [ 5885.334088]       Tainted: PF          O 3.13.0-24-generic #46-Ubuntu
Apr 28 20:27:29 nas kernel: [ 5885.334144] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 28 20:27:29 nas kernel: [ 5885.334212] rsync           D ffff88047fc14440     0 25942      1 0x00000004
Apr 28 20:27:29 nas kernel: [ 5885.334218]  ffff880402cddb90 0000000000000002 ffff880109da5fc0 ffff880402cddfd8
Apr 28 20:27:29 nas kernel: [ 5885.334223]  0000000000014440 0000000000014440 ffff880109da5fc0 ffff880464ee6350
Apr 28 20:27:29 nas kernel: [ 5885.334227]  ffff880464ee6220 ffff880464ee6358 0000000000000000 0000000000000002
Apr 28 20:27:29 nas kernel: [ 5885.334232] Call Trace:
Apr 28 20:27:29 nas kernel: [ 5885.334243]  [<ffffffff81719e89>] schedule+0x29/0x70
Apr 28 20:27:29 nas kernel: [ 5885.334263]  [<ffffffffa00d1f45>] cv_wait_common+0x105/0x1a0 [spl]
Apr 28 20:27:29 nas kernel: [ 5885.334271]  [<ffffffff810aae90>] ? prepare_to_wait_event+0x100/0x100
Apr 28 20:27:29 nas kernel: [ 5885.334282]  [<ffffffffa00d1ff5>] __cv_wait+0x15/0x20 [spl]
Apr 28 20:27:29 nas kernel: [ 5885.334326]  [<ffffffffa02d8f7b>] txg_wait_open+0x8b/0x110 [zfs]
Apr 28 20:27:29 nas kernel: [ 5885.334352]  [<ffffffffa0299f0d>] dmu_tx_wait+0xdd/0xe0 [zfs]
Apr 28 20:27:29 nas kernel: [ 5885.334389]  [<ffffffffa030fde2>] zfs_write+0x372/0xcb0 [zfs]
Apr 28 20:27:29 nas kernel: [ 5885.334395]  [<ffffffff816017ce>] ? sock_aio_read.part.8+0xfe/0x120
Apr 28 20:27:29 nas kernel: [ 5885.334432]  [<ffffffffa0324442>] zpl_write_common+0x52/0x80 [zfs]
Apr 28 20:27:29 nas kernel: [ 5885.334468]  [<ffffffffa03244d0>] zpl_write+0x60/0x90 [zfs]
Apr 28 20:27:29 nas kernel: [ 5885.334474]  [<ffffffff811b9534>] vfs_write+0xb4/0x1f0
Apr 28 20:27:29 nas kernel: [ 5885.334478]  [<ffffffff811b9f69>] SyS_write+0x49/0xa0
Apr 28 20:27:29 nas kernel: [ 5885.334483]  [<ffffffff8172663f>] tracesys+0xe1/0xe6

This time a complete stacktrace:

1:
/sbin/init#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
10:

[<ffffffff810c7e88>] rcu_nocb_kthread+0x2a8/0x2e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
1044:
smbd#-F#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811ce788>] do_sys_poll+0x428/0x540
[<ffffffff811ce975>] SyS_poll+0x65/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
10453:

[<ffffffff810846f1>] worker_thread+0x1d1/0x410
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
11:

[<ffffffff810c7e88>] rcu_nocb_kthread+0x2a8/0x2e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
1120:
/usr/bin/perl#-wT#/usr/sbin/munin-node#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
1122:
smbd#-F#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811ce788>] do_sys_poll+0x428/0x540
[<ffffffff811ce975>] SyS_poll+0x65/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
1186:
upstart-socket-bridge#--daemon#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
1189:
upstart-file-bridge#--daemon#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
12:

[<ffffffff810c9a7a>] rcu_gp_kthread+0x8a/0x880
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
12814:
/usr/sbin/apache2#-k#start#
[<ffffffff8165c480>] inet_csk_accept+0x210/0x2a0
[<ffffffff816879ad>] inet_accept+0x2d/0x100
[<ffffffff81601d33>] SYSC_accept4+0x103/0x220
[<ffffffff8160313e>] SyS_accept4+0xe/0x10
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
12972:
sshd: root@pts/5#####
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
13:

[<ffffffff810c7e88>] rcu_nocb_kthread+0x2a8/0x2e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
13019:
-bash#
[<ffffffff81069652>] do_wait+0x1e2/0x240
[<ffffffff8106a6a4>] SyS_wait4+0x64/0xe0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
136:

[<ffffffff814d89bd>] scsi_error_handler+0x8d/0x810
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
137:

[<ffffffff814d89bd>] scsi_error_handler+0x8d/0x810
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
138:

[<ffffffff814d89bd>] scsi_error_handler+0x8d/0x810
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
139:

[<ffffffff814d89bd>] scsi_error_handler+0x8d/0x810
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
14:

[<ffffffff810c7e88>] rcu_nocb_kthread+0x2a8/0x2e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
15:

[<ffffffff810c7e88>] rcu_nocb_kthread+0x2a8/0x2e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
16:

[<ffffffff810c7e88>] rcu_nocb_kthread+0x2a8/0x2e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
166:

[<ffffffff814d89bd>] scsi_error_handler+0x8d/0x810
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
167:

[<ffffffff814d89bd>] scsi_error_handler+0x8d/0x810
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
17:

[<ffffffff81092613>] smpboot_thread_fn+0xd3/0x1b0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
18:

[<ffffffff81092613>] smpboot_thread_fn+0xd3/0x1b0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
18531:
/usr/sbin/apache2#-k#start#
[<ffffffff8165c480>] inet_csk_accept+0x210/0x2a0
[<ffffffff816879ad>] inet_accept+0x2d/0x100
[<ffffffff81601d33>] SYSC_accept4+0x103/0x220
[<ffffffff8160313e>] SyS_accept4+0xe/0x10
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
189:

[<ffffffff810846f1>] worker_thread+0x1d1/0x410
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
19:

[<ffffffff81092613>] smpboot_thread_fn+0xd3/0x1b0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
2:

[<ffffffff8108bcea>] kthreadd+0x1aa/0x1c0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
20:

[<ffffffff81092613>] smpboot_thread_fn+0xd3/0x1b0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
21:

[<ffffffff81092613>] smpboot_thread_fn+0xd3/0x1b0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
22:

[<ffffffff810846f1>] worker_thread+0x1d1/0x410
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
23:

[<ffffffff810846f1>] worker_thread+0x1d1/0x410
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
231:

[<ffffffff810843ba>] rescuer_thread+0x27a/0x3e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
24:

[<ffffffff81092613>] smpboot_thread_fn+0xd3/0x1b0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
240:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
241:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
242:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
244:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
246:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
247:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
248:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
249:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
25:

[<ffffffff81092613>] smpboot_thread_fn+0xd3/0x1b0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
251:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
252:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
253:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
254:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
256:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
257:

[<ffffffffa0185cb5>] cleaner_kthread+0x155/0x180 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
258:

[<ffffffffa01888c9>] transaction_kthread+0x1f9/0x240 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
25906:

[<ffffffff810846f1>] worker_thread+0x1d1/0x410
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
25942:
rsync#-B#131072#-av#--progress#Video/#/share/media/video#
[<ffffffffa00d1f45>] cv_wait_common+0x105/0x1a0 [spl]
[<ffffffffa00d1ff5>] __cv_wait+0x15/0x20 [spl]
[<ffffffffa02d8f7b>] txg_wait_open+0x8b/0x110 [zfs]
[<ffffffffa0299f0d>] dmu_tx_wait+0xdd/0xe0 [zfs]
[<ffffffffa030fde2>] zfs_write+0x372/0xcb0 [zfs]
[<ffffffffa0324442>] zpl_write_common+0x52/0x80 [zfs]
[<ffffffffa03244d0>] zpl_write+0x60/0x90 [zfs]
[<ffffffff811b9534>] vfs_write+0xb4/0x1f0
[<ffffffff811b9f69>] SyS_write+0x49/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
26:

[<ffffffff81092613>] smpboot_thread_fn+0xd3/0x1b0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
26045:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
26046:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
26129:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
27:

[<ffffffff810846f1>] worker_thread+0x1d1/0x410
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
28:

[<ffffffff810846f1>] worker_thread+0x1d1/0x410
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
2810:
dhclient#-1#-v#-pf#/run/dhclient.em1.pid#-lf#/var/lib/dhcp/dhclient.em1.leases#em1#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
28853:
zpool#scrub#-s#share#
[<ffffffffa00d1f45>] cv_wait_common+0x105/0x1a0 [spl]
[<ffffffffa00d1ff5>] __cv_wait+0x15/0x20 [spl]
[<ffffffffa02d8d03>] txg_wait_synced+0xb3/0x190 [zfs]
[<ffffffffa02bae13>] dsl_sync_task_group_wait+0x193/0x270 [zfs]
[<ffffffffa02bb0be>] dsl_sync_task_do+0x4e/0x70 [zfs]
[<ffffffffa02b9268>] dsl_scan_cancel+0x38/0x40 [zfs]
[<ffffffffa02c9a7c>] spa_scan_stop+0x3c/0x50 [zfs]
[<ffffffffa02fc92d>] zfs_ioc_pool_scan+0x5d/0x70 [zfs]
[<ffffffffa02fdfaa>] zfsdev_ioctl+0xfa/0x1a0 [zfs]
[<ffffffff811cc6e0>] do_vfs_ioctl+0x2e0/0x4c0
[<ffffffff811cc941>] SyS_ioctl+0x81/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
28980:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
28986:

[<ffffffff810846f1>] worker_thread+0x1d1/0x410
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
29:

[<ffffffff81092613>] smpboot_thread_fn+0xd3/0x1b0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
2920:
/usr/sbin/winbindd#-F#
[<ffffffff81201742>] ep_poll+0x262/0x340
[<ffffffff81202975>] SyS_epoll_wait+0xd5/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
2945:
/usr/sbin/winbindd#-F#
[<ffffffff81201742>] ep_poll+0x262/0x340
[<ffffffff81202975>] SyS_epoll_wait+0xd5/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
2947:
nmbd#-D#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811ce788>] do_sys_poll+0x428/0x540
[<ffffffff811ce975>] SyS_poll+0x65/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
2958:
/bin/sh#-e#/proc/self/fd/9#
[<ffffffff81069652>] do_wait+0x1e2/0x240
[<ffffffff8106a6a4>] SyS_wait4+0x64/0xe0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
2961:
/bin/sh#/usr/sbin/start_pms#
[<ffffffff81069652>] do_wait+0x1e2/0x240
[<ffffffff8106a6a4>] SyS_wait4+0x64/0xe0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
2966:
./Plex Media Server#
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff81201742>] ep_poll+0x262/0x340
[<ffffffff81202975>] SyS_epoll_wait+0xd5/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff81201742>] ep_poll+0x262/0x340
[<ffffffff81202975>] SyS_epoll_wait+0xd5/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff81201742>] ep_poll+0x262/0x340
[<ffffffff81202975>] SyS_epoll_wait+0xd5/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff81610e6c>] __skb_recv_datagram+0x40c/0x4b0
[<ffffffff81610f42>] skb_recv_datagram+0x32/0x40
[<ffffffff81645213>] netlink_recvmsg+0x53/0x3a0
[<ffffffff816017ce>] sock_aio_read.part.8+0xfe/0x120
[<ffffffff81601811>] sock_aio_read+0x21/0x40
[<ffffffff811b8d1a>] do_sync_read+0x5a/0x90
[<ffffffff811b945d>] vfs_read+0x13d/0x160
[<ffffffff811b9ec9>] SyS_read+0x49/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff81201742>] ep_poll+0x262/0x340
[<ffffffff81202975>] SyS_epoll_wait+0xd5/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff81069652>] do_wait+0x1e2/0x240
[<ffffffff8106a6a4>] SyS_wait4+0x64/0xe0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3:

[<ffffffff81092613>] smpboot_thread_fn+0xd3/0x1b0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
30:

[<ffffffff81092613>] smpboot_thread_fn+0xd3/0x1b0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
3029:
/sbin/getty#-8#38400#tty4#
[<ffffffff814467c4>] n_tty_read+0x504/0xba0
[<ffffffff81441b0d>] tty_read+0x8d/0x100
[<ffffffff811b93b5>] vfs_read+0x95/0x160
[<ffffffff811b9ec9>] SyS_read+0x49/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3032:
/sbin/getty#-8#38400#tty5#
[<ffffffff814467c4>] n_tty_read+0x504/0xba0
[<ffffffff81441b0d>] tty_read+0x8d/0x100
[<ffffffff811b93b5>] vfs_read+0x95/0x160
[<ffffffff811b9ec9>] SyS_read+0x49/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3040:
/sbin/getty#-8#38400#tty2#
[<ffffffff814467c4>] n_tty_read+0x504/0xba0
[<ffffffff81441b0d>] tty_read+0x8d/0x100
[<ffffffff811b93b5>] vfs_read+0x95/0x160
[<ffffffff811b9ec9>] SyS_read+0x49/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3043:
/sbin/getty#-8#38400#tty3#
[<ffffffff814467c4>] n_tty_read+0x504/0xba0
[<ffffffff81441b0d>] tty_read+0x8d/0x100
[<ffffffff811b93b5>] vfs_read+0x95/0x160
[<ffffffff811b9ec9>] SyS_read+0x49/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3046:
Plex Plug-in [com.plexapp.system]#/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py#--server-version#0.9.9.7.429-f80a8d6#/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/System.bundle#
[<ffffffff811c1b41>] pipe_wait+0x61/0xa0
[<ffffffff811c2380>] pipe_read+0x280/0x400
[<ffffffff811b8d1a>] do_sync_read+0x5a/0x90
[<ffffffff811b93b5>] vfs_read+0x95/0x160
[<ffffffff811b9ec9>] SyS_read+0x49/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff81201742>] ep_poll+0x262/0x340
[<ffffffff81202975>] SyS_epoll_wait+0xd5/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3049:
/sbin/getty#-8#38400#tty6#
[<ffffffff814467c4>] n_tty_read+0x504/0xba0
[<ffffffff81441b0d>] tty_read+0x8d/0x100
[<ffffffff811b93b5>] vfs_read+0x95/0x160
[<ffffffff811b9ec9>] SyS_read+0x49/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3068:
/usr/sbin/sshd#-D#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3082:
cron#
[<ffffffff8108f42d>] hrtimer_nanosleep+0xad/0x170
[<ffffffff8108f556>] SyS_nanosleep+0x66/0x80
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3083:
atd#
[<ffffffff8108f42d>] hrtimer_nanosleep+0xad/0x170
[<ffffffff8108f556>] SyS_nanosleep+0x66/0x80
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
309:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
3093:
/usr/sbin/irqbalance#
[<ffffffff8108f42d>] hrtimer_nanosleep+0xad/0x170
[<ffffffff8108f556>] SyS_nanosleep+0x66/0x80
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
31:

[<ffffffff81092613>] smpboot_thread_fn+0xd3/0x1b0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
310:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
3106:
acpid#-c#/etc/acpi/events#-s#/var/run/acpid.socket#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
311:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
312:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
313:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
314:

[<ffffffffa00d20ee>] __cv_timedwait_common+0xae/0x210 [spl]
[<ffffffffa00d2283>] __cv_timedwait_interruptible+0x13/0x20 [spl]
[<ffffffffa02814bc>] arc_adapt_thread+0x9c/0x170 [zfs]
[<ffffffffa00ca7ea>] thread_generic_wrapper+0x7a/0x90 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
315:

[<ffffffffa00d20ee>] __cv_timedwait_common+0xae/0x210 [spl]
[<ffffffffa00d2283>] __cv_timedwait_interruptible+0x13/0x20 [spl]
[<ffffffffa027f6c6>] l2arc_feed_thread+0x56/0xa70 [zfs]
[<ffffffffa00ca7ea>] thread_generic_wrapper+0x7a/0x90 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
316:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
317:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
318:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
31814:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
31815:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
31816:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
31817:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
31821:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
31824:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
3185:
/usr/sbin/squid3#-N#-YC#-f#/etc/squid3/squid.conf#
[<ffffffff81201742>] ep_poll+0x262/0x340
[<ffffffff81202975>] SyS_epoll_wait+0xd5/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
31868:
zpool#scrub#-s#share#
[<ffffffffa00d1f45>] cv_wait_common+0x105/0x1a0 [spl]
[<ffffffffa00d1ff5>] __cv_wait+0x15/0x20 [spl]
[<ffffffffa02d8d03>] txg_wait_synced+0xb3/0x190 [zfs]
[<ffffffffa02bae13>] dsl_sync_task_group_wait+0x193/0x270 [zfs]
[<ffffffffa02bb0be>] dsl_sync_task_do+0x4e/0x70 [zfs]
[<ffffffffa02b9268>] dsl_scan_cancel+0x38/0x40 [zfs]
[<ffffffffa02c9a7c>] spa_scan_stop+0x3c/0x50 [zfs]
[<ffffffffa02fc92d>] zfs_ioc_pool_scan+0x5d/0x70 [zfs]
[<ffffffffa02fdfaa>] zfsdev_ioctl+0xfa/0x1a0 [zfs]
[<ffffffff811cc6e0>] do_vfs_ioctl+0x2e0/0x4c0
[<ffffffff811cc941>] SyS_ioctl+0x81/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
31879:

[<ffffffffa01b93b6>] worker_loop+0x346/0x5c0 [btrfs]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
3189:
/usr/sbin/mysqld#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811ce788>] do_sys_poll+0x428/0x540
[<ffffffff811ce975>] SyS_poll+0x65/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff81206481>] read_events+0x1b1/0x230
[<ffffffff81207f7d>] SyS_io_getevents+0x4d/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff81206481>] read_events+0x1b1/0x230
[<ffffffff81207f7d>] SyS_io_getevents+0x4d/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff81206481>] read_events+0x1b1/0x230
[<ffffffff81207f7d>] SyS_io_getevents+0x4d/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff81206481>] read_events+0x1b1/0x230
[<ffffffff81207f7d>] SyS_io_getevents+0x4d/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff81206481>] read_events+0x1b1/0x230
[<ffffffff81207f7d>] SyS_io_getevents+0x4d/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff81206481>] read_events+0x1b1/0x230
[<ffffffff81207f7d>] SyS_io_getevents+0x4d/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff81206481>] read_events+0x1b1/0x230
[<ffffffff81207f7d>] SyS_io_getevents+0x4d/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff81206481>] read_events+0x1b1/0x230
[<ffffffff81207f7d>] SyS_io_getevents+0x4d/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff81206481>] read_events+0x1b1/0x230
[<ffffffff81207f7d>] SyS_io_getevents+0x4d/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff81206481>] read_events+0x1b1/0x230
[<ffffffff81207f7d>] SyS_io_getevents+0x4d/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff8107b052>] do_sigtimedwait+0x1b2/0x280
[<ffffffff8107b196>] SYSC_rt_sigtimedwait+0x76/0xd0
[<ffffffff8107b1fe>] SyS_rt_sigtimedwait+0xe/0x10
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
319:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
31992:
-bash#
[<ffffffff81069652>] do_wait+0x1e2/0x240
[<ffffffff8106a6a4>] SyS_wait4+0x64/0xe0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
320:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
321:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
322:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
323:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
324:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
3244:
(logfile-daemon)#/var/log/squid3/access.log#
[<ffffffff816b12eb>] unix_stream_recvmsg+0x2eb/0x870
[<ffffffff816017ce>] sock_aio_read.part.8+0xfe/0x120
[<ffffffff81601811>] sock_aio_read+0x21/0x40
[<ffffffff811b8d1a>] do_sync_read+0x5a/0x90
[<ffffffff811b945d>] vfs_read+0x13d/0x160
[<ffffffff811b9ec9>] SyS_read+0x49/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3249:
(unlinkd)#
[<ffffffff811c1b41>] pipe_wait+0x61/0xa0
[<ffffffff811c2380>] pipe_read+0x280/0x400
[<ffffffff811b8d1a>] do_sync_read+0x5a/0x90
[<ffffffff811b93b5>] vfs_read+0x95/0x160
[<ffffffff811b9ec9>] SyS_read+0x49/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
325:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
326:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
327:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
3270:
/usr/bin/freshclam#-d#--quiet#
[<ffffffff8107be6c>] sys_pause+0x2c/0x50
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
328:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
329:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
3292:
diskd#3261444#3261445#3261446#
[<ffffffff812bb99d>] do_msgrcv+0x48d/0x580
[<ffffffff812bbaa5>] SyS_msgrcv+0x15/0x20
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3293:
(pinger)#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3294:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811ce788>] do_sys_poll+0x428/0x540
[<ffffffff811ce975>] SyS_poll+0x65/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3296:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3297:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3298:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3299:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
33:

[<ffffffff810846f1>] worker_thread+0x1d1/0x410
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
330:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
3300:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3301:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3302:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3303:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3304:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3305:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
331:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
332:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
3327:
/usr/sbin/hd-idle#-i#600#-l#/var/log/hd-idle.log#
[<ffffffff8108f42d>] hrtimer_nanosleep+0xad/0x170
[<ffffffff8108f556>] SyS_nanosleep+0x66/0x80
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
333:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
334:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
335:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
336:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
337:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
338:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
339:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
34:

[<ffffffff810843ba>] rescuer_thread+0x27a/0x3e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
340:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
341:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
342:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
343:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
3435:
/usr/lib/postfix/master#
[<ffffffff81201742>] ep_poll+0x262/0x340
[<ffffffff81202975>] SyS_epoll_wait+0xd5/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
344:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
3442:
pickup#-l#-t#unix#-u#-c#
[<ffffffff81201742>] ep_poll+0x262/0x340
[<ffffffff81202975>] SyS_epoll_wait+0xd5/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3443:
qmgr#-l#-t#unix#-u#
[<ffffffff81201742>] ep_poll+0x262/0x340
[<ffffffff81202975>] SyS_epoll_wait+0xd5/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
345:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
3453:
/usr/sbin/privoxy#--pidfile#/var/run/privoxy.pid#--user#privoxy#/etc/privoxy/config#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
346:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
347:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
348:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
3488:
pure-ftpd (SERVER)###########################################################################################################
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
35:

[<ffffffff81493958>] devtmpfsd+0x168/0x180
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
3500:
/usr/bin/rsync#--no-detach#--daemon#--config#/etc/rsyncd.conf#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
36:

[<ffffffff810843ba>] rescuer_thread+0x27a/0x3e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
3622:
/usr/sbin/smartd#--pidfile#/var/run/smartd.pid#
[<ffffffff8108f42d>] hrtimer_nanosleep+0xad/0x170
[<ffffffff8108f556>] SyS_nanosleep+0x66/0x80
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3642:
/usr/lib/plexmediaserver/Plex DLNA Server#
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3693:
/usr/bin/ziproxy#-c#/etc/ziproxy/ziproxy.conf#-u#ziproxy#-g#ziproxy#-d#-p#/var/run/ziproxy.pid#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
37:

[<ffffffff810843ba>] rescuer_thread+0x27a/0x3e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
3740:
/sbin/apcupsd#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff8107b052>] do_sigtimedwait+0x1b2/0x280
[<ffffffff8107b196>] SYSC_rt_sigtimedwait+0x76/0xd0
[<ffffffff8107b1fe>] SyS_rt_sigtimedwait+0xe/0x10
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff8165c480>] inet_csk_accept+0x210/0x2a0
[<ffffffff816879ad>] inet_accept+0x2d/0x100
[<ffffffff81601d33>] SYSC_accept4+0x103/0x220
[<ffffffff81603150>] SyS_accept+0x10/0x20
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
38:

[<ffffffff810843ba>] rescuer_thread+0x27a/0x3e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
3817:
/usr/sbin/apache2#-k#start#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3822:
/usr/sbin/apache2#-k#start#
[<ffffffff8165c480>] inet_csk_accept+0x210/0x2a0
[<ffffffff816879ad>] inet_accept+0x2d/0x100
[<ffffffff81601d33>] SYSC_accept4+0x103/0x220
[<ffffffff8160313e>] SyS_accept4+0xe/0x10
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3824:
/usr/sbin/apache2#-k#start#
[<ffffffff8165c480>] inet_csk_accept+0x210/0x2a0
[<ffffffff816879ad>] inet_accept+0x2d/0x100
[<ffffffff81601d33>] SYSC_accept4+0x103/0x220
[<ffffffff8160313e>] SyS_accept4+0xe/0x10
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3826:
/usr/sbin/apache2#-k#start#
[<ffffffff8165c480>] inet_csk_accept+0x210/0x2a0
[<ffffffff816879ad>] inet_accept+0x2d/0x100
[<ffffffff81601d33>] SYSC_accept4+0x103/0x220
[<ffffffff8160313e>] SyS_accept4+0xe/0x10
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3866:
/sbin/getty#-8#38400#tty1#
[<ffffffff814467c4>] n_tty_read+0x504/0xba0
[<ffffffff81441b0d>] tty_read+0x8d/0x100
[<ffffffff811b93b5>] vfs_read+0x95/0x160
[<ffffffff811b9ec9>] SyS_read+0x49/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
39:

[<ffffffff810843ba>] rescuer_thread+0x27a/0x3e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
3903:
/usr/sbin/ntpd#-p#/var/run/ntpd.pid#-g#-c#/var/lib/ntp/ntp.conf.dhcp#-u#111:119#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3922:
sshd: root@pts/0#####
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3924:

[<ffffffff810f64d4>] kauditd_thread+0x164/0x1c0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
3943:
/usr/sbin/apache2#-k#start#
[<ffffffff8165c480>] inet_csk_accept+0x210/0x2a0
[<ffffffff816879ad>] inet_accept+0x2d/0x100
[<ffffffff81601d33>] SYSC_accept4+0x103/0x220
[<ffffffff8160313e>] SyS_accept4+0xe/0x10
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
3971:
-bash#
[<ffffffff81069652>] do_wait+0x1e2/0x240
[<ffffffff8106a6a4>] SyS_wait4+0x64/0xe0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
4:

[<ffffffff810846f1>] worker_thread+0x1d1/0x410
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
40:

[<ffffffff810846f1>] worker_thread+0x1d1/0x410
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
4049:
/usr/sbin/apache2#-k#start#
[<ffffffff8165c480>] inet_csk_accept+0x210/0x2a0
[<ffffffff816879ad>] inet_accept+0x2d/0x100
[<ffffffff81601d33>] SYSC_accept4+0x103/0x220
[<ffffffff8160313e>] SyS_accept4+0xe/0x10
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
4050:
/usr/sbin/apache2#-k#start#
[<ffffffff8165c480>] inet_csk_accept+0x210/0x2a0
[<ffffffff816879ad>] inet_accept+0x2d/0x100
[<ffffffff81601d33>] SYSC_accept4+0x103/0x220
[<ffffffff8160313e>] SyS_accept4+0xe/0x10
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
4051:
/usr/sbin/apache2#-k#start#
[<ffffffff8165c480>] inet_csk_accept+0x210/0x2a0
[<ffffffff816879ad>] inet_accept+0x2d/0x100
[<ffffffff81601d33>] SYSC_accept4+0x103/0x220
[<ffffffff8160313e>] SyS_accept4+0xe/0x10
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
4052:
/usr/sbin/apache2#-k#start#
[<ffffffff8165c480>] inet_csk_accept+0x210/0x2a0
[<ffffffff816879ad>] inet_accept+0x2d/0x100
[<ffffffff81601d33>] SYSC_accept4+0x103/0x220
[<ffffffff8160313e>] SyS_accept4+0xe/0x10
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
4060:
sshd: root@pts/3#####
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
41:

[<ffffffff810843ba>] rescuer_thread+0x27a/0x3e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
4141:
-bash#
[<ffffffff81069652>] do_wait+0x1e2/0x240
[<ffffffff8106a6a4>] SyS_wait4+0x64/0xe0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
42:

[<ffffffff810843ba>] rescuer_thread+0x27a/0x3e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
43:

[<ffffffff8153827b>] hub_thread+0xdb/0x160
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
437:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
438:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
439:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
44:

[<ffffffff810843ba>] rescuer_thread+0x27a/0x3e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
440:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
441:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
442:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
443:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
444:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
445:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
446:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
447:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
448:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
449:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
45:

[<ffffffff810843ba>] rescuer_thread+0x27a/0x3e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
450:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
451:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
452:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
453:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
454:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
455:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
456:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
457:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
458:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
459:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
46:

[<ffffffff810846f1>] worker_thread+0x1d1/0x410
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
460:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
461:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
462:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
463:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
464:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
465:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
466:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
467:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
468:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
469:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
47:

[<ffffffff810846f1>] worker_thread+0x1d1/0x410
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
470:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
471:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
472:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
473:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
474:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
475:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
476:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
477:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
478:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
479:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
480:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
481:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
482:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
483:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
484:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
485:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
486:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
487:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
488:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
489:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
49:

[<ffffffff8110ca3c>] watchdog+0x3c/0x2e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
490:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
491:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
492:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
493:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
494:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
495:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
496:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
497:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
498:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
499:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
5:

[<ffffffff810846f1>] worker_thread+0x1d1/0x410
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
50:

[<ffffffff81163e8c>] kswapd+0x37c/0x410
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
500:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
501:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
502:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
503:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
504:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
505:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
506:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
507:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
508:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
509:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
51:

[<ffffffff8119bb7b>] ksm_scan_thread+0x12b/0x200
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
510:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
511:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
512:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
513:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
514:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
515:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
516:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
517:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
52:

[<ffffffff811a8a1b>] khugepaged+0x28b/0x460
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
524:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
525:

[<ffffffffa00d1f45>] cv_wait_common+0x105/0x1a0 [spl]
[<ffffffffa00d2015>] __cv_wait_interruptible+0x15/0x20 [spl]
[<ffffffffa02d990a>] txg_quiesce_thread+0x2da/0x380 [zfs]
[<ffffffffa00ca7ea>] thread_generic_wrapper+0x7a/0x90 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
526:

[<ffffffffa00d1edd>] cv_wait_common+0x9d/0x1a0 [spl]
[<ffffffffa00d2038>] __cv_wait_io+0x18/0x20 [spl]
[<ffffffffa031eb53>] zio_wait+0x103/0x1c0 [zfs]
[<ffffffffa02876bb>] dbuf_read+0x2cb/0x870 [zfs]
[<ffffffffa028ea39>] dmu_buf_hold+0xe9/0x1b0 [zfs]
[<ffffffffa02e8526>] zap_get_leaf_byblk+0x46/0x2c0 [zfs]
[<ffffffffa02e8804>] zap_deref_leaf+0x64/0x80 [zfs]
[<ffffffffa02e9850>] fzap_remove+0x30/0x90 [zfs]
[<ffffffffa02eefb2>] zap_remove_uint64+0x72/0xc0 [zfs]
[<ffffffffa028d1b6>] ddt_zap_remove+0x16/0x20 [zfs]
[<ffffffffa028cad3>] ddt_sync+0x2b3/0x6d0 [zfs]
[<ffffffffa02ca395>] spa_sync+0x465/0xad0 [zfs]
[<ffffffffa02d9cb2>] txg_sync_thread+0x302/0x580 [zfs]
[<ffffffffa00ca7ea>] thread_generic_wrapper+0x7a/0x90 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
527:

[<ffffffff810846f1>] worker_thread+0x1d1/0x410
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
53:

[<ffffffff811fc3ca>] fsnotify_mark_destroy+0x10a/0x150
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
54:

[<ffffffff812a370b>] ecryptfs_threadfn+0x17b/0x1e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
55:

[<ffffffff810843ba>] rescuer_thread+0x27a/0x3e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
645:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
650:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
654:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
659:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
67:

[<ffffffff810843ba>] rescuer_thread+0x27a/0x3e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
670:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
680:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
69:

[<ffffffff810846f1>] worker_thread+0x1d1/0x410
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
6912:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
6913:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
6914:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
6915:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
6916:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
6917:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
699:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
6995:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
6996:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
6997:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
6998:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
6999:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
7:

[<ffffffff810c9cbf>] rcu_gp_kthread+0x2cf/0x880
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
7000:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
702:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
705:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
708:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
711:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
714:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
717:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
720:

[<ffffffffa00cc9c9>] taskq_thread+0x3b9/0x4b0 [spl]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
738:
upstart-udev-bridge#--daemon#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
742:
/lib/systemd/systemd-udevd#--daemon#
[<ffffffff81201742>] ep_poll+0x262/0x340
[<ffffffff81202975>] SyS_epoll_wait+0xd5/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
7665:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
7666:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
7667:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
7668:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
7669:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
7672:
/usr/sbin/dansguardian#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
782:
dbus-daemon#--system#--fork#
[<ffffffff81201742>] ep_poll+0x262/0x340
[<ffffffff81202975>] SyS_epoll_wait+0xd5/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
8:

[<ffffffff810c7d3e>] rcu_nocb_kthread+0x15e/0x2e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
801:

[<ffffffff810843ba>] rescuer_thread+0x27a/0x3e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
820:
/lib/systemd/systemd-logind#
[<ffffffff81201742>] ep_poll+0x262/0x340
[<ffffffff81202975>] SyS_epoll_wait+0xd5/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
822:

[<ffffffffa040825e>] ipmi_thread+0x9e/0x1a0 [ipmi_si]
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
830:
rsyslogd#
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811cdb86>] do_select+0x5b6/0x780
[<ffffffff811cdf1c>] core_sys_select+0x1cc/0x2e0
[<ffffffff811ce0db>] SyS_select+0xab/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810bdf7a>] do_syslog+0x4fa/0x5c0
[<ffffffff8122da44>] kmsg_read+0x44/0x60
[<ffffffff8121f5ad>] proc_reg_read+0x3d/0x80
[<ffffffff811b93b5>] vfs_read+0x95/0x160
[<ffffffff811b9ec9>] SyS_read+0x49/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
[<ffffffff810d7a0d>] futex_wait_queue_me+0xdd/0x140
[<ffffffff810d8412>] futex_wait+0x182/0x290
[<ffffffff810da9ee>] do_futex+0xde/0x770
[<ffffffff810db0f1>] SyS_futex+0x71/0x150
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
851:
avahi-daemon: running [nas-2.local]
[<ffffffff811cd199>] poll_schedule_timeout+0x49/0x70
[<ffffffff811ce788>] do_sys_poll+0x428/0x540
[<ffffffff811ce975>] SyS_poll+0x65/0x100
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
863:
avahi-daemon: chroot helper
[<ffffffff816b12eb>] unix_stream_recvmsg+0x2eb/0x870
[<ffffffff816017ce>] sock_aio_read.part.8+0xfe/0x120
[<ffffffff81601811>] sock_aio_read+0x21/0x40
[<ffffffff811b8d1a>] do_sync_read+0x5a/0x90
[<ffffffff811b945d>] vfs_read+0x13d/0x160
[<ffffffff811b9ec9>] SyS_read+0x49/0xa0
[<ffffffff8172663f>] tracesys+0xe1/0xe6
[<ffffffffffffffff>] 0xffffffffffffffff
===
899:

[<ffffffff810843ba>] rescuer_thread+0x27a/0x3e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
9:

[<ffffffff810c7d3e>] rcu_nocb_kthread+0x15e/0x2e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
90:

[<ffffffff810843ba>] rescuer_thread+0x27a/0x3e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
91:

[<ffffffff810846f1>] worker_thread+0x1d1/0x410
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
92:

[<ffffffff810843ba>] rescuer_thread+0x27a/0x3e0
[<ffffffff8108b312>] kthread+0xd2/0xf0
[<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff
===
glenritchie commented 10 years ago

I also experienced similar stack traces and apparent hangs on exactly the same kernel and zfs version while trying to use two instances of rsync to copy to a ZFS volume at the same time.

[1812600.524039] INFO: task \x18:30375 blocked for more than 120 seconds.
[1812600.524080]       Tainted: PF          O 3.13.0-24-generic #46-Ubuntu
[1812600.524114] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1812600.524154] \x18               D ffff88012fc14440     0 30375      2 0x00000000
[1812600.524159]  ffff880126acbd70 0000000000000002 ffff8800365fc7d0 ffff880126acbfd8
[1812600.524163]  0000000000014440 0000000000014440 ffff8800365fc7d0 ffffc900107780b8
[1812600.524167]  ffffc90010778000 ffffc900107780c0 0000000000000000 0000000000000002
[1812600.524171] Call Trace:
[1812600.524180]  [<ffffffff81719e89>] schedule+0x29/0x70
[1812600.524202]  [<ffffffffa009ef45>] cv_wait_common+0x105/0x1a0 [spl]
[1812600.524208]  [<ffffffff810aae90>] ? prepare_to_wait_event+0x100/0x100
[1812600.524219]  [<ffffffffa009eff5>] __cv_wait+0x15/0x20 [spl]
[1812600.524263]  [<ffffffffa0191853>] txg_quiesce_thread+0x223/0x380 [zfs]
[1812600.524298]  [<ffffffffa0191630>] ? txg_fini+0x280/0x280 [zfs]
[1812600.524307]  [<ffffffffa00977ea>] thread_generic_wrapper+0x7a/0x90 [spl]
[1812600.524316]  [<ffffffffa0097770>] ? __thread_exit+0xa0/0xa0 [spl]
[1812600.524321]  [<ffffffff8108b312>] kthread+0xd2/0xf0
[1812600.524324]  [<ffffffff8108b240>] ? kthread_create_on_node+0x1d0/0x1d0
[1812600.524328]  [<ffffffff8172637c>] ret_from_fork+0x7c/0xb0
[1812600.524332]  [<ffffffff8108b240>] ? kthread_create_on_node+0x1d0/0x1d0
[1812600.524337] INFO: task rsync:30675 blocked for more than 120 seconds.
[1812600.524370]       Tainted: PF          O 3.13.0-24-generic #46-Ubuntu
[1812600.524403] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1812600.524443] rsync           D ffff88012fc94440     0 30675  30674 0x00000000
[1812600.524447]  ffff8801187abbd8 0000000000000002 ffff8800be0edfc0 ffff8801187abfd8
[1812600.524451]  0000000000014440 0000000000014440 ffff8800be0edfc0 ffff8801259559a0
[1812600.524455]  ffff880125955840 ffff8801259559a8 0000000000000000 0000000000000002
[1812600.524459] Call Trace:
[1812600.524462]  [<ffffffff81719e89>] schedule+0x29/0x70
[1812600.524473]  [<ffffffffa009ef45>] cv_wait_common+0x105/0x1a0 [spl]
[1812600.524477]  [<ffffffff810aae90>] ? prepare_to_wait_event+0x100/0x100
[1812600.524487]  [<ffffffffa009eff5>] __cv_wait+0x15/0x20 [spl]
[1812600.524512]  [<ffffffffa0151eab>] dmu_tx_wait+0x7b/0xe0 [zfs]
[1812600.524550]  [<ffffffffa01c6434>] zfs_setattr+0x1414/0x1910 [zfs]
[1812600.524555]  [<ffffffff812cfd0c>] ? security_inode_permission+0x1c/0x30
[1812600.524560]  [<ffffffff811c3fdb>] ? lookup_fast+0x18b/0x2c0
[1812600.524564]  [<ffffffff811a001f>] ? __kmalloc+0x15f/0x190
[1812600.524573]  [<ffffffffa00961b6>] ? kmem_alloc_debug+0x96/0x3c0 [spl]
[1812600.524582]  [<ffffffffa00961b6>] ? kmem_alloc_debug+0x96/0x3c0 [spl]
[1812600.524585]  [<ffffffff811c937f>] ? getname_flags+0x4f/0x190
[1812600.524622]  [<ffffffffa01dc73d>] zpl_setattr+0xdd/0x110 [zfs]
[1812600.524627]  [<ffffffff811d5bd1>] notify_change+0x231/0x390
[1812600.524631]  [<ffffffff811b708c>] chmod_common+0x10c/0x130
[1812600.524634]  [<ffffffff811b8165>] SyS_chmod+0x45/0xa0
[1812600.524638]  [<ffffffff8172663f>] tracesys+0xe1/0xe6
dweeezil commented 10 years ago

The messed-up process names are caused by an early version of a 3.13-related patch (in zfsonlinux/pkg-spl@17a527c) which is in the version of spl you're using.

The zpl_setattr() in these traces makes me wonder if there could be some SA involvement. Are you (@hudu and @glenritchie) running with xattr=sa? Your version of zfs still has the symlink workaround which disables SA xattrs in symlinks (zfsonlinux/zfs@6a7c0cc) but the chown()/chmod() operations may very well modify the ZPL_DACL_ACES SA which is variable-sized and could trigger another bug which wasn't fixed until zfsonlinux/zfs@83021b4 which is included in the latest daily snapshot. Could you both please try the daily snapshot and see whether you can reproduce the problem?

Unfortunately, if you've triggered the bug fixed by zfsonlinux/zfs@83021b4, the filesystem on which it occurred may be permanently damaged and, if so, must be destroyed and re-created. It's not very easy to detect such damage.

Finally, I'll note that the hangs you're seeing are not typical behavior of this bug. If you do not have `xattr=sa, it should be impossible to trigger this bug with the version of zfs you're using.

glenritchie commented 10 years ago 
:~# zfs get xattr
NAME  PROPERTY  VALUE  SOURCE
tank  xattr     on     default

Havn't tried the daily snapshot yet, will update/add comment when i do.

alek-p commented 8 years ago

This has probably been resolved by https://github.com/zfsonlinux/zfs/commit/4bda3bd0e72d582a785b6552ce16b99e04414fbe see https://github.com/zfsonlinux/zfs/issues/1939

behlendorf commented 8 years ago

It sure looks that way, thanks for pointing this out. Closing this as likely resolved.