search

openzfs / spl

A shim layer which adds the core interfaces required for OpenZFS.
https://zfsonlinux.org/
GNU General Public License v2.0
281 stars 180 forks source link

Poor benchmark performance #223

Closed mengesb closed 8 years ago

mengesb commented 11 years ago

Here are some historical notes from 3/7 when I was attempting to gather info and help through the IRC channel. http://pastebin.com/SVyA4JTf

Initial testing results where substantially better than current testing results. Here are some details:

############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 4
Writes: 106368
Reads: 101197
Re-Reads: 2094634
Random Reads: 2221879
Random Writes: 1612742
#################################################

Beginning iozone test with 16 GB file and 256 KB record length.

############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 256
Writes: 104548
Reads: 115470
Re-Reads: 2084149
Random Reads: 2294355
Random Writes: 2277694
#################################################

A second pass through results in the following:

############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 4
Writes: 9648
Reads: 8468
Re-Reads: 114021
Random Reads: 2081966
Random Writes: 1564358
#################################################

Beginning iozone test with 16 GB file and 256 KB record length.

############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 256
Writes: 7185
Reads: 8015
Re-Reads: 7602
Random Reads: 8089
Random Writes: 6324
#################################################

The only change made between the first test and subsequent tests with poor performance is:

root@storage-01:~# cat /etc/rc.local
#!/bin/sh -e
setterm -powersave off -blank 0
echo 1 >/sys/module/zfs/parameters/zfs_arc_memory_throttle_disable
exit 0

root@storage-01:~# cat /etc/modprobe.d/zfs.conf
options zfs zfs_arc_memory_throttle_disable=1

Requested a 'perf top', here is a run while the system appears sluggish but testing is still running (everything above .50%):

 27.63%  [kernel]            [k] isolate_lru_pages
 22.33%  [kernel]            [k] __isolate_lru_page
  9.19%  [kernel]            [k] shrink_page_list
  5.05%  [kernel]            [k] putback_lru_pages
  3.50%  [kernel]            [k] page_waitqueue
  3.10%  [kernel]            [k] unlock_page
  2.24%  [kernel]            [k] test_and_set_bit
  2.11%  [kernel]            [k] put_page_testzero
  2.02%  [kernel]            [k] page_evictable
  1.95%  [kernel]            [k] page_lru
  1.73%  [kernel]            [k] release_pages
  1.69%  [kernel]            [k] __wake_up_bit
  1.57%  [kernel]            [k] hpage_nr_pages
  1.53%  [kernel]            [k] list_move
  1.34%  [kernel]            [k] mod_state
  1.19%  [kernel]            [k] page_mapping
  1.12%  [kernel]            [k] inc_zone_page_state
  0.96%  [kernel]            [k] list_del
  0.86%  [kernel]            [k] __mod_zone_page_state
  0.86%  [kernel]            [k] page_referenced
  0.85%  [kernel]            [k] update_isolated_counts
  0.82%  [kernel]            [k] page_lru
  0.75%  [kernel]            [k] mem_cgroup_add_lru_list
  0.58%  [kernel]            [k] should_resched

Full top output

root@storage-01:~# top -bn1
top - 07:28:58 up 18:08,  2 users,  load average: 66.00, 66.01, 66.00
Tasks: 229 total,   2 running, 227 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us, 26.0 sy,  0.0 ni, 50.5 id, 23.1 wa,  0.0 hi,  0.4 si,  0.0 st
KiB Mem:  65340244 total, 59343864 used,  5996380 free,    36060 buffers
KiB Swap:  1631228 total,        0 used,  1631228 free,  1453868 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
 1510 root      39  19     0    0    0 R 101.3  0.0 487:12.86 spl_kmem_cache/
 1820 root      20   0 23296 1508 1072 R   6.3  0.0   0:00.01 top
    1 root      20   0 10648  728  588 S   0.0  0.0   0:02.60 init
    2 root      20   0     0    0    0 S   0.0  0.0   0:00.01 kthreadd
    3 root      20   0     0    0    0 S   0.0  0.0   0:00.22 ksoftirqd/0
    6 root      rt   0     0    0    0 S   0.0  0.0   0:02.69 migration/0
    7 root      rt   0     0    0    0 S   0.0  0.0   0:04.08 watchdog/0
    8 root      rt   0     0    0    0 S   0.0  0.0   0:07.47 migration/1
   10 root      20   0     0    0    0 S   0.0  0.0   0:00.26 ksoftirqd/1
   12 root      rt   0     0    0    0 S   0.0  0.0   0:03.74 watchdog/1
   13 root      rt   0     0    0    0 S   0.0  0.0   0:03.06 migration/2
   15 root      20   0     0    0    0 S   0.0  0.0   0:00.28 ksoftirqd/2
   16 root      rt   0     0    0    0 S   0.0  0.0   0:03.82 watchdog/2
   17 root      rt   0     0    0    0 S   0.0  0.0   0:05.60 migration/3
   19 root      20   0     0    0    0 S   0.0  0.0   0:00.16 ksoftirqd/3
   20 root      rt   0     0    0    0 S   0.0  0.0   0:03.88 watchdog/3
   21 root      rt   0     0    0    0 S   0.0  0.0   0:05.22 migration/4
   22 root      20   0     0    0    0 S   0.0  0.0   0:06.95 kworker/4:0
   23 root      20   0     0    0    0 S   0.0  0.0   0:00.20 ksoftirqd/4
   24 root      rt   0     0    0    0 S   0.0  0.0   0:03.70 watchdog/4
   25 root      rt   0     0    0    0 S   0.0  0.0   0:05.57 migration/5
   27 root      20   0     0    0    0 S   0.0  0.0   0:00.23 ksoftirqd/5
   28 root      rt   0     0    0    0 S   0.0  0.0   0:04.54 watchdog/5
   29 root      rt   0     0    0    0 S   0.0  0.0   0:04.54 migration/6
   30 root      20   0     0    0    0 S   0.0  0.0   0:00.33 kworker/6:0
   31 root      20   0     0    0    0 S   0.0  0.0   0:00.22 ksoftirqd/6
   32 root      rt   0     0    0    0 S   0.0  0.0   0:04.05 watchdog/6
   33 root      rt   0     0    0    0 S   0.0  0.0   0:05.34 migration/7
   35 root      20   0     0    0    0 S   0.0  0.0   0:00.22 ksoftirqd/7
   36 root      rt   0     0    0    0 S   0.0  0.0   0:04.26 watchdog/7
   37 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 cpuset
   38 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 khelper
   39 root      20   0     0    0    0 S   0.0  0.0   0:00.00 kdevtmpfs
   40 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 netns
   41 root      20   0     0    0    0 S   0.0  0.0   0:00.62 sync_supers
   42 root      20   0     0    0    0 S   0.0  0.0   0:00.00 bdi-default
   43 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 kintegrityd
   44 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 kblockd
   46 root      20   0     0    0    0 S   0.0  0.0   0:02.14 kworker/2:1
   52 root      20   0     0    0    0 S   0.0  0.0   0:00.05 khungtaskd
   53 root      20   0     0    0    0 D   0.0  0.0  11:59.68 kswapd0
   54 root      25   5     0    0    0 S   0.0  0.0   0:00.00 ksmd
   55 root      39  19     0    0    0 S   0.0  0.0   0:00.00 khugepaged
   56 root      20   0     0    0    0 S   0.0  0.0   0:00.02 fsnotify_mark
   57 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 crypto
  153 root      20   0     0    0    0 S   0.0  0.0   0:00.00 scsi_eh_0
  167 root      20   0     0    0    0 S   0.0  0.0   0:00.00 khubd
  174 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 ata_sff
  179 root      20   0     0    0    0 S   0.0  0.0   0:00.00 scsi_eh_1
  180 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 fw_event0
  181 root      20   0     0    0    0 S   0.0  0.0   0:00.00 scsi_eh_2
  182 root      20   0     0    0    0 S   0.0  0.0   0:00.00 scsi_eh_3
  186 root      20   0     0    0    0 S   0.0  0.0   0:00.00 scsi_eh_4
  187 root      20   0     0    0    0 S   0.0  0.0   0:00.00 scsi_eh_5
  205 root      20   0     0    0    0 S   0.0  0.0   0:04.24 kworker/1:2
  220 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 poll_0_status
  352 root      39  19     0    0    0 D   0.0  0.0   7:55.31 z_null_iss/0
  353 root      39  19     0    0    0 D   0.0  0.0  15:17.92 z_null_int/0
  354 root      39  19     0    0    0 S   0.0  0.0   0:03.16 z_rd_iss/0
  355 root      39  19     0    0    0 S   0.0  0.0   0:03.15 z_rd_iss/1
  356 root      39  19     0    0    0 S   0.0  0.0   0:03.02 z_rd_iss/2
  357 root      39  19     0    0    0 S   0.0  0.0   0:02.94 z_rd_iss/3
  358 root      39  19     0    0    0 S   0.0  0.0   0:02.97 z_rd_iss/4
  359 root      39  19     0    0    0 S   0.0  0.0   0:03.19 z_rd_iss/5
  360 root      39  19     0    0    0 S   0.0  0.0   0:03.25 z_rd_iss/6
  361 root      39  19     0    0    0 S   0.0  0.0   0:03.20 z_rd_iss/7
  362 root      39  19     0    0    0 S   0.0  0.0   7:32.88 z_rd_int/0
  363 root      39  19     0    0    0 S   0.0  0.0   7:35.16 z_rd_int/1
  364 root      39  19     0    0    0 S   0.0  0.0   7:11.60 z_rd_int/2
  365 root      39  19     0    0    0 S   0.0  0.0   7:25.91 z_rd_int/3
  366 root      39  19     0    0    0 S   0.0  0.0   7:29.56 z_rd_int/4
  367 root      39  19     0    0    0 S   0.0  0.0   7:33.04 z_rd_int/5
  368 root      39  19     0    0    0 S   0.0  0.0   7:22.05 z_rd_int/6
  369 root      39  19     0    0    0 S   0.0  0.0   7:23.94 z_rd_int/7
  370 root      39  19     0    0    0 D   0.0  0.0  70:58.73 z_wr_iss/0
  371 root      39  19     0    0    0 D   0.0  0.0  61:16.94 z_wr_iss/1
  372 root      39  19     0    0    0 D   0.0  0.0  55:31.75 z_wr_iss/2
  373 root      39  19     0    0    0 D   0.0  0.0  60:53.37 z_wr_iss/3
  374 root      39  19     0    0    0 D   0.0  0.0  61:29.72 z_wr_iss/4
  375 root      39  19     0    0    0 D   0.0  0.0  61:24.10 z_wr_iss/5
  376 root      39  19     0    0    0 D   0.0  0.0  61:22.01 z_wr_iss/6
  377 root      39  19     0    0    0 D   0.0  0.0  60:49.48 z_wr_iss/7
  378 root      39  19     0    0    0 S   0.0  0.0   0:01.88 z_wr_iss_h/0
  379 root      39  19     0    0    0 S   0.0  0.0   0:01.95 z_wr_iss_h/1
  380 root      39  19     0    0    0 S   0.0  0.0   0:01.87 z_wr_iss_h/2
  381 root      39  19     0    0    0 S   0.0  0.0   0:01.94 z_wr_iss_h/3
  382 root      39  19     0    0    0 S   0.0  0.0   0:02.06 z_wr_iss_h/4
  383 root      39  19     0    0    0 D   0.0  0.0  42:27.83 z_wr_int/0
  384 root      39  19     0    0    0 D   0.0  0.0  41:12.62 z_wr_int/1
  385 root      39  19     0    0    0 D   0.0  0.0  42:22.27 z_wr_int/2
  386 root      39  19     0    0    0 D   0.0  0.0  41:32.51 z_wr_int/3
  387 root      39  19     0    0    0 D   0.0  0.0  43:19.57 z_wr_int/4
  388 root      39  19     0    0    0 D   0.0  0.0  41:07.73 z_wr_int/5
  389 root      39  19     0    0    0 D   0.0  0.0  43:39.27 z_wr_int/6
  390 root      39  19     0    0    0 D   0.0  0.0  41:32.56 z_wr_int/7
  391 root      39  19     0    0    0 D   0.0  0.0  42:28.16 z_wr_int/8
  392 root      39  19     0    0    0 D   0.0  0.0  41:02.98 z_wr_int/9
  393 root      39  19     0    0    0 D   0.0  0.0  42:27.15 z_wr_int/10
  394 root      39  19     0    0    0 D   0.0  0.0  41:25.42 z_wr_int/11
  395 root      39  19     0    0    0 D   0.0  0.0  43:20.54 z_wr_int/12
  396 root      39  19     0    0    0 D   0.0  0.0  41:07.37 z_wr_int/13
  397 root      39  19     0    0    0 D   0.0  0.0  43:35.69 z_wr_int/14
  398 root      39  19     0    0    0 D   0.0  0.0  41:29.69 z_wr_int/15
  399 root      39  19     0    0    0 S   0.0  0.0   0:00.58 z_wr_int_h/0
  400 root      39  19     0    0    0 S   0.0  0.0   0:00.61 z_wr_int_h/1
  401 root      39  19     0    0    0 S   0.0  0.0   0:00.41 z_wr_int_h/2
  402 root      39  19     0    0    0 S   0.0  0.0   0:00.56 z_wr_int_h/3
  403 root      39  19     0    0    0 S   0.0  0.0   0:00.68 z_wr_int_h/4
  404 root      39  19     0    0    0 S   0.0  0.0   6:59.42 z_fr_iss/0
  405 root      39  19     0    0    0 S   0.0  0.0   8:16.51 z_fr_iss/1
  406 root      39  19     0    0    0 S   0.0  0.0   7:02.28 z_fr_iss/2
  407 root      39  19     0    0    0 S   0.0  0.0   9:24.57 z_fr_iss/3
  408 root      39  19     0    0    0 S   0.0  0.0   7:39.42 z_fr_iss/4
  409 root      39  19     0    0    0 S   0.0  0.0   8:08.86 z_fr_iss/5
  410 root      39  19     0    0    0 S   0.0  0.0   7:39.12 z_fr_iss/6
  411 root      39  19     0    0    0 S   0.0  0.0   8:15.93 z_fr_iss/7
  412 root      39  19     0    0    0 S   0.0  0.0   0:00.00 z_fr_int/0
  413 root      39  19     0    0    0 S   0.0  0.0   0:00.00 z_cl_iss/0
  414 root      39  19     0    0    0 S   0.0  0.0   0:00.00 z_cl_int/0
  415 root      39  19     0    0    0 S   0.0  0.0   0:01.80 z_ioctl_iss/0
  416 root      39  19     0    0    0 S   0.0  0.0   0:00.00 z_ioctl_int/0
  441 root      20   0     0    0    0 S   0.0  0.0   0:00.11 kjournald
  474 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 zfs_iput_taskq/
  475 root       0 -20     0    0    0 S   0.0  0.0   0:37.01 txg_quiesce
  476 root       0 -20     0    0    0 D   0.0  0.0  81:58.34 txg_sync
  479 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 zil_clean/0
  579 root      20   0 21456 1252  568 S   0.0  0.0   0:00.67 udevd
  937 root      20   0 10792 1064  828 S   0.0  0.0   0:00.02 bash
  949 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 edac-poller
  953 root      20   0 79720 2300 1424 S   0.0  0.0   0:04.02 sshd
  958 root      20   0 20564 2872 1208 S   0.0  0.0   0:00.07 bash
 1072 root       0 -20     0    0    0 S   0.0  0.0   0:03.86 zil_clean/0
 1073 root      20   0     0    0    0 D   0.0  0.0   0:59.19 kjournald
 1074 root      20   0 26148 3924  904 S   0.0  0.0   0:00.08 perl
 1358 root      20   0     0    0    0 D   0.0  0.0   0:27.73 flush-230:0
 1388 root      20   0  4176  576  488 S   0.0  0.0   0:00.00 sh
 1389 root      20   0 47132  17m  736 D   0.0  0.0   1:56.82 iozone
 1390 root      20   0  5624  660  564 S   0.0  0.0   0:00.00 tail
 1391 root      20   0  5600  648  556 S   0.0  0.0   0:00.00 head
 1392 root      20   0 12744  908  748 S   0.0  0.0   0:00.00 awk
 1402 root      20   0     0    0    0 S   0.0  0.0   0:00.01 flush-8:32
 1423 root      20   0     0    0    0 S   0.0  0.0   0:00.55 kworker/u:1
 1427 root      20   0     0    0    0 S   0.0  0.0   0:00.15 kworker/u:0
 1511 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 spl_system_task
 1512 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 spl_system_task
 1514 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 spl_system_task
 1517 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 spl_system_task
 1518 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 spl_system_task
 1519 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 spl_system_task
 1520 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 spl_system_task
 1521 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 spl_system_task
 1535 root       0 -20     0    0    0 D   0.0  0.0  45:58.64 arc_adapt
 1536 root       0 -20     0    0    0 D   0.0  0.0  21:04.10 l2arc_feed
 1537 root      39  19     0    0    0 S   0.0  0.0   0:00.00 z_unmount/0
 1538 root      39  19     0    0    0 D   0.0  0.0  10:15.45 zvol/0
 1539 root      39  19     0    0    0 D   0.0  0.0  11:14.65 zvol/1
 1540 root      39  19     0    0    0 D   0.0  0.0  10:38.10 zvol/2
 1541 root      39  19     0    0    0 D   0.0  0.0  11:00.60 zvol/3
 1542 root      39  19     0    0    0 D   0.0  0.0  11:11.22 zvol/4
 1543 root      39  19     0    0    0 D   0.0  0.0  11:18.11 zvol/5
 1544 root      39  19     0    0    0 D   0.0  0.0  10:58.50 zvol/6
 1545 root      39  19     0    0    0 D   0.0  0.0  10:59.77 zvol/7
 1546 root      39  19     0    0    0 D   0.0  0.0  10:20.48 zvol/8
 1547 root      39  19     0    0    0 D   0.0  0.0  11:18.51 zvol/9
 1548 root      39  19     0    0    0 D   0.0  0.0  10:32.67 zvol/10
 1549 root      39  19     0    0    0 D   0.0  0.0  11:04.96 zvol/11
 1550 root      39  19     0    0    0 D   0.0  0.0  11:14.92 zvol/12
 1551 root      39  19     0    0    0 D   0.0  0.0  11:17.28 zvol/13
 1552 root      39  19     0    0    0 D   0.0  0.0  11:03.14 zvol/14
 1553 root      39  19     0    0    0 D   0.0  0.0  10:56.68 zvol/15
 1554 root      39  19     0    0    0 D   0.0  0.0  10:18.08 zvol/16
 1555 root      39  19     0    0    0 D   0.0  0.0  11:13.37 zvol/17
 1556 root      39  19     0    0    0 D   0.0  0.0  10:38.68 zvol/18
 1557 root      39  19     0    0    0 D   0.0  0.0  11:04.12 zvol/19
 1558 root      39  19     0    0    0 D   0.0  0.0  11:12.59 zvol/20
 1559 root      39  19     0    0    0 D   0.0  0.0  11:15.70 zvol/21
 1560 root      39  19     0    0    0 D   0.0  0.0  10:55.24 zvol/22
 1561 root      39  19     0    0    0 D   0.0  0.0  11:01.85 zvol/23
 1562 root      39  19     0    0    0 D   0.0  0.0  10:21.50 zvol/24
 1563 root      39  19     0    0    0 D   0.0  0.0  11:18.96 zvol/25
 1564 root      39  19     0    0    0 D   0.0  0.0  10:33.85 zvol/26
 1565 root      39  19     0    0    0 D   0.0  0.0  11:01.54 zvol/27
 1566 root      39  19     0    0    0 D   0.0  0.0  11:12.36 zvol/28
 1567 root      39  19     0    0    0 D   0.0  0.0  11:16.28 zvol/29
 1568 root      39  19     0    0    0 D   0.0  0.0  11:00.37 zvol/30
 1569 root      39  19     0    0    0 D   0.0  0.0  11:02.26 zvol/31
 3044 root      20   0 18968  700  436 S   0.0  0.0   0:00.16 rpcbind
 3075 statd     20   0 23340 1076  612 S   0.0  0.0   0:00.00 rpc.statd
 3080 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 rpciod
 3082 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 nfsiod
 3089 root      20   0 25288  304   84 S   0.0  0.0   0:00.00 rpc.idmapd
 3296 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 iscsi_eh
 3298 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 ib_mcast
 3299 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 ib_cm
 3300 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 iw_cm_wq
 3301 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 ib_addr
 3302 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 rdma_cm
 3304 root      20   0  4992  480  376 S   0.0  0.0   0:06.35 iscsid
 3305 root      10 -10  5492 3288 2200 S   0.0  0.0   0:28.61 iscsid
 3469 root      20   0 53300 1552  588 S   0.0  0.0   0:00.18 rsyslogd
 3610 daemon    20   0 16668  332  172 S   0.0  0.0   0:00.00 atd
 3620 root      20   0 20404  892  648 S   0.0  0.0   0:00.16 cron
 3701 messageb  20   0 29800  528  208 S   0.0  0.0   0:00.00 dbus-daemon
 3886 root      20   0  4112  608  468 S   0.0  0.0   0:00.00 acpid
 3918 root      20   0 49848  940  336 S   0.0  0.0   0:00.00 sshd
 3947 Debian-e  20   0 46800  876  424 S   0.0  0.0   0:00.02 exim4
 3976 root      20   0 16252  796  640 S   0.0  0.0   0:00.00 getty
 3977 root      20   0 16252  796  640 S   0.0  0.0   0:00.00 getty
 3978 root      20   0 16252  796  640 S   0.0  0.0   0:00.00 getty
 3979 root      20   0 16252  800  640 S   0.0  0.0   0:00.00 getty
 3980 root      20   0 16252  796  640 S   0.0  0.0   0:00.00 getty
 3981 root      20   0 16252  792  640 S   0.0  0.0   0:00.00 getty
 3985 root      20   0 79720 1716  856 S   0.0  0.0   0:03.66 sshd
 3988 root      20   0  188m 1744  684 S   0.0  0.0   0:00.02 console-kit-dae
 4057 root      20   0 22372 4500 1028 S   0.0  0.0   0:18.03 bash
17583 root      20   0     0    0    0 S   0.0  0.0   0:05.49 kworker/0:0
23375 root      20   0     0    0    0 S   0.0  0.0   0:08.68 kworker/7:0
29126 root      20   0     0    0    0 S   0.0  0.0   0:07.03 kworker/0:1
29187 root      20   0     0    0    0 S   0.0  0.0   0:09.98 kworker/5:1
29222 root      20   0     0    0    0 S   0.0  0.0   0:06.51 kworker/3:2
29748 root      20   0     0    0    0 S   0.0  0.0   0:05.99 kworker/1:1
29837 root      20   0     0    0    0 S   0.0  0.0   0:03.58 kworker/4:2
29856 root      20   0     0    0    0 S   0.0  0.0   0:00.96 kworker/7:2
29922 root      20   0     0    0    0 S   0.0  0.0   0:00.37 kworker/5:2
30093 root      20   0     0    0    0 S   0.0  0.0   0:03.76 kworker/3:1
30603 root      20   0     0    0    0 S   0.0  0.0   0:09.92 kworker/6:2
31281 root      20   0     0    0    0 S   0.0  0.0   0:09.11 kworker/2:11
31326 root      20   0 21292  936  404 S   0.0  0.0   0:00.01 udevd
31327 root      20   0 21292  960  432 S   0.0  0.0   0:00.00 udevd

Recent history in /var/log/messages:

Mar  7 14:42:19 storage-01 kernel: [ 4921.026387] intel_rng: FWH not detected
Mar  7 14:43:11 storage-01 kernel: [ 4972.962984]  sdd: sdd1 sdd9
Mar  7 14:43:12 storage-01 kernel: [ 4973.107937]  sde: sde1 sde9
Mar  7 14:43:12 storage-01 kernel: [ 4973.315635]  sdf: sdf1 sdf9
Mar  7 14:43:12 storage-01 kernel: [ 4973.483068]  sdg: sdg1 sdg9
Mar  7 14:43:12 storage-01 kernel: [ 4973.626431]  sdh: sdh1 sdh9
Mar  7 14:43:12 storage-01 kernel: [ 4973.809414]  sdi: sdi1 sdi9
Mar  7 14:43:12 storage-01 kernel: [ 4973.938781]  sdj: sdj1 sdj9
Mar  7 14:43:13 storage-01 kernel: [ 4974.098591]  sdk: sdk1 sdk9
Mar  7 14:43:13 storage-01 kernel: [ 4974.284223]  sdl: sdl1 sdl9
Mar  7 14:43:13 storage-01 kernel: [ 4974.406795]  sdm: sdm1 sdm9
Mar  7 14:43:13 storage-01 kernel: [ 4974.525794]  sdn: sdn1 sdn9
Mar  7 14:43:13 storage-01 kernel: [ 4974.704827]  sdo: sdo1 sdo9
Mar  7 14:43:13 storage-01 kernel: [ 4974.819423]  sdp: sdp1 sdp9
Mar  7 14:43:13 storage-01 kernel: [ 4974.996740]  sdq: sdq1 sdq9
Mar  7 14:43:14 storage-01 kernel: [ 4975.166350]  sdr: sdr1 sdr9
Mar  7 14:43:14 storage-01 kernel: [ 4975.340149]  sds: sds1 sds9
Mar  7 14:43:14 storage-01 kernel: [ 4975.507954]  sdt: sdt1 sdt9
Mar  7 14:43:14 storage-01 kernel: [ 4975.694291]  sdu: sdu1 sdu9
Mar  7 14:43:14 storage-01 kernel: [ 4975.857239]  sdv: sdv1 sdv9
Mar  7 14:43:15 storage-01 kernel: [ 4976.238210]  sdw: sdw1 sdw9
Mar  7 14:43:15 storage-01 kernel: [ 4976.634149]  sdx: sdx1 sdx9
Mar  7 14:43:15 storage-01 kernel: [ 4976.807552]  sdy: sdy1 sdy9
Mar  7 14:43:15 storage-01 kernel: [ 4976.932656]  sdz: sdz1 sdz9
Mar  7 14:43:16 storage-01 kernel: [ 4977.090749]  sdaa: sdaa1 sdaa9
Mar  7 14:43:16 storage-01 kernel: [ 4977.150750]  sdb: sdb1 sdb9
Mar  7 14:43:16 storage-01 kernel: [ 4977.313906] SPL: using hostid 0xa8c00902
Mar  7 14:44:33 storage-01 kernel: [ 5054.313750]  zd0: unknown partition table
Mar  7 14:44:33 storage-01 kernel: [ 5054.511903]  zd16: unknown partition table
Mar  7 14:44:33 storage-01 kernel: [ 5054.793147]  zd32: unknown partition table
Mar  7 14:44:33 storage-01 kernel: [ 5054.998966]  zd48: unknown partition table
Mar  7 14:44:34 storage-01 kernel: [ 5055.116743]  zd64: unknown partition table
Mar  7 14:44:34 storage-01 kernel: [ 5055.438710]  zd80: unknown partition table
Mar  7 14:44:34 storage-01 kernel: [ 5055.607664]  zd96: unknown partition table
Mar  7 14:44:34 storage-01 kernel: [ 5055.935195]  zd112: unknown partition table
Mar  7 14:44:35 storage-01 kernel: [ 5056.178689]  zd128: unknown partition table
Mar  7 15:06:58 storage-01 kernel: [ 6399.510933] kjournald starting.  Commit interval 5 seconds
Mar  7 15:06:58 storage-01 kernel: [ 6399.511508] EXT3-fs (zd0): using internal journal
Mar  7 15:06:58 storage-01 kernel: [ 6399.511514] EXT3-fs (zd0): mounted filesystem with ordered data mode
Mar  7 23:52:20 storage-01 kernel: [37921.104143] kswapd0         D ffff880fffd53780     0    53      2 0x00000000
Mar  7 23:52:20 storage-01 kernel: [37921.104147]  ffff880ff8974af0 0000000000000046 0000000000000000 ffff880fbea8c0c0
Mar  7 23:52:20 storage-01 kernel: [37921.104152]  0000000000013780 ffff880fbe7abfd8 ffff880fbe7abfd8 ffff880ff8974af0
Mar  7 23:52:20 storage-01 kernel: [37921.104156]  000000050fcbf018 0000000000000010 0000000000000000 ffffffffa09c0768
Mar  7 23:52:20 storage-01 kernel: [37921.104159] Call Trace:
Mar  7 23:52:20 storage-01 kernel: [37921.104174]  [] ? __mutex_lock_common.isra.5+0xff/0x164
Mar  7 23:52:20 storage-01 kernel: [37921.104178]  [] ? sched_clock+0x5/0x8
Mar  7 23:52:20 storage-01 kernel: [37921.104181]  [] ? mutex_lock+0x1a/0x2d
Mar  7 23:52:20 storage-01 kernel: [37921.104204]  [] ? arc_evict+0x6c/0x49e [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.104208]  [] ? __wake_up_common+0x40/0x77
Mar  7 23:52:20 storage-01 kernel: [37921.104211]  [] ? should_resched+0x5/0x23
Mar  7 23:52:20 storage-01 kernel: [37921.104218]  [] ? spl_slab_reclaim+0x86/0x32e [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.104224]  [] ? bit_waitqueue+0x17/0x75
Mar  7 23:52:20 storage-01 kernel: [37921.104227]  [] ? wake_up_bit+0xd/0x1e
Mar  7 23:52:20 storage-01 kernel: [37921.104237]  [] ? arc_adjust+0x63/0x16e [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.104247]  [] ? arc_kmem_reap_now+0x13/0x91 [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.104258]  [] ? arc_shrinker_func+0x6b/0xed [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.104263]  [] ? shrink_slab+0x18f/0x24d
Mar  7 23:52:20 storage-01 kernel: [37921.104266]  [] ? balance_pgdat+0x274/0x4a3
Mar  7 23:52:20 storage-01 kernel: [37921.104269]  [] ? kswapd+0x2d3/0x303
Mar  7 23:52:20 storage-01 kernel: [37921.104272]  [] ? add_wait_queue+0x3c/0x3c
Mar  7 23:52:20 storage-01 kernel: [37921.104274]  [] ? balance_pgdat+0x4a3/0x4a3
Mar  7 23:52:20 storage-01 kernel: [37921.104277]  [] ? kthread+0x76/0x7e
Mar  7 23:52:20 storage-01 kernel: [37921.104280]  [] ? kernel_thread_helper+0x4/0x10
Mar  7 23:52:20 storage-01 kernel: [37921.104283]  [] ? kthread_worker_fn+0x139/0x139
Mar  7 23:52:20 storage-01 kernel: [37921.104286]  [] ? gs_change+0x13/0x13
Mar  7 23:52:20 storage-01 kernel: [37921.104408] arc_adapt       D ffff880fffdd3780     0  1535      2 0x00000000
Mar  7 23:52:20 storage-01 kernel: [37921.104411]  ffff880fbff069b0 0000000000000046 ffff880f00000000 ffff880ff8e850a0
Mar  7 23:52:20 storage-01 kernel: [37921.104415]  0000000000013780 ffff880f9a25ffd8 ffff880f9a25ffd8 ffff880fbff069b0
Mar  7 23:52:20 storage-01 kernel: [37921.104418]  00000000ffffffff 000000019a25fdc8 ffffffffa09c0660 ffffffffa09c0660
Mar  7 23:52:20 storage-01 kernel: [37921.104422] Call Trace:
Mar  7 23:52:20 storage-01 kernel: [37921.104425]  [] ? __mutex_lock_common.isra.5+0xff/0x164
Mar  7 23:52:20 storage-01 kernel: [37921.104428]  [] ? mutex_lock+0x1a/0x2d
Mar  7 23:52:20 storage-01 kernel: [37921.104433]  [] ? __cv_timedwait_common+0x135/0x1df [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.104436]  [] ? should_resched+0x5/0x23
Mar  7 23:52:20 storage-01 kernel: [37921.104439]  [] ? add_wait_queue+0x3c/0x3c
Mar  7 23:52:20 storage-01 kernel: [37921.104450]  [] ? arc_adjust_meta+0x171/0x171 [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.104461]  [] ? arc_adapt_thread+0x86/0xb4 [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.104466]  [] ? thread_generic_wrapper+0x6a/0x75 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.104471]  [] ? __thread_create+0x2be/0x2be [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.104473]  [] ? kthread+0x76/0x7e
Mar  7 23:52:20 storage-01 kernel: [37921.104476]  [] ? kernel_thread_helper+0x4/0x10
Mar  7 23:52:20 storage-01 kernel: [37921.104479]  [] ? kthread_worker_fn+0x139/0x139
Mar  7 23:52:20 storage-01 kernel: [37921.104481]  [] ? gs_change+0x13/0x13
Mar  7 23:52:20 storage-01 kernel: [37921.104598] zvol/0          D ffff880fffc13780     0  1538      2 0x00000000
Mar  7 23:52:20 storage-01 kernel: [37921.104601]  ffff880fbe4cb750 0000000000000046 0000000000000003 ffff880fbf05e380
Mar  7 23:52:20 storage-01 kernel: [37921.104605]  0000000000013780 ffff880f9a06dfd8 ffff880f9a06dfd8 ffff880fbe4cb750
Mar  7 23:52:20 storage-01 kernel: [37921.104608]  0000000000000202 ffffffff8134d03c ffff880f996b9b28 ffff880f996b9b20
Mar  7 23:52:20 storage-01 kernel: [37921.104612] Call Trace:
Mar  7 23:52:20 storage-01 kernel: [37921.104615]  [] ? _raw_spin_lock_irqsave+0x9/0x25
Mar  7 23:52:20 storage-01 kernel: [37921.104620]  [] ? cv_wait_common+0xdc/0x171 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.104623]  [] ? __wake_up+0x35/0x46
Mar  7 23:52:20 storage-01 kernel: [37921.104625]  [] ? add_wait_queue+0x3c/0x3c
Mar  7 23:52:20 storage-01 kernel: [37921.104652]  [] ? txg_wait_open+0xbe/0xf4 [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.104655]  [] ? should_resched+0x5/0x23
Mar  7 23:52:20 storage-01 kernel: [37921.104670]  [] ? dmu_tx_assign+0xf9/0x345 [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.104685]  [] ? zvol_write+0xc3/0x3be [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.104688]  [] ? __wake_up+0x35/0x46
Mar  7 23:52:20 storage-01 kernel: [37921.104693]  [] ? taskq_thread+0x290/0x3e6 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.104696]  [] ? try_to_wake_up+0x197/0x197
Mar  7 23:52:20 storage-01 kernel: [37921.104701]  [] ? task_done+0x110/0x110 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.104704]  [] ? kthread+0x76/0x7e
Mar  7 23:52:20 storage-01 kernel: [37921.104706]  [] ? kernel_thread_helper+0x4/0x10
Mar  7 23:52:20 storage-01 kernel: [37921.104709]  [] ? kthread_worker_fn+0x139/0x139
Mar  7 23:52:20 storage-01 kernel: [37921.104711]  [] ? gs_change+0x13/0x13
Mar  7 23:52:20 storage-01 kernel: [37921.104828] zvol/1          D ffff880fffc53780     0  1539      2 0x00000000
Mar  7 23:52:20 storage-01 kernel: [37921.104831]  ffff880fbe4642c0 0000000000000046 0000000000000003 ffff880fbf508180
Mar  7 23:52:20 storage-01 kernel: [37921.104835]  0000000000013780 ffff880f9a06ffd8 ffff880f9a06ffd8 ffff880fbe4642c0
Mar  7 23:52:20 storage-01 kernel: [37921.104838]  0000000000000202 ffffffff8134d03c ffff880f996b9b28 ffff880f996b9b20
Mar  7 23:52:20 storage-01 kernel: [37921.104842] Call Trace:
Mar  7 23:52:20 storage-01 kernel: [37921.104844]  [] ? _raw_spin_lock_irqsave+0x9/0x25
Mar  7 23:52:20 storage-01 kernel: [37921.104849]  [] ? cv_wait_common+0xdc/0x171 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.104851]  [] ? __wake_up+0x35/0x46
Mar  7 23:52:20 storage-01 kernel: [37921.104854]  [] ? add_wait_queue+0x3c/0x3c
Mar  7 23:52:20 storage-01 kernel: [37921.104872]  [] ? txg_wait_open+0xbe/0xf4 [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.104875]  [] ? should_resched+0x5/0x23
Mar  7 23:52:20 storage-01 kernel: [37921.104889]  [] ? dmu_tx_assign+0xf9/0x345 [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.104903]  [] ? zvol_write+0xc3/0x3be [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.104906]  [] ? __wake_up+0x35/0x46
Mar  7 23:52:20 storage-01 kernel: [37921.104910]  [] ? taskq_thread+0x290/0x3e6 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.104913]  [] ? try_to_wake_up+0x197/0x197
Mar  7 23:52:20 storage-01 kernel: [37921.104918]  [] ? task_done+0x110/0x110 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.104921]  [] ? kthread+0x76/0x7e
Mar  7 23:52:20 storage-01 kernel: [37921.104923]  [] ? kernel_thread_helper+0x4/0x10
Mar  7 23:52:20 storage-01 kernel: [37921.104926]  [] ? kthread_worker_fn+0x139/0x139
Mar  7 23:52:20 storage-01 kernel: [37921.104928]  [] ? gs_change+0x13/0x13
Mar  7 23:52:20 storage-01 kernel: [37921.105045] zvol/2          D ffff880fffc93780     0  1540      2 0x00000000
Mar  7 23:52:20 storage-01 kernel: [37921.105048]  ffff880fbff07790 0000000000000046 0000000000000003 ffff880f9ec7f550
Mar  7 23:52:20 storage-01 kernel: [37921.105051]  0000000000013780 ffff880f9a775fd8 ffff880f9a775fd8 ffff880fbff07790
Mar  7 23:52:20 storage-01 kernel: [37921.105055]  0000000000000202 ffffffff8134d03c ffff880f996b9b28 ffff880f996b9b20
Mar  7 23:52:20 storage-01 kernel: [37921.105058] Call Trace:
Mar  7 23:52:20 storage-01 kernel: [37921.105061]  [] ? _raw_spin_lock_irqsave+0x9/0x25
Mar  7 23:52:20 storage-01 kernel: [37921.105066]  [] ? cv_wait_common+0xdc/0x171 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.105068]  [] ? __wake_up+0x35/0x46
Mar  7 23:52:20 storage-01 kernel: [37921.105071]  [] ? add_wait_queue+0x3c/0x3c
Mar  7 23:52:20 storage-01 kernel: [37921.105089]  [] ? txg_wait_open+0xbe/0xf4 [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.105092]  [] ? should_resched+0x5/0x23
Mar  7 23:52:20 storage-01 kernel: [37921.105106]  [] ? dmu_tx_assign+0xf9/0x345 [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.105119]  [] ? zvol_write+0xc3/0x3be [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.105122]  [] ? __wake_up+0x35/0x46
Mar  7 23:52:20 storage-01 kernel: [37921.105127]  [] ? taskq_thread+0x290/0x3e6 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.105130]  [] ? try_to_wake_up+0x197/0x197
Mar  7 23:52:20 storage-01 kernel: [37921.105134]  [] ? task_done+0x110/0x110 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.105137]  [] ? kthread+0x76/0x7e
Mar  7 23:52:20 storage-01 kernel: [37921.105140]  [] ? kernel_thread_helper+0x4/0x10
Mar  7 23:52:20 storage-01 kernel: [37921.105142]  [] ? kthread_worker_fn+0x139/0x139
Mar  7 23:52:20 storage-01 kernel: [37921.105145]  [] ? gs_change+0x13/0x13
Mar  7 23:52:20 storage-01 kernel: [37921.105261] zvol/3          D ffff880fffcd3780     0  1541      2 0x00000000
Mar  7 23:52:20 storage-01 kernel: [37921.105264]  ffff880fbd10c770 0000000000000046 0000000000000000 ffff880ff8deef60
Mar  7 23:52:20 storage-01 kernel: [37921.105268]  0000000000013780 ffff880f9a777fd8 ffff880f9a777fd8 ffff880fbd10c770
Mar  7 23:52:20 storage-01 kernel: [37921.105271]  0000000000000202 000000018134d03c ffff880f996b9b28 ffff880f996b9b20
Mar  7 23:52:20 storage-01 kernel: [37921.105275] Call Trace:
Mar  7 23:52:20 storage-01 kernel: [37921.105280]  [] ? cv_wait_common+0xdc/0x171 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.105282]  [] ? __wake_up+0x35/0x46
Mar  7 23:52:20 storage-01 kernel: [37921.105285]  [] ? add_wait_queue+0x3c/0x3c
Mar  7 23:52:20 storage-01 kernel: [37921.105303]  [] ? txg_wait_open+0xbe/0xf4 [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.105306]  [] ? should_resched+0x5/0x23
Mar  7 23:52:20 storage-01 kernel: [37921.105320]  [] ? dmu_tx_assign+0xf9/0x345 [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.105334]  [] ? zvol_write+0xc3/0x3be [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.105336]  [] ? __wake_up+0x35/0x46
Mar  7 23:52:20 storage-01 kernel: [37921.105341]  [] ? taskq_thread+0x290/0x3e6 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.105344]  [] ? try_to_wake_up+0x197/0x197
Mar  7 23:52:20 storage-01 kernel: [37921.105349]  [] ? task_done+0x110/0x110 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.105351]  [] ? kthread+0x76/0x7e
Mar  7 23:52:20 storage-01 kernel: [37921.105354]  [] ? kernel_thread_helper+0x4/0x10
Mar  7 23:52:20 storage-01 kernel: [37921.105357]  [] ? kthread_worker_fn+0x139/0x139
Mar  7 23:52:20 storage-01 kernel: [37921.105359]  [] ? gs_change+0x13/0x13
Mar  7 23:52:20 storage-01 kernel: [37921.105490] zvol/4          D ffff880fffd13780     0  1542      2 0x00000000
Mar  7 23:52:20 storage-01 kernel: [37921.105494]  ffff880fbe495160 0000000000000046 0000000000000000 ffff880ff8e008b0
Mar  7 23:52:20 storage-01 kernel: [37921.105497]  0000000000013780 ffff880f9a6f3fd8 ffff880f9a6f3fd8 ffff880fbe495160
Mar  7 23:52:20 storage-01 kernel: [37921.105501]  0000000000000202 000000018134d03c ffff880f996b9b28 ffff880f996b9b20
Mar  7 23:52:20 storage-01 kernel: [37921.105504] Call Trace:
Mar  7 23:52:20 storage-01 kernel: [37921.105509]  [] ? cv_wait_common+0xdc/0x171 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.105511]  [] ? __wake_up+0x35/0x46
Mar  7 23:52:20 storage-01 kernel: [37921.105514]  [] ? add_wait_queue+0x3c/0x3c
Mar  7 23:52:20 storage-01 kernel: [37921.105532]  [] ? txg_wait_open+0xbe/0xf4 [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.105535]  [] ? should_resched+0x5/0x23
Mar  7 23:52:20 storage-01 kernel: [37921.105549]  [] ? dmu_tx_assign+0xf9/0x345 [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.105563]  [] ? zvol_write+0xc3/0x3be [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.105565]  [] ? __wake_up+0x35/0x46
Mar  7 23:52:20 storage-01 kernel: [37921.105570]  [] ? taskq_thread+0x290/0x3e6 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.105573]  [] ? try_to_wake_up+0x197/0x197
Mar  7 23:52:20 storage-01 kernel: [37921.105578]  [] ? task_done+0x110/0x110 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.105580]  [] ? kthread+0x76/0x7e
Mar  7 23:52:20 storage-01 kernel: [37921.105583]  [] ? kernel_thread_helper+0x4/0x10
Mar  7 23:52:20 storage-01 kernel: [37921.105586]  [] ? kthread_worker_fn+0x139/0x139
Mar  7 23:52:20 storage-01 kernel: [37921.105588]  [] ? gs_change+0x13/0x13
Mar  7 23:52:20 storage-01 kernel: [37921.105746] zvol/5          D ffff880fffd53780     0  1543      2 0x00000000
Mar  7 23:52:20 storage-01 kernel: [37921.105749]  ffff880fbfe0aa70 0000000000000046 0000000000000003 ffff880fbfcaf710
Mar  7 23:52:20 storage-01 kernel: [37921.105753]  0000000000013780 ffff880f9a65dfd8 ffff880f9a65dfd8 ffff880fbfe0aa70
Mar  7 23:52:20 storage-01 kernel: [37921.105756]  0000000000000202 ffffffff8134d03c ffff880f996b9b28 ffff880f996b9b20
Mar  7 23:52:20 storage-01 kernel: [37921.105760] Call Trace:
Mar  7 23:52:20 storage-01 kernel: [37921.105762]  [] ? _raw_spin_lock_irqsave+0x9/0x25
Mar  7 23:52:20 storage-01 kernel: [37921.105767]  [] ? cv_wait_common+0xdc/0x171 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.105769]  [] ? __wake_up+0x35/0x46
Mar  7 23:52:20 storage-01 kernel: [37921.105772]  [] ? add_wait_queue+0x3c/0x3c
Mar  7 23:52:20 storage-01 kernel: [37921.105790]  [] ? txg_wait_open+0xbe/0xf4 [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.105793]  [] ? should_resched+0x5/0x23
Mar  7 23:52:20 storage-01 kernel: [37921.105807]  [] ? dmu_tx_assign+0xf9/0x345 [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.105820]  [] ? zvol_write+0xc3/0x3be [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.105823]  [] ? __wake_up+0x35/0x46
Mar  7 23:52:20 storage-01 kernel: [37921.105828]  [] ? taskq_thread+0x290/0x3e6 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.105831]  [] ? try_to_wake_up+0x197/0x197
Mar  7 23:52:20 storage-01 kernel: [37921.105835]  [] ? task_done+0x110/0x110 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.105838]  [] ? kthread+0x76/0x7e
Mar  7 23:52:20 storage-01 kernel: [37921.105841]  [] ? kernel_thread_helper+0x4/0x10
Mar  7 23:52:20 storage-01 kernel: [37921.105843]  [] ? kthread_worker_fn+0x139/0x139
Mar  7 23:52:20 storage-01 kernel: [37921.105846]  [] ? gs_change+0x13/0x13
Mar  7 23:52:20 storage-01 kernel: [37921.106003] zvol/6          D ffff880fffd93780     0  1544      2 0x00000000
Mar  7 23:52:20 storage-01 kernel: [37921.106007]  ffff880fbee729f0 0000000000000046 0000000000000003 ffff880f991d96d0
Mar  7 23:52:20 storage-01 kernel: [37921.106010]  0000000000013780 ffff880f9a65ffd8 ffff880f9a65ffd8 ffff880fbee729f0
Mar  7 23:52:20 storage-01 kernel: [37921.106014]  0000000000000202 ffffffff8134d03c ffff880f996b9b28 ffff880f996b9b20
Mar  7 23:52:20 storage-01 kernel: [37921.106017] Call Trace:
Mar  7 23:52:20 storage-01 kernel: [37921.106020]  [] ? _raw_spin_lock_irqsave+0x9/0x25
Mar  7 23:52:20 storage-01 kernel: [37921.106024]  [] ? cv_wait_common+0xdc/0x171 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.106027]  [] ? __wake_up+0x35/0x46
Mar  7 23:52:20 storage-01 kernel: [37921.106030]  [] ? add_wait_queue+0x3c/0x3c
Mar  7 23:52:20 storage-01 kernel: [37921.106048]  [] ? txg_wait_open+0xbe/0xf4 [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.106050]  [] ? should_resched+0x5/0x23
Mar  7 23:52:20 storage-01 kernel: [37921.106065]  [] ? dmu_tx_assign+0xf9/0x345 [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.106078]  [] ? zvol_write+0xc3/0x3be [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.106081]  [] ? __wake_up+0x35/0x46
Mar  7 23:52:20 storage-01 kernel: [37921.106086]  [] ? taskq_thread+0x290/0x3e6 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.106088]  [] ? try_to_wake_up+0x197/0x197
Mar  7 23:52:20 storage-01 kernel: [37921.106093]  [] ? task_done+0x110/0x110 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.106096]  [] ? kthread+0x76/0x7e
Mar  7 23:52:20 storage-01 kernel: [37921.106099]  [] ? kernel_thread_helper+0x4/0x10
Mar  7 23:52:20 storage-01 kernel: [37921.106101]  [] ? kthread_worker_fn+0x139/0x139
Mar  7 23:52:20 storage-01 kernel: [37921.106104]  [] ? gs_change+0x13/0x13
Mar  7 23:52:20 storage-01 kernel: [37921.106261] zvol/7          D ffff880fffdd3780     0  1545      2 0x00000000
Mar  7 23:52:20 storage-01 kernel: [37921.106264]  ffff880fbf8c6870 0000000000000046 0000000000000003 ffff880fbeaf2ab0
Mar  7 23:52:20 storage-01 kernel: [37921.106268]  0000000000013780 ffff880f9a543fd8 ffff880f9a543fd8 ffff880fbf8c6870
Mar  7 23:52:20 storage-01 kernel: [37921.106271]  0000000000000202 ffffffff8134d03c ffff880f996b9b28 ffff880f996b9b20
Mar  7 23:52:20 storage-01 kernel: [37921.106275] Call Trace:
Mar  7 23:52:20 storage-01 kernel: [37921.106277]  [] ? _raw_spin_lock_irqsave+0x9/0x25
Mar  7 23:52:20 storage-01 kernel: [37921.106282]  [] ? cv_wait_common+0xdc/0x171 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.106284]  [] ? __wake_up+0x35/0x46
Mar  7 23:52:20 storage-01 kernel: [37921.106287]  [] ? add_wait_queue+0x3c/0x3c
Mar  7 23:52:20 storage-01 kernel: [37921.106305]  [] ? txg_wait_open+0xbe/0xf4 [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.106308]  [] ? should_resched+0x5/0x23
Mar  7 23:52:20 storage-01 kernel: [37921.106322]  [] ? dmu_tx_assign+0xf9/0x345 [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.106336]  [] ? zvol_write+0xc3/0x3be [zfs]
Mar  7 23:52:20 storage-01 kernel: [37921.106338]  [] ? __wake_up+0x35/0x46
Mar  7 23:52:20 storage-01 kernel: [37921.106343]  [] ? taskq_thread+0x290/0x3e6 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.106346]  [] ? try_to_wake_up+0x197/0x197
Mar  7 23:52:20 storage-01 kernel: [37921.106351]  [] ? task_done+0x110/0x110 [spl]
Mar  7 23:52:20 storage-01 kernel: [37921.106353]  [] ? kthread+0x76/0x7e
Mar  7 23:52:20 storage-01 kernel: [37921.106356]  [] ? kernel_thread_helper+0x4/0x10
Mar  7 23:52:20 storage-01 kernel: [37921.106359]  [] ? kthread_worker_fn+0x139/0x139
Mar  7 23:52:20 storage-01 kernel: [37921.106361]  [] ? gs_change+0x13/0x13

Test construct (setup zvols):

COUNT=0; for i in {512,1k,2k,4k,8k,16k,32k,64k,128k}; do echo "zfs create -V 1T -b ${i} storage01/zvol${COUNT}" && zfs create -V 1T -b ${i} storage01/zvol${COUNT}; COUNT=`expr $COUNT + 1`; done

Test script:

ZDLIST='zd0 zd16 zd32 zd48 zd64 zd80 zd96 zd112 zd128'
for i in ${ZDLIST[@]}
do
  if [ "$BLOCKSIZE" == "1024" ]; then
    BLOCKSIZE=1
    BLOCKTYPE='k'
  fi

  echo "*** Performing tests on ${i}, blocksize = ${BLOCKSIZE}${BLOCKTYPE} ***"
  echo "Syncing file systems"
  sync
  echo "Droping pagecache, dentries, and inodes"
  echo '3' > /proc/sys/vm/drop_caches
  echo "Sleeping for 5 seconds"
  sleep 5
  echo "Making EXT3 file system with DEFAULTS on /dev/${i}"
  #echo "mke2fs -j -E lazy_itable_init=0 -E lazy_journal_init=0 /dev/${i}"
  mke2fs -j -E lazy_itable_init=0 -E lazy_journal_init=0 /dev/${i}
  echo "Finished making EXT3 file system on /dev/${i}"
  echo "Sleeping for 5 seconds"
  sleep 5
  echo "Making mount point and mounting file system /dev/${i} at /mnt/ggio-${i}"

  if [ ! -d /mnt/ggio-${i} ]; then
    mkdir /mnt/ggio-${i}
    echo "/mnt/ggio-${i} created"
  else
    echo "/mnt/ggio-${i} exists"
  fi
  if [ $(cat /proc/self/mounts|grep -c ${i}) ]; then
    mount /dev/${i} /mnt/ggio-${i} || exit 1
    echo "/dev/${i} mounted at /mnt/ggio-${i}"
  else
    echo "/dev/${i} is already mounted... exiting"
    exit 1
  fi

  echo "*** Starting performance tests on ${i} ***"
  #echo "perl battery_of_tests.pl ${i}"
  perl battery_of_tests.pl ${i}
  echo "*** Completed performance tests on ${i} ***"
  echo "Allowing file system to settle for 5 seconds"
  sleep 5
  echo "Syncing file systems"
  sync
  echo "Droping pagecache, dentries, and inodes"
  echo '3' > /proc/sys/vm/drop_caches
  echo "Sleeping for 5 seconds"
  sleep 5
  echo "Unmounting /dev/${i}"

  if [ $(cat /proc/self/mounts|grep -c ${i}) ]; then
    umount /dev/${i} || exit 1
    echo "/dev/${i} successfully unmounted"
  else
    echo "/dev/${i} cannot be unmounted... exiting"
    exit 1
  fi

  if [ -d /mnt/ggio-${i} ]; then
    rmdir /mnt/ggio-${i} || exit 1
    echo "/mnt/ggio-${i} successfully removed"
  else
    echo "/mnt/ggio-${i} doesn't exist"
  fi

  BLOCKSIZE=`expr $BLOCKSIZE \* 2`

  echo "Cleanup tasks completed..."
done

Current iostat activity:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00   12.51   25.01    0.00   62.48

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdc               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdd               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sde               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdf               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdg               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdh               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdj               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdm               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdi               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdo               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdn               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdl               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdp               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdt               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdr               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdq               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sds               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdv               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdu               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdk               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdy               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdaa              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdx               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdz               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdw               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
zd0               0.00     0.00    0.00    0.00     0.00     0.00     0.00   193.00    0.00    0.00    0.00   0.00 100.00
zd16              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
zd32              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
zd48              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
zd64              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
zd80              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
zd96              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
zd112             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
zd128             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

Current arcstats:

# cat /proc/spl/kstat/zfs/arcstats
4 1 0x01 80 3840 14257871852 65995615989596
name                            type data
hits                            4    430990416
misses                          4    112183821
demand_data_hits                4    189327529
demand_data_misses              4    61192921
demand_metadata_hits            4    223860770
demand_metadata_misses          4    6600571
prefetch_data_hits              4    17802063
prefetch_data_misses            4    44390092
prefetch_metadata_hits          4    54
prefetch_metadata_misses        4    237
mru_hits                        4    112122760
mru_ghost_hits                  4    8905488
mfu_hits                        4    304807750
mfu_ghost_hits                  4    2673143
deleted                         4    383285795
recycle_miss                    4    216194510
mutex_miss                      4    430387754
evict_skip                      4    37337854743
evict_l2_cached                 4    137713318912
evict_l2_eligible               4    163665937408
evict_l2_ineligible             4    3784519168
hash_elements                   4    149877787
hash_elements_max               4    149910430
hash_collisions                 4    440449975
hash_chains                     4    1048576
hash_chain_max                  4    205
p                               4    270384325
c                               4    4315837952
c_min                           4    4294967296
c_max                           4    34359738368
size                            4    40878767824
hdr_size                        4    119932704
data_size                       4    56556032
other_size                      4    22746800
anon_size                       4    38455296
anon_evict_data                 4    0
anon_evict_metadata             4    0
mru_size                        4    6908416
mru_evict_data                  4    4637184
mru_evict_metadata              4    999424
mru_ghost_size                  4    4292341760
mru_ghost_evict_data            4    21389312
mru_ghost_evict_metadata        4    4270952448
mfu_size                        4    11192320
mfu_evict_data                  4    0
mfu_evict_metadata              4    3735552
mfu_ghost_size                  4    49664
mfu_ghost_evict_data            4    0
mfu_ghost_evict_metadata        4    49664
l2_hits                         4    18422025
l2_misses                       4    93761796
l2_feeds                        4    41332
l2_rw_clash                     4    190185
l2_read_bytes                   4    49812476928
l2_write_bytes                  4    111612105216
l2_writes_sent                  4    39659
l2_writes_done                  4    39659
l2_writes_error                 4    0
l2_writes_hdr_miss              4    890620
l2_evict_lock_retry             4    0
l2_evict_reading                4    0
l2_free_on_write                4    11161858
l2_abort_lowmem                 4    12
l2_cksum_bad                    4    0
l2_io_error                     4    0
l2_size                         4    81021397504
l2_hdr_size                     4    43072445952
memory_throttle_count           4    0
duplicate_buffers               4    0
duplicate_buffers_size          4    0
duplicate_reads                 4    0
memory_direct_count             4    344
memory_indirect_count           4    458036
arc_no_grow                     4    0
arc_tempreserve                 4    0
arc_loaned_bytes                4    0
arc_prune                       4    12319824
arc_meta_used                   4    40837831888
arc_meta_limit                  4    8589934592
arc_meta_max                    4    40838002976

zpool events -v:

root@storage-01:~# zpool events -v
TIME                           CLASS
Mar  7 2013 14:43:16.118692000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0x58069f8b8fd748
        vdev_state = 0x7
        time = 0x51391804 0x71318a0

Mar  7 2013 14:43:16.119159000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0x90d7716fa3c0e340
        vdev_state = 0x7
        time = 0x51391804 0x71a38d8

Mar  7 2013 14:43:16.119177000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0x1c3f83b4d2280807
        vdev_state = 0x7
        time = 0x51391804 0x71a7f28

Mar  7 2013 14:43:16.119175000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0x6a5a5eb959419ffd
        vdev_state = 0x7
        time = 0x51391804 0x71a7758

Mar  7 2013 14:43:16.119178000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0x568ba95286cb3647
        vdev_state = 0x7
        time = 0x51391804 0x71a8310

Mar  7 2013 14:43:16.119207000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0x6503b63e371cc63
        vdev_state = 0x7
        time = 0x51391804 0x71af458

Mar  7 2013 14:43:16.119220000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0x4aeca755841b5cbd
        vdev_state = 0x7
        time = 0x51391804 0x71b2720

Mar  7 2013 14:43:16.119384000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0x326b4ce394ae3962
        vdev_state = 0x7
        time = 0x51391804 0x71da7c0

Mar  7 2013 14:43:16.119389000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0xefb4bdfd3a245fa4
        vdev_state = 0x7
        time = 0x51391804 0x71dbb48

Mar  7 2013 14:43:16.119393000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0x5ab65342a07bc2f1
        vdev_state = 0x7
        time = 0x51391804 0x71dcae8

Mar  7 2013 14:43:16.119395000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0x5dee7a4a59bb92a8
        vdev_state = 0x7
        time = 0x51391804 0x71dd2b8

Mar  7 2013 14:43:16.119399000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0x945398f143098e45
        vdev_state = 0x7
        time = 0x51391804 0x71de258

Mar  7 2013 14:43:16.119406000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0x56322ab3eb8d11f4
        vdev_state = 0x7
        time = 0x51391804 0x71dfdb0

Mar  7 2013 14:43:16.119499000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0x7c9d471c1faea7b4
        vdev_state = 0x7
        time = 0x51391804 0x71f68f8

Mar  7 2013 14:43:16.119520000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0x4ca5e4e939da033d
        vdev_state = 0x7
        time = 0x51391804 0x71fbb00

Mar  7 2013 14:43:16.119534000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0x21ddbdd536dda72f
        vdev_state = 0x7
        time = 0x51391804 0x71ff1b0

Mar  7 2013 14:43:16.119540000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0xbf7a8d7530f37dc7
        vdev_state = 0x7
        time = 0x51391804 0x7200920

Mar  7 2013 14:43:16.119542000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0xfcc6ab344db5d17b
        vdev_state = 0x7
        time = 0x51391804 0x72010f0

Mar  7 2013 14:43:16.119551000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0x9e99ba25cf93f0d6
        vdev_state = 0x7
        time = 0x51391804 0x7203418

Mar  7 2013 14:43:16.119707000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0xdc2043ac6c8aa0c
        vdev_state = 0x7
        time = 0x51391804 0x7229578

Mar  7 2013 14:43:16.119726000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0x4fd0e7304edaaf2
        vdev_state = 0x7
        time = 0x51391804 0x722dfb0

Mar  7 2013 14:43:16.119755000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0x9509bae2deb68f31
        vdev_state = 0x7
        time = 0x51391804 0x72350f8

Mar  7 2013 14:43:16.119772000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0xef2ce736d563b9bc
        vdev_state = 0x7
        time = 0x51391804 0x7239360

Mar  7 2013 14:43:16.119789000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0x6f36b4058be3e08c
        vdev_state = 0x7
        time = 0x51391804 0x723d5c8

Mar  7 2013 14:43:16.119790000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0xb606b1b8d7783859
        vdev_state = 0x7
        time = 0x51391804 0x723d9b0

Mar  7 2013 14:43:17.469870000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0x8cc2d6b4196ef8c6
        vdev_state = 0x7
        time = 0x51391805 0x1c01a5b0

Mar  7 2013 14:43:17.498584000 resource.fs.zfs.statechange
        version = 0x0
        class = "resource.fs.zfs.statechange"
        pool_guid = 0x7e383b6c6102fd1c
        vdev_guid = 0x8cc2d6b4196ef8c6
        vdev_state = 0x7
        time = 0x51391805 0x1db7c9c0

Mar  7 2013 14:43:17.707312000 ereport.fs.zfs.config.sync
        class = "ereport.fs.zfs.config.sync"
        ena = 0x487373b60b401801
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x7e383b6c6102fd1c
        (end detector)
        pool = "storage01"
        pool_guid = 0x7e383b6c6102fd1c
        pool_context = 0x0
        pool_failmode = "wait"
        time = 0x51391805 0x2a28b980
mengesb commented 11 years ago 
root@storage-01:~# zdb -i storage01
Dataset mos [META], ID 0, cr_txg 4, 2.21M, 98 objects
Dataset storage01/zvol4 [ZVOL], ID 70, cr_txg 28, 32.0K, 2 objects
Dataset storage01/zvol3 [ZVOL], ID 64, cr_txg 26, 32.0K, 2 objects
Dataset storage01/zvol7 [ZVOL], ID 89, cr_txg 34, 32.0K, 2 objects
Dataset storage01/zvol0 [ZVOL], ID 45, cr_txg 20, 66.6G, 2 objects
    ZIL header: claim_txg 0, claim_blk_seq 0, claim_lr_seq 0 replay_seq 0, flags 0x0
Dataset storage01/zvol8 [ZVOL], ID 95, cr_txg 36, 32.0K, 2 objects
Dataset storage01/zvol2 [ZVOL], ID 58, cr_txg 24, 32.0K, 2 objects
Dataset storage01/zvol5 [ZVOL], ID 76, cr_txg 30, 32.0K, 2 objects
Dataset storage01/zvol1 [ZVOL], ID 52, cr_txg 22, 32.0K, 2 objects
Dataset storage01/zvol6 [ZVOL], ID 82, cr_txg 32, 32.0K, 2 objects
Dataset storage01 [ZPL], ID 21, cr_txg 1, 53.9K, 6 objects

root@storage-01:~# zdb -v
storage01:
    version: 5000
    name: 'storage01'
    state: 0
    txg: 4
    pool_guid: 9095084784144809244
    hostid: 2831157506
    hostname: 'storage-01'
    vdev_children: 5
    vdev_tree:
        type: 'root'
        id: 0
        guid: 9095084784144809244
        create_txg: 4
        children[0]:
            type: 'raidz'
            id: 0
            guid: 9282155731958840736
            nparity: 2
            metaslab_array: 40
            metaslab_shift: 36
            ashift: 9
            asize: 12002308128768
            is_log: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 991359518464584204
                path: '/dev/disk/by-vdev/slot00-part1'
                whole_disk: 1
                create_txg: 4
            children[1]:
                type: 'disk'
                id: 1
                guid: 359459432426220274
                path: '/dev/disk/by-vdev/slot01-part1'
                whole_disk: 1
                create_txg: 4
            children[2]:
                type: 'disk'
                id: 2
                guid: 10739320270003539761
                path: '/dev/disk/by-vdev/slot02-part1'
                whole_disk: 1
                create_txg: 4
            children[3]:
                type: 'disk'
                id: 3
                guid: 13116366372210227289
                path: '/dev/disk/by-vdev/slot03-part1'
                whole_disk: 1
                create_txg: 4
            children[4]:
                type: 'disk'
                id: 4
                guid: 8013790522867179660
                path: '/dev/disk/by-vdev/slot04-part1'
                whole_disk: 1
                create_txg: 4
            children[5]:
                type: 'disk'
                id: 5
                guid: 17234404096734378428
                path: '/dev/disk/by-vdev/slot05-part1'
                whole_disk: 1
                create_txg: 4
        children[1]:
            type: 'raidz'
            id: 1
            guid: 9244192410313874945
            nparity: 2
            metaslab_array: 38
            metaslab_shift: 36
            ashift: 9
            asize: 12002308128768
            is_log: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 6536503454800265969
                path: '/dev/disk/by-vdev/slot06-part1'
                whole_disk: 1
                create_txg: 4
            children[1]:
                type: 'disk'
                id: 1
                guid: 3633082064740170082
                path: '/dev/disk/by-vdev/slot07-part1'
                whole_disk: 1
                create_txg: 4
            children[2]:
                type: 'disk'
                id: 2
                guid: 17272639366172532644
                path: '/dev/disk/by-vdev/slot08-part1'
                whole_disk: 1
                create_txg: 4
            children[3]:
                type: 'disk'
                id: 3
                guid: 10688054502659558981
                path: '/dev/disk/by-vdev/slot09-part1'
                whole_disk: 1
                create_txg: 4
            children[4]:
                type: 'disk'
                id: 4
                guid: 6211073788336738804
                path: '/dev/disk/by-vdev/slot10-part1'
                whole_disk: 1
                create_txg: 4
            children[5]:
                type: 'disk'
                id: 5
                guid: 6768481749736067752
                path: '/dev/disk/by-vdev/slot11-part1'
                whole_disk: 1
                create_txg: 4
        children[2]:
            type: 'raidz'
            id: 2
            guid: 8775653542431708525
            nparity: 2
            metaslab_array: 37
            metaslab_shift: 36
            ashift: 9
            asize: 12002308128768
            is_log: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 8979411418162833332
                path: '/dev/disk/by-vdev/slot12-part1'
                whole_disk: 1
                create_txg: 4
            children[1]:
                type: 'disk'
                id: 1
                guid: 13797495942874889671
                path: '/dev/disk/by-vdev/slot13-part1'
                whole_disk: 1
                create_txg: 4
            children[2]:
                type: 'disk'
                id: 2
                guid: 18214433984076960123
                path: '/dev/disk/by-vdev/slot14-part1'
                whole_disk: 1
                create_txg: 4
            children[3]:
                type: 'disk'
                id: 3
                guid: 5523072208388883261
                path: '/dev/disk/by-vdev/slot15-part1'
                whole_disk: 1
                create_txg: 4
            children[4]:
                type: 'disk'
                id: 4
                guid: 11428370200988479702
                path: '/dev/disk/by-vdev/slot16-part1'
                whole_disk: 1
                create_txg: 4
            children[5]:
                type: 'disk'
                id: 5
                guid: 2440315296550856495
                path: '/dev/disk/by-vdev/slot17-part1'
                whole_disk: 1
                create_txg: 4
        children[3]:
            type: 'raidz'
            id: 3
            guid: 16092284661228798865
            nparity: 2
            metaslab_array: 35
            metaslab_shift: 36
            ashift: 9
            asize: 12002308128768
            is_log: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 10436935385757049664
                path: '/dev/disk/by-vdev/slot18-part1'
                whole_disk: 1
                create_txg: 4
            children[1]:
                type: 'disk'
                id: 1
                guid: 7663541866083753981
                path: '/dev/disk/by-vdev/slot19-part1'
                whole_disk: 1
                create_txg: 4
            children[2]:
                type: 'disk'
                id: 2
                guid: 6236264280938460743
                path: '/dev/disk/by-vdev/slot20-part1'
                whole_disk: 1
                create_txg: 4
            children[3]:
                type: 'disk'
                id: 3
                guid: 2035490369237944327
                path: '/dev/disk/by-vdev/slot21-part1'
                whole_disk: 1
                create_txg: 4
            children[4]:
                type: 'disk'
                id: 4
                guid: 5398874039040826557
                path: '/dev/disk/by-vdev/slot22-part1'
                whole_disk: 1
                create_txg: 4
            children[5]:
                type: 'disk'
                id: 5
                guid: 454928862568107107
                path: '/dev/disk/by-vdev/slot23-part1'
                whole_disk: 1
                create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 24777080261564232
            path: '/dev/disk/by-vdev/sata0-part1'
            whole_disk: 0
            metaslab_array: 34
            metaslab_shift: 30
            ashift: 9
            asize: 127993905152
            is_log: 1
            create_txg: 4
    features_for_read:

root@storage-01:~# zpool status -v
  pool: storage01
 state: ONLINE
  scan: none requested
config:
        NAME           STATE     READ WRITE CKSUM
        storage01      ONLINE       0     0     0
          raidz2-0     ONLINE       0     0     0
            slot00     ONLINE       0     0     0
            slot01     ONLINE       0     0     0
            slot02     ONLINE       0     0     0
            slot03     ONLINE       0     0     0
            slot04     ONLINE       0     0     0
            slot05     ONLINE       0     0     0
          raidz2-1     ONLINE       0     0     0
            slot06     ONLINE       0     0     0
            slot07     ONLINE       0     0     0
            slot08     ONLINE       0     0     0
            slot09     ONLINE       0     0     0
            slot10     ONLINE       0     0     0
            slot11     ONLINE       0     0     0
          raidz2-2     ONLINE       0     0     0
            slot12     ONLINE       0     0     0
            slot13     ONLINE       0     0     0
            slot14     ONLINE       0     0     0
            slot15     ONLINE       0     0     0
            slot16     ONLINE       0     0     0
            slot17     ONLINE       0     0     0
          raidz2-3     ONLINE       0     0     0
            slot18     ONLINE       0     0     0
            slot19     ONLINE       0     0     0
            slot20     ONLINE       0     0     0
            slot21     ONLINE       0     0     0
            slot22     ONLINE       0     0     0
            slot23     ONLINE       0     0     0
        logs
          sata0-part1  ONLINE       0     0     0
        cache
          sata2        ONLINE       0     0     0
errors: No known data errors
mengesb commented 11 years ago

The 'battery_of_tests.pl' script runs several IOZone tests and a bonnie++ test. The important segment of the script:

&single_thread_iozone(4,4);
&single_thread_iozone(4,64);
&single_thread_iozone(4,256);
&single_thread_iozone(16,4);
&single_thread_iozone(16,64);
&single_thread_iozone(16,256);
&multi_thread_iozone(4,64);
&bonnie_test();

The first digit is the size of the test in GB, the second is the size of the record in KB.

The first pass I was able to get through several zvols while testing, then my system locked up more or less during a drive replacement ... when testing resumed, I cannot even get past the first zvol (zd0)... and after 24 hours I haven't even completed the 16G/256K record test... whereas with my first pass I was able to get almost complete (died on the multi threaded iozon test on zd128 (the last zvol)

mengesb commented 11 years ago

Here are all the results from the first pass

root@storage-01:~# bash ext3.tests
##########
Sleeping for 5 seconds
Dropping pagecache, dentries, and inodes
Sleeping for 5 seconds
Making EXT3 file system with DEFAULTS on /dev/zd0
mke2fs 1.42.5 (29-Jul-2012)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=2 blocks, Stripe width=2 blocks
67108864 inodes, 268435456 blocks
13421772 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
8192 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
Testing /dev/zd0
\n\n
KERNEL:
3.2.0-4-amd64
DISK CONTROLLER:
3ware Inc 9650SE SATA-II RAID PCIe (rev 01)
HBA CONTROLLER:
LSI Logic / Symbios Logic SAS2004 PCI-Express Fusion-MPT SAS-2 [Spitfire] (rev 03)
NET INTERFACE(S):
Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
CPU TYPE (8 Cores):
Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz
I/O SCHEDULER:
[noop] deadline cfq
Beginning iozone test with 4 GB file and 4 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 4
Writes: 165683
Reads: 1055246
Re-Reads: 2218387
Random Reads: 1669245
Random Writes: 1684677
#################################################
Beginning iozone test with 4 GB file and 64 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 64
Writes: 345360
Reads: 1060578
Re-Reads: 1559039
Random Reads: 2340211
Random Writes: 2127273
#################################################
Beginning iozone test with 4 GB file and 256 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 256
Writes: 394878
Reads: 1011778
Re-Reads: 2335070
Random Reads: 1529441
Random Writes: 2259175
#################################################
Beginning iozone test with 16 GB file and 4 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 4
Writes: 106368
Reads: 101197
Re-Reads: 2094634
Random Reads: 2221879
Random Writes: 1612742
#################################################
Beginning iozone test with 16 GB file and 64 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 64
Writes: 101729
Reads: 113528
Re-Reads: 2132065
Random Reads: 2333749
Random Writes: 2158364
#################################################
Beginning iozone test with 16 GB file and 256 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 256
Writes: 104548
Reads: 115470
Re-Reads: 2084149
Random Reads: 2294355
Random Writes: 2277694
#################################################
Beginning Multi-Thread iozone test with 4 GB file and 64 KB record length.
This can take a while, depending on specified file size...DONE
####### IOZONE MULTI-THREAD TEST RESULTS ########
File Size (GB): 4
Record Length (KB): 64
Writes:    11862.15 KB/sec
Reads:   270673.70 KB/sec
Re-Reads:   152070.65 KB/sec
Random Reads:   864900.91 KB/sec
Random Writes:    88912.71 KB/sec
#################################################
Beginning the Bonnie++ Test.
/usr/sbin/bonnie++ -u root -d /mnt/ggio-zd0 2> /dev/null
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-                                                                                                  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--                                                                                                  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP                                                                                                  storage-01     125G   633  98 82560  47 31352   6  3433  95 59208   4  1199  12                                                                                                  Latency             19112us   23045ms    1038ms   10737us     577ms     116ms Ve                                                                                                 rsion  1.96       ------Sequential Create------ --------Random Create-------- st                                                                                                 orage-01          -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--                                                                                                                files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP                                                                                                                   16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ La                                                                                                 tency             25473us    9385us    4125us   25607us      29us    1382us 1.96                                                                                                 ,1.96,storage-01,1,1362086310,125G,,633,98,82560,47,31352,6,3433,95,59208,4,1199                                                                                                 ,12,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,19112us,2                                                                                                 3045ms,1038ms,10737us,577ms,116ms,25473us,9385us,4125us,25607us,29us,1382us
DONE
############### Bonnie++ RESULTS ################
Bonnie Version: 1.96
Hostname: storage-01
Date Run: 1362086310
Test File Size: 125G
Seq Block Writes KB p/s: 82560
Seq Block Re-Writes KB p/s: 31352
Seq Block Reads KB p/s: 59208
Random Seeks p/s: 1199
#################################################
Sleeping for 5 seconds
Sleeping for 5 seconds
ext3.tests: line 1: echo: write error: Invalid argument
##########
##########
Sleeping for 5 seconds
Dropping pagecache, dentries, and inodes
Sleeping for 5 seconds
Making EXT3 file system with DEFAULTS on /dev/zd16
mke2fs 1.42.5 (29-Jul-2012)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
67108864 inodes, 268435456 blocks
13421772 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
8192 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
Testing /dev/zd16
\n\n
KERNEL:
3.2.0-4-amd64
DISK CONTROLLER:
3ware Inc 9650SE SATA-II RAID PCIe (rev 01)
HBA CONTROLLER:
LSI Logic / Symbios Logic SAS2004 PCI-Express Fusion-MPT SAS-2 [Spitfire] (rev 0                                                                                                 3)
NET INTERFACE(S):
Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
CPU TYPE (8 Cores):
Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz
I/O SCHEDULER:
[noop] deadline cfq
Beginning iozone test with 4 GB file and 4 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 4
Writes: 105059
Reads: 156192
Re-Reads: 1848429
Random Reads: 2167985
Random Writes: 1401666
#################################################
Beginning iozone test with 4 GB file and 64 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 64
Writes: 160090
Reads: 166447
Re-Reads: 2217220
Random Reads: 2218409
Random Writes: 1436243
#################################################
Beginning iozone test with 4 GB file and 256 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 256
Writes: 200322
Reads: 249114
Re-Reads: 1848250
Random Reads: 2258555
Random Writes: 2012976
#################################################
Beginning iozone test with 16 GB file and 4 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 4
Writes: 68070
Reads: 49320
Re-Reads: 1944531
Random Reads: 2108160
Random Writes: 1569759
#################################################
Beginning iozone test with 16 GB file and 64 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 64
Writes: 63147
Reads: 63704
Re-Reads: 2145691
Random Reads: 2228298
Random Writes: 2045594
#################################################
Beginning iozone test with 16 GB file and 256 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 256
Writes: 61422
Reads: 66714
Re-Reads: 2021250
Random Reads: 2232056
Random Writes: 2141735
#################################################
Beginning Multi-Thread iozone test with 4 GB file and 64 KB record length.
This can take a while, depending on specified file size...DONE
####### IOZONE MULTI-THREAD TEST RESULTS ########
File Size (GB): 4
Record Length (KB): 64
Writes:    49891.62 KB/sec
Reads:  1132279.34 KB/sec
Re-Reads:  1175232.88 KB/sec
Random Reads:  1152682.16 KB/sec
Random Writes:    59362.64 KB/sec
#################################################
Beginning the Bonnie++ Test.
/usr/sbin/bonnie++ -u root -d /mnt/ggio-zd16 2> /dev/null
Version  1.96       ------Sequential Output------ --Sequential Input- --Random- Concu                                                                                            rrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine                                                                                                    Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP storage-01                                                                                                 125G   624  99 58236  22 22174   4  3480  96 46810   3  1128  12 Latency                                                                                                         21028us    4601ms    1488ms   17291us     529ms     104ms Version  1.96       ------S                                                                                            equential Create------ --------Random Create-------- storage-01          -Create-- --                                                                                            Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec %                                                                                            CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP                  16 +++++ +++ +++++ +++ ++                                                                                            +++ +++ +++++ +++ +++++ +++ 27971  30 Latency             22698us     530us    1159us                                                                                               38189us      12us    1333us 1.96,1.96,storage-01,1,1362100445,125G,,624,99,58236,2                                                                                            2,22174,4,3480,96,46810,3,1128,12,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,++++                                                                                            +,+++,27971,30,21028us,4601ms,1488ms,17291us,529ms,104ms,22698us,530us,1159us,38189us                                                                                            ,12us,1333us
DONE
############### Bonnie++ RESULTS ################
Bonnie Version: 1.96
Hostname: storage-01
Date Run: 1362100445
Test File Size: 125G
Seq Block Writes KB p/s: 58236
Seq Block Re-Writes KB p/s: 22174
Seq Block Reads KB p/s: 46810
Random Seeks p/s: 1128
#################################################
Sleeping for 5 seconds
Sleeping for 5 seconds
ext3.tests: line 1: echo: write error: Invalid argument
##########
##########
Sleeping for 5 seconds
Dropping pagecache, dentries, and inodes
Sleeping for 5 seconds
Making EXT3 file system with DEFAULTS on /dev/zd32
mke2fs 1.42.5 (29-Jul-2012)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=8 blocks, Stripe width=8 blocks
67108864 inodes, 268435456 blocks
13421772 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
8192 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
Testing /dev/zd32
\n\n
KERNEL:
3.2.0-4-amd64
DISK CONTROLLER:
3ware Inc 9650SE SATA-II RAID PCIe (rev 01)
HBA CONTROLLER:
LSI Logic / Symbios Logic SAS2004 PCI-Express Fusion-MPT SAS-2 [Spitfire] (rev 03)
NET INTERFACE(S):
Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
CPU TYPE (8 Cores):
Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz
I/O SCHEDULER:
[noop] deadline cfq
Beginning iozone test with 4 GB file and 4 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 4
Writes: 170840
Reads: 951868
Re-Reads: 2124946
Random Reads: 2009463
Random Writes: 1623289
#################################################
Beginning iozone test with 4 GB file and 64 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 64
Writes: 300507
Reads: 1004303
Re-Reads: 2192760
Random Reads: 1878416
Random Writes: 2083751
#################################################
Beginning iozone test with 4 GB file and 256 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 256
Writes: 344520
Reads: 915413
Re-Reads: 2242882
Random Reads: 1877826
Random Writes: 2190649
#################################################
Beginning iozone test with 16 GB file and 4 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 4
Writes: 165193
Reads: 171756
Re-Reads: 1975286
Random Reads: 2144172
Random Writes: 1578734
#################################################
Beginning iozone test with 16 GB file and 64 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 64
Writes: 157848
Reads: 167283
Re-Reads: 2180261
Random Reads: 2231735
Random Writes: 2087969
#################################################
Beginning iozone test with 16 GB file and 256 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 256
Writes: 159713
Reads: 183303
Re-Reads: 2037627
Random Reads: 2226426
Random Writes: 2157262
#################################################
Beginning Multi-Thread iozone test with 4 GB file and 64 KB record length.
This can take a while, depending on specified file size...DONE
####### IOZONE MULTI-THREAD TEST RESULTS ########
File Size (GB): 4
Record Length (KB): 64
Writes:   110707.82 KB/sec
Reads:   272999.58 KB/sec
Re-Reads:   981149.86 KB/sec
Random Reads:   917143.93 KB/sec
Random Writes:   127475.25 KB/sec
#################################################
Beginning the Bonnie++ Test.
/usr/sbin/bonnie++ -u root -d /mnt/ggio-zd32 2> /dev/null
Version  1.96       ------Sequential Output------ --Sequential Input- --Random- Concu                                                                                            rrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine                                                                                                    Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP storage-01                                                                                                 125G   630  98 155178  63 66886  17  3350  98 117553  10 762.3   9 Latency                                                                                                         21279us    4739ms    1179ms   13093us     802ms     102ms Version  1.96       -----                                                                                            -Sequential Create------ --------Random Create-------- storage-01          -Create--                                                                                             --Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec                                                                                             %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP                  16 +++++ +++ +++++ +++                                                                                             +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency             16807us     494us    1389                                                                                            us   54907us      12us    1465us 1.96,1.96,storage-01,1,1362121396,125G,,630,98,15517                                                                                            8,63,66886,17,3350,98,117553,10,762.3,9,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,++                                                                                            +,+++++,+++,+++++,+++,21279us,4739ms,1179ms,13093us,802ms,102ms,16807us,494us,1389us,                                                                                            54907us,12us,1465us
DONE
############### Bonnie++ RESULTS ################
Bonnie Version: 1.96
Hostname: storage-01
Date Run: 1362121396
Test File Size: 125G
Seq Block Writes KB p/s: 155178
Seq Block Re-Writes KB p/s: 66886
Seq Block Reads KB p/s: 117553
Random Seeks p/s: 762.3
#################################################
Sleeping for 5 seconds
Sleeping for 5 seconds
ext3.tests: line 1: echo: write error: Invalid argument
##########
##########
Sleeping for 5 seconds
Dropping pagecache, dentries, and inodes
Sleeping for 5 seconds
Making EXT3 file system with DEFAULTS on /dev/zd48
mke2fs 1.42.5 (29-Jul-2012)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
67108864 inodes, 268435456 blocks
13421772 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
8192 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
Testing /dev/zd48
\n\n
KERNEL:
3.2.0-4-amd64
DISK CONTROLLER:
3ware Inc 9650SE SATA-II RAID PCIe (rev 01)
HBA CONTROLLER:
LSI Logic / Symbios Logic SAS2004 PCI-Express Fusion-MPT SAS-2 [Spitfire] (rev 03)
NET INTERFACE(S):
Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
CPU TYPE (8 Cores):
Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz
I/O SCHEDULER:
[noop] deadline cfq
Beginning iozone test with 4 GB file and 4 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 4
Writes: 417
Reads: 1060503
Re-Reads: 1459879
Random Reads: 1905908
Random Writes: 1582240
#################################################
Beginning iozone test with 4 GB file and 64 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 64
Writes: 508738
Reads: 1082349
Re-Reads: 1409180
Random Reads: 1864345
Random Writes: 1751620
#################################################
Beginning iozone test with 4 GB file and 256 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 256
Writes: 427867
Reads: 844410
Re-Reads: 1422103
Random Reads: 1914403
Random Writes: 1980361
#################################################
Beginning iozone test with 16 GB file and 4 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 4
Writes: 610
Reads: 1746
Re-Reads: 1071
Random Reads: 59617
Random Writes: 38279
#################################################
Beginning iozone test with 16 GB file and 64 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 64
Writes: 8633
Reads: 11074
Re-Reads: 5947
Random Reads: 14776
Random Writes: 23977
#################################################
Beginning iozone test with 16 GB file and 256 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 256
Writes: 10965
Reads: 8591
Re-Reads: 12378
Random Reads: 2458
Random Writes: 2921
#################################################
Beginning Multi-Thread iozone test with 4 GB file and 64 KB record length.
This can take a while, depending on specified file size...DONE
####### IOZONE MULTI-THREAD TEST RESULTS ########
File Size (GB): 4
Record Length (KB): 64
Writes:    11623.77 KB/sec
Reads:    26422.91 KB/sec
Re-Reads:    24655.59 KB/sec
Random Reads:    10238.90 KB/sec
Random Writes:     8150.52 KB/sec
#################################################
Beginning the Bonnie++ Test.
/usr/sbin/bonnie++ -u root -d /mnt/ggio-zd48 2> /dev/null
Version  1.96       ------Sequential Output------ --Sequential Input- --Random- Concu                                                                                            rrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine                                                                                                    Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP storage-01                                                                                                 125G   593  96 11958   5  5085   1  3138  88 12374   1 267.1   3 Latency                                                                                                         40621us   19472ms    9227ms   41598us     728ms     163ms Version  1.96       ------S                                                                                            equential Create------ --------Random Create-------- storage-01          -Create-- --                                                                                            Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec %                                                                                            CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP                  16 19744  26 +++++ +++ ++                                                                                            +++ +++ 21982  30 +++++ +++ 13677  27 Latency             15105us     512us    1313us                                                                                                 916us    3535us    4291us 1.96,1.96,storage-01,1,1362238559,125G,,593,96,11958,5                                                                                            ,5085,1,3138,88,12374,1,267.1,3,16,,,,,19744,26,+++++,+++,+++++,+++,21982,30,+++++,++                                                                                            +,13677,27,40621us,19472ms,9227ms,41598us,728ms,163ms,15105us,512us,1313us,916us,3535                                                                                            us,4291us
DONE
############### Bonnie++ RESULTS ################
Bonnie Version: 1.96
Hostname: storage-01
Date Run: 1362238559
Test File Size: 125G
Seq Block Writes KB p/s: 11958
Seq Block Re-Writes KB p/s: 5085
Seq Block Reads KB p/s: 12374
Random Seeks p/s: 267.1
#################################################
Sleeping for 5 seconds
Sleeping for 5 seconds
ext3.tests: line 1: echo: write error: Invalid argument
##########
##########
Sleeping for 5 seconds
Dropping pagecache, dentries, and inodes
Sleeping for 5 seconds
Making EXT3 file system with DEFAULTS on /dev/zd64
mke2fs 1.42.5 (29-Jul-2012)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=16 blocks, Stripe width=16 blocks
67108864 inodes, 268435456 blocks
13421772 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
8192 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
Testing /dev/zd64
\n\n
KERNEL:
3.2.0-4-amd64
DISK CONTROLLER:
3ware Inc 9650SE SATA-II RAID PCIe (rev 01)
HBA CONTROLLER:
LSI Logic / Symbios Logic SAS2004 PCI-Express Fusion-MPT SAS-2 [Spitfire] (rev 03)
NET INTERFACE(S):
Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
CPU TYPE (8 Cores):
Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz
I/O SCHEDULER:
[noop] deadline cfq
Beginning iozone test with 4 GB file and 4 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 4
Writes: 160901
Reads: 442658
Re-Reads: 1867403
Random Reads: 1295997
Random Writes: 1593548
#################################################
Beginning iozone test with 4 GB file and 64 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 64
Writes: 285596
Reads: 236359
Re-Reads: 1968329
Random Reads: 1590488
Random Writes: 1974030
#################################################
Beginning iozone test with 4 GB file and 256 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 256
Writes: 241475
Reads: 352732
Re-Reads: 1533206
Random Reads: 1986929
Random Writes: 1960327
#################################################
Beginning iozone test with 16 GB file and 4 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 4
Writes: 122593
Reads: 162349
Re-Reads: 356888
Random Reads: 1895003
Random Writes: 1553830
#################################################
Beginning iozone test with 16 GB file and 64 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 64
Writes: 133728
Reads: 148519
Re-Reads: 1994558
Random Reads: 2049710
Random Writes: 1964604
#################################################
Beginning iozone test with 16 GB file and 256 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 256
Writes: 133212
Reads: 155937
Re-Reads: 1948007
Random Reads: 2032862
Random Writes: 1989223
#################################################
Beginning Multi-Thread iozone test with 4 GB file and 64 KB record length.
This can take a while, depending on specified file size...DONE
####### IOZONE MULTI-THREAD TEST RESULTS ########
File Size (GB): 4
Record Length (KB): 64
Writes:    69681.59 KB/sec
Reads:   492534.58 KB/sec
Re-Reads:   276598.36 KB/sec
Random Reads:   241286.22 KB/sec
Random Writes:   106966.64 KB/sec
#################################################
Beginning the Bonnie++ Test.
/usr/sbin/bonnie++ -u root -d /mnt/ggio-zd64 2> /dev/null
Version  1.96       ------Sequential Output------ --Sequential Input- --Random- Concu                                                                                            rrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine                                                                                                    Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP storage-01                                                                                                 125G   626  99 143076  57 71225  19  3213  87 121097  12 495.5   5 Latency                                                                                                         13103us    6632ms    1827ms     108ms     476ms     172ms Version  1.96       -----                                                                                            -Sequential Create------ --------Random Create-------- storage-01          -Create--                                                                                             --Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec                                                                                             %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP                  16 +++++ +++ +++++ +++                                                                                             27067  29 +++++ +++ +++++ +++ +++++ +++ Latency             16613us     537us    1355                                                                                            us   51069us      13us    1292us 1.96,1.96,storage-01,1,1362308082,125G,,626,99,14307                                                                                            6,57,71225,19,3213,87,121097,12,495.5,5,16,,,,,+++++,+++,+++++,+++,27067,29,+++++,+++                                                                                            ,+++++,+++,+++++,+++,13103us,6632ms,1827ms,108ms,476ms,172ms,16613us,537us,1355us,510                                                                                            69us,13us,1292us
DONE
############### Bonnie++ RESULTS ################
Bonnie Version: 1.96
Hostname: storage-01
Date Run: 1362308082
Test File Size: 125G
Seq Block Writes KB p/s: 143076
Seq Block Re-Writes KB p/s: 71225
Seq Block Reads KB p/s: 121097
Random Seeks p/s: 495.5
#################################################
Sleeping for 5 seconds
Sleeping for 5 seconds
ext3.tests: line 1: echo: write error: Invalid argument
##########
##########
Sleeping for 5 seconds
Dropping pagecache, dentries, and inodes
Sleeping for 5 seconds
Making EXT3 file system with DEFAULTS on /dev/zd80
mke2fs 1.42.5 (29-Jul-2012)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=4 blocks, Stripe width=4 blocks
67108864 inodes, 268435456 blocks
13421772 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
8192 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
Testing /dev/zd80
\n\n
KERNEL:
3.2.0-4-amd64
DISK CONTROLLER:
3ware Inc 9650SE SATA-II RAID PCIe (rev 01)
HBA CONTROLLER:
LSI Logic / Symbios Logic SAS2004 PCI-Express Fusion-MPT SAS-2 [Spitfire] (rev 03)
NET INTERFACE(S):
Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
CPU TYPE (8 Cores):
Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz
I/O SCHEDULER:
[noop] deadline cfq
Beginning iozone test with 4 GB file and 4 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 4
Writes: 190220
Reads: 949710
Re-Reads: 1935808
Random Reads: 1489639
Random Writes: 1602871
#################################################
Beginning iozone test with 4 GB file and 64 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 64
Writes: 370476
Reads: 1305811
Re-Reads: 2007991
Random Reads: 1759173
Random Writes: 1964328
#################################################
Beginning iozone test with 4 GB file and 256 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 256
Writes: 402300
Reads: 1150006
Re-Reads: 1923982
Random Reads: 1560099
Random Writes: 2000029
#################################################
Beginning iozone test with 16 GB file and 4 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 4
Writes: 125297
Reads: 134160
Re-Reads: 1953420
Random Reads: 2069949
Random Writes: 1578695
#################################################
Beginning iozone test with 16 GB file and 64 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 64
Writes: 117568
Reads: 145334
Re-Reads: 2000634
Random Reads: 2158810
Random Writes: 2048914
#################################################
Beginning iozone test with 16 GB file and 256 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 256
Writes: 112749
Reads: 144719
Re-Reads: 2022986
Random Reads: 2134001
Random Writes: 2093222
#################################################
Beginning Multi-Thread iozone test with 4 GB file and 64 KB record length.
This can take a while, depending on specified file size...DONE
####### IOZONE MULTI-THREAD TEST RESULTS ########
File Size (GB): 4
Record Length (KB): 64
Writes:   100666.01 KB/sec
Reads:   211496.13 KB/sec
Re-Reads:   290872.05 KB/sec
Random Reads:  1534482.75 KB/sec
Random Writes:   110928.56 KB/sec
#################################################
Beginning the Bonnie++ Test.
/usr/sbin/bonnie++ -u root -d /mnt/ggio-zd80 2> /dev/null
Version  1.96       ------Sequential Output------ --Sequential Input- --Random- Concu                                                                                            rrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine                                                                                                    Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP storage-01                                                                                                 125G   620  99 133245  54 44868  10  3496  97 70811   5 883.6   9 Latency                                                                                                         21312us    5539ms     896ms   10929us     578ms     145ms Version  1.96       ------                                                                                            Sequential Create------ --------Random Create-------- storage-01          -Create-- -                                                                                            -Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec                                                                                             %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP                  16 +++++ +++ +++++ +++ +                                                                                            ++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency             35331us     518us    1846u                                                                                            s   56406us      14us    1442us 1.96,1.96,storage-01,1,1362298719,125G,,620,99,133245                                                                                            ,54,44868,10,3496,97,70811,5,883.6,9,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+                                                                                            ++++,+++,+++++,+++,21312us,5539ms,896ms,10929us,578ms,145ms,35331us,518us,1846us,5640                                                                                            6us,14us,1442us
DONE
############### Bonnie++ RESULTS ################
Bonnie Version: 1.96
Hostname: storage-01
Date Run: 1362298719
Test File Size: 125G
Seq Block Writes KB p/s: 133245
Seq Block Re-Writes KB p/s: 44868
Seq Block Reads KB p/s: 70811
Random Seeks p/s: 883.6
#################################################
Sleeping for 5 seconds
Sleeping for 5 seconds
ext3.tests: line 1: echo: write error: Invalid argument
##########
##########
Sleeping for 5 seconds
Dropping pagecache, dentries, and inodes
Sleeping for 5 seconds
Making EXT3 file system with DEFAULTS on /dev/zd96
mke2fs 1.42.5 (29-Jul-2012)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=2 blocks, Stripe width=2 blocks
67108864 inodes, 268435456 blocks
13421772 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
8192 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
Testing /dev/zd96
\n\n
KERNEL:
3.2.0-4-amd64
DISK CONTROLLER:
3ware Inc 9650SE SATA-II RAID PCIe (rev 01)
HBA CONTROLLER:
LSI Logic / Symbios Logic SAS2004 PCI-Express Fusion-MPT SAS-2 [Spitfire] (rev 03)
NET INTERFACE(S):
Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
CPU TYPE (8 Cores):
Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz
I/O SCHEDULER:
[noop] deadline cfq
Beginning iozone test with 4 GB file and 4 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 4
Writes: 135806
Reads: 1017187
Re-Reads: 1962084
Random Reads: 1724675
Random Writes: 1609299
#################################################
Beginning iozone test with 4 GB file and 64 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 64
Writes: 377094
Reads: 1090548
Re-Reads: 2072642
Random Reads: 1844925
Random Writes: 2005823
#################################################
Beginning iozone test with 4 GB file and 256 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 256
Writes: 389472
Reads: 1147269
Re-Reads: 2021222
Random Reads: 1532128
Random Writes: 2010638
#################################################
Beginning iozone test with 16 GB file and 4 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 4
Writes: 75866
Reads: 103253
Re-Reads: 1946121
Random Reads: 2102262
Random Writes: 1578542
#################################################
Beginning iozone test with 16 GB file and 64 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 64
Writes: 96525
Reads: 119462
Re-Reads: 2037643
Random Reads: 2219448
Random Writes: 2086093
#################################################
Beginning iozone test with 16 GB file and 256 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 256
Writes: 98198
Reads: 116075
Re-Reads: 1975138
Random Reads: 2205056
Random Writes: 2142556
#################################################
Beginning Multi-Thread iozone test with 4 GB file and 64 KB record length.
This can take a while, depending on specified file size...DONE
####### IOZONE MULTI-THREAD TEST RESULTS ########
File Size (GB): 4
Record Length (KB): 64
Writes:    12864.36 KB/sec
Reads:   263433.59 KB/sec
Re-Reads:   145807.26 KB/sec
Random Reads:   815088.65 KB/sec
Random Writes:    86919.98 KB/sec
#################################################
Beginning the Bonnie++ Test.
/usr/sbin/bonnie++ -u root -d /mnt/ggio-zd96 2> /dev/null
Version  1.96       ------Sequential Output------ --Sequential Input- --Random- Concu                                                                                            rrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine                                                                                                    Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP storage-01                                                                                                 125G   641  98 80696  47 30231   6  2798  80 55669   4  1085  11 Latency                                                                                                         19243us   40861ms    1045ms   68880us     435ms     176ms Version  1.96       ------S                                                                                            equential Create------ --------Random Create-------- storage-01          -Create-- --                                                                                            Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec %                                                                                            CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP                  16 +++++ +++ +++++ +++ ++                                                                                            +++ +++ +++++ +++ +++++ +++ +++++ +++ Latency               638us     515us    1336us                                                                                               37722us    3051us    4767us 1.96,1.96,storage-01,1,1362323836,125G,,641,98,80696,4                                                                                            7,30231,6,2798,80,55669,4,1085,11,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,++++                                                                                            +,+++,+++++,+++,19243us,40861ms,1045ms,68880us,435ms,176ms,638us,515us,1336us,37722us                                                                                            ,3051us,4767us
DONE
############### Bonnie++ RESULTS ################
Bonnie Version: 1.96
Hostname: storage-01
Date Run: 1362323836
Test File Size: 125G
Seq Block Writes KB p/s: 80696
Seq Block Re-Writes KB p/s: 30231
Seq Block Reads KB p/s: 55669
Random Seeks p/s: 1085
#################################################
Sleeping for 5 seconds
Sleeping for 5 seconds
ext3.tests: line 1: echo: write error: Invalid argument
##########
##########
Sleeping for 5 seconds
Dropping pagecache, dentries, and inodes
Sleeping for 5 seconds
Making EXT3 file system with DEFAULTS on /dev/zd112
mke2fs 1.42.5 (29-Jul-2012)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=32 blocks, Stripe width=32 blocks
67108864 inodes, 268435456 blocks
13421772 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
8192 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
Testing /dev/zd112
\n\n
KERNEL:
3.2.0-4-amd64
DISK CONTROLLER:
3ware Inc 9650SE SATA-II RAID PCIe (rev 01)
HBA CONTROLLER:
LSI Logic / Symbios Logic SAS2004 PCI-Express Fusion-MPT SAS-2 [Spitfire] (rev 03)
NET INTERFACE(S):
Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
CPU TYPE (8 Cores):
Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz
I/O SCHEDULER:
[noop] deadline cfq
Beginning iozone test with 4 GB file and 4 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 4
Writes: 205673
Reads: 947014
Re-Reads: 1817843
Random Reads: 1992061
Random Writes: 1606978
#################################################
Beginning iozone test with 4 GB file and 64 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 64
Writes: 331675
Reads: 979658
Re-Reads: 1760764
Random Reads: 2073619
Random Writes: 1988485
#################################################
Beginning iozone test with 4 GB file and 256 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 256
Writes: 363405
Reads: 978292
Re-Reads: 1667724
Random Reads: 2082841
Random Writes: 2036630
#################################################
Beginning iozone test with 16 GB file and 4 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 4
Writes: 173424
Reads: 210917
Re-Reads: 2061139
Random Reads: 2117643
Random Writes: 1573413
#################################################
Beginning iozone test with 16 GB file and 64 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 64
Writes: 191903
Reads: 190881
Re-Reads: 2156064
Random Reads: 2205047
Random Writes: 2059621
#################################################
Beginning iozone test with 16 GB file and 256 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 256
Writes: 193423
Reads: 190507
Re-Reads: 2165561
Random Reads: 2192645
Random Writes: 2155172
#################################################
Beginning Multi-Thread iozone test with 4 GB file and 64 KB record length.
This can take a while, depending on specified file size...DONE
####### IOZONE MULTI-THREAD TEST RESULTS ########
File Size (GB): 4
Record Length (KB): 64
Writes:   118542.51 KB/sec
Reads:  1869317.94 KB/sec
Re-Reads:  1869103.41 KB/sec
Random Reads:   991320.56 KB/sec
Random Writes:    95948.42 KB/sec
#################################################
Beginning the Bonnie++ Test.
/usr/sbin/bonnie++ -u root -d /mnt/ggio-zd112 2> /dev/null
Version  1.96       ------Sequential Output------ --Sequential Input- --Random- Concu                                                                                            rrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine                                                                                                    Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP storage-01                                                                                                 125G   637  99 170072  67 115315  33  3439  97 266356  30 706.8   7 Latency                                                                                                         19634us    5597ms    2666ms   22423us     224ms     234ms Version  1.96       ----                                                                                            --Sequential Create------ --------Random Create-------- storage-01          -Create--                                                                                             --Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /se                                                                                            c %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP                  16 19790  27 +++++ +++                                                                                             +++++ +++ 27389  36 +++++ +++ +++++ +++ Latency             28702us     493us    142                                                                                            0us     240ms      14us    1425us 1.96,1.96,storage-01,1,1362317665,125G,,637,99,1700                                                                                            72,67,115315,33,3439,97,266356,30,706.8,7,16,,,,,19790,27,+++++,+++,+++++,+++,27389,3                                                                                            6,+++++,+++,+++++,+++,19634us,5597ms,2666ms,22423us,224ms,234ms,28702us,493us,1420us,                                                                                            240ms,14us,1425us
DONE
############### Bonnie++ RESULTS ################
Bonnie Version: 1.96
Hostname: storage-01
Date Run: 1362317665
Test File Size: 125G
Seq Block Writes KB p/s: 170072
Seq Block Re-Writes KB p/s: 115315
Seq Block Reads KB p/s: 266356
Random Seeks p/s: 706.8
#################################################
Sleeping for 5 seconds
Sleeping for 5 seconds
ext3.tests: line 1: echo: write error: Invalid argument
##########
##########
Sleeping for 5 seconds
Dropping pagecache, dentries, and inodes
Sleeping for 5 seconds
Making EXT3 file system with DEFAULTS on /dev/zd128
mke2fs 1.42.5 (29-Jul-2012)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
67108864 inodes, 268435456 blocks
13421772 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
8192 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
Testing /dev/zd128
\n\n
KERNEL:
3.2.0-4-amd64
DISK CONTROLLER:
3ware Inc 9650SE SATA-II RAID PCIe (rev 01)
HBA CONTROLLER:
LSI Logic / Symbios Logic SAS2004 PCI-Express Fusion-MPT SAS-2 [Spitfire] (rev 03)
NET INTERFACE(S):
Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
Intel Corporation 80003ES2LAN Gigabit Ethernet Controller (Copper) (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
Intel Corporation 82576 Gigabit Network Connection (rev 01)
CPU TYPE (8 Cores):
Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz
I/O SCHEDULER:
[noop] deadline cfq
Beginning iozone test with 4 GB file and 4 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 4
Writes: 16807
Reads: 999792
Re-Reads: 1161069
Random Reads: 2143606
Random Writes: 1615629
#################################################
Beginning iozone test with 4 GB file and 64 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 64
Writes: 428155
Reads: 763001
Re-Reads: 991255
Random Reads: 1911306
Random Writes: 2022679
#################################################
Beginning iozone test with 4 GB file and 256 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 4
Record Length (KB): 256
Writes: 456834
Reads: 854554
Re-Reads: 1059920
Random Reads: 2094527
Random Writes: 2105862
#################################################
Beginning iozone test with 16 GB file and 4 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 4
Writes: 8962
Reads: 7750
Re-Reads: 34240
Random Reads: 1457317
Random Writes: 1541325
#################################################
Beginning iozone test with 16 GB file and 64 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 64
Writes: 6997
Reads: 8827
Re-Reads: 16041
Random Reads: 154097
Random Writes: 1960091
#################################################
Beginning iozone test with 16 GB file and 256 KB record length.
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 256
Writes: 7076
Reads: 7678
Re-Reads: 7536
Random Reads: 8002
Random Writes: 9040
#################################################

zvols tested were nearly the same... except I skipped testing a volblock size of 2k:

root@storage-01:~# zfs list
NAME              USED  AVAIL  REFER  MOUNTPOINT
storage01        9.91T  18.6T  53.9K  /storage01
storage01/zvol0  1.50T  20.1T  34.0G  -
storage01/zvol1  1.25T  19.9T  32.0K  -
storage01/zvol2  1.06T  19.7T  32.0K  -
storage01/zvol3  1.03T  19.6T  32.0K  -
storage01/zvol4  1.02T  19.6T  32.0K  -
storage01/zvol5  1.01T  19.6T  32.0K  -
storage01/zvol6  1.00T  19.6T  32.0K  -
storage01/zvol7  1.00T  19.6T  32.0K  -
storage01/zvol8  1.03T  19.6T   515M  -

Testing zvols in this scenario where zvol0-7, zvol8 was the control (created with defaults...)

COUNT=0; for i in {512,1k,4k,8k,16k,32k,64k,128k}; do echo "zfs create -V 1T -b ${i} storage01/zvol${COUNT}" && zfs create -V 1T -b ${i} storage01/zvol${COUNT}; COUNT=`expr $COUNT + 1`; done && zfs create -V 1T storage01/zvol${COUNT}
mengesb commented 11 years ago

Hardware:

Manufacturer: Supermicro Product Name: X7DB8 Memory: 64G DDR2 @ 667 MHz - 16x4096 MB - 4 Banks x 4 Slots CPU: 2x Intel(R) Xeon(R) CPU E5430 @ 2.66GHz HBA:

[    3.344497] mpt2sas0: LSISAS2008: FWVersion(07.00.00.00), ChipRevision(0x03), BiosVersion(07.11.00.00)
[   11.189711] scsi 1:0:24:0: Enclosure         LSILOGIC SASX36 A.1       7015 PQ: 0 ANSI: 3
mengesb commented 11 years ago

after 48 hours, My test still hasn't completed the 16G / 256k record length test on the 512b zvol dev.

Here's perf top:

 25.68%  [kernel]            [k] __isolate_lru_page
 18.20%  [kernel]            [k] isolate_lru_pages
  8.31%  [kernel]            [k] shrink_page_list
  5.74%  [kernel]            [k] putback_lru_pages
  4.09%  [kernel]            [k] page_waitqueue
  3.29%  [kernel]            [k] unlock_page
  2.56%  [kernel]            [k] test_and_set_bit
  2.36%  [kernel]            [k] put_page_testzero
  2.31%  [kernel]            [k] page_lru
  2.22%  [kernel]            [k] page_evictable
  2.07%  [kernel]            [k] release_pages
  2.04%  [kernel]            [k] list_move
  1.96%  [kernel]            [k] __wake_up_bit
  1.58%  [kernel]            [k] hpage_nr_pages
  1.15%  [kernel]            [k] page_mapping
  1.13%  [kernel]            [k] page_referenced
  1.10%  [kernel]            [k] update_isolated_counts
  1.07%  [kernel]            [k] list_del
  1.02%  [kernel]            [k] __mod_zone_page_state
  0.94%  [kernel]            [k] mod_state
  0.85%  [kernel]            [k] mem_cgroup_add_lru_list
  0.83%  [kernel]            [k] page_lru
  0.74%  [kernel]            [k] inc_zone_page_state
  0.70%  [kernel]            [k] should_resched
  0.63%  [kernel]            [k] __page_check_address
  0.62%  [kernel]            [k] do_raw_spin_lock
  0.58%  [kernel]            [k] _cond_resched
  0.54%  [kernel]            [k] mem_cgroup_disabled
  0.53%  [kernel]            [k] lru_add_drain
  0.53%  [kernel]            [k] mem_cgroup_del_lru_list

Looks like it is constantly cycling pages.

root@storage-01:~# free -m
             total       used       free     shared    buffers     cached
Mem:         63808      61761       2047          0         43        234
-/+ buffers/cache:      61483       2325
Swap:         1592          0       1592

As noted before, spl_kmem_cache appears to be locking a CPU up:

top - 08:07:03 up 3 days, 17:46,  2 users,  load average: 53.28, 58.64, 62.38
Tasks: 234 total,   2 running, 232 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us, 12.5 sy,  0.0 ni, 50.0 id, 37.5 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  65340244 total, 63245284 used,  2094960 free,    44660 buffers
KiB Swap:  1631228 total,        4 used,  1631224 free,   240324 cached
  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
 1510 root      39  19     0    0    0 R 100.0  0.0   4776:48 spl_kmem_cache/
  476 root       0 -20     0    0    0 D   0.3  0.0  83:17.25 txg_sync
    1 root      20   0 10648  728  588 S   0.0  0.0   0:03.26 init
    2 root      20   0     0    0    0 S   0.0  0.0   0:00.01 kthreadd
    3 root      20   0     0    0    0 S   0.0  0.0   0:00.24 ksoftirqd/0
    6 root      rt   0     0    0    0 S   0.0  0.0   0:02.69 migration/0
    7 root      rt   0     0    0    0 S   0.0  0.0   0:04.79 watchdog/0
    8 root      rt   0     0    0    0 S   0.0  0.0   0:07.47 migration/1
   10 root      20   0     0    0    0 S   0.0  0.0   0:00.28 ksoftirqd/1
   12 root      rt   0     0    0    0 S   0.0  0.0   0:03.96 watchdog/1
   13 root      rt   0     0    0    0 S   0.0  0.0   0:03.06 migration/2
   15 root      20   0     0    0    0 S   0.0  0.0   0:00.32 ksoftirqd/2
   16 root      rt   0     0    0    0 S   0.0  0.0   0:04.26 watchdog/2
   17 root      rt   0     0    0    0 S   0.0  0.0   0:05.60 migration/3
   19 root      20   0     0    0    0 S   0.0  0.0   0:00.17 ksoftirqd/3
   20 root      rt   0     0    0    0 S   0.0  0.0   0:04.07 watchdog/3
   21 root      rt   0     0    0    0 S   0.0  0.0   0:05.22 migration/4
   22 root      20   0     0    0    0 S   0.0  0.0   0:09.18 kworker/4:0
   23 root      20   0     0    0    0 S   0.0  0.0   0:00.22 ksoftirqd/4
   24 root      rt   0     0    0    0 S   0.0  0.0   0:03.90 watchdog/4

No activity on my devs recorded by iostat except for a 300 on avgqu-sz for zd0 (the tested dev for the last 72 hours), and reports of 100% disk usage.

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00   12.51   37.43    0.00   50.06
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdb               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdc               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdd               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sde               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdf               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdg               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdh               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdj               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdm               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdi               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdo               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdn               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdl               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdp               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdt               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdr               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdq               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sds               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdv               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdu               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdk               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdy               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdaa              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdx               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdz               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdw               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
zd0               0.00     0.00    0.00    0.00     0.00     0.00     0.00   300.00    0.00    0.00    0.00   0.00 100.00
zd16              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
zd32              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
zd48              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
zd64              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
zd80              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
zd96              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
zd112             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
zd128             0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mengesb commented 11 years ago 
root@storage-01:~# cat /proc/spl/kstat/zfs/arcstats
4 1 0x01 80 3840 14257871852 327315182515129
name                            type data
hits                            4    437295646
misses                          4    112414715
demand_data_hits                4    189385304
demand_data_misses              4    61310435
demand_metadata_hits            4    230107971
demand_metadata_misses          4    6705511
prefetch_data_hits              4    17802317
prefetch_data_misses            4    44398453
prefetch_metadata_hits          4    54
prefetch_metadata_misses        4    316
mru_hits                        4    112204225
mru_ghost_hits                  4    9078344
mfu_hits                        4    311031277
mfu_ghost_hits                  4    2717298
deleted                         4    389153020
recycle_miss                    4    220709750
mutex_miss                      4    431216907
evict_skip                      4    37419709527
evict_l2_cached                 4    139514239488
evict_l2_eligible               4    166815066624
evict_l2_ineligible             4    3788137984
hash_elements                   4    151500302
hash_elements_max               4    151587980
hash_collisions                 4    446823727
hash_chains                     4    1048576
hash_chain_max                  4    208
p                               4    294448160
c                               4    4564022784
c_min                           4    4294967296
c_max                           4    34359738368
size                            4    41305799760
hdr_size                        4    236638976
data_size                       4    48081408
other_size                      4    21222480
anon_size                       4    40369152
anon_evict_data                 4    0
anon_evict_metadata             4    0
mru_size                        4    796160
mru_evict_data                  4    384000
mru_evict_metadata              4    49152
mru_ghost_size                  4    4536396800
mru_ghost_evict_data            4    252975616
mru_ghost_evict_metadata        4    4283421184
mfu_size                        4    6916096
mfu_evict_data                  4    0
mfu_evict_metadata              4    81920
mfu_ghost_size                  4    3889664
mfu_ghost_evict_data            4    6144
mfu_ghost_evict_metadata        4    3883520
l2_hits                         4    18505561
l2_misses                       4    93909154
l2_feeds                        4    42203
l2_rw_clash                     4    190831
l2_read_bytes                   4    50935898112
l2_write_bytes                  4    112967882752
l2_writes_sent                  4    40167
l2_writes_done                  4    40167
l2_writes_error                 4    0
l2_writes_hdr_miss              4    898654
l2_evict_lock_retry             4    0
l2_evict_reading                4    0
l2_free_on_write                4    11359835
l2_abort_lowmem                 4    18
l2_cksum_bad                    4    0
l2_io_error                     4    0
l2_size                         4    81666772992
l2_hdr_size                     4    43411613184
memory_throttle_count           4    0
duplicate_buffers               4    0
duplicate_buffers_size          4    0
duplicate_reads                 4    0
memory_direct_count             4    1121
memory_indirect_count           4    510631
arc_no_grow                     4    0
arc_tempreserve                 4    851968
arc_loaned_bytes                4    0
arc_prune                       4    12367496
arc_meta_used                   4    41268161104
arc_meta_limit                  4    8589934592
arc_meta_max                    4    41287783136
mengesb commented 11 years ago

zfs/spl 0.6.1 installed from repository

   966  Mar 27 16:58:21 storage-01 kernel: [  584.971637] SPL: using hostid 0xa8c00902
   967  Mar 27 16:58:25 storage-01 kernel: [  588.806353]  zd0: unknown partition table
   968  Mar 27 16:58:26 storage-01 kernel: [  589.138735]  zd16: unknown partition table
   969  Mar 27 16:58:26 storage-01 kernel: [  589.258381]  zd32: unknown partition table
   970  Mar 27 16:58:26 storage-01 kernel: [  589.651478]  zd48: unknown partition table
   971  Mar 27 16:58:27 storage-01 kernel: [  590.142868]  zd64: unknown partition table
   972  Mar 27 16:58:27 storage-01 kernel: [  590.701137]  zd80: unknown partition table
   973  Mar 27 16:58:27 storage-01 kernel: [  591.046921]  zd96: unknown partition table
   974  Mar 27 16:58:27 storage-01 kernel: [  591.067719]  zd112: unknown partition table
   975  Mar 27 16:58:28 storage-01 kernel: [  591.161893]  zd128: unknown partition table
   976  Mar 27 17:04:58 storage-01 kernel: [  981.750485]  zd0: unknown partition table
   977  Mar 27 17:05:03 storage-01 kernel: [  986.317452]  zd16: unknown partition table
   978  Mar 27 17:05:07 storage-01 kernel: [  990.871750]  zd32: unknown partition table
   979  Mar 27 17:05:12 storage-01 kernel: [  995.480102]  zd48: unknown partition table
   980  Mar 27 17:20:38 storage-01 kernel: [ 1921.177449] EXT4-fs (zd48): mounted filesystem with ordered data mode. Opts: (null)
   981  Mar 27 17:24:38 storage-01 kernel: [ 2161.100333] jbd2/zd48-8     D ffff880fffd13780     0  6420      2 0x00000000
   982  Mar 27 17:24:38 storage-01 kernel: [ 2161.100337]  ffff880fbd45a830 0000000000000046 0000000000000000 ffff880ff8e008b0
   983  Mar 27 17:24:38 storage-01 kernel: [ 2161.100342]  0000000000013780 ffff8802d320dfd8 ffff8802d320dfd8 ffff880fbd45a830
   984  Mar 27 17:24:38 storage-01 kernel: [ 2161.100345]  0000000000000246 000000018134d97c ffff880fbbe108e0 ffff880fbbe10800
   985  Mar 27 17:24:38 storage-01 kernel: [ 2161.100349] Call Trace:
   986  Mar 27 17:24:38 storage-01 kernel: [ 2161.100360]  [] ? jbd2_journal_commit_transaction+0x1a3/0x1057 [jbd2]
   987  Mar 27 17:24:38 storage-01 kernel: [ 2161.100367]  [] ? load_TLS+0x7/0xa
   988  Mar 27 17:24:38 storage-01 kernel: [ 2161.100370]  [] ? __switch_to+0x133/0x258
   989  Mar 27 17:24:38 storage-01 kernel: [ 2161.100374]  [] ? finish_task_switch+0x88/0xb9
   990  Mar 27 17:24:38 storage-01 kernel: [ 2161.100378]  [] ? arch_local_irq_save+0x11/0x17
   991  Mar 27 17:24:38 storage-01 kernel: [ 2161.100383]  [] ? add_wait_queue+0x3c/0x3c
   992  Mar 27 17:24:38 storage-01 kernel: [ 2161.100387]  [] ? kjournald2+0xc0/0x20a [jbd2]
   993  Mar 27 17:24:38 storage-01 kernel: [ 2161.100390]  [] ? add_wait_queue+0x3c/0x3c
   994  Mar 27 17:24:38 storage-01 kernel: [ 2161.100393]  [] ? commit_timeout+0x5/0x5 [jbd2]
   995  Mar 27 17:24:38 storage-01 kernel: [ 2161.100396]  [] ? kthread+0x76/0x7e
   996  Mar 27 17:24:38 storage-01 kernel: [ 2161.100409]  [] ? kernel_thread_helper+0x4/0x10
   997  Mar 27 17:24:38 storage-01 kernel: [ 2161.100412]  [] ? kthread_worker_fn+0x139/0x139
   998  Mar 27 17:24:38 storage-01 kernel: [ 2161.100414]  [] ? gs_change+0x13/0x13
   999  Mar 27 18:32:37 storage-01 kernel: [ 6240.652174] Modules linked in: ext4 crc16 jbd2 ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop zfs(P) zunicode(P) zavl(P) zcommon(P) znvpair(P) spl(O) zlib_deflate i5000_edac edac_core radeon iTCO_wdt iTCO_vendor_support snd_pcm snd_page_alloc ttm snd_timer i5k_amb snd drm_kms_helper coretemp soundcore drm power_supply i2c_algo_bit ioatdma i2c_i801 serio_raw pcspkr i2c_core evdev parport_pc parport shpchp container processor button thermal_sys ext3 mbcache jbd ses enclosure sg sd_mod crc_t10dif ata_generic uhci_hcd floppy ata_piix libata ehci_hcd 3w_9xxx usbcore igb mpt2sas dca usb_common raid_class scsi_transport_sas scsi_mod e1000e [last unloaded: scsi_wait_scan]
  1000  Mar 27 18:32:37 storage-01 kernel: [ 6240.652237] CPU 6
  1001  Mar 27 18:32:37 storage-01 kernel: [ 6240.652238] Modules linked in: ext4 crc16 jbd2 ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop zfs(P) zunicode(P) zavl(P) zcommon(P) znvpair(P) spl(O) zlib_deflate i5000_edac edac_core radeon iTCO_wdt iTCO_vendor_support snd_pcm snd_page_alloc ttm snd_timer i5k_amb snd drm_kms_helper coretemp soundcore drm power_supply i2c_algo_bit ioatdma i2c_i801 serio_raw pcspkr i2c_core evdev parport_pc parport shpchp container processor button thermal_sys ext3 mbcache jbd ses enclosure sg sd_mod crc_t10dif ata_generic uhci_hcd floppy ata_piix libata ehci_hcd 3w_9xxx usbcore igb mpt2sas dca usb_common raid_class scsi_transport_sas scsi_mod e1000e [last unloaded: scsi_wait_scan]
  1002  Mar 27 18:32:37 storage-01 kernel: [ 6240.652277]
  1003  Mar 27 18:32:37 storage-01 kernel: [ 6240.652280] Pid: 53, comm: kswapd0 Tainted: P           O 3.2.0-4-amd64 #1 Debian 3.2.39-2 Supermicro X7DB8/X7DB8
  1004  Mar 27 18:32:37 storage-01 kernel: [ 6240.652284] RIP: 0010:[]  [] remove_vm_area+0x45/0x7f
  1005  Mar 27 18:32:37 storage-01 kernel: [ 6240.652292] RSP: 0018:ffff880fbdcbfb50  EFLAGS: 00000287
  1006  Mar 27 18:32:37 storage-01 kernel: [ 6240.652294] RAX: ffff880a49ab9fc0 RBX: 0000000000000000 RCX: ffffc90aade2f000
  1007  Mar 27 18:32:37 storage-01 kernel: [ 6240.652296] RDX: ffff880a4eac37c0 RSI: 000000000000059f RDI: ffffffff81629110
  1008  Mar 27 18:32:37 storage-01 kernel: [ 6240.652298] RBP: ffff880e22de57c0 R08: ffff880e23bfc600 R09: ffff880e2462be00
  1009  Mar 27 18:32:37 storage-01 kernel: [ 6240.652300] R10: ffff880e2462be00 R11: ffff880773142780 R12: 0000000000000001
  1010  Mar 27 18:32:37 storage-01 kernel: [ 6240.652302] R13: ffffffff810ca110 R14: ffff880fffffbe00 R15: 0000000000016668
  1011  Mar 27 18:32:37 storage-01 kernel: [ 6240.652305] FS:  0000000000000000(0000) GS:ffff880fffd80000(0000) knlGS:0000000000000000
  1012  Mar 27 18:32:37 storage-01 kernel: [ 6240.652307] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
  1013  Mar 27 18:32:37 storage-01 kernel: [ 6240.652309] CR2: 00007fffe7609fac CR3: 0000000001605000 CR4: 00000000000006e0
  1014  Mar 27 18:32:37 storage-01 kernel: [ 6240.652311] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  1015  Mar 27 18:32:37 storage-01 kernel: [ 6240.652313] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
  1016  Mar 27 18:32:37 storage-01 kernel: [ 6240.652316] Process kswapd0 (pid: 53, threadinfo ffff880fbdcbe000, task ffff880ff8974af0)
  1017  Mar 27 18:32:37 storage-01 kernel: [ 6240.652357]  ffff880e23bfc600 ffff880fbf00e000 ffffc90aaf402000 ffffffff810dba31
  1018  Mar 27 18:32:37 storage-01 kernel: [ 6240.652362]  ffff880fbdcbfbb8 ffffc90aade2f000 ffff880fbf00e000 ffffc90aaf402000
  1019  Mar 27 18:32:37 storage-01 kernel: [ 6240.652366]  ffff880fbf00f040 ffffffffa03d0e16 0000000000000000 0000000000000000
  1020  Mar 27 18:32:37 storage-01 kernel: [ 6240.652413]  [] ? __vunmap+0x35/0xb8
  1021  Mar 27 18:32:37 storage-01 kernel: [ 6240.652421]  [] ? spl_slab_reclaim+0x2c0/0x324 [spl]
  1022  Mar 27 18:32:37 storage-01 kernel: [ 6240.652427]  [] ? spl_kmem_cache_reap_now+0x188/0x1f5 [spl]
  1023  Mar 27 18:32:37 storage-01 kernel: [ 6240.652448]  [] ? arc_kmem_reap_now+0x5a/0x91 [zfs]
  1024  Mar 27 18:32:37 storage-01 kernel: [ 6240.652459]  [] ? arc_shrinker_func+0x6b/0xed [zfs]
  1025  Mar 27 18:32:37 storage-01 kernel: [ 6240.652463]  [] ? shrink_slab+0x18f/0x24d
  1026  Mar 27 18:32:37 storage-01 kernel: [ 6240.652466]  [] ? balance_pgdat+0x283/0x4b7
  1027  Mar 27 18:32:37 storage-01 kernel: [ 6240.652468]  [] ? kswapd+0x2d3/0x303
  1028  Mar 27 18:32:37 storage-01 kernel: [ 6240.652473]  [] ? add_wait_queue+0x3c/0x3c
  1029  Mar 27 18:32:37 storage-01 kernel: [ 6240.652476]  [] ? balance_pgdat+0x4b7/0x4b7
  1030  Mar 27 18:32:37 storage-01 kernel: [ 6240.652479]  [] ? kthread+0x76/0x7e
  1031  Mar 27 18:32:37 storage-01 kernel: [ 6240.652483]  [] ? kernel_thread_helper+0x4/0x10
  1032  Mar 27 18:32:37 storage-01 kernel: [ 6240.652486]  [] ? kthread_worker_fn+0x139/0x139
  1033  Mar 27 18:32:37 storage-01 kernel: [ 6240.652489]  [] ? gs_change+0x13/0x13
  1034  Mar 27 18:32:37 storage-01 kernel: [ 6240.653463] Call Trace:
  1035  Mar 27 18:32:37 storage-01 kernel: [ 6240.653465]  [] ? __vunmap+0x35/0xb8
  1036  Mar 27 18:32:37 storage-01 kernel: [ 6240.653470]  [] ? spl_slab_reclaim+0x2c0/0x324 [spl]
  1037  Mar 27 18:32:37 storage-01 kernel: [ 6240.653475]  [] ? spl_kmem_cache_reap_now+0x188/0x1f5 [spl]
  1038  Mar 27 18:32:37 storage-01 kernel: [ 6240.653485]  [] ? arc_kmem_reap_now+0x5a/0x91 [zfs]
  1039  Mar 27 18:32:37 storage-01 kernel: [ 6240.653495]  [] ? arc_shrinker_func+0x6b/0xed [zfs]
  1040  Mar 27 18:32:37 storage-01 kernel: [ 6240.653498]  [] ? shrink_slab+0x18f/0x24d
  1041  Mar 27 18:32:37 storage-01 kernel: [ 6240.653500]  [] ? balance_pgdat+0x283/0x4b7
  1042  Mar 27 18:32:37 storage-01 kernel: [ 6240.653503]  [] ? kswapd+0x2d3/0x303
  1043  Mar 27 18:32:37 storage-01 kernel: [ 6240.653506]  [] ? add_wait_queue+0x3c/0x3c
  1044  Mar 27 18:32:37 storage-01 kernel: [ 6240.653508]  [] ? balance_pgdat+0x4b7/0x4b7
  1045  Mar 27 18:32:37 storage-01 kernel: [ 6240.653511]  [] ? kthread+0x76/0x7e
  1046  Mar 27 18:32:37 storage-01 kernel: [ 6240.653513]  [] ? kernel_thread_helper+0x4/0x10
  1047  Mar 27 18:32:37 storage-01 kernel: [ 6240.653516]  [] ? kthread_worker_fn+0x139/0x139
  1048  Mar 27 18:32:37 storage-01 kernel: [ 6240.653519]  [] ? gs_change+0x13/0x13
  1049  Mar 27 18:34:13 storage-01 kernel: [ 6336.100150] Modules linked in: ext4 crc16 jbd2 ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop zfs(P) zunicode(P) zavl(P) zcommon(P) znvpair(P) spl(O) zlib_deflate i5000_edac edac_core radeon iTCO_wdt iTCO_vendor_support snd_pcm snd_page_alloc ttm snd_timer i5k_amb snd drm_kms_helper coretemp soundcore drm power_supply i2c_algo_bit ioatdma i2c_i801 serio_raw pcspkr i2c_core evdev parport_pc parport shpchp container processor button thermal_sys ext3 mbcache jbd ses enclosure sg sd_mod crc_t10dif ata_generic uhci_hcd floppy ata_piix libata ehci_hcd 3w_9xxx usbcore igb mpt2sas dca usb_common raid_class scsi_transport_sas scsi_mod e1000e [last unloaded: scsi_wait_scan]
  1050  Mar 27 18:34:13 storage-01 kernel: [ 6336.100213] CPU 0
  1051  Mar 27 18:34:13 storage-01 kernel: [ 6336.100214] Modules linked in: ext4 crc16 jbd2 ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop zfs(P) zunicode(P) zavl(P) zcommon(P) znvpair(P) spl(O) zlib_deflate i5000_edac edac_core radeon iTCO_wdt iTCO_vendor_support snd_pcm snd_page_alloc ttm snd_timer i5k_amb snd drm_kms_helper coretemp soundcore drm power_supply i2c_algo_bit ioatdma i2c_i801 serio_raw pcspkr i2c_core evdev parport_pc parport shpchp container processor button thermal_sys ext3 mbcache jbd ses enclosure sg sd_mod crc_t10dif ata_generic uhci_hcd floppy ata_piix libata ehci_hcd 3w_9xxx usbcore igb mpt2sas dca usb_common raid_class scsi_transport_sas scsi_mod e1000e [last unloaded: scsi_wait_scan]
  1052  Mar 27 18:34:13 storage-01 kernel: [ 6336.100253]
  1053  Mar 27 18:34:13 storage-01 kernel: [ 6336.100256] Pid: 53, comm: kswapd0 Tainted: P           O 3.2.0-4-amd64 #1 Debian 3.2.39-2 Supermicro X7DB8/X7DB8
  1054  Mar 27 18:34:13 storage-01 kernel: [ 6336.100260] RIP: 0010:[]  [] remove_vm_area+0x45/0x7f
  1055  Mar 27 18:34:13 storage-01 kernel: [ 6336.100269] RSP: 0018:ffff880fbdcbfb50  EFLAGS: 00000287
  1056  Mar 27 18:34:13 storage-01 kernel: [ 6336.100270] RAX: ffff8809eeafca00 RBX: 0000000000000000 RCX: ffffc90b6933d000
  1057  Mar 27 18:34:13 storage-01 kernel: [ 6336.100272] RDX: ffff880c277ac780 RSI: 000000000000059f RDI: ffffffff81629110
  1058  Mar 27 18:34:13 storage-01 kernel: [ 6336.100274] RBP: ffff880f3671fe40 R08: ffff880f34f44800 R09: ffff880f36460c00
  1059  Mar 27 18:34:13 storage-01 kernel: [ 6336.100276] R10: ffff880f36460c00 R11: ffff880650270800 R12: 0000000000000001
  1060  Mar 27 18:34:13 storage-01 kernel: [ 6336.100278] R13: ffffffff810ca110 R14: ffff880fffffbe00 R15: 0000000000016668
  1061  Mar 27 18:34:13 storage-01 kernel: [ 6336.100281] FS:  0000000000000000(0000) GS:ffff880fffc00000(0000) knlGS:0000000000000000
  1062  Mar 27 18:34:13 storage-01 kernel: [ 6336.100283] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
  1063  Mar 27 18:34:13 storage-01 kernel: [ 6336.100285] CR2: 00007f022393d56f CR3: 0000000001605000 CR4: 00000000000006f0
  1064  Mar 27 18:34:13 storage-01 kernel: [ 6336.100287] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  1065  Mar 27 18:34:13 storage-01 kernel: [ 6336.100289] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
  1066  Mar 27 18:34:13 storage-01 kernel: [ 6336.100291] Process kswapd0 (pid: 53, threadinfo ffff880fbdcbe000, task ffff880ff8974af0)
  1067  Mar 27 18:34:13 storage-01 kernel: [ 6336.100333]  ffff880f34f44800 ffff880fbf00e000 ffffc90b691f8000 ffffffff810dba31
  1068  Mar 27 18:34:13 storage-01 kernel: [ 6336.100338]  ffff880fbdcbfbb8 ffffc90b6933d000 ffff880fbf00e000 ffffc90b691f8000
  1069  Mar 27 18:34:13 storage-01 kernel: [ 6336.100341]  ffff880fbf00f040 ffffffffa03d0e16 0000000000000000 0000000000000000
  1070  Mar 27 18:34:13 storage-01 kernel: [ 6336.100389]  [] ? __vunmap+0x35/0xb8
  1071  Mar 27 18:34:13 storage-01 kernel: [ 6336.100397]  [] ? spl_slab_reclaim+0x2c0/0x324 [spl]
  1072  Mar 27 18:34:13 storage-01 kernel: [ 6336.100402]  [] ? spl_kmem_cache_reap_now+0x188/0x1f5 [spl]
  1073  Mar 27 18:34:13 storage-01 kernel: [ 6336.100432]  [] ? arc_kmem_reap_now+0x5a/0x91 [zfs]
  1074  Mar 27 18:34:13 storage-01 kernel: [ 6336.100442]  [] ? arc_shrinker_func+0x6b/0xed [zfs]
  1075  Mar 27 18:34:13 storage-01 kernel: [ 6336.100446]  [] ? shrink_slab+0x18f/0x24d
  1076  Mar 27 18:34:13 storage-01 kernel: [ 6336.100449]  [] ? balance_pgdat+0x283/0x4b7
  1077  Mar 27 18:34:13 storage-01 kernel: [ 6336.100452]  [] ? kswapd+0x2d3/0x303
  1078  Mar 27 18:34:13 storage-01 kernel: [ 6336.100457]  [] ? add_wait_queue+0x3c/0x3c
  1079  Mar 27 18:34:13 storage-01 kernel: [ 6336.100460]  [] ? balance_pgdat+0x4b7/0x4b7
  1080  Mar 27 18:34:13 storage-01 kernel: [ 6336.100462]  [] ? kthread+0x76/0x7e
  1081  Mar 27 18:34:13 storage-01 kernel: [ 6336.100467]  [] ? kernel_thread_helper+0x4/0x10
  1082  Mar 27 18:34:13 storage-01 kernel: [ 6336.100470]  [] ? kthread_worker_fn+0x139/0x139
  1083  Mar 27 18:34:13 storage-01 kernel: [ 6336.100472]  [] ? gs_change+0x13/0x13
  1084  Mar 27 18:34:13 storage-01 kernel: [ 6336.101447] Call Trace:
  1085  Mar 27 18:34:13 storage-01 kernel: [ 6336.101449]  [] ? __vunmap+0x35/0xb8
  1086  Mar 27 18:34:13 storage-01 kernel: [ 6336.101454]  [] ? spl_slab_reclaim+0x2c0/0x324 [spl]
  1087  Mar 27 18:34:13 storage-01 kernel: [ 6336.101459]  [] ? spl_kmem_cache_reap_now+0x188/0x1f5 [spl]
  1088  Mar 27 18:34:13 storage-01 kernel: [ 6336.101469]  [] ? arc_kmem_reap_now+0x5a/0x91 [zfs]
  1089  Mar 27 18:34:13 storage-01 kernel: [ 6336.101479]  [] ? arc_shrinker_func+0x6b/0xed [zfs]
  1090  Mar 27 18:34:13 storage-01 kernel: [ 6336.101482]  [] ? shrink_slab+0x18f/0x24d
  1091  Mar 27 18:34:13 storage-01 kernel: [ 6336.101484]  [] ? balance_pgdat+0x283/0x4b7
  1092  Mar 27 18:34:13 storage-01 kernel: [ 6336.101488]  [] ? kswapd+0x2d3/0x303
  1093  Mar 27 18:34:13 storage-01 kernel: [ 6336.101490]  [] ? add_wait_queue+0x3c/0x3c
  1094  Mar 27 18:34:13 storage-01 kernel: [ 6336.101493]  [] ? balance_pgdat+0x4b7/0x4b7
  1095  Mar 27 18:34:13 storage-01 kernel: [ 6336.101495]  [] ? kthread+0x76/0x7e
  1096  Mar 27 18:34:13 storage-01 kernel: [ 6336.101498]  [] ? kernel_thread_helper+0x4/0x10
  1097  Mar 27 18:34:13 storage-01 kernel: [ 6336.101501]  [] ? kthread_worker_fn+0x139/0x139
  1098  Mar 27 18:34:13 storage-01 kernel: [ 6336.101503]  [] ? gs_change+0x13/0x13
  1099  Mar 27 18:35:13 storage-01 kernel: [ 6396.376134] Modules linked in: ext4 crc16 jbd2 ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop zfs(P) zunicode(P) zavl(P) zcommon(P) znvpair(P) spl(O) zlib_deflate i5000_edac edac_core radeon iTCO_wdt iTCO_vendor_support snd_pcm snd_page_alloc ttm snd_timer i5k_amb snd drm_kms_helper coretemp soundcore drm power_supply i2c_algo_bit ioatdma i2c_i801 serio_raw pcspkr i2c_core evdev parport_pc parport shpchp container processor button thermal_sys ext3 mbcache jbd ses enclosure sg sd_mod crc_t10dif ata_generic uhci_hcd floppy ata_piix libata ehci_hcd 3w_9xxx usbcore igb mpt2sas dca usb_common raid_class scsi_transport_sas scsi_mod e1000e [last unloaded: scsi_wait_scan]
  1100  Mar 27 18:35:13 storage-01 kernel: [ 6396.376198] CPU 3
  1101  Mar 27 18:35:13 storage-01 kernel: [ 6396.376199] Modules linked in: ext4 crc16 jbd2 ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop zfs(P) zunicode(P) zavl(P) zcommon(P) znvpair(P) spl(O) zlib_deflate i5000_edac edac_core radeon iTCO_wdt iTCO_vendor_support snd_pcm snd_page_alloc ttm snd_timer i5k_amb snd drm_kms_helper coretemp soundcore drm power_supply i2c_algo_bit ioatdma i2c_i801 serio_raw pcspkr i2c_core evdev parport_pc parport shpchp container processor button thermal_sys ext3 mbcache jbd ses enclosure sg sd_mod crc_t10dif ata_generic uhci_hcd floppy ata_piix libata ehci_hcd 3w_9xxx usbcore igb mpt2sas dca usb_common raid_class scsi_transport_sas scsi_mod e1000e [last unloaded: scsi_wait_scan]
  1102  Mar 27 18:35:13 storage-01 kernel: [ 6396.376238]
  1103  Mar 27 18:35:13 storage-01 kernel: [ 6396.376241] Pid: 53, comm: kswapd0 Tainted: P           O 3.2.0-4-amd64 #1 Debian 3.2.39-2 Supermicro X7DB8/X7DB8
  1104  Mar 27 18:35:13 storage-01 kernel: [ 6396.376244] RIP: 0010:[]  [] remove_vm_area+0x45/0x7f
  1105  Mar 27 18:35:13 storage-01 kernel: [ 6396.376253] RSP: 0018:ffff880fbdcbfb50  EFLAGS: 00000287
  1106  Mar 27 18:35:13 storage-01 kernel: [ 6396.376255] RAX: ffff880de1dfd1c0 RBX: 0000000000000000 RCX: ffffc90b80ddb000
  1107  Mar 27 18:35:13 storage-01 kernel: [ 6396.376257] RDX: ffff8809ce230440 RSI: 000000000000059f RDI: ffffffff81629110
  1108  Mar 27 18:35:13 storage-01 kernel: [ 6396.376259] RBP: ffff8800be2a85c0 R08: ffff880f1c3de2c0 R09: ffff880ee6b654c0
  1109  Mar 27 18:35:13 storage-01 kernel: [ 6396.376261] R10: ffff880ee6b65180 R11: ffff880d1ecb9400 R12: 0000000000000001
  1110  Mar 27 18:35:13 storage-01 kernel: [ 6396.376263] R13: ffffffff810ca110 R14: ffff880fffffb700 R15: 0000000000016600
  1111  Mar 27 18:35:13 storage-01 kernel: [ 6396.376266] FS:  0000000000000000(0000) GS:ffff880fffcc0000(0000) knlGS:0000000000000000
  1112  Mar 27 18:35:13 storage-01 kernel: [ 6396.376268] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
  1113  Mar 27 18:35:13 storage-01 kernel: [ 6396.376270] CR2: 00007f40853c6f30 CR3: 0000000001605000 CR4: 00000000000006e0
  1114  Mar 27 18:35:13 storage-01 kernel: [ 6396.376272] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  1115  Mar 27 18:35:13 storage-01 kernel: [ 6396.376274] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
  1116  Mar 27 18:35:13 storage-01 kernel: [ 6396.376276] Process kswapd0 (pid: 53, threadinfo ffff880fbdcbe000, task ffff880ff8974af0)
  1117  Mar 27 18:35:13 storage-01 kernel: [ 6396.376318]  ffff880f1c3de2c0 ffff880fbf00e000 ffffc90b7dda4000 ffffffff810dba31
  1118  Mar 27 18:35:13 storage-01 kernel: [ 6396.376323]  ffff880fbdcbfbb8 ffffc90b80ddb000 ffff880fbf00e000 ffffc90b7dda4000
  1119  Mar 27 18:35:13 storage-01 kernel: [ 6396.376326]  ffff880fbf00f040 ffffffffa03d0e16 00000000ffffffff ffffc909c0057000
  1120  Mar 27 18:35:13 storage-01 kernel: [ 6396.376374]  [] ? __vunmap+0x35/0xb8
  1121  Mar 27 18:35:13 storage-01 kernel: [ 6396.376382]  [] ? spl_slab_reclaim+0x2c0/0x324 [spl]
  1122  Mar 27 18:35:13 storage-01 kernel: [ 6396.376387]  [] ? spl_kmem_cache_reap_now+0x188/0x1f5 [spl]
  1123  Mar 27 18:35:13 storage-01 kernel: [ 6396.376392]  [] ? wake_up_bit+0xd/0x1e
  1124  Mar 27 18:35:13 storage-01 kernel: [ 6396.376415]  [] ? arc_kmem_reap_now+0x5a/0x91 [zfs]
  1125  Mar 27 18:35:13 storage-01 kernel: [ 6396.376425]  [] ? arc_shrinker_func+0x6b/0xed [zfs]
  1126  Mar 27 18:35:13 storage-01 kernel: [ 6396.376429]  [] ? shrink_slab+0x18f/0x24d
  1127  Mar 27 18:35:13 storage-01 kernel: [ 6396.376432]  [] ? balance_pgdat+0x283/0x4b7
  1128  Mar 27 18:35:13 storage-01 kernel: [ 6396.376435]  [] ? kswapd+0x2d3/0x303
  1129  Mar 27 18:35:13 storage-01 kernel: [ 6396.376438]  [] ? add_wait_queue+0x3c/0x3c
  1130  Mar 27 18:35:13 storage-01 kernel: [ 6396.376440]  [] ? balance_pgdat+0x4b7/0x4b7
  1131  Mar 27 18:35:13 storage-01 kernel: [ 6396.376443]  [] ? kthread+0x76/0x7e
  1132  Mar 27 18:35:13 storage-01 kernel: [ 6396.376448]  [] ? kernel_thread_helper+0x4/0x10
  1133  Mar 27 18:35:13 storage-01 kernel: [ 6396.376451]  [] ? kthread_worker_fn+0x139/0x139
  1134  Mar 27 18:35:13 storage-01 kernel: [ 6396.376453]  [] ? gs_change+0x13/0x13
  1135  Mar 27 18:35:13 storage-01 kernel: [ 6396.377427] Call Trace:
  1136  Mar 27 18:35:13 storage-01 kernel: [ 6396.377430]  [] ? __vunmap+0x35/0xb8
  1137  Mar 27 18:35:13 storage-01 kernel: [ 6396.377434]  [] ? spl_slab_reclaim+0x2c0/0x324 [spl]
  1138  Mar 27 18:35:13 storage-01 kernel: [ 6396.377439]  [] ? spl_kmem_cache_reap_now+0x188/0x1f5 [spl]
  1139  Mar 27 18:35:13 storage-01 kernel: [ 6396.377442]  [] ? wake_up_bit+0xd/0x1e
  1140  Mar 27 18:35:13 storage-01 kernel: [ 6396.377452]  [] ? arc_kmem_reap_now+0x5a/0x91 [zfs]
  1141  Mar 27 18:35:13 storage-01 kernel: [ 6396.377462]  [] ? arc_shrinker_func+0x6b/0xed [zfs]
  1142  Mar 27 18:35:13 storage-01 kernel: [ 6396.377465]  [] ? shrink_slab+0x18f/0x24d
  1143  Mar 27 18:35:13 storage-01 kernel: [ 6396.377467]  [] ? balance_pgdat+0x283/0x4b7
  1144  Mar 27 18:35:13 storage-01 kernel: [ 6396.377470]  [] ? kswapd+0x2d3/0x303
  1145  Mar 27 18:35:13 storage-01 kernel: [ 6396.377473]  [] ? add_wait_queue+0x3c/0x3c
  1146  Mar 27 18:35:13 storage-01 kernel: [ 6396.377475]  [] ? balance_pgdat+0x4b7/0x4b7
  1147  Mar 27 18:35:13 storage-01 kernel: [ 6396.377478]  [] ? kthread+0x76/0x7e
  1148  Mar 27 18:35:13 storage-01 kernel: [ 6396.377480]  [] ? kernel_thread_helper+0x4/0x10
  1149  Mar 27 18:35:13 storage-01 kernel: [ 6396.377483]  [] ? kthread_worker_fn+0x139/0x139
  1150  Mar 27 18:35:13 storage-01 kernel: [ 6396.377486]  [] ? gs_change+0x13/0x13
  1151  Mar 27 18:36:29 storage-01 kernel: [ 6472.744158] Modules linked in: ext4 crc16 jbd2 ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop zfs(P) zunicode(P) zavl(P) zcommon(P) znvpair(P) spl(O) zlib_deflate i5000_edac edac_core radeon iTCO_wdt iTCO_vendor_support snd_pcm snd_page_alloc ttm snd_timer i5k_amb snd drm_kms_helper coretemp soundcore drm power_supply i2c_algo_bit ioatdma i2c_i801 serio_raw pcspkr i2c_core evdev parport_pc parport shpchp container processor button thermal_sys ext3 mbcache jbd ses enclosure sg sd_mod crc_t10dif ata_generic uhci_hcd floppy ata_piix libata ehci_hcd 3w_9xxx usbcore igb mpt2sas dca usb_common raid_class scsi_transport_sas scsi_mod e1000e [last unloaded: scsi_wait_scan]
  1152  Mar 27 18:36:29 storage-01 kernel: [ 6472.744221] CPU 7
  1153  Mar 27 18:36:29 storage-01 kernel: [ 6472.744222] Modules linked in: ext4 crc16 jbd2 ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop zfs(P) zunicode(P) zavl(P) zcommon(P) znvpair(P) spl(O) zlib_deflate i5000_edac edac_core radeon iTCO_wdt iTCO_vendor_support snd_pcm snd_page_alloc ttm snd_timer i5k_amb snd drm_kms_helper coretemp soundcore drm power_supply i2c_algo_bit ioatdma i2c_i801 serio_raw pcspkr i2c_core evdev parport_pc parport shpchp container processor button thermal_sys ext3 mbcache jbd ses enclosure sg sd_mod crc_t10dif ata_generic uhci_hcd floppy ata_piix libata ehci_hcd 3w_9xxx usbcore igb mpt2sas dca usb_common raid_class scsi_transport_sas scsi_mod e1000e [last unloaded: scsi_wait_scan]
  1154  Mar 27 18:36:29 storage-01 kernel: [ 6472.744262]
  1155  Mar 27 18:36:29 storage-01 kernel: [ 6472.744265] Pid: 53, comm: kswapd0 Tainted: P           O 3.2.0-4-amd64 #1 Debian 3.2.39-2 Supermicro X7DB8/X7DB8
  1156  Mar 27 18:36:29 storage-01 kernel: [ 6472.744268] RIP: 0010:[]  [] remove_vm_area+0x45/0x7f
  1157  Mar 27 18:36:29 storage-01 kernel: [ 6472.744277] RSP: 0018:ffff880fbdcbfb50  EFLAGS: 00000283
  1158  Mar 27 18:36:29 storage-01 kernel: [ 6472.744279] RAX: ffff88051a009240 RBX: 0000000000000000 RCX: ffffc90ba1cf1000
  1159  Mar 27 18:36:29 storage-01 kernel: [ 6472.744281] RDX: ffff880989109380 RSI: 000000000000059f RDI: ffffffff81629110
  1160  Mar 27 18:36:29 storage-01 kernel: [ 6472.744283] RBP: ffff880091964640 R08: ffff880094850a00 R09: ffff880092e93e00
  1161  Mar 27 18:36:29 storage-01 kernel: [ 6472.744285] R10: ffff880092e93e00 R11: ffff8800b5707d40 R12: 0000000000000001
  1162  Mar 27 18:36:29 storage-01 kernel: [ 6472.744287] R13: ffffffff810ca110 R14: ffff880fffffb700 R15: 0000000000016600
  1163  Mar 27 18:36:29 storage-01 kernel: [ 6472.744290] FS:  0000000000000000(0000) GS:ffff880fffdc0000(0000) knlGS:0000000000000000
  1164  Mar 27 18:36:29 storage-01 kernel: [ 6472.744292] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
  1165  Mar 27 18:36:29 storage-01 kernel: [ 6472.744294] CR2: 00007f5b8c287e02 CR3: 0000000001605000 CR4: 00000000000006e0
  1166  Mar 27 18:36:29 storage-01 kernel: [ 6472.744296] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  1167  Mar 27 18:36:29 storage-01 kernel: [ 6472.744298] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
  1168  Mar 27 18:36:29 storage-01 kernel: [ 6472.744300] Process kswapd0 (pid: 53, threadinfo ffff880fbdcbe000, task ffff880ff8974af0)
  1169  Mar 27 18:36:29 storage-01 kernel: [ 6472.744342]  ffff880094850a00 ffff880fbf00e000 ffffc90ba0104000 ffffffff810dba31
  1170  Mar 27 18:36:29 storage-01 kernel: [ 6472.744347]  ffff880fbdcbfbb8 ffffc90ba1cf1000 ffff880fbf00e000 ffffc90ba0104000
  1171  Mar 27 18:36:29 storage-01 kernel: [ 6472.744350]  ffff880fbf00f040 ffffffffa03d0e16 0000000000000000 0000000000000000
  1172  Mar 27 18:36:29 storage-01 kernel: [ 6472.744398]  [] ? __vunmap+0x35/0xb8
  1173  Mar 27 18:36:29 storage-01 kernel: [ 6472.744406]  [] ? spl_slab_reclaim+0x2c0/0x324 [spl]
  1174  Mar 27 18:36:29 storage-01 kernel: [ 6472.744411]  [] ? spl_kmem_cache_reap_now+0x188/0x1f5 [spl]
  1175  Mar 27 18:36:29 storage-01 kernel: [ 6472.744440]  [] ? arc_kmem_reap_now+0x5a/0x91 [zfs]
  1176  Mar 27 18:36:29 storage-01 kernel: [ 6472.744451]  [] ? arc_shrinker_func+0x6b/0xed [zfs]
  1177  Mar 27 18:36:29 storage-01 kernel: [ 6472.744455]  [] ? shrink_slab+0x18f/0x24d
  1178  Mar 27 18:36:29 storage-01 kernel: [ 6472.744458]  [] ? balance_pgdat+0x283/0x4b7
  1179  Mar 27 18:36:29 storage-01 kernel: [ 6472.744461]  [] ? kswapd+0x2d3/0x303
  1180  Mar 27 18:36:29 storage-01 kernel: [ 6472.744466]  [] ? add_wait_queue+0x3c/0x3c
  1181  Mar 27 18:36:29 storage-01 kernel: [ 6472.744469]  [] ? balance_pgdat+0x4b7/0x4b7
  1182  Mar 27 18:36:29 storage-01 kernel: [ 6472.744471]  [] ? kthread+0x76/0x7e
  1183  Mar 27 18:36:29 storage-01 kernel: [ 6472.744476]  [] ? kernel_thread_helper+0x4/0x10
  1184  Mar 27 18:36:29 storage-01 kernel: [ 6472.744479]  [] ? kthread_worker_fn+0x139/0x139
  1185  Mar 27 18:36:29 storage-01 kernel: [ 6472.744481]  [] ? gs_change+0x13/0x13
  1186  Mar 27 18:36:29 storage-01 kernel: [ 6472.745455] Call Trace:
  1187  Mar 27 18:36:29 storage-01 kernel: [ 6472.745458]  [] ? __vunmap+0x35/0xb8
  1188  Mar 27 18:36:29 storage-01 kernel: [ 6472.745462]  [] ? spl_slab_reclaim+0x2c0/0x324 [spl]
  1189  Mar 27 18:36:29 storage-01 kernel: [ 6472.745467]  [] ? spl_kmem_cache_reap_now+0x188/0x1f5 [spl]
  1190  Mar 27 18:36:29 storage-01 kernel: [ 6472.745477]  [] ? arc_kmem_reap_now+0x5a/0x91 [zfs]
  1191  Mar 27 18:36:29 storage-01 kernel: [ 6472.745487]  [] ? arc_shrinker_func+0x6b/0xed [zfs]
  1192  Mar 27 18:36:29 storage-01 kernel: [ 6472.745490]  [] ? shrink_slab+0x18f/0x24d
  1193  Mar 27 18:36:29 storage-01 kernel: [ 6472.745493]  [] ? balance_pgdat+0x283/0x4b7
  1194  Mar 27 18:36:29 storage-01 kernel: [ 6472.745495]  [] ? kswapd+0x2d3/0x303
  1195  Mar 27 18:36:29 storage-01 kernel: [ 6472.745498]  [] ? add_wait_queue+0x3c/0x3c
  1196  Mar 27 18:36:29 storage-01 kernel: [ 6472.745501]  [] ? balance_pgdat+0x4b7/0x4b7
  1197  Mar 27 18:36:29 storage-01 kernel: [ 6472.745503]  [] ? kthread+0x76/0x7e
  1198  Mar 27 18:36:29 storage-01 kernel: [ 6472.745506]  [] ? kernel_thread_helper+0x4/0x10
  1199  Mar 27 18:36:29 storage-01 kernel: [ 6472.745509]  [] ? kthread_worker_fn+0x139/0x139
  1200  Mar 27 18:36:29 storage-01 kernel: [ 6472.745511]  [] ? gs_change+0x13/0x13
  1201  Mar 27 18:36:57 storage-01 kernel: [ 6500.744172] Modules linked in: ext4 crc16 jbd2 ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop zfs(P) zunicode(P) zavl(P) zcommon(P) znvpair(P) spl(O) zlib_deflate i5000_edac edac_core radeon iTCO_wdt iTCO_vendor_support snd_pcm snd_page_alloc ttm snd_timer i5k_amb snd drm_kms_helper coretemp soundcore drm power_supply i2c_algo_bit ioatdma i2c_i801 serio_raw pcspkr i2c_core evdev parport_pc parport shpchp container processor button thermal_sys ext3 mbcache jbd ses enclosure sg sd_mod crc_t10dif ata_generic uhci_hcd floppy ata_piix libata ehci_hcd 3w_9xxx usbcore igb mpt2sas dca usb_common raid_class scsi_transport_sas scsi_mod e1000e [last unloaded: scsi_wait_scan]
  1202  Mar 27 18:36:57 storage-01 kernel: [ 6500.744236] CPU 7
  1203  Mar 27 18:36:57 storage-01 kernel: [ 6500.744237] Modules linked in: ext4 crc16 jbd2 ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop zfs(P) zunicode(P) zavl(P) zcommon(P) znvpair(P) spl(O) zlib_deflate i5000_edac edac_core radeon iTCO_wdt iTCO_vendor_support snd_pcm snd_page_alloc ttm snd_timer i5k_amb snd drm_kms_helper coretemp soundcore drm power_supply i2c_algo_bit ioatdma i2c_i801 serio_raw pcspkr i2c_core evdev parport_pc parport shpchp container processor button thermal_sys ext3 mbcache jbd ses enclosure sg sd_mod crc_t10dif ata_generic uhci_hcd floppy ata_piix libata ehci_hcd 3w_9xxx usbcore igb mpt2sas dca usb_common raid_class scsi_transport_sas scsi_mod e1000e [last unloaded: scsi_wait_scan]
  1204  Mar 27 18:36:57 storage-01 kernel: [ 6500.744276]
  1205  Mar 27 18:36:57 storage-01 kernel: [ 6500.744279] Pid: 53, comm: kswapd0 Tainted: P           O 3.2.0-4-amd64 #1 Debian 3.2.39-2 Supermicro X7DB8/X7DB8
  1206  Mar 27 18:36:57 storage-01 kernel: [ 6500.744283] RIP: 0010:[]  [] remove_vm_area+0x45/0x7f
  1207  Mar 27 18:36:57 storage-01 kernel: [ 6500.744300] RSP: 0018:ffff880fbdcbfb50  EFLAGS: 00000287
  1208  Mar 27 18:36:57 storage-01 kernel: [ 6500.744302] RAX: ffff880c08c4df40 RBX: 0000000000000000 RCX: ffffc90bb9a5c000
  1209  Mar 27 18:36:57 storage-01 kernel: [ 6500.744304] RDX: ffff8805dae7ca00 RSI: 000000000000059f RDI: ffffffff81629110
  1210  Mar 27 18:36:57 storage-01 kernel: [ 6500.744306] RBP: ffff8800717e0340 R08: ffff88006ff7c000 R09: ffff88006f38b800
  1211  Mar 27 18:36:57 storage-01 kernel: [ 6500.744308] R10: ffff8800712a3e00 R11: ffff8800b5829b80 R12: 0000000000000001
  1212  Mar 27 18:36:57 storage-01 kernel: [ 6500.744310] R13: ffffffff810ca110 R14: ffff880fffffb700 R15: 0000000000016600
  1213  Mar 27 18:36:57 storage-01 kernel: [ 6500.744313] FS:  0000000000000000(0000) GS:ffff880fffdc0000(0000) knlGS:0000000000000000
  1214  Mar 27 18:36:57 storage-01 kernel: [ 6500.744315] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
  1215  Mar 27 18:36:57 storage-01 kernel: [ 6500.744317] CR2: 00007f5b8c287e02 CR3: 0000000001605000 CR4: 00000000000006e0
  1216  Mar 27 18:36:57 storage-01 kernel: [ 6500.744319] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  1217  Mar 27 18:36:57 storage-01 kernel: [ 6500.744321] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
  1218  Mar 27 18:36:57 storage-01 kernel: [ 6500.744323] Process kswapd0 (pid: 53, threadinfo ffff880fbdcbe000, task ffff880ff8974af0)
  1219  Mar 27 18:36:57 storage-01 kernel: [ 6500.744365]  ffff88006ff7c000 ffff880fbf00e000 ffffc90bb968e000 ffffffff810dba31
  1220  Mar 27 18:36:57 storage-01 kernel: [ 6500.744370]  ffff880fbdcbfbb8 ffffc90bb9a5c000 ffff880fbf00e000 ffffc90bb968e000
  1221  Mar 27 18:36:57 storage-01 kernel: [ 6500.744373]  ffff880fbf00f040 ffffffffa03d0e16 0000000000000000 0000000000000000
  1222  Mar 27 18:36:57 storage-01 kernel: [ 6500.744420]  [] ? __vunmap+0x35/0xb8
  1223  Mar 27 18:36:57 storage-01 kernel: [ 6500.744428]  [] ? spl_slab_reclaim+0x2c0/0x324 [spl]
  1224  Mar 27 18:36:57 storage-01 kernel: [ 6500.744433]  [] ? spl_kmem_cache_reap_now+0x188/0x1f5 [spl]
  1225  Mar 27 18:36:57 storage-01 kernel: [ 6500.744463]  [] ? arc_kmem_reap_now+0x5a/0x91 [zfs]
  1226  Mar 27 18:36:57 storage-01 kernel: [ 6500.744473]  [] ? arc_shrinker_func+0x6b/0xed [zfs]
  1227  Mar 27 18:36:57 storage-01 kernel: [ 6500.744477]  [] ? shrink_slab+0x18f/0x24d
  1228  Mar 27 18:36:57 storage-01 kernel: [ 6500.744480]  [] ? balance_pgdat+0x283/0x4b7
  1229  Mar 27 18:36:57 storage-01 kernel: [ 6500.744483]  [] ? kswapd+0x2d3/0x303
  1230  Mar 27 18:36:57 storage-01 kernel: [ 6500.744488]  [] ? add_wait_queue+0x3c/0x3c
  1231  Mar 27 18:36:57 storage-01 kernel: [ 6500.744491]  [] ? balance_pgdat+0x4b7/0x4b7
  1232  Mar 27 18:36:57 storage-01 kernel: [ 6500.744494]  [] ? kthread+0x76/0x7e
  1233  Mar 27 18:36:57 storage-01 kernel: [ 6500.744498]  [] ? kernel_thread_helper+0x4/0x10
  1234  Mar 27 18:36:57 storage-01 kernel: [ 6500.744501]  [] ? kthread_worker_fn+0x139/0x139
  1235  Mar 27 18:36:57 storage-01 kernel: [ 6500.744503]  [] ? gs_change+0x13/0x13
  1236  Mar 27 18:36:57 storage-01 kernel: [ 6500.745478] Call Trace:
  1237  Mar 27 18:36:57 storage-01 kernel: [ 6500.745480]  [] ? __vunmap+0x35/0xb8
  1238  Mar 27 18:36:57 storage-01 kernel: [ 6500.745485]  [] ? spl_slab_reclaim+0x2c0/0x324 [spl]
  1239  Mar 27 18:36:57 storage-01 kernel: [ 6500.745490]  [] ? spl_kmem_cache_reap_now+0x188/0x1f5 [spl]
  1240  Mar 27 18:36:57 storage-01 kernel: [ 6500.745500]  [] ? arc_kmem_reap_now+0x5a/0x91 [zfs]
  1241  Mar 27 18:36:57 storage-01 kernel: [ 6500.745510]  [] ? arc_shrinker_func+0x6b/0xed [zfs]
  1242  Mar 27 18:36:57 storage-01 kernel: [ 6500.745513]  [] ? shrink_slab+0x18f/0x24d
  1243  Mar 27 18:36:57 storage-01 kernel: [ 6500.745516]  [] ? balance_pgdat+0x283/0x4b7
  1244  Mar 27 18:36:57 storage-01 kernel: [ 6500.745519]  [] ? kswapd+0x2d3/0x303
  1245  Mar 27 18:36:57 storage-01 kernel: [ 6500.745521]  [] ? add_wait_queue+0x3c/0x3c
  1246  Mar 27 18:36:57 storage-01 kernel: [ 6500.745524]  [] ? balance_pgdat+0x4b7/0x4b7
  1247  Mar 27 18:36:57 storage-01 kernel: [ 6500.745526]  [] ? kthread+0x76/0x7e
  1248  Mar 27 18:36:57 storage-01 kernel: [ 6500.745529]  [] ? kernel_thread_helper+0x4/0x10
  1249  Mar 27 18:36:57 storage-01 kernel: [ 6500.745532]  [] ? kthread_worker_fn+0x139/0x139
  1250  Mar 27 18:36:57 storage-01 kernel: [ 6500.745534]  [] ? gs_change+0x13/0x13
  1251  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002] sending NMI to all CPUs:
  1252  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235] NMI backtrace for cpu 1
  1253  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235] CPU 1
  1254  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235] Modules linked in: ext4 crc16 jbd2 ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop zfs(P) zunicode(P) zavl(P) zcommon(P) znvpair(P) spl(O) zlib_deflate i5000_edac edac_core radeon iTCO_wdt iTCO_vendor_support snd_pcm snd_page_alloc ttm snd_timer i5k_amb snd drm_kms_helper coretemp soundcore drm power_supply i2c_algo_bit ioatdma i2c_i801 serio_raw pcspkr i2c_core evdev parport_pc parport shpchp container processor button thermal_sys ext3 mbcache jbd ses enclosure sg sd_mod crc_t10dif ata_generic uhci_hcd floppy ata_piix libata ehci_hcd 3w_9xxx usbcore igb mpt2sas dca usb_common raid_class scsi_transport_sas scsi_mod e1000e [last unloaded: scsi_wait_scan]
  1255  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235]
  1256  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235] Pid: 0, comm: swapper/1 Tainted: P           O 3.2.0-4-amd64 #1 Debian 3.2.39-2 Supermicro X7DB8/X7DB8
  1257  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235] RIP: 0010:[]  [] mwait_idle+0x7f/0xac
  1258  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235] RSP: 0018:ffff880ff8dadef8  EFLAGS: 00000246
  1259  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235] RAX: 0000000000000000 RBX: ffff880ff8dac000 RCX: 0000000000000000
  1260  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff880ff8dac000
  1261  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235] RBP: ffffffff8168ef80 R08: 0000000000000000 R09: 0000000000000000
  1262  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235] R10: 0000000000000246 R11: ffff880ff8dac000 R12: 0000000000000000
  1263  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
  1264  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235] FS:  0000000000000000(0000) GS:ffff880fffc40000(0000) knlGS:0000000000000000
  1265  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
  1266  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235] CR2: 00007fd388008108 CR3: 0000000fbd580000 CR4: 00000000000006e0
  1267  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  1268  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
  1269  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235] Process swapper/1 (pid: 0, threadinfo ffff880ff8dac000, task ffff880ff8d8e0c0)
  1270  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235]  00000000ffffffed ffffffff8100d24d 0000000000000000 b6e7c3d7bf1c1f06
  1271  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235]  0000000044809df0 0000000000000001 0000000000000000 ffffffff8133fc5c
  1272  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235]  0000000000000000 0ace77318d5d6dec 0000000000000000 0000000000000000
  1273  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235]  [] ? cpu_idle+0xaf/0xf2
  1274  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235]  [] ? start_secondary+0x1d5/0x1db
  1275  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235] Call Trace:
  1276  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235]  [] ? cpu_idle+0xaf/0xf2
  1277  Mar 27 18:37:02 storage-01 kernel: [ 6505.809235]  [] ? start_secondary+0x1d5/0x1db
  1278  Mar 27 18:37:02 storage-01 kernel: [ 6505.913326] NMI backtrace for cpu 3
  1279  Mar 27 18:37:02 storage-01 kernel: [ 6505.913327] CPU 3
  1280  Mar 27 18:37:02 storage-01 kernel: [ 6505.913329] Modules linked in: ext4 crc16 jbd2 ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop zfs(P) zunicode(P) zavl(P) zcommon(P) znvpair(P) spl(O) zlib_deflate i5000_edac edac_core radeon iTCO_wdt iTCO_vendor_support snd_pcm snd_page_alloc ttm snd_timer i5k_amb snd drm_kms_helper coretemp soundcore drm power_supply i2c_algo_bit ioatdma i2c_i801 serio_raw pcspkr i2c_core evdev parport_pc parport shpchp container processor button thermal_sys ext3 mbcache jbd ses enclosure sg sd_mod crc_t10dif ata_generic uhci_hcd floppy ata_piix libata ehci_hcd 3w_9xxx usbcore igb mpt2sas dca usb_common raid_class scsi_transport_sas scsi_mod e1000e [last unloaded: scsi_wait_scan]
  1281  Mar 27 18:37:02 storage-01 kernel: [ 6505.913369]
  1282  Mar 27 18:37:02 storage-01 kernel: [ 6505.913371] Pid: 0, comm: swapper/3 Tainted: P           O 3.2.0-4-amd64 #1 Debian 3.2.39-2 Supermicro X7DB8/X7DB8
  1283  Mar 27 18:37:02 storage-01 kernel: [ 6505.913375] RIP: 0010:[]  [] mwait_idle+0x7f/0xac
  1284  Mar 27 18:37:02 storage-01 kernel: [ 6505.913379] RSP: 0018:ffff880ff8df7ef8  EFLAGS: 00000246
  1285  Mar 27 18:37:02 storage-01 kernel: [ 6505.913381] RAX: 0000000000000000 RBX: ffff880ff8df6000 RCX: 0000000000000000
  1286  Mar 27 18:37:02 storage-01 kernel: [ 6505.913383] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff880ff8df6000
  1287  Mar 27 18:37:02 storage-01 kernel: [ 6505.913384] RBP: ffffffff8168ef80 R08: 0000000000000000 R09: ffffffff8168ef80
  1288  Mar 27 18:37:02 storage-01 kernel: [ 6505.913386] R10: ffff880c50daf100 R11: ffff880ff8df6000 R12: 0000000000000000
  1289  Mar 27 18:37:02 storage-01 kernel: [ 6505.913388] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
  1290  Mar 27 18:37:02 storage-01 kernel: [ 6505.913390] FS:  0000000000000000(0000) GS:ffff880fffcc0000(0000) knlGS:0000000000000000
  1291  Mar 27 18:37:02 storage-01 kernel: [ 6505.913393] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
  1292  Mar 27 18:37:02 storage-01 kernel: [ 6505.913394] CR2: 00007f40853c6f30 CR3: 0000000001605000 CR4: 00000000000006e0
  1293  Mar 27 18:37:02 storage-01 kernel: [ 6505.913396] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  1294  Mar 27 18:37:02 storage-01 kernel: [ 6505.913398] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
  1295  Mar 27 18:37:02 storage-01 kernel: [ 6505.913401] Process swapper/3 (pid: 0, threadinfo ffff880ff8df6000, task ffff880ff8deef60)
  1296  Mar 27 18:37:02 storage-01 kernel: [ 6505.913443]  00000000ffffffed ffffffff8100d24d 0000000000000000 497a78de89677bff
  1297  Mar 27 18:37:02 storage-01 kernel: [ 6505.913447]  00000000cfad5d94 0000000000000003 0000000000000000 ffffffff8133fc5c
  1298  Mar 27 18:37:02 storage-01 kernel: [ 6505.913450]  0000000000000000 cd959d69a7aa94a2 0000000000000000 0000000000000000
  1299  Mar 27 18:37:02 storage-01 kernel: [ 6505.913496]  [] ? cpu_idle+0xaf/0xf2
  1300  Mar 27 18:37:02 storage-01 kernel: [ 6505.913499]  [] ? start_secondary+0x1d5/0x1db
  1301  Mar 27 18:37:02 storage-01 kernel: [ 6505.914473] Call Trace:
  1302  Mar 27 18:37:02 storage-01 kernel: [ 6505.914476]  [] ? cpu_idle+0xaf/0xf2
  1303  Mar 27 18:37:02 storage-01 kernel: [ 6505.914478]  [] ? start_secondary+0x1d5/0x1db
  1304  Mar 27 18:37:02 storage-01 kernel: [ 6505.914482] NMI backtrace for cpu 5
  1305  Mar 27 18:37:02 storage-01 kernel: [ 6505.914484] CPU 5
  1306  Mar 27 18:37:02 storage-01 kernel: [ 6505.914486] Modules linked in: ext4 crc16 jbd2 ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop zfs(P) zunicode(P) zavl(P) zcommon(P) znvpair(P) spl(O) zlib_deflate i5000_edac edac_core radeon iTCO_wdt iTCO_vendor_support snd_pcm snd_page_alloc ttm snd_timer i5k_amb snd drm_kms_helper coretemp soundcore drm power_supply i2c_algo_bit ioatdma i2c_i801 serio_raw pcspkr i2c_core evdev parport_pc parport shpchp container processor button thermal_sys ext3 mbcache jbd ses enclosure sg sd_mod crc_t10dif ata_generic uhci_hcd floppy ata_piix libata ehci_hcd 3w_9xxx usbcore igb mpt2sas dca usb_common raid_class scsi_transport_sas scsi_mod e1000e [last unloaded: scsi_wait_scan]
  1307  Mar 27 18:37:02 storage-01 kernel: [ 6505.914556]
  1308  Mar 27 18:37:02 storage-01 kernel: [ 6505.914559] Pid: 0, comm: swapper/5 Tainted: P           O 3.2.0-4-amd64 #1 Debian 3.2.39-2 Supermicro X7DB8/X7DB8
  1309  Mar 27 18:37:02 storage-01 kernel: [ 6505.914563] RIP: 0010:[]  [] mwait_idle+0x7f/0xac
  1310  Mar 27 18:37:02 storage-01 kernel: [ 6505.914572] RSP: 0018:ffff880ff8e3fef8  EFLAGS: 00000246
  1311  Mar 27 18:37:02 storage-01 kernel: [ 6505.914574] RAX: 0000000000000000 RBX: ffff880ff8e3e000 RCX: 0000000000000000
  1312  Mar 27 18:37:02 storage-01 kernel: [ 6505.914575] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff880ff8e3e000
  1313  Mar 27 18:37:02 storage-01 kernel: [ 6505.914577] RBP: ffffffff8168ef80 R08: 0000000000000000 R09: ffffffff8168ef80
  1314  Mar 27 18:37:02 storage-01 kernel: [ 6505.914579] R10: ffff880b00000018 R11: ffff880ff8e3e000 R12: 0000000000000000
  1315  Mar 27 18:37:02 storage-01 kernel: [ 6505.914581] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
  1316  Mar 27 18:37:02 storage-01 kernel: [ 6505.914584] FS:  0000000000000000(0000) GS:ffff880fffd40000(0000) knlGS:0000000000000000
  1317  Mar 27 18:37:02 storage-01 kernel: [ 6505.914586] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
  1318  Mar 27 18:37:02 storage-01 kernel: [ 6505.914588] CR2: 000000000060a3a0 CR3: 0000000001605000 CR4: 00000000000006e0
  1319  Mar 27 18:37:02 storage-01 kernel: [ 6505.914590] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  1320  Mar 27 18:37:02 storage-01 kernel: [ 6505.914593] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
  1321  Mar 27 18:37:02 storage-01 kernel: [ 6505.914595] Process swapper/5 (pid: 0, threadinfo ffff880ff8e3e000, task ffff880ff8e2e200)
  1322  Mar 27 18:37:02 storage-01 kernel: [ 6505.914638]  00000000ffffffed ffffffff8100d24d 0000000000000000 3350923e16940454
  1323  Mar 27 18:37:02 storage-01 kernel: [ 6505.914643]  0000000019d76d22 0000000000000005 0000000000000000 ffffffff8133fc5c
  1324  Mar 27 18:37:02 storage-01 kernel: [ 6505.914646]  0000000000000000 8281c18f0be4ccf6 0000000000000000 0000000000000000
  1325  Mar 27 18:37:02 storage-01 kernel: [ 6505.914695]  [] ? cpu_idle+0xaf/0xf2
  1326  Mar 27 18:37:02 storage-01 kernel: [ 6505.914700]  [] ? start_secondary+0x1d5/0x1db
  1327  Mar 27 18:37:02 storage-01 kernel: [ 6505.915675] Call Trace:
  1328  Mar 27 18:37:02 storage-01 kernel: [ 6505.915677]  [] ? cpu_idle+0xaf/0xf2
  1329  Mar 27 18:37:02 storage-01 kernel: [ 6505.915680]  [] ? start_secondary+0x1d5/0x1db
  1330  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002] NMI backtrace for cpu 7
  1331  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002] CPU 7
  1332  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002] Modules linked in: ext4 crc16 jbd2 ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop zfs(P) zunicode(P) zavl(P) zcommon(P) znvpair(P) spl(O) zlib_deflate i5000_edac edac_core radeon iTCO_wdt iTCO_vendor_support snd_pcm snd_page_alloc ttm snd_timer i5k_amb snd drm_kms_helper coretemp soundcore drm power_supply i2c_algo_bit ioatdma i2c_i801 serio_raw pcspkr i2c_core evdev parport_pc parport shpchp container processor button thermal_sys ext3 mbcache jbd ses enclosure sg sd_mod crc_t10dif ata_generic uhci_hcd floppy ata_piix libata ehci_hcd 3w_9xxx usbcore igb mpt2sas dca usb_common raid_class scsi_transport_sas scsi_mod e1000e [last unloaded: scsi_wait_scan]
  1333  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]
  1334  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002] Pid: 53, comm: kswapd0 Tainted: P           O 3.2.0-4-amd64 #1 Debian 3.2.39-2 Supermicro X7DB8/X7DB8
  1335  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002] RIP: 0010:[]  [] __const_udelay+0x9/0x20
  1336  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002] RSP: 0018:ffff880fffdc3e20  EFLAGS: 00000046
  1337  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002] RAX: 0000000000000000 RBX: 0000000000002710 RCX: 0000000000000040
  1338  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002] RDX: 0000000000a2c561 RSI: 0000000000000200 RDI: 0000000000418958
  1339  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002] RBP: ffffffff81622280 R08: 0000000000000002 R09: 0000000000000000
  1340  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002] R10: ffffffff81433340 R11: ffffffff81433340 R12: ffffffff81622380
  1341  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002] R13: ffff880fffdc3f50 R14: ffff880fffdcea78 R15: ffffffff8106bed4
  1342  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002] FS:  0000000000000000(0000) GS:ffff880fffdc0000(0000) knlGS:0000000000000000
  1343  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
  1344  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002] CR2: 00007f5b8c287e02 CR3: 0000000001605000 CR4: 00000000000006e0
  1345  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  1346  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
  1347  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002] Process kswapd0 (pid: 53, threadinfo ffff880fbdcbe000, task ffff880ff8974af0)
  1348  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  ffffffff81024bf1 ffff880fffdcedc0 ffffffff81095a40 0000000000000048
  1349  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  ffffffff810398a4 0000000000000001 0000000000000007 0000000000000000
  1350  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  0000000000000007 ffff880fffdc3f50 ffff880fffdcea78 ffffffff8106bed4
  1351  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? arch_trigger_all_cpu_backtrace+0x6c/0x7b
  1352  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? __rcu_pending+0x82/0x358
  1353  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? cpuacct_update_stats+0x3f/0x50
  1354  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? tick_nohz_handler+0xd0/0xd0
  1355  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? rcu_check_callbacks+0x90/0xcc
  1356  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? update_process_times+0x31/0x63
  1357  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? tick_sched_timer+0x6a/0x90
  1358  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? __run_hrtimer+0xac/0x135
  1359  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? hrtimer_interrupt+0xd5/0x1af
  1360  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? call_softirq+0x1c/0x30
  1361  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? smp_apic_timer_interrupt+0x6f/0x82
  1362  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? apic_timer_interrupt+0x6e/0x80
  1363  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? remove_vm_area+0x45/0x7f
  1364  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? __vunmap+0x35/0xb8
  1365  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? spl_slab_reclaim+0x2c0/0x324 [spl]
  1366  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? spl_kmem_cache_reap_now+0x188/0x1f5 [spl]
  1367  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? arc_kmem_reap_now+0x5a/0x91 [zfs]
  1368  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? arc_shrinker_func+0x6b/0xed [zfs]
  1369  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? shrink_slab+0x18f/0x24d
  1370  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? balance_pgdat+0x283/0x4b7
  1371  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? kswapd+0x2d3/0x303
  1372  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? add_wait_queue+0x3c/0x3c
  1373  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? balance_pgdat+0x4b7/0x4b7
  1374  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? kthread+0x76/0x7e
  1375  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? kernel_thread_helper+0x4/0x10
  1376  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? kthread_worker_fn+0x139/0x139
  1377  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? gs_change+0x13/0x13
  1378  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002] Call Trace:
  1379  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]    [] ? arch_trigger_all_cpu_backtrace+0x6c/0x7b
  1380  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? __rcu_pending+0x82/0x358
  1381  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? cpuacct_update_stats+0x3f/0x50
  1382  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? tick_nohz_handler+0xd0/0xd0
  1383  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? rcu_check_callbacks+0x90/0xcc
  1384  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? update_process_times+0x31/0x63
  1385  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? tick_sched_timer+0x6a/0x90
  1386  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? __run_hrtimer+0xac/0x135
  1387  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? hrtimer_interrupt+0xd5/0x1af
  1388  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? call_softirq+0x1c/0x30
  1389  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? smp_apic_timer_interrupt+0x6f/0x82
  1390  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? apic_timer_interrupt+0x6e/0x80
  1391  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]    [] ? remove_vm_area+0x45/0x7f
  1392  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? __vunmap+0x35/0xb8
  1393  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? spl_slab_reclaim+0x2c0/0x324 [spl]
  1394  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? spl_kmem_cache_reap_now+0x188/0x1f5 [spl]
  1395  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? arc_kmem_reap_now+0x5a/0x91 [zfs]
  1396  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? arc_shrinker_func+0x6b/0xed [zfs]
  1397  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? shrink_slab+0x18f/0x24d
  1398  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? balance_pgdat+0x283/0x4b7
  1399  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? kswapd+0x2d3/0x303
  1400  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? add_wait_queue+0x3c/0x3c
  1401  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? balance_pgdat+0x4b7/0x4b7
  1402  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? kthread+0x76/0x7e
  1403  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? kernel_thread_helper+0x4/0x10
  1404  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? kthread_worker_fn+0x139/0x139
  1405  Mar 27 18:37:02 storage-01 kernel: [ 6505.912002]  [] ? gs_change+0x13/0x13
  1406  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003] NMI backtrace for cpu 2
  1407  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003] CPU 2
  1408  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003] Modules linked in: ext4 crc16 jbd2 ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop zfs(P) zunicode(P) zavl(P) zcommon(P) znvpair(P) spl(O) zlib_deflate i5000_edac edac_core radeon iTCO_wdt iTCO_vendor_support snd_pcm snd_page_alloc ttm snd_timer i5k_amb snd drm_kms_helper coretemp soundcore drm power_supply i2c_algo_bit ioatdma i2c_i801 serio_raw pcspkr i2c_core evdev parport_pc parport shpchp container processor button thermal_sys ext3 mbcache jbd ses enclosure sg sd_mod crc_t10dif ata_generic uhci_hcd floppy ata_piix libata ehci_hcd 3w_9xxx usbcore igb mpt2sas dca usb_common raid_class scsi_transport_sas scsi_mod e1000e [last unloaded: scsi_wait_scan]
  1409  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003]
  1410  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003] Pid: 0, comm: swapper/2 Tainted: P           O 3.2.0-4-amd64 #1 Debian 3.2.39-2 Supermicro X7DB8/X7DB8
  1411  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003] RIP: 0010:[]  [] mwait_idle+0x7f/0xac
  1412  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003] RSP: 0018:ffff880ff8de3ef8  EFLAGS: 00000246
  1413  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003] RAX: 0000000000000000 RBX: ffff880ff8de2000 RCX: 0000000000000000
  1414  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff880ff8de2000
  1415  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003] RBP: ffffffff8168ef80 R08: 0000000000000000 R09: 000000000007d8aa
  1416  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003] R10: 000000000007d8aa R11: ffff880ff8de2000 R12: 0000000000000000
  1417  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
  1418  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003] FS:  0000000000000000(0000) GS:ffff880fffc80000(0000) knlGS:0000000000000000
  1419  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
  1420  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003] CR2: 00007fb76e758f30 CR3: 0000000a046e6000 CR4: 00000000000006e0
  1421  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  1422  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
  1423  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003] Process swapper/2 (pid: 0, threadinfo ffff880ff8de2000, task ffff880ff8de1610)
  1424  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003]  00000000ffffffed ffffffff8100d24d 0000000000000000 4373871c848559e3
  1425  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003]  000000009e70c749 0000000000000002 0000000000000000 ffffffff8133fc5c
  1426  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003]  0000000000000000 e319de52095d4e99 0000000000000000 0000000000000000
  1427  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003]  [] ? cpu_idle+0xaf/0xf2
  1428  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003]  [] ? start_secondary+0x1d5/0x1db
  1429  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003] Call Trace:
  1430  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003]  [] ? cpu_idle+0xaf/0xf2
  1431  Mar 27 18:37:02 storage-01 kernel: [ 6505.916003]  [] ? start_secondary+0x1d5/0x1db
  1432  Mar 27 18:37:02 storage-01 kernel: [ 6505.912013] NMI backtrace for cpu 4
  1433  Mar 27 18:37:02 storage-01 kernel: [ 6505.912013] CPU 4
  1434  Mar 27 18:37:02 storage-01 kernel: [ 6505.912013] Modules linked in: ext4 crc16 jbd2 ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc loop zfs(P) zunicode(P) zavl(P) zcommon(P) znvpair(P) spl(O) zlib_deflate i5000_edac edac_core radeon iTCO_wdt iTCO_vendor_support snd_pcm snd_page_alloc ttm snd_timer i5k_amb snd drm_kms_helper coretemp soundcore drm power_supply i2c_algo_bit ioatdma i2c_i801 serio_raw pcspkr i2c_core evdev parport_pc parport shpchp container processor button thermal_sys ext3 mbcache jbd ses enclosure sg sd_mod crc_t10dif ata_generic uhci_hcd floppy ata_piix libata ehci_hcd 3w_9xxx usbcore igb mpt2sas dca usb_common raid_class scsi_transport_sas scsi_mod e1000e [last unloaded: scsi_wait_scan]
  1435  Mar 27 18:37:02 storage-01 kernel: [ 6505.912013]
  1436  Mar 27 18:37:02 storage-01 kernel: [ 6505.912013] Pid: 0, comm: swapper/4 Tainted: P           O 3.2.0-4-amd64 #1 Debian 3.2.39-2 Supermicro X7DB8/X7DB8
  1437  Mar 27 18:37:02 storage-01 kernel: [ 6505.912013] RIP: 0010:[]  [] mwait_idle+0x7f/0xac
  1438  Mar 27 18:37:02 storage-01 kernel: [ 6505.912013] RSP: 0018:ffff880ff8e2bef8  EFLAGS: 00000246
  1439  Mar 27 18:37:02 storage-01 kernel: [ 6505.912013] RAX: 0000000000000000 RBX: ffff880ff8e2a000 RCX: 0000000000000000
  1440  Mar 27 18:37:02 storage-01 kernel: [ 6505.912013] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff880ff8e2a000
  1441  Mar 27 18:37:02 storage-01 kernel: [ 6505.912013] RBP: ffffffff8168ef80 R08: 0000000000000000 R09: 0000000000000000
  1442  Mar 27 18:37:02 storage-01 kernel: [ 6505.912013] R10: ffff880ff8e05620 R11: ffff880ff8e2a000 R12: 0000000000000000
  1443  Mar 27 18:37:02 storage-01 kernel: [ 6505.912013] R13: 0000000000
mengesb commented 11 years ago

Here are some of the poor performance benchmarks that I'm seeing still:

*** Performing tests on zd48, blocksize = 4k ***
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 4
Writes: 12652
Reads: 36236
Re-Reads: 1666547
Random Reads: 1997763
Random Writes: 1466757
#################################################
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 64
Writes: 16448
Reads: 34041
Re-Reads: 1567571
Random Reads: 2044404
Random Writes: 1882307
#################################################
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 256
Writes: 18021
Reads: 30887
Re-Reads: 888381
Random Reads: 2024703
Random Writes: 1941902
#################################################
####### IOZONE MULTI-THREAD TEST RESULTS ########
File Size (GB): 4
Record Length (KB): 64
Writes:    14679.34 KB/sec
Reads:  1076848.34 KB/sec
Re-Reads:  1187719.03 KB/sec
Random Reads:  1172257.69 KB/sec
Random Writes:    56898.93 KB/sec
#################################################
############### Bonnie++ RESULTS ################
Bonnie Version: 1.96
Hostname: storage-01
Date Run: 1364437551
Test File Size: 125G
Seq Block Writes KB p/s: 57815
Seq Block Re-Writes KB p/s: 22718
Seq Block Reads KB p/s: 47361
Random Seeks p/s: 1206
#################################################
*** Performing tests on zd64, blocksize = 8k ***
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 4
Writes: 15287
Reads: 78785
Re-Reads: 1599585
Random Reads: 1825014
Random Writes: 1354681
#################################################
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 64
Writes: 54286
Reads: 58605
Re-Reads: 1489750
Random Reads: 1897617
Random Writes: 1880008
#################################################
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 64
Writes: 54286
Reads: 58605
Re-Reads: 1489750
Random Reads: 1897617
Random Writes: 1880008
#################################################
############## IOZONE TEST RESULTS ##############
File Size (GB): 16
Record Length (KB): 256
Writes: 34189
Reads: 60424
Re-Reads: 149303
Random Reads: 1449086
Random Writes: 659444
#################################################
####### IOZONE MULTI-THREAD TEST RESULTS ########
File Size (GB): 4
Record Length (KB): 64
Writes:    29651.81 KB/sec
Reads:   327992.78 KB/sec
Re-Reads:  1293782.94 KB/sec
Random Reads:   998739.78 KB/sec
Random Writes:    50520.75 KB/sec
#################################################

Tests ran on zvols with block sizes incrementing from 4k to 128k (doubling each step).

behlendorf commented 8 years ago

Closing as stale, a lot has canged in the last 3 years since this was filed.