search

zfsonlinux / pkg-zfs

Native ZFS packaging for Debian and Ubuntu
https://launchpad.net/~zfs-native/+archive/daily
308 stars 55 forks source link

Ubuntu 14.04.3, 0.6.5.3-1~trusty on 4.1.15: arc stuck very small, cpu spinning on txt_sync during writes #189

Closed gdevenyi closed 8 years ago

gdevenyi commented 8 years ago

Just installed a new Ubuntu 14.04.3 and added the ubuntu-zfs PPA.

System is a 4-core atom ( https://www.asus.com/us/Commercial-Servers-Workstations/P9AIC2750SAS4L/specifications/ ) with 16GB ECC ram.

Built the following pool:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            sdd     ONLINE       0     0     0
            sdc     ONLINE       0     0     0
            sdi     ONLINE       0     0     0
            sdj     ONLINE       0     0     0
            sdm     ONLINE       0     0     0
            sdn     ONLINE       0     0     0
            sdq     ONLINE       0     0     0
            sdf     ONLINE       0     0     0

errors: No known data errors

Disks are Segate 2TB SAS drives.

I was rsyncing over from another sever and noticed that the io write is very very slow. Running performance testing iotest.sh I find the writing really slow.

While running iotest: Examining the arc:

cat /proc/spl/kstat/zfs/arcstats
6 1 0x01 91 4368 16849188538 285078069665
name                            type data
hits                            4    2569
misses                          4    3749
demand_data_hits                4    0
demand_data_misses              4    0
demand_metadata_hits            4    2476
demand_metadata_misses          4    3631
prefetch_data_hits              4    0
prefetch_data_misses            4    0
prefetch_metadata_hits          4    93
prefetch_metadata_misses        4    118
mru_hits                        4    2350
mru_ghost_hits                  4    0
mfu_hits                        4    126
mfu_ghost_hits                  4    0
deleted                         4    17
mutex_miss                      4    0
evict_skip                      4    3
evict_not_enough                4    0
evict_l2_cached                 4    0
evict_l2_eligible               4    183296
evict_l2_ineligible             4    0
evict_l2_skip                   4    0
hash_elements                   4    245
hash_elements_max               4    246
hash_collisions                 4    0
hash_chains                     4    0
hash_chain_max                  4    0
p                               4    4197946368
c                               4    8395892736
c_min                           4    33554432
c_max                           4    8395892736
size                            4    769235872
hdr_size                        4    2580040
data_size                       4    762970112
metadata_size                   4    1792000
other_size                      4    1893720
anon_size                       4    763232256
anon_evictable_data             4    0
anon_evictable_metadata         4    0
mru_size                        4    1527808
mru_evictable_data              4    0
mru_evictable_metadata          4    1277952
mru_ghost_size                  4    0
mru_ghost_evictable_data        4    0
mru_ghost_evictable_metadata    4    0
mfu_size                        4    2048
mfu_evictable_data              4    0
mfu_evictable_metadata          4    2048
mfu_ghost_size                  4    0
mfu_ghost_evictable_data        4    0
mfu_ghost_evictable_metadata    4    0
l2_hits                         4    0
l2_misses                       4    0
l2_feeds                        4    0
l2_rw_clash                     4    0
l2_read_bytes                   4    0
l2_write_bytes                  4    0
l2_writes_sent                  4    0
l2_writes_done                  4    0
l2_writes_error                 4    0
l2_writes_lock_retry            4    0
l2_evict_lock_retry             4    0
l2_evict_reading                4    0
l2_evict_l1cached               4    0
l2_free_on_write                4    0
l2_cdata_free_on_write          4    0
l2_abort_lowmem                 4    0
l2_cksum_bad                    4    0
l2_io_error                     4    0
l2_size                         4    0
l2_asize                        4    0
l2_hdr_size                     4    0
l2_compress_successes           4    0
l2_compress_zeros               4    0
l2_compress_failures            4    0
memory_throttle_count           4    0
duplicate_buffers               4    0
duplicate_buffers_size          4    0
duplicate_reads                 4    0
memory_direct_count             4    0
memory_indirect_count           4    0
arc_no_grow                     4    0
arc_tempreserve                 4    0
arc_loaned_bytes                4    0
arc_prune                       4    0
arc_meta_used                   4    6266480
arc_meta_limit                  4    6296919552
arc_meta_max                    4    8970256
arc_meta_min                    4    16777216
arc_need_free                   4    0
arc_sys_free                    4    262369280

And looking at iotop:

 1554 be/4 root        0.00 B/s    0.00 B/s  0.00 % 87.79 % [txg_sync]
  207 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.40 % [kworker/u8:6]
    6 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.09 % [kworker/u8:0]
 3147 be/4 gdevenyi    0.00 B/s  346.38 M/s  0.00 %  0.00 % dd if=/dev/zero of=/storage/iotest.sh-dd-write.1 bs=64k count=8404992
 3148 be/4 gdevenyi    0.00 B/s  316.30 M/s  0.00 %  0.00 % dd if=/dev/zero of=/storage/iotest.sh-dd-write.3 bs=64k count=8404992
 3149 be/4 gdevenyi    0.00 B/s  371.01 M/s  0.00 %  0.00 % dd if=/dev/zero of=/storage/iotest.sh-dd-write.2 bs=64k count=8404992
 3150 be/4 gdevenyi    0.00 B/s  329.10 M/s  0.00 %  0.00 % dd if=/dev/zero of=/storage/iotest.sh-dd-write.4 bs=64k count=8404992
kernelOfTruth commented 8 years ago

please close this and re-open against:

https://github.com/zfsonlinux/zfs/issues

kernelOfTruth commented 8 years ago

Meanwhile:

try to raise zfs_arc_min to 2 GiB and see whether that helps in any way

options zfs zfs_arc_min=0x80000000
gdevenyi commented 8 years ago

Whoops, wrong repo thanks.