search

dlangille / zfs_benchmarks

Compare resilver times on various ZFS configurations
3 stars 1 forks source link

First test - copy #8

Open dlangille opened 9 years ago

dlangille commented 9 years ago

The first test, which isn't really a test, will be copying a 1.4TB data set from one pool to other. The original pool will be later involved in destructive tests I need to move to data to a new location.

These are the existing pools: https://gist.github.com/dlangille/58412e268c464297cd45

It might be interesting to compare cp with zfs send | zfs recv.

dlangille commented 9 years ago

The benchmark has started. 2042 UTC on 17 Aug.

[root@varm /music/dan]# time cp -Rp Touhou\ lossless\ music\ collection/ /benchmarking/

[dan@varm:~] $ zpool list
NAME            SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
benchmarking   21.8T  1.71T  20.0T         -     3%     7%  1.00x  ONLINE  -
music            30T  1.86T  28.1T         -     3%     6%  1.00x  ONLINE  -
random_mirror  2.72T   480K  2.72T         -     0%     0%  1.00x  ONLINE  -
zroot           220G  1.52G   218G         -     0%     0%  1.00x  ONLINE  -
[dan@varm:~] $ zpool status
  pool: benchmarking
 state: ONLINE
  scan: resilvered 202G in 0h39m with 0 errors on Fri Aug 14 19:00:07 2015
config:

    NAME                    STATE     READ WRITE CKSUM
    benchmarking            ONLINE       0     0     0
      raidz3-0              ONLINE       0     0     0
        gpt/disk_Z2T2UJJAS  ONLINE       0     0     0
        gpt/disk_13Q8U6GYS  ONLINE       0     0     0
        gpt/disk_256BWVLGS  ONLINE       0     0     0
        gpt/disk_256BY66GS  ONLINE       0     0     0
        gpt/disk_255BV69GS  ONLINE       0     0     0
        gpt/disk_255BS4NGS  ONLINE       0     0     0
        gpt/disk_255BUT1GS  ONLINE       0     0     0
        gpt/disk_35AL161GS  ONLINE       0     0     0

errors: No known data errors

  pool: music
 state: ONLINE
  scan: resilvered 173G in 1h25m with 0 errors on Fri Aug 14 19:15:59 2015
config:

    NAME                       STATE     READ WRITE CKSUM
    music                      ONLINE       0     0     0
      raidz3-0                 ONLINE       0     0     0
        gpt/disk_653AK2MXFS9A  ONLINE       0     0     0
        gpt/disk_653EK93XFS9A  ONLINE       0     0     0
        gpt/disk_653DK7WCFS9A  ONLINE       0     0     0
        gpt/disk_6525K2DGFS9A  ONLINE       0     0     0
        gpt/disk_652FK58FFS9A  ONLINE       0     0     0
        gpt/disk_653BK12IFS9A  ONLINE       0     0     0
        gpt/disk_653EK93QFS9A  ONLINE       0     0     0
        gpt/disk_653IK1IBFS9A  ONLINE       0     0     0
        gpt/disk_6539K3OJFS9A  ONLINE       0     0     0
        gpt/disk_653BK12FFS9A  ONLINE       0     0     0
        gpt/disk_653DK7WBFS9A  ONLINE       0     0     0

errors: No known data errors

  pool: random_mirror
 state: ONLINE
  scan: none requested
config:

    NAME                    STATE     READ WRITE CKSUM
    random_mirror           ONLINE       0     0     0
      mirror-0              ONLINE       0     0     0
        gpt/disk_255BUT4GS  ONLINE       0     0     0
        gpt/disk_256BWJ1GS  ONLINE       0     0     0
        gpt/disk_256BY5SGS  ONLINE       0     0     0

errors: No known data errors

  pool: zroot
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Sat Jul 25 14:50:11 2015
config:

    NAME          STATE     READ WRITE CKSUM
    zroot         ONLINE       0     0     0
      mirror-0    ONLINE       0     0     0
        gpt/zfs0  ONLINE       0     0     0
        gpt/zfs1  ONLINE       0     0     0

errors: No known data errors
[dan@varm:~] $
dlangille commented 9 years ago

The time was 

real    67m49.591s
user    0m1.020s
sys     8m48.490s

# zpool list
NAME            SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
benchmarking   21.8T  4.08T  17.7T         -     6%    18%  1.00x  ONLINE  -
music            30T  1.86T  28.1T         -     3%     6%  1.00x  ONLINE  -
random_mirror  2.72T   480K  2.72T         -     0%     0%  1.00x  ONLINE  -
zroot           220G  1.52G   218G         -     0%     0%  1.00x  ONLINE  -
dlangille commented 9 years ago

Please check my math:

[root@varm /music/dan]# du -ck -d 0
1450756166      .
1450756166      total
[root@varm /music/dan]#

That total is the number of 1024-byte blocks.

Thus total bytes is 1450756166 * 1024 = 1485574313984

Total time in seconds is 67 * 60 + 49.591 = 4069.591

Bytes / second = 1485574313984 / 4069.591 = 365042657.599744

Which translates to 365.042658 MB / s

dlangille commented 9 years ago

Copy from the mirror to the 11-disk raidz3:

[root@varm /benchmarking]# time cp -Rp music /music/COPY-music

real    50m41.279s
user    0m1.514s
sys     8m7.768s

Which translates to 488 MB / s

dlangille commented 9 years ago

Set up for a zfs-send test:

[root@varm /benchmarking]# zfs create benchmarking/zfs-test
[root@varm /benchmarking]# ls -l
total 1048882745
drwxr-xr-x  1345 root  dan             1346 Aug 18 01:14 music
-rw-r--r--     1 root  wheel  1073741824000 Jul 24 22:44 test-1000G
drwxr-xr-x     2 root  wheel              2 Aug 25 20:18 zfs-test
[root@varm /benchmarking]# time cp -Rp music zfs-test/
real    170m18.543s
user    0m1.113s
sys     11m39.716s
dlangille commented 9 years ago

Take the snapshot:

[dan@varm:~] $ sudo zfssnapshot / benchmarking/zfs-^Cst@ForSending
[dan@varm:~] $

Then send it

[root@varm ~]# time zfs send benchmarking/zfs-test@ForSending | zfs receive -d music
real    37m37.752s
user    0m0.000s
sys     16m49.575s

Wow, that takes only 20% of the time. Clearly, block copying is much faster. No checksum to compute, etc.

dlangille commented 9 years ago

Compression is on both datasets used above.

dlangille commented 8 years ago

Did another copy:

# time zfs send benchmarking/zfs-test@ForSending | zfs receive -d music_save
real    124m2.069s
user    0m0.000s
sys 17m5.255s

To a new zpool:

.90G   218G         -     0%     0%  1.00x  ONLINE  -
[root@varm /music_save]# zpool status music_save
  pool: music_save
 state: ONLINE
  scan: none requested
config:
    NAME                       STATE     READ WRITE CKSUM
    music_save                 ONLINE       0     0     0
      mirror-0                 ONLINE       0     0     0
        gpt/disk_653EK93PFS9A  ONLINE       0     0     0
        gpt/disk_653DK7WAFS9A  ONLINE       0     0     0
errors: No known data errors