radiasoft / devops

sysadmin
Apache License 2.0
11 stars 4 forks source link

RAID & SSD performance #142

Closed robnagler closed 6 years ago

robnagler commented 6 years ago

Documenting performance of performance of PERC H310 (non-RAID) vs motherboard S110 in AHCI and RAID modes. The summary is that SSD performance varies wildly (2x). Perhaps its about wear, because most of these disks had been used 3K+ hours. However, there doesn't seem to be a correlation.

The SMART stats have been saved in full on f1's db.

Benchmark code:

lvcreate -y --wipesignatures y -L 20G -n test centos
mkfs.xfs /dev/mapper/centos-test
mount /dev/mapper/centos-test /mnt
cd /mnt
# sync on close (every 10G with enough RAM (64G))
i=$(date +%s); dd bs=1M count=10240 if=/dev/zero of=test conv=fdatasync; echo $(( $(date +%s) - $i ))
rm -f test
sync
# sync on write (every 1M)
i=$(date +%s); dd bs=1M count=10240 if=/dev/zero of=test oflag=dsync; echo $(( $(date +%s) - $i ))
cd
umount /mnt
lvremove /dev/mapper/centos-test

Swapped slots and disks f2 & f3:

Swapped disks f2 & f3:

Removed sdb from f2:

Remove sda put sdb in sda slot from f2:

Removed H310 from f2 136s disk in slot 0:

Some configuration:

SMART info for drives (Writes=Total_LBAs_Written, Hours=Power_On_Hours):

Mach-Dev Writes Hours
f11-sda   119E6   306
f2-sda   2857E6  3620
f2-sdb   2709E6  3616
f3-sda   4237E6  3658
f3-sdb    263E6  3658
f4-sda    533E6  3659
f4-sdb   6459E6  3659
f7-sda   1615E6 36350
f7-sdb   4610E6 35015

iostat

Bit of a red herring. iostat 1 while running dsync test on f2:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 58743840   3380 6405120    0    0     0 74830 3545 6096  0  1 99  0  0
 0  0      0 58668548   3380 6480976    0    0     0 73805 3302 6540  0  1 99  0  0
 0  0      0 58591740   3380 6558976    0    0     0 75860 3718 5751  0  1 99  1  0
 0  1      0 58511940   3380 6637740    0    0     0 76879 3618 6019  0  1 99  1  0
 1  0      0 58433568   3380 6716860    0    0     0 76881 3540 6884  0  1 99  0  0
 0  0      0 58349632   3380 6800896    0    0     0 82005 3768 7112  0  1 99  1  0
 0  1      0 58260360   3380 6888652    0    0     0 85079 3977 6599  0  1 99  1  0
 0  1      0 58182408   3380 6966532    0    0     0 75860 3582 5736  0  1 99  1  0
 0  0      0 58101428   3380 7048000    0    0     0 79956 3783 6722  0  1 99  1  0
 0  0      0 58019780   3380 7130376    0    0     0 79955 3867 6587  0  1 99  1  0
 0  0      0 57939324   3380 7209944    0    0     0 77905 3813 6205  0  1 99  1  0
 0  1      0 57852468   3380 7297048    0    0     0 84054 4001 7072  0  1 99  1  0
 0  0      0 57774636   3380 7374944    0    0     0 75861 3556 6544  0  1 99  1  0

on f3:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 62882420   3356 2266916    0    0     0 117881 5315 10353  0  1 98  1  0
 0  1      0 62759968   3356 2390028    0    0     0 119934 5155 10148  0  1 98  1  0
 0  0      0 62638844   3356 2510308    0    0     0 116862 5148 9382  0  1 98  1  0
 0  1      0 62514976   3356 2634508    0    0     0 120959 5285 10501  0  1 98  1  0
 1  0      0 62399292   3356 2751804    0    0     0 113786 4940 9064  0  1 98  1  0
 0  0      0 62278528   3356 2871444    0    0     0 116855 5264 9215  0  1 98  1  0
 0  1      0 62160376   3356 2989464    0    0     0 114809 5124 9057  0  1 98  1  0
 0  0      0 62040028   3356 3110248    0    0     0 117887 5236 10282  0  1 98  1  0
 0  1      0 61914328   3356 3235740    0    0     0 121984 5156 10772  0  1 98  1  0
 0  1      0 61798476   3356 3351284    0    0     0 112760 5134 9541  0  1 98  1  0
 0  0      0 61674668   3356 3475632    0    0     0 120982 5354 9564  0  1 98  1  0
 1  1      0 61553560   3356 3595340    0    0     0 116859 5027 10058  0  1 98  1  0
 0  1      0 61435180   3356 3715728    0    0     0 116860 5252 9298  0  1 98  1  0

References on performance of hardware RAID vs madam:

robnagler commented 6 years ago

We are stable.