HW versus SW raid and iotune - Githubissues

scylladb / seastar

High performance server-side application framework

http://seastar.io

Apache License 2.0

8.3k stars 1.54k forks source link

HW versus SW raid and iotune #810

Closed tarzanek closed 4 months ago

tarzanek commented 3 years ago

2 SSDs on the same hardware, with different aproaches to raid:

Hardware raid0

:::::::::::::: 
/etc/scylla.d/cpuset.conf 
:::::::::::::: 
CPUSET="--cpuset 1-23,25-47 " 
:::::::::::::: 
/etc/scylla.d/io.conf 
:::::::::::::: 
SEASTAR_IO="--num-io-queues=1 --io-properties-file=/etc/scylla.d/io_properties.yaml" 
:::::::::::::: 
/etc/scylla.d/io_properties.yaml 
:::::::::::::: 
disks: 
 - mountpoint: / 
   read_iops: 63143 
   read_bandwidth: 733626944 
   write_iops: 45624 
   write_bandwidth: 248903968 
:::::::::::::: 
::::::::::::::

Software raid0

:::::::::::::: 
:::::::::::::: 
/etc/scylla.d/cpuset.conf 
:::::::::::::: 
CPUSET="--cpuset 1-23,25-47 " 
:::::::::::::: 
/etc/scylla.d/io.conf 
:::::::::::::: 
SEASTAR_IO="--num-io-queues=46 --io-properties-file=/etc/scylla.d/io_properties.yaml" 
:::::::::::::: 
/etc/scylla.d/io_properties.yaml 
:::::::::::::: 
disks: 
 - mountpoint: /data 
   read_iops: 138689 
   read_bandwidth: 963001664 
   write_iops: 127368 
   write_bandwidth: 867488384

note that I don't know why the autodetection from iotune used only 1 queue, shouldn't we use more and not default to 1?
Shouldn't iotune be smarted in the hw raid situation?

@xemul any ideas about above situation?

tarzanek commented 3 years ago

HW raid used is coming from PERC H740P

xemul commented 3 years ago

@tarzanek , how did you generate the io.conf? This repo's iotune doesn't write the num-io-queues parameter.

gcarmin commented 3 years ago

@xemul scylla version 2019.1.18. and we run scylla_setup

xemul commented 3 years ago

HW raid used is coming from PERC H740P

And the software raid is ... ?

xemul commented 3 years ago

@gcarmin , 2019.1.18 is claimed not to start yet, is it 2019.1.17?

dyasny commented 3 years ago

And the software raid is ... ?

mdraid, same 2 ssd disks

xemul commented 3 years ago

OK. Let's clarify the version. The 2019.1.18 seems not to exist, and 2019.1.17 has seastar with buggy iotune which may report semi-random results. The fix is scylladb/seastar@83919516c9

xemul commented 3 years ago

Also there's scylladb/scylla#7146 that shows that IOPS evaluation is still problematic.

gcarmin commented 3 years ago

@xemul sorry for the type -scylla version is 2019.1.8

xemul commented 3 years ago

@gcarmin , @tarzanek , the iotune in 2019.1.8 has a BUG too and it's totally worth backporting the fix before making any assumptions about the numbers.

gcarmin commented 3 years ago

@xemul - is this BUG 8391951 is solved in 2020.1.3? (I may ask one of my customer to give it a shoot)

xemul commented 3 years ago

@gcarmin , no it's not yet there.

gcarmin commented 3 years ago

@xemul - Is it in latest OSS? Is there a time-line to backport it to 2020.X

xemul commented 3 years ago

@gcarmin , yes 4.1.8 does have it. About backporting I think we should ask ... @bhalevy ?

bhalevy commented 3 years ago

@gcarmin , yes 4.1.8 does have it. About backporting I think we should ask ... @bhalevy ?

I'm all for it. @avikivity ?

mykaul commented 4 months ago

Closing, either irrelevant, fixed or the HW raid is not the best path - we prefer SW raid.