iotune can be wrong on EBS drive

scylladb / seastar

High performance server-side application framework

http://seastar.io

Apache License 2.0

8.39k stars 1.55k forks source link

iotune can be wrong on EBS drive #1030

Open xemul opened 2 years ago

xemul commented 2 years ago

In scylla issue #9906 there's such an io-properties.yaml file

disks:
  - mountpoint: /var/lib/scylla
    read_iops: 32004
    read_bandwidth: 262607760
    write_iops: 34608
    write_bandwidth: 265524816

however, the drive was created with default bandwidth, which is 128MB/s. This number also shows up if re-running iotune by hands

fruch commented 1 year ago

@xemul

what is the recommendation for such a case ? manually crafted io-properties.yaml file ?

for example for 2TB gp3 with 16K IOPS ? https://aws.amazon.com/ebs/general-purpose/

disks:
  - mountpoint: /var/lib/scylla
    read_iops: 16000
    read_bandwidth: 262607760
    write_iops: 16000
    write_bandwidth: 265524816

fruch commented 1 year ago

@xemul

what is the recommendation for such a case ? manually crafted io-properties.yaml file ?

for example for 2TB gp3 with 16K IOPS ? https://aws.amazon.com/ebs/general-purpose/
disks:
  - mountpoint: /var/lib/scylla
    read_iops: 16000
    read_bandwidth: 262607760
    write_iops: 16000
    write_bandwidth: 265524816

We have such a case, but it's keep failing cause of this iotune overshooting

mykaul commented 1 year ago

@xemul what is the recommendation for such a case ? manually crafted io-properties.yaml file ? for example for 2TB gp3 with 16K IOPS ? https://aws.amazon.com/ebs/general-purpose/
disks:
  - mountpoint: /var/lib/scylla
    read_iops: 16000
    read_bandwidth: 262607760
    write_iops: 16000
    write_bandwidth: 265524816
We have such a case, but it's keep failing cause of this iotune overshooting

I wonder what the interaction between gp3 (which has consistent performance vs gp2 burst) and the EBS network limitations of some nodes - which is bursty - can they sustain the same IO bandwidth. What does fio say, for example)?
Are we creating those EBS disks from scratch or from a snapshot?

xemul commented 1 year ago

@xemul

what is the recommendation for such a case ? manually crafted io-properties.yaml file ?

for example for 2TB gp3 with 16K IOPS ? https://aws.amazon.com/ebs/general-purpose/
disks:
  - mountpoint: /var/lib/scylla
    read_iops: 16000
    read_bandwidth: 262607760
    write_iops: 16000
    write_bandwidth: 265524816

These numbers are burst-values, on sustained workload the drive won't work like that. I'd check it with fio. Well -- with diskplorer :D #1297 was supposed to facilitate that

mykaul commented 1 year ago

@xemul what is the recommendation for such a case ? manually crafted io-properties.yaml file ? for example for 2TB gp3 with 16K IOPS ? https://aws.amazon.com/ebs/general-purpose/
disks:
  - mountpoint: /var/lib/scylla
    read_iops: 16000
    read_bandwidth: 262607760
    write_iops: 16000
    write_bandwidth: 265524816
These numbers are burst-values, on sustained workload the drive won't work like that. I'd check it with fio. Well -- with diskplorer :D #1297 was supposed to facilitate that

gp3 does not use burst though. That's why I was wondering if it's something with the node limits.

regevran commented 4 weeks ago

Closing as there is no interest in EBS. If you think differently please reopen.

avikivity commented 4 weeks ago

Lack of interest by one user doesn't mean the bug is fixed.