When lightning using a cloud disk, the import speed drops and the tikv cpu cannot be full.

pingcap / tidb-lightning

This repository has been moved to https://github.com/pingcap/br

Apache License 2.0

142 stars 66 forks source link

When lightning using a cloud disk, the import speed drops and the tikv cpu cannot be full. #486

Closed shuijing198799 closed 3 years ago

shuijing198799 commented 3 years ago

Question

When I use gp2 in the aws cloud. The machine types as below,

c5.9xlarge 36 core72G mem  900G gp2 disk

The speed is rarely low than use the nvme disk and the tikv cpu can't be full

BTW, TiKV use i3.2xlarge machine, 8c 61G 1900G nvme disk.

glorv commented 3 years ago

After our investigating, we found the aws cloud disk's io bandwidth is related to the disk size (the bigger disk the higher bandwidth), so with relative small disk, the disk io bandwidth is not big enough for lightning's reading and writing, thus, lightning speed is restricted by the io bandwidth.

@cyliu0 @shuijing198799 could you add some bench result with different size cloud disk here?

cyliu0 commented 3 years ago

Since the c5.9xlarge and 3TB gp2 is working just fine, and we have some enhancements already, close this one first.