search

vesoft-inc / nebula-operator

Operation utilities for Nebula Graph
https://vesoft-inc.github.io/nebula-operator
Apache License 2.0
81 stars 30 forks source link

Benchmark local PV #517

Closed wenhaocs closed 1 month ago

wenhaocs commented 3 months ago

Customer finds that when using local PV, the CPU utilization is abnormally higher than using EBS.

(1) Under the local pv cluster, 20 CPU of storaged is being used which is so high for just 2K QPS. Because prod (with EBS) is using 50 CPU for 25k QPS... (2) The queries and dataset between two clusters are exactly the same.

Here is the configuration:

Local PV cluster Prod cluster
Storaged instance type m6id.32xlarge 128cpu, 512 GiB Mem, 4 x 1900 NVMe SSD c7a.48xlarge 192cpu, 384 GiB mem, EBS Only
# instances 18 18
# partitions 120

Dataset size: 2.5 TB

When benchmarking, please focus on the CPU utilization of storaged under the same QPS. Please use reasonable dataset size, so that dataset is not too small. Below shows the I/Os during user's test and we can see there are I/Os on disk.

Image

wenhaocs commented 1 month ago

It's not required as the performance issue has been resolved.