pingcap / go-ycsb

A Go port of Yahoo! Cloud Serving Benchmark (YCSB)
Apache License 2.0
602 stars 244 forks source link

It take a long time to compute when changing ZipfianConstant #301

Open Fischer0522 opened 3 months ago

Fischer0522 commented 3 months ago

Here is my workload for a custom kv store:

recordcount=10000
operationcount=10000
workload=core

readallfields=true

readproportion=0.5
updateproportion=0.5
scanproportion=0
insertproportion=0

requestdistribution=zipfian

When testing the performance of my kvstore for handling data skew, I noticed in generator/zipfian.go that the comments stated, "be aware: initializing this generator may take a long time if there are lots of items to choose from (e.g. over a minute for 100 million objects)." However, when I changed ZipfianConstant from 0.99 to 0.5, it took almost 10 minutes to prepare the data, which seems much slower than described.

I also noticed that most of the time was spent on the zetaStatic function, and n (i.e., itemCount) is a huge number. If I want to make it faster, can I just use a smaller n (itemCount)? Will this affect correctness?

func zetaStatic(st int64, n int64, theta float64, initialSum float64) float64 {
    sum := initialSum

    for i := st; i < n; i++ {
        sum += 1 / math.Pow(float64(i+1), theta)
    }

    return sum
}