When testing the performance of my kvstore for handling data skew, I noticed in generator/zipfian.go that the comments stated, "be aware: initializing this generator may take a long time if there are lots of items to choose from (e.g. over a minute for 100 million objects)." However, when I changed ZipfianConstant from 0.99 to 0.5, it took almost 10 minutes to prepare the data, which seems much slower than described.
I also noticed that most of the time was spent on the zetaStatic function, and n (i.e., itemCount) is a huge number. If I want to make it faster, can I just use a smaller n (itemCount)? Will this affect correctness?
func zetaStatic(st int64, n int64, theta float64, initialSum float64) float64 {
sum := initialSum
for i := st; i < n; i++ {
sum += 1 / math.Pow(float64(i+1), theta)
}
return sum
}
Here is my workload for a custom kv store:
When testing the performance of my kvstore for handling data skew, I noticed in generator/zipfian.go that the comments stated, "be aware: initializing this generator may take a long time if there are lots of items to choose from (e.g. over a minute for 100 million objects)." However, when I changed ZipfianConstant from 0.99 to 0.5, it took almost 10 minutes to prepare the data, which seems much slower than described.
I also noticed that most of the time was spent on the zetaStatic function, and n (i.e., itemCount) is a huge number. If I want to make it faster, can I just use a smaller n (itemCount)? Will this affect correctness?