search

pingcap / go-ycsb

A Go port of Yahoo! Cloud Serving Benchmark (YCSB)
Apache License 2.0
583 stars 241 forks source link

Avoid coordinated omission in the generator #26

Open abdullin opened 6 years ago

abdullin commented 6 years ago

It appears that go-ycsb might be affected by the coordinated omission. This could mask some serious performance issues.

In order to avoid CO, the load generator could launch a new request on a fixed schedule (e.g. by spawning a new goroutine) without waiting for the previous requests to complete.

Here is an example of a load benchmark executed against FoundationDB. First chart was produced by an experiment using a conventional load tester (client runs 1000 goroutines pushing requests sequentially). Second chart was produced by the load tester after switching to avoid coordinated omission and drive the load at predefined 20kHz - it reveals a bottleneck caused by the conflict contention and actually kills the client with OOM afterwards :)

ddlznfdxuaedfny jpg-large

summary

PS: This discussion has originated at the FoundationDB forum, Alex Miller suggested filing an issue here.

siddontang commented 6 years ago

Hi @abdullin

I just port from Java YCSB, so maybe this problem still exists.

Second chart was produced by the load tester after switching to avoid coordinated omission and drive the load at predefined 20kHz

Can you show me the change for it? Then I can investigate why OOM.

abdullin commented 6 years ago

Hi @siddontang

I just port from Java YCSB, so maybe this problem still exists.

In that case, the Java code also suffers from the coordinated omission.

Can you show me the change for it? Then I can investigate why OOM.

That was my own load tester for the FoundationDB layers, not go-ycsb. OOM there was caused by the unavoidable fact that it kept on launching go routines at a fixed rate even though they were locked in a retry loop and never garbage collected. That is a behaviour one would expect in such situation

abdullin commented 6 years ago

I found the related issue in the original Java project: https://github.com/brianfrankcooper/YCSB/issues/731

It appears, they have started discussion on correcting coordinated omission in 2016, but the issue is still pending.

siddontang commented 6 years ago

@abdullin

I find that YCSB uses an intended time which can relieve this problem but still can't fix completely.

tyy1 commented 5 years ago

I want to use go-ycsb to test tikv, when make ,some packages can't be found because of timeout,who knows how to solve this problem？？ These packages are from golang. org/x/ and cloud.google.com/go.