Coordinated Omission in operation latency measurement

[nitsanw]

Looking at how latency is reported e.g.: https://github.com/Netflix/ndbench/blob/master/ndbench-core/src/main/java/com/netflix/ndbench/core/operations/ReadOperation.java#L43

            Long startTime = System.nanoTime();
            String value = client.readSingle(key);
            monitor.recordReadLatency((System.nanoTime() - startTime)/1000);

The above implies the operation is synchronous and that the harness has no notion of schedule (we don't know when we intend to start measurement). This generally leads to coordinate omission. The fix is usually to add some notion of schedule mapping each operation to it's intended start time. This is a common issue in load generators, see following blog posts(which I wrote) on the issue, it's impact and resolution in YCSB and Cassandra Stress: http://psy-lob-saw.blogspot.com/2015/03/fixing-ycsb-coordinated-omission.html http://psy-lob-saw.blogspot.com/2016/07/fixing-co-in-cstress.html

[ipapapa]

@nitsanw yes, we had this discussion back in July with @vinaykumarchella. We will get to it asap. Thanks for the letting us know.

[timiblossom]

@nitsanw You are right that we ignore coordinate omission (CO) at that level. However, most of our plugins have their own metrics as well as a scheduler that this framework has no access. In the case of Dyno (https://github.com/Netflix/Dyno) or Java Datastax Driver, they both maintain their own metrics hence can do a better measurement there.

What we put there is to measure, at a fixed throughput, the response latencies what the users can see under pressure. Our metric report in this framework is only just for a subset of the performance metrics, throughputs and response latencies. At this level, our report would fall into the category "Fixed" that you mentioned in http://psy-lob-saw.blogspot.com/2016/07/fixing-co-in-cstress.html under "Who Reports What?" section. In addition, for most of our tests, we also use many test machines and threads that CO can also become negligible to the overall response times.

As Gil Tene said, "CO is a testing methodology and reporting problem, and has nothing to do with the system under test and what it is used for, so no type of system is immune, only types of testers are. Even real systems with a single real world client are susceptible."

Already mentioned, what we are looking for is to measure what application users would see when they use a particular client or driver (Dyno, Java Datastax Driver, etc.) and we delegate CO measurements in the plugins themselves. Furthermore, we have cases that the threads of this framework execute async call to the underline drivers resulting to non-measurable CO duration at this layer.

A part from this, we collect many other metrics along the way at different stages of the testing pipelines. By comparing them together, we can detect the skewness caused by CO and repeat the tests with a different set-up if needed.

What do you think? Please feel free to give us a PR if you see where we can address it better.

Thanks.

[nitsanw]

If operations are sync then the above method of measurement is wrong. If they are async they are meaningless unless you got some Quasar like magic to make it look sync while being async. IIUC your code calls into the plugins, so it cannot delegate the notion of schedule to them. If your tests are supposed to be "at a fixed throughput, the response latencies what the users can see under pressure" then you have an implied schedule. The recorded metrics might help you reconstruct that you've missed your schedule, but it's much easier to just record the intended schedule.

[nitsanw]

In fact, looking further into the benchmark driver it seems like the measurement method is very similar to cassandra-stress before it was fixed. Using Google RateLimiter and therefore not having access to intended operation time and not measuring it.