BitFunnel / mg4j-workbench

Java tools for evaluating BitFunnel performance compared to an mg4j baseline.
GNU Lesser General Public License v3.0
1 stars 2 forks source link

Conflicting timing measurements #16

Open MikeHopcroft opened 7 years ago

MikeHopcroft commented 7 years ago

In 525d6d6965f9259646f11c7d5aa2bb64a40e4ecd, QueryLogRunner.go() implements three different strategies for measuring the time spent processing queries:

  1. Take the difference between the time the last thread finished processing and the first thread started.
  2. Take the difference between the start time of of the finishSynchronizer and that of the the performanceSynchronizer.
  3. Take the difference between the time the threads joined and the time the threads were started.

In the results, below processing 06.efficiency_topics.all, we see that method (3) measures 24.25s, a value almost twice as large as the other two methods. The difference stems from the fact that QueryProcessorThread.run() processes the entire query log twice, in order to warm up the system. Methods (1) and (2) only measure the time for the second run. We should clean up the time reporting.

Thread count: 8 Query count: 100000 Total time: 11.793248 Total time (synchronizer): 11.793267 Total time (simple): 24.249336 QPS: 8479.414311