pravega / pravega-benchmark

Performance benchmark tool for Pravega
Apache License 2.0
8 stars 22 forks source link

Issue 89: Add Batch Reader Support #92

Open maddisondavid opened 4 years ago

maddisondavid commented 4 years ago

Change log description Adds support for using the BatchReader instead of the StreamingReader within consumers

Purpose of the change Fixes #89

What the code does When reading historic data from a stream data can be read using either the Streaming Reader or the Batch (bounded) Reader. This adds a new -batchreaders option that signifies that the stream should be consumed using the BatchClient. In this mode all the segments from with a Stream bound are supplied by the Pravega Client and the consumers are free to consume them in any order.

The existing consumer has been renamed PravegaStreamingReaderWorker and a new PravegaBatchReaderWorker has been introduced. Each reader is assigned a certain number of the Batch segments with an attempt to give each consumer an equal number of the segments. If the segment count is lower than the number of specified consumers then the consumer count is reduced so that there is one consumer per Batch segment.

 -batchreaders                       signifies the consumers should all be
                                     Batch Readers rather than Streaming
                                     readers

How to verify When running with the -batchreaders option the test will log the consumer to segment assignments

>pravega-benchmark -controller tcp://localhost:9090 -consumers 10 -scope dave -stream test6 -segments 5 -time 20 -batchreaders
...
...
...
2020-01-12 13:16:49:359 +0000 [main] INFO io.pravega.perf.PravegaPerfTest - Limiting To 5 consumers due to small number of segment ranges
2020-01-12 13:16:49:359 +0000 [main] INFO io.pravega.perf.PravegaPerfTest - Segment Assignment 0 -> test6 0(0:2215032)
2020-01-12 13:16:49:359 +0000 [main] INFO io.pravega.perf.PravegaPerfTest - Segment Assignment 1 -> test6 1(0:2175864)
2020-01-12 13:16:49:359 +0000 [main] INFO io.pravega.perf.PravegaPerfTest - Segment Assignment 2 -> test6 2(0:2194224)
2020-01-12 13:16:49:359 +0000 [main] INFO io.pravega.perf.PravegaPerfTest - Segment Assignment 3 -> test6 3(0:2177904)
2020-01-12 13:16:49:359 +0000 [main] INFO io.pravega.perf.PravegaPerfTest - Segment Assignment 4 -> test6 4(0:2189328)

As the consumers progress through thier assigned segments ranges they will also log progress:

2020-01-12 13:16:49:379 +0000 [ForkJoinPool-2-worker-4] INFO io.pravega.perf.PravegaBatchReaderWorker - id:3 Starting Segment test6, 3(0:2177904)
2020-01-12 13:16:49:379 +0000 [ForkJoinPool-2-worker-3] INFO io.pravega.perf.PravegaBatchReaderWorker - id:2 Starting Segment test6, 2(0:2194224)
2020-01-12 13:16:49:379 +0000 [ForkJoinPool-2-worker-1] INFO io.pravega.perf.PravegaBatchReaderWorker - id:0 Starting Segment test6, 0(0:2215032)
2020-01-12 13:16:49:379 +0000 [ForkJoinPool-2-worker-5] INFO io.pravega.perf.PravegaBatchReaderWorker - id:4 Starting Segment test6, 4(0:2189328)
2020-01-12 13:16:49:379 +0000 [ForkJoinPool-2-worker-2] INFO io.pravega.perf.PravegaBatchReaderWorker - id:1 Starting Segment test6, 1(0:2175864)
...
...
2020-01-12 13:16:49:595 +0000 [ForkJoinPool-2-worker-2] INFO io.pravega.perf.PravegaBatchReaderWorker - id:1 Completed Segment test6, 1(0:2175864)
2020-01-12 13:16:49:595 +0000 [ForkJoinPool-2-worker-2] INFO io.pravega.perf.PravegaBatchReaderWorker - id:1 Completed all assigned assignedSegments
2020-01-12 13:16:49:601 +0000 [ForkJoinPool-2-worker-4] INFO io.pravega.perf.PravegaBatchReaderWorker - id:3 Completed Segment test6, 3(0:2177904)
2020-01-12 13:16:49:601 +0000 [ForkJoinPool-2-worker-3] INFO io.pravega.perf.PravegaBatchReaderWorker - id:2 Completed Segment test6, 2(0:2194224)
2020-01-12 13:16:49:601 +0000 [ForkJoinPool-2-worker-4] INFO io.pravega.perf.PravegaBatchReaderWorker - id:3 Completed all assigned assignedSegments
2020-01-12 13:16:49:601 +0000 [ForkJoinPool-2-worker-3] INFO io.pravega.perf.PravegaBatchReaderWorker - id:2 Completed all assigned assignedSegments
...
...

Signed-off-by: David Maddison david.maddison@dell.com

RaulGracia commented 4 years ago

@maddisondavid can you please have a look to the conflicts so we can merge this one?

RaulGracia commented 4 years ago

@maddisondavid any updates on this one?