Change log description
Adds support for using the BatchReader instead of the StreamingReader within consumers
Purpose of the change
Fixes #89
What the code does
When reading historic data from a stream data can be read using either the Streaming Reader or the Batch (bounded) Reader. This adds a new -batchreaders option that signifies that the stream should be consumed using the BatchClient. In this mode all the segments from with a Stream bound are supplied by the Pravega Client and the consumers are free to consume them in any order.
The existing consumer has been renamed PravegaStreamingReaderWorker and a new PravegaBatchReaderWorker has been introduced. Each reader is assigned a certain number of the Batch segments with an attempt to give each consumer an equal number of the segments. If the segment count is lower than the number of specified consumers then the consumer count is reduced so that there is one consumer per Batch segment.
-batchreaders signifies the consumers should all be
Batch Readers rather than Streaming
readers
How to verify
When running with the -batchreaders option the test will log the consumer to segment assignments
>pravega-benchmark -controller tcp://localhost:9090 -consumers 10 -scope dave -stream test6 -segments 5 -time 20 -batchreaders
...
...
...
2020-01-12 13:16:49:359 +0000 [main] INFO io.pravega.perf.PravegaPerfTest - Limiting To 5 consumers due to small number of segment ranges
2020-01-12 13:16:49:359 +0000 [main] INFO io.pravega.perf.PravegaPerfTest - Segment Assignment 0 -> test6 0(0:2215032)
2020-01-12 13:16:49:359 +0000 [main] INFO io.pravega.perf.PravegaPerfTest - Segment Assignment 1 -> test6 1(0:2175864)
2020-01-12 13:16:49:359 +0000 [main] INFO io.pravega.perf.PravegaPerfTest - Segment Assignment 2 -> test6 2(0:2194224)
2020-01-12 13:16:49:359 +0000 [main] INFO io.pravega.perf.PravegaPerfTest - Segment Assignment 3 -> test6 3(0:2177904)
2020-01-12 13:16:49:359 +0000 [main] INFO io.pravega.perf.PravegaPerfTest - Segment Assignment 4 -> test6 4(0:2189328)
As the consumers progress through thier assigned segments ranges they will also log progress:
2020-01-12 13:16:49:379 +0000 [ForkJoinPool-2-worker-4] INFO io.pravega.perf.PravegaBatchReaderWorker - id:3 Starting Segment test6, 3(0:2177904)
2020-01-12 13:16:49:379 +0000 [ForkJoinPool-2-worker-3] INFO io.pravega.perf.PravegaBatchReaderWorker - id:2 Starting Segment test6, 2(0:2194224)
2020-01-12 13:16:49:379 +0000 [ForkJoinPool-2-worker-1] INFO io.pravega.perf.PravegaBatchReaderWorker - id:0 Starting Segment test6, 0(0:2215032)
2020-01-12 13:16:49:379 +0000 [ForkJoinPool-2-worker-5] INFO io.pravega.perf.PravegaBatchReaderWorker - id:4 Starting Segment test6, 4(0:2189328)
2020-01-12 13:16:49:379 +0000 [ForkJoinPool-2-worker-2] INFO io.pravega.perf.PravegaBatchReaderWorker - id:1 Starting Segment test6, 1(0:2175864)
...
...
2020-01-12 13:16:49:595 +0000 [ForkJoinPool-2-worker-2] INFO io.pravega.perf.PravegaBatchReaderWorker - id:1 Completed Segment test6, 1(0:2175864)
2020-01-12 13:16:49:595 +0000 [ForkJoinPool-2-worker-2] INFO io.pravega.perf.PravegaBatchReaderWorker - id:1 Completed all assigned assignedSegments
2020-01-12 13:16:49:601 +0000 [ForkJoinPool-2-worker-4] INFO io.pravega.perf.PravegaBatchReaderWorker - id:3 Completed Segment test6, 3(0:2177904)
2020-01-12 13:16:49:601 +0000 [ForkJoinPool-2-worker-3] INFO io.pravega.perf.PravegaBatchReaderWorker - id:2 Completed Segment test6, 2(0:2194224)
2020-01-12 13:16:49:601 +0000 [ForkJoinPool-2-worker-4] INFO io.pravega.perf.PravegaBatchReaderWorker - id:3 Completed all assigned assignedSegments
2020-01-12 13:16:49:601 +0000 [ForkJoinPool-2-worker-3] INFO io.pravega.perf.PravegaBatchReaderWorker - id:2 Completed all assigned assignedSegments
...
...
Change log description Adds support for using the BatchReader instead of the StreamingReader within consumers
Purpose of the change Fixes #89
What the code does When reading historic data from a stream data can be read using either the Streaming Reader or the Batch (bounded) Reader. This adds a new
-batchreaders
option that signifies that the stream should be consumed using the BatchClient. In this mode all the segments from with a Stream bound are supplied by the Pravega Client and the consumers are free to consume them in any order.The existing consumer has been renamed
PravegaStreamingReaderWorker
and a newPravegaBatchReaderWorker
has been introduced. Each reader is assigned a certain number of the Batch segments with an attempt to give each consumer an equal number of the segments. If the segment count is lower than the number of specified consumers then the consumer count is reduced so that there is one consumer per Batch segment.How to verify When running with the
-batchreaders
option the test will log the consumer to segment assignmentsAs the consumers progress through thier assigned segments ranges they will also log progress:
Signed-off-by: David Maddison david.maddison@dell.com