search

pravega / openmessaging-benchmark

Internal version of OpenMessaging Benchmark with Pravega integration
Apache License 2.0
4 stars 6 forks source link

Load isn't getting generated in case transaction size is higher than 1. #66

Open AndreyKoltsov1997 opened 3 years ago

AndreyKoltsov1997 commented 3 years ago

In case transactions are enabled and events per transaction is set to value higher than 1, load isn't getting generated.

Example Pravega Driver manifest - standalone, enabled transactions, 3 events per transaction.

name: Pravega
driverClass: io.openmessaging.benchmark.driver.pravega.PravegaBenchmarkDriver
...
client:
  controllerURI: "tcp://localhost:9090"
....
enableTransaction: True
eventsPerTransaction: 3 # any value higher than 1
createScope: True
...

Load manifest - 1 topic, 1 producer (writer), 1 consumer (reader):

name: 1 topic / 1 partition / 100b
...
topics: 1
partitionsPerTopic: 1
subscriptionsPerTopic: 1
consumerPerSubscription: 1
producersPerTopic: 1
testDurationMinutes: 1
...

In case OpenMessaging Benchmark would be executed with parameters provided above, the actual parallel read and write wouldn't be executed. Please, note that it doesn't apply to the case when eventsPerTransaction is set to 1.

...
20:01:45.434 [clientInternal-3-10] INFO io.pravega.client.stream.impl.Pinger - Start sending transaction pings.
20:01:54.921 [clientInternal-3-5] INFO io.pravega.client.stream.impl.Pinger - Start sending transaction pings.
20:02:04.408 [clientInternal-3-7] INFO io.pravega.client.stream.impl.Pinger - Start sending transaction pings.
20:02:06.430 [pool-3-thread-1] INFO io.pravega.client.stream.impl.EventStreamReaderImpl - EventStreamReaderImpl( id=ee2096f6-e635-4a32-aee3-fdf62f207d63) at checkpoint b37e3c89-02b2-4e93-987f-ebe12183a35b
20:02:06.431 [pool-3-thread-1] INFO io.pravega.client.stream.impl.EventStreamReaderImpl - Reader ee2096f6-e635-4a32-aee3-fdf62f207d63 completed checkpoint b37e3c89-02b2-4e93-987f-ebe12183a
AndreyKoltsov1997 commented 3 years ago

The issue is related to probeProducers() method that is executed prior to the actual load generation. It expects to write 1 event from producer and read it back from the consumer. Since we commit transaction only once it's populated with the requested amount of events, the data is never sent to Pravega in case its size is higher than 1. Thus, reader (consumer) couldn't get the data from the system since it hadn't been written.