Process to use load generate to generate new load and push to Worker

[pawanjuly3]

Is there a yaml configuration which can be used to push newly generated load to Kafka,Cassandra without using Data Source?

[vajda]

No. You need to have some kind of data source. Having said that, you can still use Ranger as data source and generate data you would like to push to Kafka, Cassandra. In that case, you don't need prepared data in some data source. Take a look at examples For more info about Ranger syntax, take a look at Ranger documentation.

[pawanjuly3]

Here I created a Ranger to Kafka yaml and it failes with error while yaml is well formed. Exception: 2018-08-19 15:04:16.385 ERROR [async-worker-thread-1] i.s.b.w.InternalWorker - Error while accepting payload at worker: io.smartcat.berserker.kafka.worker.KafkaWorker. Error: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.String at io.smartcat.berserker.kafka.worker.KafkaWorker.accept(KafkaWorker.java:55) at io.smartcat.berserker.kafka.worker.KafkaWorker.accept(KafkaWorker.java:14) at io.smartcat.berserker.worker.InternalWorker.lambda$createAndInitThreadPoolExecutor$4(InternalWorker.java:215) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

Command to Run: java -jar ./berserker-runner/target/berserker-runner-0.0.13-SNAPSHOT.jar -c ./b.yaml YAML File: b.yaml load-generator-configuration: data-source-configuration-name: Ranger rate-generator-configuration-name: default worker-configuration-name: Kafka metrics-reporter-configuration-name: SimpleConsoleReporter thread-count: 1 queue-capacity: 1

data-source-configuration: values: id: uuid() firstName: random(['Peter', 'Mike', 'Steven', 'Joshua', 'John', 'Brandon']) lastName: random(['Smith', 'Johnson', 'Williams', 'Davis', 'Jackson', 'White', 'Lewis', 'Clark']) age: random(20..45)

kafkaMessage:
  key: circular(1..1000000, 1)
  value: string("{}",$firstName )
  topic: random(['topic1', 'topic2', 'topic3'])

output: $kafkaMessage

rate-generator-configuration: rates: r: 1 output: $r

worker-configuration: async: false topic: topic1 producer-configuration: bootstrap.servers: BuildVM:9092

[vajda]

See my answer at #58

[pawanjuly3]

With this change , it works with out errors. But looks it is not connecting to kafka. I do not see even topic in Kafka. While my kafka is working fine , I have tested it manually. Kafka and berserker both are on same VM.

io.smartcat.berserker.generatedThroughput count = 130 mean rate = 1.00 events/second 1-minute rate = 0.98 events/second 5-minute rate = 0.87 events/second 15-minute rate = 0.83 events/second io.smartcat.berserker.successProcessedThroughput count = 0 mean rate = 0.00 events/second 1-minute rate = 0.00 events/second 5-minute rate = 0.00 events/second 15-minute rate = 0.00 events/second io.smartcat.berserker.totalProcessedThroughput count = 2 mean rate = 0.02 events/second 1-minute rate = 0.02 events/second 5-minute rate = 0.01 events/second 15-minute rate = 0.00 events/second