Issue 39: Minor fixes to make it easier to deploy P3 experiments on OMB

[RaulGracia]

Change log description Done several minor improvements in the repo to easily execute experiments:

• Fixed Kafka client version in some yaml files.
• Fixed Kafka Terraform version requirements.
• Added vars.yaml to Pulsar and Kafka drivers to make it easier to execute experiments with P3 Driver.
• Fixed Pravega README.
• Set the both ZK endpoint properties in controller.config.properties to support versions after and vefore 0.8.

Purpose Fixes #39.

How to verify it Executed multiple experiments on AWS.

Signed-off-by: Raúl Gracia raul.gracia@emc.com

[jingerbread]

Running mvn license:format in project root this will add all required license headers. May be we should add all required license header to build project without -Dlicense.skip=true flag?

[jingerbread]

[changed] JVM_MEM for bin/benchmark-worker from -Xms12G -Xmx12G to -Xms6G -Xmx6G and for bin/benchmark from -Xmx1Gto -Xmx4G, similar to Pravega and Pulsar settings"

[jingerbread]

Successfully deployed Kafka and ran test. Approved PR.

[jingerbread]

May be we should add Readme for Kafka:

#---------------------------------------
#  INSTALL KAFKA ON AWS 
#--------------------------------------
# Generate key
yes | ssh-keygen -P "" -f  ~/.ssh/kafka_aws
cd openmessaging-benchmark/
# Run `mvn license:format` or use `-Dlicense.skip=true` flag
# will add licenses to files that are missing license header
# mvn license:format
mvn clean install -DskipTests=true -Dlicense.skip=true 
# Create AWS resources
cd driver-kafka/deploy
terraform init
# Create terraform resources
echo "yes" | terraform apply
export TF_STATE=./
# 11 min
time ansible-playbook --user ec2-user --inventory `which terraform-inventory` deploy.yaml
#-------------------
#    RUN TEST
#-------------------
ssh -i ~/.ssh/kafka_aws ec2-user@<client0 PublicIP>
sudo su
cd /opt/benchmark
# Create payload file
fallocate -l 10000 payload/payload-10k.data
# Create workload
cat << 'EOT' > workloads/1-topic-16-partitions-10k-rate-35k.yaml
name: 1 topic / 16 partition / 10kb / 50k
topics: 1
partitionsPerTopic: 16
messageSize: 10000
payloadFile: "payload/payload-10k.data"
subscriptionsPerTopic: 1
consumerPerSubscription: 1
producersPerTopic: 1
producerRate: 50000
consumerBacklogSizeGB: 0
testDurationMinutes: 3
EOT
# Run omb test
bin/benchmark --drivers driver-kafka/kafka.yaml workloads/1-topic-16-partitions-10k-rate-35k.yaml 2>&1 | tee test1.log && grep Aggregated test1.log
12:04:06.208 [main] INFO io.openmessaging.benchmark.WorkloadGenerator - ----- Aggregated Pub rate 28773.3 msg/s / 287.7 MB/s | Cons rate 28773.6 msg/s / 287.7 MB/s
12:04:06.336 [main] INFO io.openmessaging.benchmark.WorkloadGenerator - ----- Aggregated Pub Latency (ms) avg: 115.7 - 50%: 13.5 - 95%: 498.0 - 99%: 687.8 - 99.9%: 808.9 - 99.99%: 899.4 - Max: 918.5
12:04:06.341 [main] INFO io.openmessaging.benchmark.WorkloadGenerator - ----- Aggregated E2E Latency (ms) avg: 121.2 - 50%: 19.0 - 95%: 504.0 - 99%: 694.0 - 99.9%: 818.0 - 99.99%: 904.0 - Max: 955.0
#--------------------------------
# DESTROY AWS resources
#---------------------------------
cd driver-kafka/deploy
time terraform destroy -force