Pepper-Box is kafka load generator plugin for jmeter. It allows to send kafka messages of type plain text(JSON, XML, CSV or any other custom format) as well as java serialized objects.
Pepper-Box includes five main components
Pepper-Box uses Java 8 with java compiler API, hence on JMeter machine JDK 8 should be installed instead of JRE 8.
Install openjdk on Debian, Ubuntu, etc.,
sudo apt-get install openjdk-8-jdk
Install openjdk on Fedora, Oracle Linux, Red Hat Enterprise Linux, etc.,
su -c "yum install java-1.8.0-openjdk-devel"
For windows you can download oracle JDK 8 setup from here
mvn clean install -Djmeter.version=3.0 -Dkafka.version=0.9.0.1
JMeter and Kafka version can be passed dynamically.
Once build is completed, copy jar file to JMETER_HOME/lib/ext directory.
This is java sampler hence in ThreadGroup add sampler as Java Request and select class as com.gslab.pepper.sampler.PepperBoxKafkaSampler
As you can see in above screen, you can configure producer properties and topic details.
Above properties are added by default in sampler as those are more significant in terms of performance in most of the cases. But you can add other non listed kafka properties with prefix "_".
For example to enable SSL properties you can add below properties
_ssl.key.password
_ssl.keystore.location
_ssl.keystore.password
_ssl.truststore.location
_ssl.truststore.password
Pepper-Box PlainText Config is jmeter config element. It takes schema template is input and generates message for each sampler request.
You can add this config element using Thread group --> Add --> Config Element --> Pepper-Box PlainText Config
Input schema template can be in any format
JSON schema template
{
"messageId":{{SEQUENCE("messageId", 1, 1)}},
"messageBody":"{{RANDOM_ALPHA_NUMERIC("abcedefghijklmnopqrwxyzABCDEFGHIJKLMNOPQRWXYZ", 100)}}",
"messageCategory":"{{RANDOM_STRING("Finance", "Insurance", "Healthcare", "Shares")}}",
"messageStatus":"{{RANDOM_STRING("Accepted","Pending","Processing","Rejected")}}",
"messageTime":{{TIMESTAMP()}}
}
XML schema template
<message>
<messageId>{{SEQUENCE("messageId", 1, 1)}}</messageId>
<messageBody>{{RANDOM_ALPHA_NUMERIC("abcedefghijklmnopqrwxyzABCDEFGHIJKLMNOPQRWXYZ", 100)}}</messageBody>
<messageCategory>{{RANDOM_STRING("Finance", "Insurance", "Healthcare", "Shares")}}</messageCategory>
<messageStatus>{{RANDOM_STRING("Accepted","Pending","Processing","Rejected")}}</messageStatus>
<messageTime>{{TIMESTAMP()}}</messageTime>
</message>
Custom schema template
Hello {{FIRST_NAME()}}
This is sample message sending at {{DATE("dd/MM/yyyy HH:mm:ss")}}.
Thanks and Regards,
{{FIRST_NAME()}} {{LAST_NAME()}}
Java serialized objects can be sent to kafka using Pepper-Box Serialized Config Element. This config element can be added using Thread group --> Add --> Config Element --> Pepper-Box Serialized Config
Follow below steps to use this config element,
class name
section (e.g. com.gslab.pepper.Message in above screen). This class should be present in jmeter classpath folder(lib or lib/ext). You can copy jar containing required class to JMETER_HOME/lib/ext folder.Ignore
means field value will not set.Example Class,
package com.gslab.pepper;
import java.io.Serializable;
public class Message implements Serializable{
private long messageId;
private String messageBody;
private String messageStatus;
private String messageCategory;
private long messageTime;
public long getMessageId() {
return messageId;
}
public void setMessageId(long messageId) {
this.messageId = messageId;
}
public String getMessageBody() {
return messageBody;
}
public void setMessageBody(String messageBody) {
this.messageBody = messageBody;
}
public String getMessageStatus() {
return messageStatus;
}
public void setMessageStatus(String messageStatus) {
this.messageStatus = messageStatus;
}
public String getMessageCategory() {
return messageCategory;
}
public void setMessageCategory(String messageCategory) {
this.messageCategory = messageCategory;
}
public long getMessageTime() {
return messageTime;
}
public void setMessageTime(long messageTime) {
this.messageTime = messageTime;
}
}
Please make sure that function return type and field data type should be compatible with each other.
is a partner to the LoadGenerator. It currently expects the messages will be in JSON format and use the structure and content shown in various examples.
Example command-line:
java -cp target/pepper-box-1.0.jar:. com.gslab.pepper.PepperBoxLoadConsumer --consumer-config-file pblg.properties --num-consumers 5 --per-thread-topics YES --test-duration 30 --throughput-per-consumer 10
PepperBoxLoadGenerator is console plaintext load generation utility.
Command,
java -cp pepper-box-1.0.jar com.gslab.pepper.PepperBoxLoadGenerator --schema-file <schema file absolute path> --producer-config-file <producer properties absoulte path> --throughput-per-producer <throughput rate per producer> --test-duration <test duration in seconds> --num-producers <number of producers>
Example
{
"messageId":{{SEQUENCE("messageId", 1, 1)}},
"messageBody":"{{RANDOM_ALPHA_NUMERIC("abcedefghijklmnopqrwxyzABCDEFGHIJKLMNOPQRWXYZ", 100)}}",
"messageCategory":"{{RANDOM_STRING("Finance", "Insurance", "Healthcare", "Shares")}}",
"messageStatus":"{{RANDOM_STRING("Accepted","Pending","Processing","Rejected")}}",
"messageTime":{{TIMESTAMP()}}
}
bootstrap.servers=<Broker List>
zookeeper.servers=<Zookeeper List>
kafka.topic.name=<kafka topic>
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
acks=0
send.buffer.bytes=131072
receive.buffer.bytes=32768
batch.size=16384
linger.ms=0
buffer.memory=33554432
compression.type=none
security.protocol=PLAINTEXT
kerberos.auth.enabled=NO
java.security.auth.login.config=<JAAS File Location>
java.security.krb5.conf=<krb5.conf location>
sasl.kerberos.service.name=<Kerberos service name>
For schema file and producer properties file most of the features same as jmeter plain text config element.
We have also included pepper_box.jmx
jmeter sample test file which can be directly imported in jmeter.
Pepper-Box provides various template functions for random data generation,
Function | Details | Example(For serialized use without {{ }} ) |
Returns |
---|---|---|---|
TIMESTAMP() | current time in long | {{TIMESTAMP()}} |
Long |
TIMESTAMP(startDate, endDate) | Random long date between two Dates | {{TIMESTAMP("01-05-1998 10:30:12","03-03-2017 12:12:12")}} |
Long |
DATE(format) | current date with given format | {{DATE("dd-MM-yyyy HH:mm:ss")}} |
String |
RANDOM_STRING(string1, string2, string3,...) | Random string among given | {{RANDOM_STRING("ONE","TWO","THREE","FOUR")}} |
String |
RANDOM_INT(int1, int2, int3,...) | Random integer among given | {{RANDOM_INT(1, 2, 3, 4)}} |
Integer |
RANDOM_FLOAT(float1, float2, float3,...) | Random float among given | {{RANDOM_FLOAT(1.1F ,2.1F, 3.1F, 4.1F)}} |
Float |
RANDOM_DOUBLE(double1, double2, double3,...) | Random double among given | {{RANDOM_DOUBLE(1.1, 2.1, 3.1, 4.1)}} |
Double |
RANDOM_LONG(long1, long2, long3,...) | Random long among given | {{RANDOM_LONG(1, 2, 3, 4)}} |
Long |
RANDOM_INT_RANGE(min, max) | Random integer among given | {{RANDOM_INT_RANGE(1,100)}} |
Integer |
RANDOM_FLOAT_RANGE(min, max) | Random float between min and max | {{RANDOM_FLOAT_RANGE(1.0F, 100.0F)}} |
Float |
RANDOM_FLOAT_RANGE(min, max) | Random double between min and max | {{RANDOM_FLOAT_RANGE(1.0, 100.0)}} |
Double |
RANDOM_LONG_RANGE(min, max) | Random long between min and max | {{RANDOM_LONG_RANGE(1,100)}} |
Long |
FIRST_NAME() | Random first name | {{FIRST_NAME()}} |
String |
LAST_NAME() | Random last name | {{LAST_NAME()}} |
String |
RANDOM_ALPHA_NUMERIC(charSet, length) | Random string of given length from given char set | {{RANDOM_ALPHA_NUMERIC("abcdefghijklmn", 10)}} |
String |
UUID() | Random UUID | {{UUID()}} |
String |
SEQUENCE(sequenceId, startValue, incrementBy) | Generates incremental sequence | {{SEQUENCE("messageId", 1, 1)}} |
Long |
PHONE() | Random 10 digit phone number | {{PHONE()}} |
String |
GENDER() | Random gender | {{GENDER()}} |
String |
BOOLEAN() | Random boolean | {{BOOLEAN()}} |
boolean |
EMAIL(domain) | Random email id for given domain | {{EMAIL("test.com")}} |
String |
USERNAME() | Random username | {{USERNAME()}} |
String |
IPV4() | Random IPV4 address | {{IPV4()}} |
String |
IPV6() | Random IPV6 address | {{IPV6()}} |
String |
Apart from these functions, you can also add your own custom function in com.gslab.pepper.input.CustomFunctions
class. Please make sure that those are static functions.
Example
public static float AVG(float... floats){
int count = floats.length;
float sum = 0.0;
for (float number : floats){
sum += number;
}
return sum/count;
}
AVG function can be used in schema as shown below,
{{AVG(32.2, 34.5, 64.2)}}
Note: While writing custom functions, please try to keep data in memory or scale your function as much other functions otherwise your custom function itself becomes performance bottlneck. e.g. you need some record ids from RDBMS for some schema fields, instead of querying every time bring all ids inmemory and get random id from those ids.
You can also add manipulations on template functions, for example TIMESTAMP() function returns time in milliseconds but you can get time in seconds,
{{java.util.concurrent.TimeUnit.MILLISECONDS.toSeconds(TIMESTAMP())}}
We would like to special thanks to kafkameter and wrtting custom jmeter plugin blogpost which helped to understand writing custom plugins for JMeter.
We also like to thanks to InMemoryJavaCompiler which helped to understand in memory code compilation.