Open chusneowork opened 1 month ago
Hello,
Thanks for raising this issue.
I understand that you want to use a custom serializer for your testing. The current plugin suppports only the default key/value serializer
To implement custom serializer, you need to copy the serializer jar to the
I have gone through your jmeter.log file and found that you are loading 2 jars as part of the run.
If wf-components-binaryprotocol-base-70.3.0.jar is your custom serializer jar, then ensure the package path is properly given If wf-components-binaryprotocol-base-70.3.0.jar is not your custom serializer then you need to create one and place it
How to Build custom serializer : Refer here
LMK, if you need any help with this.
Thanks,
Thanks @rollno748 for your quick answer.
I added the jar files to the "Add directory or jar to classpath", but now after your suggestion, I've copied the needed JAR into the
See below the signature of the serializer which is included/implemented already:
Then into the Kakfa producer config I put the Serializer value "com.tomtomwork.webfleet.components.binaryprotocol.kafka.serialization.ProtocolBlockSerializer"
And then when executing the jmeter testplan I got error on the kafka message
"Error sending message to kafka topic : org.apache.kafka.common.errors.SerializationException: Can't convert value of class java.lang.String to class com.tomtomwork.webfleet.components.binaryprotocol.kafka.serialization.ProtocolBlockSerializer specified in value.serializer"
You mentioned that the current plugin suppports only the default key/value serializer, so I am afraid I can NOT use a different default serializer no? Or what you mean with a default serializer, that "only" any of those can be used, right?
Even, using a default serializer, such as "BytesSerializer", I see this error, so I can not use any of those default serializers...
Error sending message to kafka topic : org.apache.kafka.common.errors.SerializationException: Can't convert value of class java.lang.String to class org.apache.kafka.common.serialization.BytesSerializer specified in value.serializer
Hello @chusneowork
Thanks for the update. As per the documentation, it should support all the default serialization and deserialization methods.
With your serializer, it clearly says that the return type would be an Byte array.
so, you should be using the following configuration for the producer
key.serializer : org.apache.kafka.common.serialization.StringSerializer value.serializer : org.apache.kafka.common.serialization.ByteArraySerializer
If the above didn't work for you, you can try the following 3 ways
Using Json serializer
For JsonSerializer you can download the kafka-support jar from mvn and place it in
In the producer config you have configure the json serilizer key.serializer : org.apache.kafka.common.serialization.StringSerializer value.serializer : org.springframework.kafka.support.serializer.JsonSerializer
Writing custom jsr223 pre-processor for converting your message as string
Compiling the plugin to support your own serialization (I have designed this in such way to pick up on the jar, not sure - why, it is not working)
@rollno748 thanks for your quick answer. I tried the mentioned steps but yet not working. (except re-compiling the plugin)
We have a working kafka generation using our own serializer writted with custom jsr223, but without using your plugin, I'd like to make it work using your di-kafkameter plugin which is really great :-)
So, I've tried to use the value.serializer you mentioned, but even though the same problem:
Response code 500 and message error: "Error sending message to kafka topic : org.apache.kafka.common.errors.SerializationException: Can't convert value of class java.lang.String to class org.apache.kafka.common.serialization.ByteArraySerializer specified in value.serializer"
One doubt I've got is about the input for "Kafka message". I am introducing a json, but at the end the field is a String isn't? So, when the plugin gets the input kafka message is handling it as a String instead of a ByteArray or any of the types allowed no?
https://github.com/rollno748/di-kafkameter/blob/master/src/main/java/com/di/jmeter/kafka/sampler/KafkaProducerSampler.java#L129-L135
Hi @chusneowork
I have made some changes to the plugin, that should support other serialization. I will validate and push it over the weekend.
Thanks
Nice! Thanks @rollno748 , looking forward to that version! :-)
Hi @rollno748 , is the new plugin version available? I would like to try it but I thing changes are not yet available.
Hello @chusneowork
i have pushed the code to a branch and built the plugin
https://github.com/rollno748/di-kafkameter/releases/download/1.3/di-kafkameter-1.3.jar
Can you test this in your scenario and let me know.
If you still see any issues, kindly share the JSR223 code, which you are using in your case - If possible
HI @rollno748 , I am trying the new di-kafkameter 1.3 you generated for me, I delete the di-kafkameter-1.2.jar from the folder lib/ext and copied the v1.3, and when I try to send message, it is failing..
See logs when opening Jmeter after copied the v1.3 in the lib/ext folder
Not sure if I am doing something wrong to try that version v1.3 of di-kafkameter. Can you support here?
Note 1: I put it back the version 1.2, and the message was sent as expected. Note 2: I tried to update the Plugin via Plugin Manager, but the version 1.3 is not recognized
I've checked the commits you did under 1.3 and those looks promising, so as soon as I am able to try this new version, I have hope that it will work :-)
@chusneowork Can you validate the variable name in kafka producer sampler is right as given in the config element ?
If you still see issues, Can you please attach the jmeter.log
You can mask the hostnames, username and password with dummy values
Ei, you are right, the kafka producer name from the sampler was different than in the config element, after adjusting that, I am able to send kafka message. Let's go to validate the serialization issue. I will inform you during today Thanks @rollno748
@rollno748 I just did some tries, but it seems not to work yet:
My Kafka Producer Sampler Message looks like that (json text file)
After tried different Serializer values, none of them worked:
Only works with the String serializer (org.apache.kafka.common.serialization.StringSerializer)
@chusneowork Can you please attach the jmeter.log
You can mask the hostnames, username and password with dummy values
@chusneowork Hi, i have a similar problem: i'm able to load the confluent serializer but i have this error:
Error sending message to kafka topic : java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidConfigurationException: Schema being registered is incompatible with an earlier schema for subject "AlertingAPI-value", details: [{errorType:'TYPE_MISMATCH', description:'The type (path '/') of a field in the new schema does not match with the old schema', additionalInfo:'reader type: STRING not compatible with writer type: RECORD'}, {oldSchemaVersion: 2}, {oldSchema: '{"type":"record","name":"AlertEvent", "namespace":"...... " ; error code: 409
i think it can't convert the json i'm passing in the right way to the avro schema saved in schema registry. Can you help me?
Thank you
Hi @rollno748 , have you been able to check the jmeter log? Have you find something? Looking forward to your response.
Hi @Roby84 , can you share your jmeter file, or the json and the kafka configuration you are trying to send? I can try to check... However, as we are trying to fix in this thread, the problem is that with the current DI-KafkaMeter Plugin only one serializer works (org.apache.kafka.common.serialization.StringSerializer), any other different serializer does not work.
Hi @chusneowork, yes i know, i'm using the kafkaAvroSerializer of confluent. I have imported all lib required and fixed configuration. But the error is:
There is a way to have more detailed error message or a stacktrace? It seems to send a string and not an avro message...
Hello @chusneowork @Roby84
i have lost my saved store due to lap crash. Will push a fix soon.
Apologies for the delay.
@rollno748 thanks for the heads up. Hope you did not lost a lot of stuff. Looking forward to the update
@Roby84 have you tried to enable the debug log?
Hi @chusneowork it works and says that before error:
Checking compatibility of reader {"type":"record","name":"AlertEvent","namespace":"com.datatransferobjects.alert.avro","doc":"Event to send an alert in asynchronous way", "fields":[{"name":"header","type":{"type":"record","name":"AlertEventHeader", "fields":[{"name":"traceId","type":["null","string"],"doc":"Identifier", "default":null},{"name":"messageId","type":"string","doc":"UUID"}, {"name":"origMessageId","type":["null","string"],"doc":"UUID of original event","default":null}, {"name":"source","type":"string","doc":"Application name that produced this message"}]}},{"name":"body","type":{"type":"record","name":"AlertEventBody", "fields":[{"name":"productCode","type":"string","doc":"product code","default":"0"}, {"name":"msgDetail","type":["null",{"type":"map","values":{"type":"string","avro.java.string":"String"},"avro.java.string":"String"}], "doc":"placeholder parameters.", "default":null}]}}],"aliases":["Alerting"]} with writer "string"
so it find avro from schema registry but the writer is still string.
How i can solve?
Thank you..
@Roby84 I think the problem you have is related to the current implementation of the serialization, even you send a json the writer seems to read it as string. I would say let's wait for the changes that @rollno748 needs to do and hope that version will work. Let's wait for it.
The thing is that we need to modify the “Kafka Producer Serializer value” because for some specific message we have a different serializer, but the only value that works is “org.apache.kafka.common.serialization.StringSerializer”. I would need to use the value “org.apache.kafka.common.serialization.Serializer” or “com.tomtomwork.webfleet.components.binaryprotocol.kafka.serialization.ProtocolBlockSerializer”, but when doing so and try to send the message, there is an error
See examples of different serializers value below:
See the error when sending the kafka sampler below:![image](https://github.com/user-attachments/assets/ee913f03-3b3d-4c97-bcf3-f51c2d92e51f)
In the code plugin (https://github.com/rollno748/di-kafkameter/blob/master/src/main/java/com/di/jmeter/kafka/config/KafkaProducerConfig.java#L97-L98), it seems the interface allowed is only java.io.Serialize, but I would need to be able other serializers which use/implements “org.apache.kafka.common.serialization.Serializer” instead of “org.apache.kafka.common.serialization.StringSerializer”. It seems the plugin only allows string serializer (See he error message => Can't convert value of class java.lang.String to class org.apache.kafka.common.serialization.ByteArraySerializer specified in value.serializer)
So, oher Serializer Values such as the ones below should be able to be used:
Is there any way to add support for other serializers into the di-kafkaplugin?
Attached log files: