search

rollno748 / di-kafkameter

JMeter Plugin to load test Apache Kafka topics/brokers
34 stars 11 forks source link

Only String Serializer Values are allowed #25

Open chusneowork opened 1 month ago

chusneowork commented 1 month ago

The thing is that we need to modify the “Kafka Producer Serializer value” because for some specific message we have a different serializer, but the only value that works is “org.apache.kafka.common.serialization.StringSerializer”. I would need to use the value “org.apache.kafka.common.serialization.Serializer” or “com.tomtomwork.webfleet.components.binaryprotocol.kafka.serialization.ProtocolBlockSerializer”, but when doing so and try to send the message, there is an error

See examples of different serializers value below:

See the error when sending the kafka sampler below: image

In the code plugin (https://github.com/rollno748/di-kafkameter/blob/master/src/main/java/com/di/jmeter/kafka/config/KafkaProducerConfig.java#L97-L98), it seems the interface allowed is only java.io.Serialize, but I would need to be able other serializers which use/implements “org.apache.kafka.common.serialization.Serializer” instead of “org.apache.kafka.common.serialization.StringSerializer”. It seems the plugin only allows string serializer (See he error message => Can't convert value of class java.lang.String to class org.apache.kafka.common.serialization.ByteArraySerializer specified in value.serializer)

So, oher Serializer Values such as the ones below should be able to be used:

Is there any way to add support for other serializers into the di-kafkaplugin?

Attached log files:

rollno748 commented 1 month ago

Hello,

Thanks for raising this issue.

I understand that you want to use a custom serializer for your testing. The current plugin suppports only the default key/value serializer

To implement custom serializer, you need to copy the serializer jar to the /lib directory and provide the package/class info on the place of the value serializer

I have gone through your jmeter.log file and found that you are loading 2 jars as part of the run.

  1. kafka-clients-3.7.0.jar (di-kafkameter dependency) - which is expected
  2. wf-components-binaryprotocol-base-70.3.0.jar - Is this your custom serializer jar ?

If wf-components-binaryprotocol-base-70.3.0.jar is your custom serializer jar, then ensure the package path is properly given If wf-components-binaryprotocol-base-70.3.0.jar is not your custom serializer then you need to create one and place it

How to Build custom serializer : Refer here

LMK, if you need any help with this.

Thanks,

chusneowork commented 1 month ago

Thanks @rollno748 for your quick answer.

I added the jar files to the "Add directory or jar to classpath", but now after your suggestion, I've copied the needed JAR into the /lib folder, see pic below: image See below the signature of the serializer which is included/implemented already: image

Then into the Kakfa producer config I put the Serializer value "com.tomtomwork.webfleet.components.binaryprotocol.kafka.serialization.ProtocolBlockSerializer" image

And then when executing the jmeter testplan I got error on the kafka message "Error sending message to kafka topic : org.apache.kafka.common.errors.SerializationException: Can't convert value of class java.lang.String to class com.tomtomwork.webfleet.components.binaryprotocol.kafka.serialization.ProtocolBlockSerializer specified in value.serializer" image

You mentioned that the current plugin suppports only the default key/value serializer, so I am afraid I can NOT use a different default serializer no? Or what you mean with a default serializer, that "only" any of those can be used, right?

chusneowork commented 1 month ago

Even, using a default serializer, such as "BytesSerializer", I see this error, so I can not use any of those default serializers...

image image

Error sending message to kafka topic : org.apache.kafka.common.errors.SerializationException: Can't convert value of class java.lang.String to class org.apache.kafka.common.serialization.BytesSerializer specified in value.serializer image

rollno748 commented 1 month ago

Hello @chusneowork

Thanks for the update. As per the documentation, it should support all the default serialization and deserialization methods.

With your serializer, it clearly says that the return type would be an Byte array.

image

so, you should be using the following configuration for the producer

key.serializer : org.apache.kafka.common.serialization.StringSerializer value.serializer : org.apache.kafka.common.serialization.ByteArraySerializer

If the above didn't work for you, you can try the following 3 ways

  1. Using Json serializer

    For JsonSerializer you can download the kafka-support jar from mvn and place it in /lib/

    In the producer config you have configure the json serilizer key.serializer : org.apache.kafka.common.serialization.StringSerializer value.serializer : org.springframework.kafka.support.serializer.JsonSerializer

  2. Writing custom jsr223 pre-processor for converting your message as string

  3. Compiling the plugin to support your own serialization (I have designed this in such way to pick up on the jar, not sure - why, it is not working)

chusneowork commented 1 month ago

@rollno748 thanks for your quick answer. I tried the mentioned steps but yet not working. (except re-compiling the plugin)

We have a working kafka generation using our own serializer writted with custom jsr223, but without using your plugin, I'd like to make it work using your di-kafkameter plugin which is really great :-)

So, I've tried to use the value.serializer you mentioned, but even though the same problem:

image Response code 500 and message error: "Error sending message to kafka topic : org.apache.kafka.common.errors.SerializationException: Can't convert value of class java.lang.String to class org.apache.kafka.common.serialization.ByteArraySerializer specified in value.serializer" image image

One doubt I've got is about the input for "Kafka message". I am introducing a json, but at the end the field is a String isn't? So, when the plugin gets the input kafka message is handling it as a String instead of a ByteArray or any of the types allowed no? image image https://github.com/rollno748/di-kafkameter/blob/master/src/main/java/com/di/jmeter/kafka/sampler/KafkaProducerSampler.java#L129-L135 image

rollno748 commented 1 month ago

Hi @chusneowork

I have made some changes to the plugin, that should support other serialization. I will validate and push it over the weekend.

Thanks

chusneowork commented 1 month ago

Nice! Thanks @rollno748 , looking forward to that version! :-)

chusneowork commented 1 month ago

Hi @rollno748 , is the new plugin version available? I would like to try it but I thing changes are not yet available.

rollno748 commented 4 weeks ago

Hello @chusneowork

i have pushed the code to a branch and built the plugin

https://github.com/rollno748/di-kafkameter/releases/download/1.3/di-kafkameter-1.3.jar

Can you test this in your scenario and let me know.

If you still see any issues, kindly share the JSR223 code, which you are using in your case - If possible

chusneowork commented 3 weeks ago

HI @rollno748 , I am trying the new di-kafkameter 1.3 you generated for me, I delete the di-kafkameter-1.2.jar from the folder lib/ext and copied the v1.3, and when I try to send message, it is failing.. image image

See logs when opening Jmeter after copied the v1.3 in the lib/ext folder image

Not sure if I am doing something wrong to try that version v1.3 of di-kafkameter. Can you support here?

Note 1: I put it back the version 1.2, and the message was sent as expected. Note 2: I tried to update the Plugin via Plugin Manager, but the version 1.3 is not recognized

I've checked the commits you did under 1.3 and those looks promising, so as soon as I am able to try this new version, I have hope that it will work :-)

rollno748 commented 3 weeks ago

@chusneowork Can you validate the variable name in kafka producer sampler is right as given in the config element ?

If you still see issues, Can you please attach the jmeter.log

You can mask the hostnames, username and password with dummy values

chusneowork commented 3 weeks ago

Ei, you are right, the kafka producer name from the sampler was different than in the config element, after adjusting that, I am able to send kafka message. Let's go to validate the serialization issue. I will inform you during today Thanks @rollno748

chusneowork commented 3 weeks ago

@rollno748 I just did some tries, but it seems not to work yet:

My Kafka Producer Sampler Message looks like that (json text file) image

After tried different Serializer values, none of them worked:

Only works with the String serializer (org.apache.kafka.common.serialization.StringSerializer)

rollno748 commented 3 weeks ago

@chusneowork Can you please attach the jmeter.log

You can mask the hostnames, username and password with dummy values

chusneowork commented 3 weeks ago

Sure, here you go https://webfleet-my.sharepoint.com/:u:/p/jesus_munoz/Efgs51xH2RFJuFOWjvBB4VcBzac9ebsKxg4b9l5aVukr7w

Roby84 commented 1 week ago

@chusneowork Hi, i have a similar problem: i'm able to load the confluent serializer but i have this error:

Error sending message to kafka topic : java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidConfigurationException: Schema being registered is incompatible with an earlier schema for subject "AlertingAPI-value", details: [{errorType:'TYPE_MISMATCH', description:'The type (path '/') of a field in the new schema does not match with the old schema', additionalInfo:'reader type: STRING not compatible with writer type: RECORD'}, {oldSchemaVersion: 2}, {oldSchema: '{"type":"record","name":"AlertEvent", "namespace":"...... " ; error code: 409

i think it can't convert the json i'm passing in the right way to the avro schema saved in schema registry. Can you help me?

Thank you

chusneowork commented 4 days ago

Hi @rollno748 , have you been able to check the jmeter log? Have you find something? Looking forward to your response.

Hi @Roby84 , can you share your jmeter file, or the json and the kafka configuration you are trying to send? I can try to check... However, as we are trying to fix in this thread, the problem is that with the current DI-KafkaMeter Plugin only one serializer works (org.apache.kafka.common.serialization.StringSerializer), any other different serializer does not work.

Roby84 commented 4 days ago

Hi @chusneowork, yes i know, i'm using the kafkaAvroSerializer of confluent. I have imported all lib required and fixed configuration. But the error is: image

There is a way to have more detailed error message or a stacktrace? It seems to send a string and not an avro message...

rollno748 commented 4 days ago

Hello @chusneowork @Roby84

i have lost my saved store due to lap crash. Will push a fix soon.

Apologies for the delay.

chusneowork commented 4 days ago

@rollno748 thanks for the heads up. Hope you did not lost a lot of stuff. Looking forward to the update

@Roby84 have you tried to enable the debug log? image

Roby84 commented 4 days ago

Hi @chusneowork it works and says that before error:

Checking compatibility of reader {"type":"record","name":"AlertEvent","namespace":"com.datatransferobjects.alert.avro","doc":"Event to send an alert in asynchronous way", "fields":[{"name":"header","type":{"type":"record","name":"AlertEventHeader", "fields":[{"name":"traceId","type":["null","string"],"doc":"Identifier", "default":null},{"name":"messageId","type":"string","doc":"UUID"}, {"name":"origMessageId","type":["null","string"],"doc":"UUID of original event","default":null}, {"name":"source","type":"string","doc":"Application name that produced this message"}]}},{"name":"body","type":{"type":"record","name":"AlertEventBody", "fields":[{"name":"productCode","type":"string","doc":"product code","default":"0"}, {"name":"msgDetail","type":["null",{"type":"map","values":{"type":"string","avro.java.string":"String"},"avro.java.string":"String"}], "doc":"placeholder parameters.", "default":null}]}}],"aliases":["Alerting"]} with writer "string"

so it find avro from schema registry but the writer is still string.

How i can solve?

Thank you..

chusneowork commented 3 days ago

@Roby84 I think the problem you have is related to the current implementation of the serialization, even you send a json the writer seems to read it as string. I would say let's wait for the changes that @rollno748 needs to do and hope that version will work. Let's wait for it.