Open nrellanti opened 1 year ago
Hi @nrellanti
Thank you for your collaboration. We are looking at it and we will get back to you as soon as possible.
Hi @nrellanti , could you add the complete protobuf file as there are missing parts such as the Order.
Thank you.
@AdrianLagarteraSngular , please see the attached complete messaging_intra_order_proto.txt protobuf file referencing in the messaging_unified_order_proto.txt
@nrellanti Sorry for the delay, checking the messaging_intra_order_proto.txt
file there are some references to classes that it does not have, could it be that it is a schema in a previous version?
For example in the messaging_unified_order_proto.txt
the first reference to the .demo.Order.CustomerAccount
class does not exist in the messaging_intra_order_proto.txt
file
@AdrianLagarteraSngular , Here is the corrected messaging_intra_order_proto.txt file
@nrellanti Hi, we have added some changes in the attached branch to fix the problem, to avoid errors, if you are using schema registry to find the file, the subject and the filename must have the same name, in next versions we will fix this so the subject can be different from the filename and it will still work.
On the other hand if you don't use schema registry you can indicate the classpath in jmeter and it will also work without schema registry, just put the files you use in the same folder.
Best regards and thank you very much
Hi @AdrianLagarteraSngular , Thanks for sharing the information and trying to work around this issue. Basically, We're following the standard practices to register this schema the way @nrellanti has shared two schema files. We're using TopicNaming Strategy to create topics and schema in the confluent cloud.
Example:
Topic name: MESSAGING-UNIFIED-ORDER
Subject name of parent schema: MESSAGING-UNIFIED-ORDER-value
Child schema's subject name: messaging_intra_order.proto
(We are importing this schema in the above parent schema as import "messaging_intra_order.proto";
)
We're importing child schema like this because we have seen standard practices for google's well-known schema also registering the same way and importing in any schema.
In the same way, we're importing other custom or child schemas into the parent schema.
The latest Kloadgen jar is not supporting this use case. Can you please advise on the above scenario?
Thank you, Vishal
Hi, sorry for don't answer before
Using the given params you put on your last message @vishaldhanani I will put the steps you need to do to cover your scenario:
First of all compile the KloadGen from the attached brand: https://github.com/sngular/kloadgen/tree/332-comsngularkloadgenserializerprotobufserializer-nullpointerexception (Currently not in master because we wait until you can confirm it's working in your scenario)
When you pass your schema in your schema registry, be sure your schema have the same root name subject as your file, using the example, subject: messaging_intra_order
, name: messaging_intra_order.proto
The references of the schema should be like this:
{
"schema": "..."
"schemaType": "PROTOBUF",
"references": [
{
"name": "messaging_intra_order.proto",
"subject": "messaging_intra_order",
"version": 1
}]
}
And after that ensure your topic have the same name as the subject of the schema registry, in the example value you give the subject name of the parent not correct if your topic name its: MESSAGING-UNIFIED-ORDER
, should be exactly the same name in the subject and the topic
And finally check if your plan test have the correct Name Strategy
because in the attached one the strategy was RecordNameStrategy
instead of the TopicNameStrategy
Let me know if it has been useful for you, or if you find new issues, and thanks for your patience.
Hi @AdrianLagarteraSngular, Thank you for sharing the complete process. Basically, Google's well-known schema is registered as below, and based on those schemas we are following the same process to register custom schema. I think we cannot change this process to register all custom schema for only testing purposes the way you have shared in the above response.
Looking forward if you can handle this kind of use case in the Kloadgen plugin. { "schema": "..." "schemaType": "PROTOBUF", "references": [ { "name": "messaging_intra_order.proto", "subject": "messaging_intra_order.proto", "version": 1 }] }
Thank you, Vishal
Hi @vishaldhanani ,
We are taking a look at your last answer, sorry for the delay. Best regards.
Hi @vishaldhanani ,
We have introduced a fix to get the subject from the subject of the reference, so your scenery should be processed correctly. Please, could you check if this solution is valid for you?
Thanks in advance. Best regards
Hi @alfredo9f, I appreciate your reply. We'll check the most recent changes and let you know if there are any problems.
Thank you, Vishal
Hi @alfredo9f , Tested the new code from Branch 332-comsngularkloadgenserializerprotobufserializer-nullpointerexception
Getting below error
2023-03-23 09:32:31,932 ERROR c.s.k.l.i.ProtobufLoadGenerator: Please make sure that properties data type and expression function return type are compatible with each other io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Subject 'MESSAGING-UNIFIED-ORDER' not found.; error code: 40401.
Please suggest!
Hi @nrellanti ,
Please, could you tell me if you have the schema 'MESSAGING-UNIFIED-ORDER' loaded in your schema registry and what 'Strategy Name' are you using in your test?
Thanks in advance.
Hi @alfredo9f, Yes, I also tested with the latest changes and got the same error as @nrellanti getting it.
We are using TopicNameStragtegy to register the schemas and we already have registered schemas for MESSAGING-UNIFIED-ORDER
topic.
Please advise!
Thank you, Vishal
Hi @vishaldhanani ,
We did a small fix so now should works. Please, could you check if now is ok for you?
Thanks in advance!
@alfredo9f , Looks like the new fix was working as expected. I will test few more similar proto files and will confirm. Thanks for fix !
Hi @nrellanti ,
Thanks to all for the checks. We will wait your confirm :)
Hi @alfredo9f, Using the most recent jar, I also verified and published messages. Everything is operating exactly as expected. I really appreciate you taking care of this. Thank you all very much :)
Thank you, Vishal
Hi @alfredo9f , Tested and verified with other proto files. Thank you very much!
@jemacineiras @dhergonsngular , https://github.com/sngular/kloadgen/issues/332
Getting similar error with latest version, could you please reopen the issue 332
Thanks for the feedback @nrellanti we will take a look
Should we use the same files attached to this issue for the investigation?
@jemacineiras , yes you can use the same files
@jemacineiras, Hope you're doing great! Let us know if you need any more information related to this issue. Thank you!
Hi @vishaldhanani, is holidays season up here so we are kinda short of hands. But I will not forget you.
Cheers
@jemacineiras , Hello Jose, hope that you have enjoyed holidays season with your family. DO you have any update on this bug?
Hi @nrellanti, At the moment I couldn't enjoy holidays for now... plenty of things were jumping around. I reopened the 332 branch, update it to master and check the code... and was working for me. Do you mind check if that branch fix you problem so I can work on it?
Cheers.
Hi @vishaldhanani, I reopened the 332 branch, update it to master and check the code... and was working for me. Do you mind check if that branch fix you problem so I can work on it?
Cheers.
Hi @jemacineiras, Thank you for update it to master. Based on the latest updates, We have tested latest kloadgen-5.6.7.jar
with other topic and schema. But still it is throwing below error. Can you check why it is still throwing this error?
Thank you so much for all your help!
Hi @vishaldhanani,
had you move to the latest JMeter 5.6.2?
Cheers
Hi @jemacineiras, Yes, I just ran with JMeter 5.6.2 and it's giving us below error. Hope this might help you to troubleshoot the issue.
Thank you!
Hi @vishaldhanani ,
JMeter 5.6.2 version introduce some changes compared with 5.5, version we use to compile the plugin. I just update the branch and fix (I believe) the incompatibilities. Can you please check the branch?
Hi @jemacineiras, We have test it out based on your latest changes. It is still throwing the same error.
Hi @vishaldhanani , we don't include the google api grpc. Looks that import of yours is part of another package, like an optional. I just added in this branch in order to see if that works for you. Usually external dependencies should be provided as a jar in the classpath, since we are building the protobuff files dynamically we need to have all 3rd party or external dependencies in the classpath.
Please try this approach and keep in mind for the future.
Cheers
Hi @jemacineiras, After getting all the latest changes, We are still getting the same error as earlier : java.lang.IllegalArgumentException: cannot resolve import "google/type/money.proto";
Hi @vishaldhanani,
looks the assemble didn't took the .proto
files.. Can you add that dependency in the Jmeter classpath?
Hi @jemacineiras, Thanks for providing the above info. We have added the class path for .proto file but it is throwing new error : java.nio.BufferOverflowException: null
. Do we still missing anything on the configuration side based on latest code changes for this plugin?
Thank you so much!
HI @vishaldhanani, Looks it's generating a big object and protobuff serializer cannot handle. can you provide the generated object please? U can add the result tree component and see it there. We add the generated object to the JMeter context for debut and furthers used if it is needed.
Cheers
Hi @jemacineiras, It is not generating any generated object/Message payload in View Results Tree. It is directly throwing java.nio.BufferOverflowException: null exception in log. Our app is serializing and publishing as expected with the same protobuf schema. Also, We are using option java_multiple_files = true;
in our proto schema, so Protobuf Compiler can generate separate auto-generated class files.
Is there any Payload limitations, we have it on JMeter or Kloadgen plugin side?
Thank you!
HI @vishaldhanani, Situation here is interesting. You are using our serializer who consider the schema_id has a size of 4 bytes but in your case looks bigger so you are getting that error once you want to serialize and publish the data. Can you please enable debug mode, so I can get an example data? Cheers
Hi @jemacineiras, We have test it out all the latest changes. Looks like java.nio.BufferOverflowException: null
has been resolved now. but we are still getting java.lang.IllegalArgumentException: cannot resolve import "google/type/money.proto";
error.
Here is our use case: JMeter Testing Topic: ORDER-TEST-TOPIC Schema name in Confluent Schema Registry: ORDER-TEST-TOPIC-value
syntax = "proto3";
package demo;
import "customer_schema.proto";
option java_package = "com.order.demo.proto"; option java_outer_classname = "OrderDemo"; option java_multiple_files = true;
message Order { int32 id = 1; string name = 2; .demo.Customer customer = 3; .demo.Customer.Details details = 4; }
FYI: We have already created references for `import "customer_schema.proto";` schema in Confluent Schema registry to give references so both the schema tie together and get all reference messages from the `customer_schema.proto`.
2. import schema: `customer_schema.proto`
import schema register in Confluent Schema registry as `customer_schema.proto`
syntax = "proto3"; package demo;
import "google/type/money.proto";
option java_package = "com.demo.customer.proto"; option java_outer_classname = "CustomerDemo"; option java_multiple_files = true;
message Customer { int32 cust_id = 1; string cust_name = 2; repeated .demo.Customer.Details details = 3;
message Details { int32 detail_id = 1; .google.type.Money amount = 2; }
FYI: We are not creating any references for `import "google/type/money.proto";` because Confluent Cloud Schema Registry understand automatically if we only register `google/type/money.proto` schema into Schema registry. We don't need to create any references for Google well-known type schemas.
3. We are registering our `google/type/money.proto` file in Confluent Schema Registry as below:
<img width="495" alt="image" src="https://github.com/sngular/kloadgen/assets/29518449/08602407-af15-465c-9783-60940280187d">
FYI: This is a google well-known type schema so we are registering same as Google provides.
syntax = "proto3"; package google.type;
option cc_enable_arenas = true; option go_package = "google.golang.org/genproto/googleapis/type/money;money"; option java_multiple_files = true; option java_outer_classname = "MoneyProto"; option java_package = "com.google.type"; option objc_class_prefix = "GTP";
message Money { string currency_code = 1; int64 units = 2; int32 nanos = 3; }
Let us know if you have any questions on above use case. We are trying to testing the 1st topic: ORDER-TEST-TOPIC in JMeter and which is throwing error as mentioned on above screen shot.
Appreciate your help! Thank you so much :)
HI @vishaldhanani ,
situation is a bit complex here. At the moment we currently support 1 level deep of import, meaning that we can create a Protobuf object and one leve of imports.. I need to rewrite this pieces to support whatever level.
I will try to have asap.
Regarding the type money proto. I manage to add them to the plugin classpath so those should be find in the lastest code.
Will see how it behaves.
Cheers.
Hi @jemacineiras, You can keep consider all the google well-known types schema so it will resolve this issue entirely. Here are the list of google well known types schema.
Thank you!
Hello @jemacineiras , any update on this bug and google well-known types schema to work with protobuf?
Hi @nrellanti ,
I just added the last group of Protobuff well-know types. The branch is still under development. But if you can try it. It will help a lot to speed up.
Cheers
@jemacineiras , thank you. I will try and let you know the output. Thanks
Hello @jemacineiras,
still getting errors
java.lang.IllegalArgumentException: Could not parse Protobuf
at io.confluent.kafka.schemaregistry.protobuf.ProtobufSchema.toProtoFile(ProtobufSchema.java:462) ~[kloadgen%201.jar:73.1]
at io.confluent.kafka.schemaregistry.protobuf.ProtobufSchema.
Hi @nrellanti I see this is a different type of error of previous one. Do you mind if we move that to other issue so I can merge these changes?
Cheers
Hello @jemacineiras ,
Tested your latest branch of yesterday's change, here is the error message. The above error messages are previous branch I.e one/two week ago.
2023-11-14 10:11:33,823 ERROR o.a.j.t.JMeterThread: Error while processing sampler: 'Java Request'.
java.lang.ClassCastException: class com.squareup.wire.schema.internal.parser.ProtoFileElement cannot be cast to class org.apache.avro.Schema (com.squareup.wire.schema.internal.parser.ProtoFileElement and org.apache.avro.Schema are in unnamed module of loader org.apache.jmeter.DynamicClassLoader @5c647e05)
at com.sngular.kloadgen.processor.objectcreatorfactory.impl.AvroObjectCreatorFactory.
What happened?
Caused by: java.lang.NullPointerException: Parameter specified as non-null is null: method com.squareup.wire.schema.internal.parser.ProtoParser$Companion.parse, parameter data
KloadGen Version
kloadgen-5.2.10.jar
Relevant log output
Have you added your JMX Testplan or a replica?
Have you added your Schema or a replica?