search

pravega / schema-registry

Pravega Schema Registry repository
Apache License 2.0
15 stars 23 forks source link

Problem with protobuf deserializer #291

Open Welkin-Y opened 11 months ago

Welkin-Y commented 11 months ago

Problem description Hello, I am currently working on the flink-connector project trying to add protobuf support. I've noticed that after serializing a message to a ByteBuffer, if I convert it to a byte[] using .array(), and then convert it back to a ByteBuffer using ByteBuffer.wrap(), there will be an exception when passing it to the deserialize function. I noticed that in the test cases, there are no scenarios covering conversions between ByteBuffers and byte arrays. However, such a convertion is needed for flink-connector interfaces.

Problem location

To reproduce the exception, you can modify this file by changing line 84 as follows:, you can regenerate the exception by changing line 84 as follows:

c80e7a2f340e1b7d45c1a848cdf5c19

The error message is like this:

768133634d1465c845f564f4843f7e1

Suggestions for an improvement I'm not sure if it's related to the deserialize method. From my experiments, I found that the issue arises because, after converting a ByteBuffer to a byte[] and back, the limit information of the ByteBuffer is lost, leading to a parsing error in protobuf.

crazyzhou commented 10 months ago

@Welkin-Y When parsing bytebuffer into byte[], do you have the information of the limit? Directly using array() may lose such infomation, there are some usages like: https://github.com/pravega/flink-connectors/blob/053269788245cccd53257cd70178097d4b61bf6c/src/main/java/io/pravega/connectors/flink/util/FlinkPravegaUtils.java#L204 Maybe you can have a try.

@shshashwat Please also help to see if there are anything can be improvement from this side.