search

camunda-community-hub / zeebe-redis-exporter

Export events from Zeebe to Redis
Apache License 2.0
13 stars 3 forks source link

Exception "Protocol message end-group tag did not match expected tag" when setting args.format = json #204

Closed asalhani closed 2 months ago

asalhani commented 2 months ago

As per README.md, exporters: redis: className: io.zeebe.redis.exporter.RedisExporter jarPath: exporters/zeebe-redis-exporter-1.0.0-jar-with-dependencies.jar args: cleanupCycleInSeconds: -1 maxTimeToLiveInSeconds: -1 format: "json"

Format property can take either of 2 values [protobuf|json] When setting to protobuf the listner is working fine.

However, when setting it to "json", the following excption is being thrown when listner activating:

2024-08-17T18:53:22.173+03:00 ERROR 22244 --- [tasklistapp] [pool-5-thread-1] io.zeebe.redis.connect.java.ZeebeRedis : Consumer[group=USER_TASK_CONSUMER_GROUP, id=consumer-1] failed to read from streams 'zeebe:*' com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag. at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:114) ~[protobuf-java-3.25.1.jar:na] at com.google.protobuf.CodedInputStream$ArrayDecoder.checkLastTagWas(CodedInputStream.java:615) ~[protobuf-java-3.25.1.jar:na] at com.google.protobuf.CodedInputStream$ArrayDecoder.readGroup(CodedInputStream.java:811) ~[protobuf-java-3.25.1.jar:na] at com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:494) ~[protobuf-java-3.25.1.jar:na] at com.google.protobuf.GeneratedMessageV3$Builder.parseUnknownField(GeneratedMessageV3.java:864) ~[protobuf-java-3.25.1.jar:na] at io.zeebe.exporter.proto.Schema$Record$Builder.mergeFrom(Schema.java:995) ~[zeebe-exporter-protobuf-1.6.1.jar:1.6.1] at io.zeebe.exporter.proto.Schema$Record$1.parsePartialFrom(Schema.java:1166) ~[zeebe-exporter-protobuf-1.6.1.jar:1.6.1] at io.zeebe.exporter.proto.Schema$Record$1.parsePartialFrom(Schema.java:1158) ~[zeebe-exporter-protobuf-1.6.1.jar:1.6.1] at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:135) ~[protobuf-java-3.25.1.jar:na] at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:168) ~[protobuf-java-3.25.1.jar:na] at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:180) ~[protobuf-java-3.25.1.jar:na] at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:185) ~[protobuf-java-3.25.1.jar:na] at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:25) ~[protobuf-java-3.25.1.jar:na] at io.zeebe.exporter.proto.Schema$Record.parseFrom(Schema.java:754) ~[zeebe-exporter-protobuf-1.6.1.jar:1.6.1] at io.zeebe.redis.connect.java.ZeebeRedis.handleRecord(ZeebeRedis.java:341) ~[zeebe-redis-connector-1.0.0.jar:1.0.0] at io.zeebe.redis.connect.java.ZeebeRedis.readNext(ZeebeRedis.java:300) ~[zeebe-redis-connector-1.0.0.jar:1.0.0] at io.zeebe.redis.connect.java.ZeebeRedis.readFromStream(ZeebeRedis.java:270) ~[zeebe-redis-connector-1.0.0.jar:1.0.0] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[na:na] at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) ~[na:na] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:842) ~[na:na]

I need to set it to JSON because "protobuf" does not render the value correctly when checking record in Redis Insight

Format= json ==> value can be inspected in Redis Insight

image

Format= protobuf ==> value can't retrived in Redis insight

image
asalhani commented 2 months ago

As per README.md, exporters: redis: className: io.zeebe.redis.exporter.RedisExporter jarPath: exporters/zeebe-redis-exporter-1.0.0-jar-with-dependencies.jar args: cleanupCycleInSeconds: -1 maxTimeToLiveInSeconds: -1 format: "json"

Format property can take either of 2 values [protobuf|json] When setting to protobuf the listner is working fine.

However, when setting it to "json", the following excption is being thrown when listner activating:

2024-08-17T18:53:22.173+03:00 ERROR 22244 --- [tasklistapp] [pool-5-thread-1] io.zeebe.redis.connect.java.ZeebeRedis : Consumer[group=USER_TASK_CONSUMER_GROUP, id=consumer-1] failed to read from streams 'zeebe:*' com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag. at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:114) ~[protobuf-java-3.25.1.jar:na] at com.google.protobuf.CodedInputStream$ArrayDecoder.checkLastTagWas(CodedInputStream.java:615) ~[protobuf-java-3.25.1.jar:na] at com.google.protobuf.CodedInputStream$ArrayDecoder.readGroup(CodedInputStream.java:811) ~[protobuf-java-3.25.1.jar:na] at com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:494) ~[protobuf-java-3.25.1.jar:na] at com.google.protobuf.GeneratedMessageV3$Builder.parseUnknownField(GeneratedMessageV3.java:864) ~[protobuf-java-3.25.1.jar:na] at io.zeebe.exporter.proto.Schema$Record$Builder.mergeFrom(Schema.java:995) ~[zeebe-exporter-protobuf-1.6.1.jar:1.6.1] at io.zeebe.exporter.proto.Schema$Record$1.parsePartialFrom(Schema.java:1166) ~[zeebe-exporter-protobuf-1.6.1.jar:1.6.1] at io.zeebe.exporter.proto.Schema$Record$1.parsePartialFrom(Schema.java:1158) ~[zeebe-exporter-protobuf-1.6.1.jar:1.6.1] at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:135) ~[protobuf-java-3.25.1.jar:na] at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:168) ~[protobuf-java-3.25.1.jar:na] at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:180) ~[protobuf-java-3.25.1.jar:na] at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:185) ~[protobuf-java-3.25.1.jar:na] at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:25) ~[protobuf-java-3.25.1.jar:na] at io.zeebe.exporter.proto.Schema$Record.parseFrom(Schema.java:754) ~[zeebe-exporter-protobuf-1.6.1.jar:1.6.1] at io.zeebe.redis.connect.java.ZeebeRedis.handleRecord(ZeebeRedis.java:341) ~[zeebe-redis-connector-1.0.0.jar:1.0.0] at io.zeebe.redis.connect.java.ZeebeRedis.readNext(ZeebeRedis.java:300) ~[zeebe-redis-connector-1.0.0.jar:1.0.0] at io.zeebe.redis.connect.java.ZeebeRedis.readFromStream(ZeebeRedis.java:270) ~[zeebe-redis-connector-1.0.0.jar:1.0.0] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[na:na] at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) ~[na:na] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na] at java.base/java.lang.Thread.run(Thread.java:842) ~[na:na]

I need to set it to JSON because "protobuf" does not render the value correctly when checking record in Redis Insight

Format= json ==> value can be inspected in Redis Insight image

Format= protobuf ==> value can't retrived in Redis insight image

@VonDerBeck

VonDerBeck commented 2 months ago

Hi @asalhani,

even if the exporter itself is able to (optionally) export events as JSON, the connector is designed to use the default which is protobuf. So this is not an error.

asalhani commented 2 months ago

@VonDerBeck Is there is a way to configure both exporter and connector to transmit data as JSON instead of protobuf?

VonDerBeck commented 2 months ago

@asalhani Currently the connector has no option to read JSON. Provided connectors are based on protobuf.

While I understand that it could be a nice feature especially during development to have a look into Redis itself and have readable messages on the other hand you are able to log messages after receiving them via the connector as well. So of course this is nice-to-have, but currently it's not there. Maybe something for the future.