Fails to deserialise avro messages

[jaehyeon-kim]

Hi

The value is serialised as avro and the schema is stored in a Glue schema registry (%s-value). However values are not deserialised.

I enabled debug logs and the schema seems to be fetched correctly.

2023-03-31 03:34:50,766 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 >> "POST / HTTP/1.1[\r][\n]"
2023-03-31 03:34:50,767 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 >> "Host: glue.ap-southeast-2.amazonaws.com[\r][\n]"
2023-03-31 03:34:50,767 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 >> "amz-sdk-invocation-id: 790e1c6c-1176-d243-b474-24d1a82887a1[\r][\n]"
2023-03-31 03:34:50,768 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 >> "amz-sdk-request: attempt=1; max=4[\r][\n]"
2023-03-31 03:34:50,768 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=ASIAYS4LBXD7K3ES4SBU/20230331/ap-southeast-2/glue/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=bd21938ee1497fe8c26b73d806ae87bd6391935bac028ee5e11ce8c43ac418d8[\r][\n]"
2023-03-31 03:34:50,768 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 >> "Content-Type: application/x-amz-json-1.1[\r][\n]"
2023-03-31 03:34:50,769 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 >> "User-Agent: aws-sdk-java/2.17.122 Linux/5.10.102.1-microsoft-standard-WSL2 OpenJDK_64-Bit_Server_VM/17.0.6+10-LTS Java/17.0.6 scala/2.12.10 kotlin vendor/Azul_Systems__Inc. io/sync http/Apache cfg/retry-mode/legacy[\r][\n]"
2023-03-31 03:34:50,769 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 >> "X-Amz-Date: 20230331T033450Z[\r][\n]"
2023-03-31 03:34:50,770 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 >> "X-Amz-Security-Token: IQoJb3JpZ2luX2VjEKz//////////wEaDmFwLXNvdXRoZWFzdC0yIkgwRgIhANCgpRq4/PdWQVxw43mm+Rmyix0YrKGddHVC8194VoZHAiEAqyp3dnwcjOb89InCuRUvJWGu7jHUBw7JgFWEK7kZgHwqogMIhf//////////ARAAGgw1OTAzMTI3NDkzMTAiDIn12HDF3qbj5vHkwyr2AnBgteHAPVCNYS6Ld4WR0iqf0yAeEMZwMJmTkhBFTrzOY9KL0niUX9NQNEGYiLIxBO08Ntd06cWaax1jxKDwaMjhXYISMjjyFfw1HKMeTDRfh4nhfBjKjUj25BObW6YwJVvNDt7J+5p13v5i2rd9eIon55/gbBfp2aMZc5ft4COMOAUkEZPMjsWzwZIE4gPvLhdJ8pkRUsVPAOfVmByHt4gzpe8FI9/zN0zUNsrbo9T8zHJ4FOKRSeqLTkRckVVEd+rG447Q5duzT61ifZEMyTU+Ka7TIeW/n1jVwliBkp8DH+W5cXuM11O41R3Hc0YneXaKFQ3RbCVB+nXwRMgoOyPHJfbfIRDEBUzCOmVaXoUmb6lHzDy4LcdgcUq0BcQirr8DpctU0HK+3VcebzzjE6IbOcSOVgCpLeULo5Tk/X3ClVttwWzwdgQJZ5BaxJqR9K+JVBjcJME77DAdBAeUSgQmbOxROR1sMM937YEPtReAdRQxzgV5MKCpmaEGOqUBzAz2+n+PZCZdwvPb1qOqj9zOsz7EbBhSlxu9RTxtI88cDckxZH+gxLj9oN0mliUgf9VKrebRmsmclEwBeAr8MinbOxTEfHvUAfS3CXvtUF2g3QzdiDQYjAJmD7Q/dSSY5qkgLPFyakijuUPglcbwkEEE+7Rt8Z2ukOdiWSFN9pS/ypBBLl1YVpD1rLzr7tIkPFS3WRAtIzesZV8WXwsSAoAmQHP3[\r][\n]"
2023-03-31 03:34:50,770 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 >> "X-Amz-Target: AWSGlue.GetSchema[\r][\n]"
2023-03-31 03:34:50,770 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 >> "Content-Length: 66[\r][\n]"
2023-03-31 03:34:50,771 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
2023-03-31 03:34:50,771 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 >> "[\r][\n]"
2023-03-31 03:34:50,772 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 >> "{"SchemaId":{"SchemaName":"persons-value","RegistryName":"ccdak"}}"
2023-03-31 03:34:50,851 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
2023-03-31 03:34:50,851 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 << "Date: Fri, 31 Mar 2023 03:34:50 GMT[\r][\n]"
2023-03-31 03:34:50,852 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 << "Content-Type: application/x-amz-json-1.1[\r][\n]"
2023-03-31 03:34:50,852 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 << "Content-Length: 430[\r][\n]"
2023-03-31 03:34:50,852 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 << "Connection: keep-alive[\r][\n]"
2023-03-31 03:34:50,853 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 << "x-amzn-RequestId: 205af4e0-b246-449c-9603-1792c0fc16de[\r][\n]"
2023-03-31 03:34:50,853 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 << "[\r][\n]"
2023-03-31 03:34:50,853 DEBUG [parallel-8] o.a.h.wire: http-outgoing-0 << "{"Compatibility":"BACKWARD","CreatedTime":"2023-03-31T03:25:14.900Z","DataFormat":"AVRO","LatestSchemaVersion":1,"NextSchemaVersion":2,"RegistryArn":"arn:aws:glue:ap-southeast-2:590312749310:registry/ccdak","RegistryName":"ccdak","SchemaArn":"arn:aws:glue:ap-southeast-2:590312749310:schema/ccdak/persons-value","SchemaCheckpoint":1,"SchemaName":"persons-value","SchemaStatus":"AVAILABLE","UpdatedTime":"2023-03-31T03:25:14.900Z"}"
2023-03-31 03:34:50,853 DEBUG [parallel-8] o.a.h.headers: http-outgoing-0 << HTTP/1.1 200 OK
2023-03-31 03:34:50,854 DEBUG [parallel-8] o.a.h.headers: http-outgoing-0 << Date: Fri, 31 Mar 2023 03:34:50 GMT
2023-03-31 03:34:50,854 DEBUG [parallel-8] o.a.h.headers: http-outgoing-0 << Content-Type: application/x-amz-json-1.1
2023-03-31 03:34:50,854 DEBUG [parallel-8] o.a.h.headers: http-outgoing-0 << Content-Length: 430
2023-03-31 03:34:50,854 DEBUG [parallel-8] o.a.h.headers: http-outgoing-0 << Connection: keep-alive
2023-03-31 03:34:50,855 DEBUG [parallel-8] o.a.h.headers: http-outgoing-0 << x-amzn-RequestId: 205af4e0-b246-449c-9603-1792c0fc16de

Attached includes the full log messages.

log.txt

Can you please check it?

Cheers, Jaehyeon

[iliax]

Hello @jaehyeon-kim, thank you for rasing this. For glue serder default topic -> schema naming pattern is just 1-to-1 mapping (as is). This is done because of default deserializer's behavior (from java glue library).

To change it to smth else (like %s-value) you can set the corresponding property https://github.com/provectus/kafkaui-glue-sr-serde/blob/5a41f6937faab630efc9cf4cfcdcf2c782fab96d/docker-compose/setup-example.yaml#L27

[jaehyeon-kim]

Hi @iliax

I update it as you mentioned. Here is the docker compose file.

version: "3.5"

services:
  kafka-ui-glue:
    image: provectuslabs/kafka-ui:master
    container_name: kafka-ui-glue
    ports:
      - "8080:8080"
    networks:
      - kafkanet
    environment:
      LOGGING_LEVEL_ROOT: debug
      AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
      AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
      AWS_SESSION_TOKEN: $AWS_SESSION_TOKEN
      KAFKA.clusters.0.name: local
      KAFKA.clusters.0.bootstrapServers: kafka-0:9092,kafka-1:9092,kafka-2:9092
      kafka.clusters.0.serde.0.name: GlueSchemaRegistry
      kafka.clusters.0.serde.0.filePath: /glue-serde/kafkaui-glue-serde-1.0-SNAPSHOT-jar-with-dependencies.jar
      kafka.clusters.0.serde.0.className: com.provectus.kafka.ui.serdes.glue.GlueSerde
      kafka.clusters.0.serde.0.properties.region: $AWS_DEFAULT_REGION #required
      kafka.clusters.0.serde.0.properties.registry: ccdak #required, name of Glue Schema Registry
      # template that will be used to find schema name for topic key. Optional, default is null (not set).
      kafka.clusters.0.serde.0.properties.keySchemaNameTemplate: "%s-key"
      # template that will be used to find schema name for topic value. Optional, default is  '%s'
      kafka.clusters.0.serde.0.properties.valueSchemaNameTemplate: "%s-value"
      # # schema name -> topics pattern where it will be used for keys. Optional.
      # kafka.clusters.0.serde.0.properties.topicKeysSchemas.some-topic-key: "some-topic1|some-topic2"
      # # schema name -> topics pattern where it will be used for values. Optional.
      # kafka.clusters.0.serde.0.properties.topicValuesSchemas.some-topic-value: "some-topic1|some-topic2"
      # kafka.clusters.0.serde.0.properties.topicValuesSchemas.another-topic-val: "another-topic-value"

    volumes:
      - ../misc:/glue-serde

networks:
  kafkanet:
    external: true
    name: kafka-network

[Haarolean]

@jaehyeon-kim doesn't it still work?

[Haarolean]

Also, try the 1.0.1 release.

[jaehyeon-kim]

Hi @Haarolean

I tried with v1.0.3 and it works.

Cheers, Jaehyeon