Avro deser with schema-registry do not work

[raphaelauv]

Avro deser with schema-registry do not work

my setup

  kafka-ui:
    image: provectuslabs/kafka-ui
    container_name: kafka-ui
    ports:
      - "9025:8080"
    restart: always
    environment:
      - KAFKA_CLUSTERS_0_NAME=local
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka-broker:29092
      - KAFKA_CLUSTERS_0_ZOOKEEPER=zookeeper:2181
      - KAFKA_CLUSTERS_0_SCHEMAREGISTRY=schema-registry:8081
      - KAFKA_CLUSTERS_0_SCHEMANAMETEMPLATE="%s-value"

it work with the control-center of confluent

[germanosin]

@raphaelauv thanks for creating issue. It looks strange. Could you please provide some logs from kafka-ui at the moment of query?

[germanosin]

And could you also please share topic name and schema name

[raphaelauv]

3 empty msg , then 1 message with avro content

alos would be nice to show key's in the UI ?

10:20:28.160 [boundedElastic-2] WARN  com.provectus.kafka.ui.deserialization.SchemaRegistryRecordDeserializer - Failed to get Schema for topic aaaaa
io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Subject '"aaaaa-value"' not found.; error code: 40401
    at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:292) ~[kafka-schema-registry-client-5.5.1.jar!/:?]
    at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:351) ~[kafka-schema-registry-client-5.5.1.jar!/:?]
    at io.confluent.kafka.schemaregistry.client.rest.RestService.getAllVersions(RestService.java:768) ~[kafka-schema-registry-client-5.5.1.jar!/:?]
    at io.confluent.kafka.schemaregistry.client.rest.RestService.getAllVersions(RestService.java:759) ~[kafka-schema-registry-client-5.5.1.jar!/:?]
    at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getAllVersions(CachedSchemaRegistryClient.java:364) ~[kafka-schema-registry-client-5.5.1.jar!/:?]
    at com.provectus.kafka.ui.deserialization.SchemaRegistryRecordDeserializer.detectFormat(SchemaRegistryRecordDeserializer.java:106) ~[classes!/:?]
    at com.provectus.kafka.ui.deserialization.SchemaRegistryRecordDeserializer.lambda$getMessageFormat$1(SchemaRegistryRecordDeserializer.java:99) ~[classes!/:?]
    at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705) ~[?:?]
    at com.provectus.kafka.ui.deserialization.SchemaRegistryRecordDeserializer.getMessageFormat(SchemaRegistryRecordDeserializer.java:99) ~[classes!/:?]
    at com.provectus.kafka.ui.deserialization.SchemaRegistryRecordDeserializer.deserialize(SchemaRegistryRecordDeserializer.java:71) ~[classes!/:?]
    at com.provectus.kafka.ui.util.ClusterUtil.mapToTopicMessage(ClusterUtil.java:195) ~[classes!/:?]
    at com.provectus.kafka.ui.service.ConsumingService.lambda$loadMessages$2(ConsumingService.java:61) ~[classes!/:?]
    at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:199) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
    at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.onNext(FluxSubscribeOn.java:151) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
    at reactor.core.publisher.FluxCreate$BufferAsyncSink.drain(FluxCreate.java:793) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
    at reactor.core.publisher.FluxCreate$BufferAsyncSink.next(FluxCreate.java:718) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
    at reactor.core.publisher.FluxCreate$SerializedSink.next(FluxCreate.java:153) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
    at com.provectus.kafka.ui.service.ConsumingService$RecordEmitter.accept(ConsumingService.java:146) [classes!/:?]
    at com.provectus.kafka.ui.service.ConsumingService$RecordEmitter.accept(ConsumingService.java:128) [classes!/:?]
    at reactor.core.publisher.FluxCreate.subscribe(FluxCreate.java:94) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
    at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
    at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
    at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:830) [?:?]
10:20:28.205 [boundedElastic-2] INFO  com.provectus.kafka.ui.deserialization.SchemaRegistryRecordDeserializer - Failed to parse json from topic aaaaa
10:20:28.213 [boundedElastic-2] DEBUG org.springframework.http.codec.json.Jackson2JsonEncoder - [638f3606] Encoding [[class TopicMessage {
    partition: 0
    offset: 0
    timestamp: 2021-05-16T01:43:41.004Z
    tim (truncated)...]

[raphaelauv]

I replaced schema-registry:8081 by http://schema-registry:8081

cause I was having a java.net.MalformedURLException: unknown protocol: schema-registry in message UI even if schema-registry UI was working

but the problem is still there

[Flyur]

@raphaelauv Hi, for some reason it's unable to read KAFKA_CLUSTERS_0_SCHEMAREGISTRY property, could you please try to list environment variables by using mapping syntax (with a colon as separator):

environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: broker:29092
      KAFKA_CLUSTERS_0_ZOOKEEPER: zookeeper:2181
      KAFKA_CLUSTERS_0_SCHEMAREGISTRY: http://schema-registry:8081
      KAFKA_CLUSTERS_0_SCHEMANAMETEMPLATE: "%s-value"

[raphaelauv]

It work with this syntax ! Thank you

also it would be nice to show key's in the UI

[Flyur]

@raphaelauv sure, this issue already assigned https://github.com/provectus/kafka-ui/issues/377

[raphaelauv]

@germanosin the examples still use the syntax with =

it should be correct

https://github.com/provectus/kafka-ui/blob/master/docker-compose.md