search

provectus / kafka-ui

Open-Source Web UI for Apache Kafka Management
Apache License 2.0
9.69k stars 1.17k forks source link

Request for CloudEvent serde #4210

Open supriya-albal-polestar opened 1 year ago

supriya-albal-polestar commented 1 year ago

Issue submitter TODO list

Is your proposal related to a problem?

No response

Describe the feature you're interested in

We use CloudEvent to wrap our kafka messages. As a result, we are seeing encoded values in Provectus, that we are unable to read. Here is link to CloudEvent library - https://cloudevents.github.io/sdk-java/kafka.html.

Describe alternatives you've considered

No response

Version you're running

56fa824 v0.7.1

Additional context

No response

github-actions[bot] commented 1 year ago

Hello there supriya-albal-polestar! 👋

Thank you and congratulations 🎉 for opening your very first issue in this project! 💖

In case you want to claim this issue, please comment down below! We will try to get back to you as soon as we can. 👀

supriya-albal-polestar commented 1 year ago

I would like to get inputs on this issue.

Haarolean commented 1 year ago

Hi, we made the serde API for this exact purpose -- so the community can create pluggable SerDes. Currently, we don't have any plans to develop any other serdes, especially the ones with narrow use-cases. Feel free to reach out for professional services for Apache Kafka by Provectus if you wish to get one developed for you personally, though.

supriya-albal-polestar commented 1 year ago

@Haarolean Is there any documentation or example for using protobuf? I see that protoBuf is supported. Can I use proto object for CloudEvent with kafka-ui? I tried by referring to one yml example in your code, but it did not help. Have also asked on stackoverflow - https://stackoverflow.com/questions/77077386/how-to-use-custom-serde-with-provectus-kafka-ui?noredirect=1#comment135879983_77077386

Haarolean commented 1 year ago

Here should be all the info on the protobuf serde.

supriya-albal-polestar commented 1 year ago

thanks @Haarolean

I referred to the documents and created following docker compose -

kafka: image: confluentinc/cp-kafka:6.0.2 container_name: kafka restart: always depends_on:

  • zookeeper ports:
  • "9092:9092"
  • "29092:29092" extra_hosts:

  • "host.docker.internal:host-gateway" environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092 KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

    kafka-ui: container_name: kafka-ui image: provectuslabs/kafka-ui:latest ports:

  • "8080:8080" depends_on:
  • zookeeper
  • kafka environment: KAFKA_CLUSTERS_0_NAME: local KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:29092 KAFKA_CLUSTERS_0_ZOOKEEPER: zookeeper:2181 KAFKA_CLUSTERS_0_JMXPORT: 9997 kafka.clusters.0.serde.0.name: ProtobufFile kafka.clusters.0.serde.0.className: com.provectus.kafka.ui.serdes.builtin.ProtobufRawSerde kafka.clusters.0.serde.0.properties.protobufFilesDir: /protofiles/ kafka.clusters.0.serde.0.properties.protobufMessageName: io.cloudevents.v1.CloudEvent # default type for values kafka.clusters.0.serde.0.properties.protobufMessageNameByTopic.ais-fused: io.cloudevents.v1.CloudEvent volumes:
  • "./proto/:/protofiles"

However I dont see Protobuf getting picked up by UI.

Haarolean commented 1 year ago

@supriya-albal-polestar let's keep the discussion on topic, feel free to raise a discussion or join us on discord for unrelated things.