search

asyncapi / glee

Glee — The AsyncAPI framework that will make you smile again :)
87 stars 62 forks source link

Improve kafka adapter #411

Open KhudaDad414 opened 1 year ago

KhudaDad414 commented 1 year ago

Reason/Context

The current implementation of kafka adapter provides the bare minimum functionality and I think we can improve upon that by using bindings.

What are bindings

AsyncAPI is protocol agnostic, it means that it supports a wide range of protocols and doesn't "assume" the protocol when you are writing your AsyncAPI file. but if it doesn't assume protocols, how are we supposed to write our file accustomed to an specific protocols? Behold, the answer is bindings. If you look at kafka bindings you will see a wide range of options that can be used to customize every part of your kafka client.

The Problem

Currently kafka adapter ignore lots of this options. for example. instead of using clientId from bindings for clinetId it just sets it to glee-client.

The Solution

Kafka bindings should be examined and we should support as much features as the bindings support.

KhudaDad414 commented 1 year ago

/gfi ts

shubham-y commented 1 year ago

Hey @KhudaDad414 , I would like to work on this issue

KhudaDad414 commented 1 year ago

@shubham-y sure. Just let me know if you have any questions 🙂

shubham-y commented 1 year ago

I see there is a similar draft PR #381 opened 2 months ago. So is there any discussion that I can refer to from that time?

KhudaDad414 commented 1 year ago

Not that I know of. this issue is also kinda related to https://github.com/asyncapi/glee/issues/27

shubham-y commented 1 year ago

Hey @KhudaDad414 , I went through the current implementation of kafka adapter and kafka bindings document. I have a doubt In Server Binding Object there is a field schemaRegistryVendor, which vendors will we be supporting? Because I could only find this package for confluent @kafkajs/confluent-schema-registry which uses the avsc library for Avro encoding/decoding. Also, will we be also supporting other serializers than Avro?

For further doubts in the future, please let me know which platform is preferred for such doubts Slack or GitHub comments

KhudaDad414 commented 1 year ago

Hi @shubham-y,

In Server Binding Object there is a field schemaRegistryVendor, which vendors will we be supporting?

Based on a recent discussion that I had with @fmvilas, We think it is better to ignore the schemaRegistryVendor for now. This is something that needs to be done in parser-js. we basically get the parsed schema in glee. so we do not need to worry about it.

KhudaDad414 commented 1 year ago

Hey @shubham-y, Have you started working on this issue? 😄 @Ruchip16 can continue to work on it or give a hand if you need some help.

shubham-y commented 1 year ago

Hey @shubham-y, Have you started working on this issue? 😄 @Ruchip16 can continue to work on it or give a hand if you need some help.

Hey @KhudaDad414, I had started working on it but had to take a break due to unavoidable personal circumstances. Sorry, I did not communicate earlier. I have resumed working on it and it will great if @Ruchip16 can help me with this as I am struggling a bit due to limited docs/tutorials for Glee.

Ruchip16 commented 1 year ago

Hey @shubham-y, Have you started working on this issue? smile @Ruchip16 can continue to work on it or give a hand if you need some help.

Hey @KhudaDad414, I had started working on it but had to take a break due to unavoidable personal circumstances. Sorry, I did not communicate earlier. I have resumed working on it and it will great if @Ruchip16 can help me with this as I am struggling a bit due to limited docs/tutorials for Glee.

hey I can initiate a PR for you and you can take it from there, if that sounds good?

shubham-y commented 1 year ago

Hey @shubham-y, Have you started working on this issue? smile @Ruchip16 can continue to work on it or give a hand if you need some help.

Hey @KhudaDad414, I had started working on it but had to take a break due to unavoidable personal circumstances. Sorry, I did not communicate earlier. I have resumed working on it and it will great if @Ruchip16 can help me with this as I am struggling a bit due to limited docs/tutorials for Glee.

hey I can initiate a PR for you and you can take it from there, if that sounds good?

Sounds good. Thank you for that.

kaushik-rishi commented 1 year ago

Me and @shubham-y have agreed to work on it together :slightly_smiling_face:

UPDATE (May 26) Ruchi, said she'll work on it and we are not working on it.

sudoshreyansh commented 1 year ago

Hi, is this issue available to be taken?

kaushik-rishi commented 1 year ago

@Ruchip16 is working on it @sudoshreyansh 🙂

I wanted to take it up but she said she was working on it.

sudoshreyansh commented 1 year ago

@kaushik-rishi 👍

oviecodes commented 1 year ago

/progress 10 I'm starting to work on this.

Souvikns commented 1 year ago

any updates on this @oviecodes?

oviecodes commented 1 year ago

@Souvikns I'm sorry I got so busy, please can you take this, I'll join you once I'm free. Thanks

Amzani commented 1 year ago

@KhudaDad414 will take a look

KhudaDad414 commented 1 year ago

/progress 10 started looking into the Kafka bindings.

oviecodes commented 1 year ago

@KhudaDad414 any updates on this?

KhudaDad414 commented 1 year ago

@oviecodes I am busy with spec v3 support in glee. might do it later. I don't have any progress in it. so feel free to pick it up.

oviecodes commented 1 year ago

Alright man 👍...

oviecodes commented 11 months ago

@fmvilas @Souvikns @KhudaDad414 could anyone please explain this issue further?

KhudaDad414 commented 11 months ago

@oviecodes to summarise, you have to look through the bindings here: https://github.com/asyncapi/bindings/tree/next-major-spec/kafka, parse them from the asyncapi document and apply them whenever necessary. we are using https://kafka.js.org/ in Glee, so all you have to do is parse the bindings and pass them to the kafkajs. that's it.

Ruchip16 commented 11 months ago

Hey @KhudaDad414 can i start working on this again, I promise to complete it this time 😅?

oviecodes commented 11 months ago

@Ruchip16 wanna work on this one together? 😃

Ruchip16 commented 11 months ago

yes yes suree 😊

oviecodes commented 11 months ago

that's great, we'll open a pull request to update our progress on this issue 🙂

oviecodes commented 11 months ago

/progress 20 looking into kafka bindings and kafka.js

oviecodes commented 10 months ago

The Kafka Adapter has to be updated to use spec v3. I'll open an issues as appropriate @Souvikns @KhudaDad414

github-actions[bot] commented 6 months ago

This issue has been automatically marked as stale because it has not had recent activity :sleeping:

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience :heart:

KhudaDad414 commented 6 months ago

still relevant.

github-actions[bot] commented 2 months ago

This issue has been automatically marked as stale because it has not had recent activity :sleeping:

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience :heart: