Spec generator based on Kafka data and traffic

fpighi commented 1 year ago

Reason/Context

This comes out of a message I initially posted on Slack.

I have cases of Kafka clusters, used for service to service messaging, and it would be nice to have an automated way of building an AsyncAPI spec by listening to who's consuming/producing and what types of messages are sent to each topic. This would become even more useful when a Schema Registry is in place, which might contain schemas in multiple formats (JSON, Protobuf, AVRO). In my case those are all Protobuf, so I would prefer if that was the first choice, but I know Protobuf poses some challenges in terms of dynamically deserializing messages in some languages (I'm not sure about any of the TS SDKs for Schema Registry).

The alternatives to this approach that came to mind, both with downsides:

Manually building a spec file. This has the main downside of duplicating the source of truth between Schema Registry and the spec
Building some tool that doesn't rely on listening to live traffic, but instead does that through some common metadata that is part of the schema. This could work, provided those metadata are strictly checked by our code. It also requires migrating a lot of stuff, if that metadata format changes or if we have legacy message schemas that are hard to change.

For this reason, I think having a tool based on live traffic could be a good thing.

Description

This is speculative, because I'm not familiar with the Studio codebase, nor a TS expert. I would prefer somebody more familiar to provide any details that could be useful here.

The closest thing I found to this idea is https://github.com/SolaceLabs/event-discovery-agent

github-actions[bot] commented 1 year ago

Welcome to AsyncAPI. Thanks a lot for reporting your first issue. Please check out our contributors guide and the instructions about a basic recommended setup useful for opening a pull request.
Keep in mind there are also other channels you can use to interact with AsyncAPI community. For more details check out this issue.

github-actions[bot] commented 8 months ago

This issue has been automatically marked as stale because it has not had recent activity :sleeping:

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience :heart:

KhudaDad414 commented 7 months ago

still relevant.

github-actions[bot] commented 3 months ago