search

lerenn / asyncapi-codegen

An AsyncAPI Golang Code generator that generates all Go code from the broker to the application/user. Just plug your application to your favorite message broker!
Apache License 2.0
89 stars 23 forks source link

Schema Evolution #63

Closed gedw99 closed 11 months ago

gedw99 commented 1 year ago

Do you have a game plan of how your going to support Schema Evolution ?

We have things in 2 places .. There is a schema description in the yaml and the data living inside nats that conforms to a version of the types described by the yaml schema.

this is not at all unique to your project but in fact an issue with nats itself in my opinion.

The problem is reduced if protobufs are used instead of json . It async-api wants to use json . So then a work around is if we can store the data inside nats as protobuf and convert it to json on the way in and out … So then you won’t need to go back and adjust the pending data inside nats.

just an idea..

anyways it’s a cool project @lerenn

lerenn commented 1 year ago

Hello @gedw99, thanks for your interest in this project ! :)

To be honest, I don't have plan for this (part of it being because I'm not that familiar with event broker yet and haven't been confronted to it yet), but I'm familiar with the concept of migration for SQL databases, so not completely lost. I'll take a look to existing solutions to see what's could be the best.

However, I would be glad to have some advice on how I could handle it if you're familiar with it.

For now, only NATS and JSON are implemented, but we could see other brokers (Kafka, RabbitMQ, etc) and other protocols (such as protobuf) as well (even if a part of me feels that even with that, we should have a solution for classic JSON transmission).

What do you think ?

gedw99 commented 1 year ago

I added a possible solution using protobufs while you were writing your answer :)

gedw99 commented 1 year ago

And regarding other messsge brokers … I prefer nats due to operational simplicity.

that’s just my own bias though.

lerenn commented 1 year ago

Well, I think I'll take a look how to add protobuf to this project soon ! I'll have to check also how NATS handles binary messages (I'm more familiar with text in NATS).

I'll keep you updated, but feel free to keep in touch with me if you have any idea or advice :)

gedw99 commented 1 year ago

Nats has built in proto support and json support.

https://github.com/nats-io/natscli#translating-message-data-using-a-converter-command Was added to allow devs to convert to other types

lerenn commented 1 year ago

Thanks ! I was about to say 'nevermind, I forgot that I already send text as []byte{} in the NATS implementation', it's already in binary :smile:

It should be too hard then. I'll keep you updated !

lerenn commented 11 months ago

I added the versioning support in the last version of this repository: https://github.com/lerenn/asyncapi-codegen#versioning

So I won't be implementing the protobuf support for now, but let me know if that's a problem !