LGouellec / kafka-streams-dotnet

.NET Stream Processing Library for Apache Kafka 🚀

https://lgouellec.github.io/kafka-streams-dotnet/

MIT License

452 stars 73 forks source link

Topology involving topics spread accross multiple clusters #286

Closed kevin-mcmanus closed 9 months ago

kevin-mcmanus commented 9 months ago

Description

I have a use case to read data from cluster (BootstrapServers) A, process the data and write it to a topic in cluster B. I'm not aware of any features that enable this behavior in the current implementation due to the use of a single StreamConfig object. Would it be possible to add this as a feature request, perhaps enabling overload to StreamBuilder.Stream() and StreamBuilder.Table() that take in a subset of the current configuration object.

LGouellec commented 9 months ago

@kevin-mcmanus,

No for now this feature is not available because this library is designed to process data from Kafka to Kafka with a single cluster. Technically it's possible but you loose the Exactly Once Semantics ..

Best regards,

kevin-mcmanus commented 9 months ago

Thanks for the response.