search

confluentinc / kafka-streams-examples

Demo applications and code examples for Apache Kafka's Streams API.
Apache License 2.0
25 stars 1.15k forks source link

Join operation is not working after update to 1.2.0 #421

Closed alexandraprisecaru closed 2 years ago

alexandraprisecaru commented 2 years ago

Hello,

We are currently using the Streamiz package in our project. We are using it mainly for joining streams in a .NET 6 microservice.

I updated to the latest package version to benefit from the Schema Registry integration, but the join operation is no longer working.

Initially, it was failing because it didn't have a replication factor of 3. I set that in the stream config and then the console logs looked fine, but the join operation was not applied.

I set the log level to debug and attached the logs here. Were there any major changes between the two versions in this area? streamiz_update_1.2.0_logs.txt.txt

Also, I noticed that after the update it auto-creates topics. It also adds additional topics for other operations such as SelectKey. image

This didn't happen before. Is there any way we could disable this, but keep the code up & running? I set the AllowAutoCreateTopics property to false, but it didn't help.

Any help is much appreciated! :) Thanks!

alexandraprisecaru commented 2 years ago

One more question, sorry for the trouble.

I noticed that if I use the SchemaAvroSerDes I am forced to use a value type or an object that implements ISpecificRecord or inherits SpecificFixed. This means, that for my models I need to attach the Schema even though I am communicating with Schema Registry. image

In my implementation of SerDes I skipped this step by using Chr.Avro.Confluent serializer & deserializer. image

Would it be possible for the Streamiz SerDes to have this integrated somehow at one point? I think it would be very useful to rely only on Schema Registry for data governance.

Thanks!