LGouellec / kafka-streams-dotnet

.NET Stream Processing Library for Apache Kafka 🚀
https://lgouellec.github.io/kafka-streams-dotnet/
MIT License
453 stars 73 forks source link

Authentication #256

Closed TechMz closed 1 year ago

TechMz commented 1 year ago

Hey, I can't seem to find a way to get this to work with OAuth like Confluent Kafka.

They expose a method in the Builders called 'SetOAuthBearerTokenRefreshHandler' which takes a callback function providing the bearer token, there does not seem to be an equivalent in this library. Is there something I am missing? As there are OAuth related properties in the config.

Please let me know. Much appreciated, thanks.

LGouellec commented 1 year ago

Hi @TechMz,

Unfortunately, you can't easily to customize this handler.

One solution is to implement your own Kafka supplier and set the handler inside.

Please have a look to :

https://github.com/LGouellec/kafka-streams-dotnet/blob/a6189c8882e1220c82e9f3931bb38d360eb5156c/core/KafkaStream.cs#L287

https://github.com/LGouellec/kafka-streams-dotnet/blob/a6189c8882e1220c82e9f3931bb38d360eb5156c/core/Kafka/Internal/DefaultKafkaClientSupplier.cs#L21

TechMz commented 1 year ago

Thanks for this. I have created the supplier, and am able to connect.

However, it appears to fail in the STARTING state, after 15 to 60 seconds it returns this:

[test-streamer-stream-thread-0] ERROR Streamiz.Kafka.Net.Processors.StreamThread - Unable to locate active task for received-record partition source-topic [[1]]. Current tasks: . Current Consumer Assignment : source-topic-[[0]],source-topic-[[1]]

[test-streamer-stream-thread-0] ERROR Streamiz.Kafka.Net.Processors.StreamThread - stream-thread[test-streamer-stream-thread-0] Encountered the following error during processing: System.NullReferenceException: Task was unexpectedly missing for partition source-topic [[1]] at Streamiz.Kafka.Net.Processors.StreamThread.AddToTasks(IEnumerable`1 records) at Streamiz.Kafka.Net.Processors.StreamThread.Run()

Any idea on why the active task is missing?

TechMz commented 1 year ago

Hey, ignore the above message, worked it out. All working now.

Thanks!