Need to Support Kafka Streams APIs

[liu-hongwei]

Description

Start from version 0.10.0.X, Kafka itself supports the Streams APIs, however, seems except the Java version clients support this feature, the .net client still does not support Kafka Streams APIs.

enabling this feature in .net client will be really helpfully when we use Kafka on .net platform , especially for real time data processing.

How to reproduce

not bug, but feature implementation.

Checklist

Please provide the following information:

• [ ] Confluent.Kafka nuget version:
• [ ] Apache Kafka version:
• [ ] Client configuration:
• [ ] Operating system:
• [ ] Provide logs (with "debug" : "..." as necessary in configuration)
• [ ] Provide broker log excerpts
• [ ] Critical issue

[mhowlett]

Unfortunately, we don't have near term plans to implement a Kafka Streams API in .NET (it's a very large amount of work) though we're happy to facilitate other efforts to do so.

Confluent have recently launched KSQL, which effectively allows you to use the Streams API without Java and has a REST API that you can call from .NET. It's more limited, but perhaps it satisfies your use case.

https://www.confluent.io/blog/ksql-open-source-streaming-sql-for-apache-kafka/

[pavel-agarkov]

That contradicts with what is written earlier:

Supporting languages beyond Java

One of the nice things about this approach is that the amount of code is quite small and we think it should be possible to build out a really nice implementation in a number of major languages and still have them feel like natives of their ecosystem. We’re focusing on getting the normal clients in shape there first, but we’ll pick up working on stream processing support shortly after.

[alessandroabba]

Still no intention to publish a client for the kafka streams api for .net?

[mhowlett]

not at this stage. it's a lot of work and we need to implement more foundational things like the AdminClient and exactly-once-semantics first.

[raskolnikoov]

Any update on this nowdays? Would be really cool feature to use in .Net-land :)

[mhowlett]

we're still focussed on getting to feature parity with the java producer/consumer (admin client currently in progress, and work on exactly once about to begin).

at this stage, take a look at KSQL (which is production ready in Confluent Platform 4.1, just released), to see if it meets your needs.

[smitty-codes]

@mhowlett Just making sure I understand your response - Confluent is focused on making sure the existing .net client library (which supports producer/consumer only) is equivalent to the Java version, then Kafka Streams will come? Or you guys are already working on Kafka Streams for .net? Thanks

[mhowlett]

(as much as i would like to be working on this) we don't currently have a timeframe for any non-java streams implementation.

[smitty-codes]

Bummer. Thanks for the follow up.

[ghost]

@mhowlett Any news/update about this? :)

[mhowlett]

keep the up-votes coming and I'll periodically ping the relevant decision makers with this thread :-)

[srinathrangaramanujam]

Any Updates ?

[RTodorov]

Hi @mhowlett ! A few months have passed, just checking if there have been any changes to this topic. Thank you!

[mhowlett]

We don't have any definite plans for streams in .net., but 72 upvotes ... I'll make sure product are aware of this.

It's worth noting that KSQL now has UDFs and UDAFs. I believe we are going to have a good story here for non-java languages relatively soon, which would cover-off many of the use cases of native streams in other languages. KSQL is worth checking out.

https://www.confluent.io/blog/build-udf-udaf-ksql-5-0

[almazik]

I'm eagerly waiting for .NET implementation of KStreams.

I tried to use KSQL for simple data processing/filtering, but I constantly hit roadblocks: Cannot escape reserved words as column names https://github.com/confluentinc/ksql/issues/677 (or ttps://github.com/confluentinc/ksql/issues/1213, same issue) Cannot parse ISO-formatted timestamp https://github.com/confluentinc/ksql/issues/1439 (fixed, waiting for release) Cannot map values https://github.com/confluentinc/ksql/issues/620 Cannot (could not find how to) delete rows from a table based on stream data (e.g. "remove a row if the field "status = 'deleted'") - just created an issue https://github.com/confluentinc/ksql/issues/1945

That's just from our last attempt to add KSQL to our technology stack

[dengia]

Can we push/help in any way for .net kafka-streams? Kafka usage would definitely expand in .net world with streams implemented ...

[SquaLLpt]

Any news about this?

[caryhocker]

New year, new hopes, any news to report on this request?

[smitty-codes]

In the confluent 5.2 release, I see this:

"Confluent Platform 5.2 proudly introduces librdkafka 1.0. This is a big milestone, because it brings this popular client library closer to parity with the Java client for Kafka."

I assume when we're past close and librdkafka is in parity with Java implementation, then we'll see streams for .NET client soon after?

[fred070]

Any updates on Streaming library? It has 122 votes as of this writing, reckon this feature can be added in the near future?

[budibong]

I am learning KAFKA for my green field project and very excited about streaming processing with KAFKA STREAM API, but when i know it only for java, I am confused if I must continue to adopt KAFKA or not :(

[k-tomak]

@mhowlett any news? 144 votes

[mhowlett]

the basic non-java clients are still playing catch-up - achieving feature parity here is our priority (and a prerequisite for a streams implementation). in the meantime, KSQL development is being pushed very heavily as a language agnostic alternative for many use cases.

related, i've been playing with some ideas for stream processing in my spare time independent of confluent, however this is just a start, not for production use, no guarantees etc. etc.

https://www.matthowlett.com/2019-06-02-stream-processing-pt1-what-and-why.html https://github.com/mhowlett/howlett-kafka-extensions

[ghost]

@mhowlett Thanks for your preliminary efforts on this. Perhaps a new issue should be opened to actually track Streams API support, rather than encouraging users who still want this feature to keep necroing this 2-yr old issue?

[ahernandez-allegion]

Is there a reason this issue was closed or was it moved to another issue? Any updates on this?

[mhowlett]

it was closed 2 years ago, i'll open it again as a place to accumulate votes.

this is unlikely to be prioritized in the short term - LOE is too high to match the java streams api and there are too many other pressing things.

Worth mentioning that when transactions are out, I think it will be feasible to roll your own stream processing framework with exactly once semantics that gets you 80% of the way there (some commentary on that in previously referenced links).

[jerotas2005]

Question for mhowlett or anyone that understands the predicament here.

in the meantime, KSQL development is being pushed very heavily as a language agnostic alternative for many use cases.

I'm really not sure how/if KSQL helps in the absence of a StreamBuilder .NET client. Yes I can create and drop streams calling the KSQL API, but I cannot return a stream calling it because it will never return unless I do "LIMIT X" which defeats the purpose of using a stream...I want to get ALL records, but since that's an API it obviously can't return all unless it's a very small amount.

[jerotas2005]

Update to my comment yesterday, we figured out how to query with the "headersonlyreceived" option for SendAsync.

[fabiomarreco]

hi @mhowlett ! Thanks for the effort.

I currently need to mix information from 2 different topics, which have the same key, much the same way as is described in the SO question.

In .Net, assuming I won´t have streams any time soon, what is the recomended way to do that ?

I was thinking about re-assign the partitions on topic 2 manually every time I have a re-assigment on topic 1, Is that something what people do ?

[mhowlett]

@fabiomarreco - yep, you can definitely use the rebalance events of one consumer to control the assignment of partitions on additional consumers.

[mhowlett]

interesting project: https://github.com/LGouellec/kafka-stream-net

[lemkepf]

@mhowlett We have a large .net project and my very first use case for Kakfa was dynamic streams.... alas my hopes are dashed. Any update on where this might be at in the priority list?

[LGouellec]

@lemkepf Hi, my project https://github.com/LGouellec/kafka-stream-net aims to offer a stream processing framework around kafka like Kafka Stream. The goal is to provide the same functionality. if you want contribute it, i’m Ok ;)

[iam-gopi]

Do we have any update on this? I've a .net framework 4.7.2 REST API Project where I'm trying to implement confluent Kafka there. But, unfortunately it's not working. but, the same implementation works fine with SOAP API. Can I get any guidance on this please @mhowlett ?

[smitty-codes]

I keep up with Confluent's blog posts - I know they are marketing in disguise but they do offer alot of good info as well. Yesterday's blog post titled Kafka Streams Interactive Queries Go Prime Time has the first sentence "What is stopping you from using Kafka Streams as your data layer for building applications?"

Answer: C# support :-)

We're using the Microsoft stack and we're likely going to use Azure EventHub, despite its limitations, instead of Confluent Cloud solely because Kafka Streams is not supported in C#. It doesn't make sense for me to retrain and retool my team to use NetBeans/IntelliJ/Eclipse and learning Java when our team is already skilled in C# and Visual Studio and all of our code and processes are built around that.

I'm really hoping you consider C# support.

As some kind of middle ground, I raised on one of Confluent's webinars invoking KSQL like we would using a regular SqlConnection and SqlCommand classes/objects to query Kafka but that was not supported. Maybe this is the path Confluent is pursuing rather than porting Kafka Streams for every language?

Thanks

[mhowlett]

we have a big focus on making ksqldb fully featured and powerful and i think use cases for streams itself will become increasingly niche. on the client side, we're committed to feature parity of the basic clients with java, including adding a lot of things that make streams powerful, like exactly once semantics (including kip-447) and incremental rebalancing. in summary, it's very feasible to write a lot of applications with just the basic clients, and a lot of the applications you'd use the higher level streams api for are the types of things you can easily express in ksql. we'd like to support kafka streams in different languages, but it's a relatively large amount of work, and my perception is the payoff probably isn't there - we're better off putting the focus into ksqldb (disclaimer, i'm not on the product team!)

[smitty-codes]

@mhowlett Thanks Matt. I'll revisit KSQL with my team and see if we can blend it with the regular .NET Kafka client.

I think I'll keep an eye on https://github.com/confluentinc/ksql/issues/734 which appears to be closer to that middle ground I was describing.

[ahernandez-allegion]

@smitty-codes We're also largely an Azure/C# shop. We're currently using Event Hubs with Connect as a stop gap measure and we're doing our stream processing strictly through KSQL. So far the basic C# client support as of 1.4+ (with the updated Schema Registry serdes support) has been pretty good against Event Hubs.

[devopg]

vote C# kafka streams

[amitla]

we are also looking forward for KStreams in .net

[dariengit]

Vote Kafka Streaming for .Net Core

[tolgabalci]

In the absence of Kafka Streaming API support in .Net It would be helpful if a guide and examples could be provided to how best use KSQL in .Net.

[jasoncouture]

+1 here too.

[suadev]

Knock knock for the .Net core 🤓

[kvansaders]

Can the subscriber listen to a ksql stream by name (as it does a topic)?

[almazik]

Can the subscriber listen to a ksql stream by name (as it does a topic)?

KSQL stream IS a topic. Each KSQL stream and table has a topic behind it.

[efeozyer]

Happy new year -.-

[Asif-Mahmood]

any update on stream for .NET core?

[LGouellec]

Hi @Asif-Mahmood, community package exist to support Kafka Streams in .NET Standard 2.1. It's begin in written. All features are not available yet (but soon I hope). You can check here for more details.

[Asif-Mahmood]

that's great and thanks for quick reply.. i hope that it works on .NET core as well because its mentioned on the repo