Streamiz and KNet

[mariomastrodicasa]

Description

Hi @LGouellec, we are working, more or less, on the same topic. If you take look at KNet you can see our project. Currently we cover almost everything the official Apache Kafka does, Apache Kafka Streams too. We use a complete different approach to get access to Kafka Streams from a .NET environment. We focus on the APIs and not on the implementation of them. If there are new features, or bug solved, in new versions of Apache Kafka, they are immediately available. Do you think can be a convergence point?

[LGouellec]

Hi @mariomastrodicasa ,

I had a look to your project few weeks ago. Streamiz's aim is multiple :

• release a .Net stream processing library for Apache Kafka
• provide the same API than Kafka streams Java
• Add more features, just an example asynchronous processing #147 or support FasterStore #100 for the next release

If I'm not wrong, you release a gateway between the Clr and the JVM. So for running a KNet application, you have to run clr and Java runtime in the same process ?

What convergence point are you thinking ?

Best regards,

[mariomastrodicasa]

Hi @LGouellec

If I'm not wrong, you release a gateway between the Clr and the JVM. So for running a KNet application, you have to run clr and Java runtime in the same process ?

You got the technical point, our little drawback is the embedded JVM within the CLR process: with this approach we try to avoid to do again what it was done in Apache Foundation group (development, debug, testing) on each release. The project started as a gateway, however during the development other features were added. One, still open, is masesgroup/KNet#43 which tries to create APIs more closed to .NET style.

I try to write something more for completeness.

I had a look to your project few weeks ago. Streamiz's aim is multiple :

• release a .Net stream processing library for Apache Kafka

KNet is more than stream processing. It covers Administration (in your case some functions are covered from librdkafka) and Apache Connect too, moreover it is able to run the server side application (a Linux OS is recommended from Apache Kafka); the full featured CLI interface gives access to all command-line classes.

• provide the same API than Kafka streams Java

It was a driver for our design.

• Add more features, just an example asynchronous processing External call inside topology #147 or support FasterStore FASTER State store #100 for the next release

I read #147 and from my understanding:

• since KNet has its own KNet Connect SDK it is possible to use Kafka Connect with an extra sink topic;
• KNet comes with a support Java code in the JVM; maybe it is possible to extend some classes and add extra features: the same approach was used to create KNet specific Producer/Consumer able to speed up the library, see performance. Anyway, we need to verify the feasibility of this approach for stream processing,

About #100: until now it is out of project scope; anyway, if it is fast more than RocksDB, there is a C++ version of the library and maybe it can be added in KNet creating some dedicated Java classes with native access (like RocksDB do).

What convergence point are you thinking ?

Since our project aligns to latest Kafka version with little effort, some ideas are:

• Currently you use Confluent.Kafka (which is based on librdkafka) for poducer/consumer, maybe you can try to switch to KNetProducer/KNetConsumer to test their efficiency.
• Also you can try to use KNet Apache Kafka Streams interfaces to:
  • create .NET processor starting from the generic Processor interface (not DSL)
  • together we can improve the usage where there are some lacks.
• Other things can come later

BR

[LGouellec]

Hi @mariomastrodicasa,

Good idea, I will test your KNetProducer/KNetConsumer to test their efficiency, and I will send to you my feedbacks :)

BR