KNet: .NET suite for Apache Kafka™

KNet is a comprehensive .NET suite for Apache Kafka™ providing access to all APIs and features: Producer, Consumer, Admin, Streams, Connect, backends (ZooKeeper and Kafka).

Libraries and Tools

KNet	KNetCLI	KNet.Templates	KNetPS	KNetConnect

Pipelines

Project disclaimer

KNet is a suite for Apache Kafka™, curated by MASES Group, can be supported by the open-source community.

Its primary scope is to support other, public or internal, MASES Group projects: open-source community and commercial entities can use it for their needs and support this project, moreover there are dedicated community and commercial subscription plans.

The repository code and releases may contain bugs, the release cycle depends from Apache Kafka™ release cycle, critical discovered issues and/or enhancement requested from this or other projects.

Looking for the help of Apache Kafka™ experts? MASES Group can help you design, build, deploy, and manage Apache Kafka™ clusters and streaming applications.

Scope of the project

This project aims to create a set of libraries and tools to direct access, from .NET, all the features available in the Apache Kafka™ binary distribution.

There are many client libraries written to manage communication with Apache Kafka™. Conversely, this project use directly the Java packages released from The Apache Foundation giving more than one benefit:

• all implemented features are availables at no extra implementation costs, see KNet usage;
• avoids any third party communication protocol implementation;
• access all features made available from Apache Kafka™: the most important are Apache Kafka™ Streams and Apache Kafka™ Connect which does not have any C# implementation;
• measured high performance in many operating conditions.

Currently the project tries to support, at our best, the supported Apache Kafka™ binary distribution:

• Apache Kafka™ version 3.8.*:
  • branch master
  • KNet version 2.8.*
• Apache Kafka™ version 3.7.*:
  • branch release/2.7.X
  • KNet version 2.7.*
• Apache Kafka™ version 3.6.*:
  • branch release/2.6.X
  • KNet version 2.6.*

The Apache Kafka™ packages are downloaded from:

kafka-clients	kafka-streams	kafka-tools	kafka_2.13

connect-runtime	connect-mirror	connect-file	connect-basic-auth-extension

Community and Contribution

Do you like the project?

• Request your free community subscription.

Do you want to help us?

• put a :star: on this project
• open issues to request features or report bugs :bug:
• improves the project with Pull Requests

This project adheres to the Contributor Covenant code of conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to coc_reporting@masesgroup.com.

Summary

• Roadmap
• Current state
• Performance
• Connect SDK
• Streams SDK
• KNet usage
• KNet APIs extensibility
• KNet Serializer/Deserializer
• KNet CLI usage
• KNet Connect usage
• KNet Docker usage
• KNet PowerShell usage
• KNet Template usage
• How to build from scratch

News

• V1.4.4+: From version 1.4.4 there is a new project, named KNetPS, which permits to write PowerShell client scripts for an Apache Kafka™ cluster and many other things, here full usage.
• V1.4.7+: From version 1.4.7 there is a new project, named KNetConnect, to execute Apache Kafka™ Connect related jobs, here full usage.
• V1.5.4+: From version 1.5.4 there are new packages dedicated to KNet Serializer/Deserializer
• V2.0.0+: From version 2.0.0 the code base is fully reflected from the JARs of the Apache Kafka™ distribution downloaded from Maven; some developed classes still remains beside the specific KNet implementations
• V2.4.0+: From version 2.4.0 it is available the new KNet Streams SDK
• V2.5.0+: From version 2.5.0 there are two breaking changes: uses Java.Lang.String instead of string (System.String) in generated classes and KNet Streams SDK manages the counter-part JVM types
• V2.7.0+: From version 2.7.0:
  • all classes KNetProducer, KNetConsumer and KNet Streams SDK manage the counter-part JVM types
  • serializers supports data exchange based on byte array and ByteBuffer
  • version 2.7.2 introduces ISerDesSelector to optimize serialization selection based on byte array or ByteBuffer
• V2.8.0+: From version 2.8.0: supports Apache Kafka™ version 3.8.*

Runtime engine

KNet uses JNet, and indeed JCOBridge with its features, to obtain many benefits:

• Cyber-security:
  • JVM and CLR, or CoreCLR, runs in the same process, but are insulated from each other;
  • JCOBridge does not make any code injection into JVM;
  • JCOBridge does not use any other communication mechanism than JNI;
  • .NET (CLR) inherently inherits the cyber-security levels of running JVM and Apache Kafka™;
• Direct access the JVM from any .NET application:
  • Any Java/Scala class behind Apache Kafka™ can be directly managed: Consumer, Producer, Administration, Streams, Server-side, and so on;
  • No need to learn new APIs: we try to expose the same APIs in C# style;
  • No extra validation cycle on protocol and functionality: bug fix, improvements, new features are immediately available;
  • Documentation is shared;
• Dynamic code: it helps to write a Java/Scala/Kotlin/etc seamless language code directly inside a standard .NET application written in C#/VB.NET: look at this simple example and KNet APIs extensibility.

JCOBridge resources

Have a look at the following JCOBridge resources:

• Release notes
• Community Edition
• Commercial Edition
• Latest release:

KAFKA is a registered trademark of The Apache Software Foundation. KNet has no affiliation with and is not endorsed by The Apache Software Foundation.