Implement missing stage types from alpakka-kafka project

[IgorFedchenko]

There are lots of consumer and provider stage types already implemented in this project: https://github.com/akka/alpakka-kafka

We need to implement consumers from here: https://doc.akka.io/docs/alpakka-kafka/current/consumer.html And producers from here: https://doc.akka.io/docs/alpakka-kafka/current/producer.html

Some of them are going to be based on brand new flowWithContext-type stages implemented, so until Akka.NET v1.4.0-beta2 will be released, this can be based on nightly Akka builds.

This is not critical to implement all stages from alpakka right now, so the concrete must-have list to implement will be specified/discussed later.

TODO List

Consumers

• [x] PlainSource
• [x] PlainExternalSource
• [x] CommittableSource
• [x] CommittableExternalSource
• [x] CommitWithMetadataSource
• [x] AtMostOnceSource
• [x] SourceWithOffsetContext
• [x] PlainPartitionedSource
• [x] PlainPartitionedManualOffsetSource
• [x] CommittablePartitionedSource
• [x] CommitWithMetadataPartitionedSource

Transactional consumers

• [x] Source
• [ ] SourceWithOffsetContext

Producers

• [x] PlainSink
• [x] FlexiFlow
• [x] FlowWithContext

Transactional producers

• [x] Sink (not working until #85 is resolved )
• [x] Flow (not working until #85 is resolved )
• [ ] SinkWithOffsetContext
• [ ] FlowWithOffsetContext

[IgorFedchenko]

I wrote some short overview about how current scala implementation uses buffering and separate actor for stages implementation here

It looks like we will need to implement something similar in our project. This is going to be a series of PRs with

1. Code refactoring to use base classes containing shared logic between stages.
2. Implementing of missing stages (or some of them)
3. Looking at the external actor usage to consume messages in batch.

The one more thing I noticed in scala imlementation is that consumer.Poll method they use returns list of messages - but our confluent driver returns only one message in a time. This may result in somewhat different implementation on our side.

[Aaronontheweb]

@IgorFedchenko let's add the following stages next so we can do a beta release of this package:

• FlowWithContext
• Flow
• Sink

[IgorFedchenko]

@Aaronontheweb So you want to implement some of Transactional Producers? Also, I suspect I will need FlexiFlow producer implemented to implement FlowWithContext - I will check that out.

[Aaronontheweb]

Ok, got it - thank you!

[IgorFedchenko]

All right, now all non-transactional stages in review, and transactional producers are almost implemented, but blocked until #85 is resolved.

[IgorFedchenko]

The rest of the transactional stages (*WithContext ones) will be implemented after stages without context will work well.

[Aaronontheweb]

@IgorFedchenko so only 3 more stages left?

[IgorFedchenko]

@Aaronontheweb Technically yes. But the thing is that transactional sources are ment to be used in pair with transactional producers. And transactional producers are based on IProducer of Confluent driver, which does not support transactions yet (see our issue #85 ). So, until they will add support for transactions, there is no much point in implementing the rest of the stages.

But if I will not have any other work to do, I will work on this, so that after Confluent driver will fix their issue we could just make minor changes in code and make it all work.