search

akka / alpakka

Alpakka is a Reactive Enterprise Integration library for Java and Scala, based on Reactive Streams and Akka.
https://doc.akka.io/libraries/alpakka/current/
Other
1.26k stars 643 forks source link

Kinesis Support for KPL and KCL Libraries #433

Closed jaymell closed 4 years ago

jaymell commented 7 years ago

Hello: I wanted to see whether anyone has investigated the feasibility of writing a Kinesis connector that supports AWS's Producer and Consumer libraries.

The advantage is that the producer library already supports aggregation of records both by batching records in requests and by aggregating multiple records into single requests. The consumer library supports the required de-aggregation as well as coordinating multiple consumers via checkpointing with DynamoDB.

We have been POC'ing Kafka with Akka streams apps for the past couple months at my company and are considering Kinesis as an alternative. Without the additional features that the KPL and KCL libraries offer, however, Kinesis is much less attractive.

aserrallerios commented 7 years ago

I've opened a PR of a Flow that uses the low-level sdk to PUT records. After that, I'll open a PR of a Source using KCL; i'ts already tested and implemented, I'm just finishing the migration to alpakka.

So the only missing piece would be a Flow that PUTs records using KPL. I guess I can have a look at it after finishing the KCL Source.

aserrallerios commented 7 years ago

The PR: https://github.com/akka/alpakka/pull/434

There's still work to do but feedback would be much appreciated :)

jaymell commented 7 years ago

@aserrallerios I was planning to work on the consumer side this week, so I will use your branch as a starting point. Thanks so much for the link!

aserrallerios commented 7 years ago

Oh, I'll keep working on it, just minor tweaks and near trivial tests are missing.

jaymell commented 7 years ago

@aserrallerios Sounds great. I will work to integrate our existing app using your branch -- will be glad to provide any feedback as I do so.

eduardojld commented 7 years ago

Hi @aserrallerios @jaymell , is there any progress on the sink stage with the put request, I have a working version that uses the sdk directly (no KPL) based on the sqs one, was about to rise the pull request when I bump into your ticket, I cant find a PR with that feature, is there one?

aserrallerios commented 7 years ago

Here it is the sink sdk PR (merged): https://github.com/akka/alpakka/pull/408 KCL PR: https://github.com/akka/alpakka/pull/434

YikSanChan commented 4 years ago

KCL support is released in 2.0.0. See https://akka.io/blog/news/2020/04/30/alpakka-2.0.0-released

ennru commented 4 years ago

Thank you for the ping. Available as of 2.0.0.