search

akka / alpakka

Alpakka is a Reactive Enterprise Integration library for Java and Scala, based on Reactive Streams and Akka.
https://doc.akka.io/libraries/alpakka/current/
Other
1.26k stars 645 forks source link

Implement Kinesis KPL Flow/Sink #2309

Open aserrallerios opened 4 years ago

aserrallerios commented 4 years ago

Alpakka 2.0.0 introduced KCL Source, but it still relies on "manual" Kinesis Flow/Sink to publish records to Kinesis, using AWS Kinesis SDK.

Although this flow works and is pretty optimized, KPL library allows clients to reach maximum publication throughput at minimum complexity (as the library totally assumes the batching and aggregation logic).

Existing KCL Source should be fully compatible with KPL-published records.

YikSanChan commented 4 years ago

@aserrallerios KPL is on AWS SDK v1 unfortunately.

A thought from my team at Tubi (credit to @touchdown) is:

  1. Port the AggRecord to SDK v2 (just some API changes).
  2. Turn records in this aggregated record before writing to Kinesis. Akka streams can help with throttle and batch control.

We're experimenting with this approach and will update here if we see a good outcome.

rptr94 commented 2 years ago

Hello Any news about using akka with KPL ?