search

akka / alpakka

Alpakka is a Reactive Enterprise Integration library for Java and Scala, based on Reactive Streams and Akka.
https://doc.akka.io/libraries/alpakka/current/
Other
1.26k stars 643 forks source link

AWS Kinesis: add support for enhanced fan-out #2037

Closed regiskuckaertz closed 4 years ago

regiskuckaertz commented 5 years ago

Short description

From the documentation

This feature enables consumers to receive records from a stream with throughput of up to 2 MiB of data per second per shard

Details

As an alternative to the current poll-based mechanism, this one has Kinesis pushing records over an HTTP2 pipeline, speeding up message propagation. Enhanced fan-out also guarantees throughput per shard no matter how many consumers, i.e. goodbye to throughput exceeded exceptions. Details

YikSanChan commented 4 years ago

@aserrallerios I am not familiar with this, do you mind confirming enhanced fan-out is not covered by your contribution to alpakka-kinesis? Thanks

aserrallerios commented 4 years ago

Thanks for the mention @YikSanChan.

From the documentation:

The following diagram shows the enhanced fan-out architecture. If you use version 2.0 or later of the Amazon Kinesis Client Library (KCL) to build a consumer, the KCL sets up the consumer to use enhanced fan-out to receive data from all the shards of the stream

Alpakka has KCL v2.0 support since 2.0.0, so I think we're covered on that regard.

YikSanChan commented 4 years ago

@regiskuckaertz Shall we close?

regiskuckaertz commented 4 years ago

@YikSanChan yes, great stuff !