search

roncemer / spark-sql-kinesis

Kinesis Connector for Spark Structured Streaming
http://www.roncemer.com
Apache License 2.0
11 stars 6 forks source link

Support for Enhanced-Fan-Out for Kinesis Data Streams #4

Open achals opened 1 year ago

achals commented 1 year ago

Kinesis supports something called Enhanced-Fan-Out (https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-api.html) which allows participating consumers to consume data with their own dedicated bandwidth.

I think consumers need to call RegisterStreamConsumer before calling SubscribeToShard. Consumers also need to call DeregisterStreamConsumer since pending consumers may have a cost associated with them.

achals commented 1 year ago

I'd be happy to try to contribute this functionality with some guidance on testing and help publishing a new version of the library.