Amazon Kinesis Support Enhanced Fan-Out

[pauldheinrichs]

Description

Feel free to close this if there is a better place to put this or if I'm incorrect. But I feel like there are improvements that could be made to the kinesis ingestion to resolve the main "known issue" under the kinesis druid docs.

Before you deploy the Kinesis extension to production, consider the following known issues: Avoid implementing more than one Kinesis supervisor that reads from the same Kinesis stream for ingestion. Kinesis has a per-shard read throughput limit and having multiple supervisors on the same stream can reduce available read throughput for an individual supervisor's tasks.

What does Enhanced fan out do?

A consumer that uses enhanced fan-out doesn't have to contend with other consumers who are receiving data from the stream

Source Docs:

• https://aws.amazon.com/blogs/aws/kds-enhanced-fanout/
  • V1 docs: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesis/AmazonKinesis.html#putRecords-com.amazonaws.services.kinesis.model.PutRecordsRequest-

https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-api.html

---

I have not dug into the kinesis consuming code enough to say what the lift would be to support this, but thought posting this for visibility might be worth while? :shrug:

[yurmix]

This will require solving https://github.com/apache/druid/issues/16903