asnare
commented
5 years ago We've got AWS Kinesis support working internally for a client, but haven't merged it yet for a few reasons. In the interests of being open about this:
- With each additional sink we've added the maintenance and testing burden for the future rises in a non-linear manner. In particular we can't easily test these things ourselves as scale. As such we have to carefully consider the "cost" against the benefit of having it.
- The KPL has dependencies that conflict with ours. This is always painful, but it's now at the point where we want to put it in a separate class loader (or something similar) to avoid this problem in general. It's a fair chunk of work.
- Since we put together the internal build, AWS has announced native support for Kafka. As such we're sceptical about whether it's worth supporting Kinesis with Kafka also available on AWS as a managed service.
Some of this has been informed by our experiences supporting the Google Cloud services. These have been fairly high-maintenance in comparison to HDFS and Kafka.
Any chance of support AWS Kinesis other than Hadoop / Kafka in future? :)