search

awslabs / amazon-kinesis-producer

Amazon Kinesis Producer Library
Apache License 2.0
397 stars 330 forks source link

Will KPL support writing to kinesis firehose? #29

Open loneknightpy opened 8 years ago

gharikum commented 8 years ago

We do plan to update the KPL to support Firehose. Unfortunately we do not have a date yet.

findchris commented 8 years ago

Still no date here?

benalta commented 8 years ago

Hi Chris, can you send me email to benaltar@amazon.com? would love to hear about your use case.

eusebeus commented 8 years ago

Any update re: target date for this feature?

marcelomalcher commented 8 years ago

Hey there, any updates on this?

kippster commented 8 years ago

I was told by my AWS rep on July 2, 2016 that KPL does not support firehose and there is not intention for Amazon engineers to do that...they are leaving it up to the community to write that piece of code...

pfifer commented 7 years ago

It seems there is interest in adding support for Firehose to the KPL, and it's something we're investigating. We are prioritizing the adding the support with other customer requests.

Other please comment, or add reactions to assist us in prioritizing the issue.

meowcakes commented 7 years ago

Would be great if it was supported for Firehose, to save me re-implementing all the logic myself.

muhufuk commented 6 years ago

any good news ?

RyPeck commented 5 years ago

If there was a Lambda Blueprint for transforming KPL records in a Firehose Stream that would be helpful.

wsargent commented 5 years ago

Is this about writing directly to Firehose, without going through a data stream? Instead of "KPL → Kinesis Data Stream → Kinesis Data Firehose" you have "KPL → Kinesis Data Firehose"?

It seems a bit odd to do that, because with a direct PUT you still have the 5000 limit, where if you used the stream you'd not be subject to that.

https://docs.aws.amazon.com/firehose/latest/dev/limits.html

"When Kinesis Data Streams is configured as the data source, this limit doesn't apply, and Kinesis Data Firehose scales up and down with no limit."

EDIT: Ah, I see the pricing difference between Kinesis Data Streams and Firehose now...

wsargent commented 5 years ago

Probably the best thing to do is use https://github.com/awslabs/amazon-kinesis-producer/blob/master/aggregation-format.md and https://github.com/awslabs/kinesis-aggregation/tree/master/java/KinesisAggregator with your own streaming implementation.

soreko commented 4 years ago

Would be great if it was supported for Firehose, to save me re-implementing all the logic myself.