search

aws / aws-sdk-ruby

The official AWS SDK for Ruby.
https://aws.amazon.com/sdk-for-ruby/
Apache License 2.0
3.56k stars 1.22k forks source link

Asynchronous producers in Kinesis #1930

Open kamaljoshi opened 5 years ago

kamaljoshi commented 5 years ago

Issue description

Currently Kinesis API within the AWS Ruby SDK only supports synchronous producers. While benchmarking internally we found the latency of put to be around 8ms on average. This seems a lot and ideally an application using Kinesis a lot will this quite fast. So, I was looking for asynchronous producers on Kinesis, it doesn't seem like Ruby MRI has a library that supports that. From Kinesis docs it seems like the performant way to use that is KPL but that has interfaces available only for Java so that rules it out.

Do you guys have any suggestion on this? I was thinking of implementing a side extension to the main lib that comes with aws-sdk-kinesis where the main thread just pushes the job onto a list which has a worker thread which takes care of publishing messages to Kinesis asynchronously. The current approach is a fire and forget approach where you won't get acknowledgement of message deliveries and can lead to dropped messages in case like when Kinesis's APIs are down.

Gem name

aws-sdk, aws-sdk-kinesis

Version of Ruby

Ruby: 2.4/2.5

srchase commented 5 years ago

Thanks for opening this issue.

Are you looking for something like this, but for use with Ruby?

Amazon Kinesis Producer

I've marked this as a feature request for the team to consider.

kamaljoshi commented 5 years ago

Yes, Something like KPL would be the best option. Considering that KPL's core lib is in C++, one option is to extend it using something like FFI and use that to expose Ruby interfaces.

Right now, I think the two options to do here are to do a futures like pattern with a lib like concurrent-ruby or the separate thread approach as mentioned.

If you guys don't have bandwidth, I can take this up if we can agree on the implementation.

conrosebraugh commented 5 years ago

Is this something that is under active development? My organization has similar needs and could benefit from this feature.

pchaganti commented 5 years ago

👍

github-actions[bot] commented 3 years ago

Greetings! We’re closing this issue because it has been open a long time and hasn’t been updated in a while and may not be getting the attention it deserves. We encourage you to check if this is still an issue in the latest release and if you find that this is still a problem, please feel free to comment or open a new issue.

github-actions[bot] commented 2 years ago

Greetings! We’re closing this issue because it has been open a long time and hasn’t been updated in a while and may not be getting the attention it deserves. We encourage you to check if this is still an issue in the latest release and if you find that this is still a problem, please feel free to comment or open a new issue.

github-actions[bot] commented 1 year ago

Greetings! We’re closing this issue because it has been open a long time and hasn’t been updated in a while and may not be getting the attention it deserves. We encourage you to check if this is still an issue in the latest release and if you find that this is still a problem, please feel free to comment or open a new issue.