search

ex-aws / ex_aws

A flexible, easy to use set of clients AWS APIs for Elixir
https://hex.pm/packages/ex_aws
MIT License
1.29k stars 529 forks source link

[Kinesis] Demand-based streaming with GenStage #273

Closed nealfax closed 7 years ago

nealfax commented 8 years ago

One option might be to pipe get_records into a stream:

ExAws.Kinesis.get_records(shard_iterator) |> ExAws.stream!

But that'd still be request/poll-based.

Ideally we could leverage something like GenStage to setup a producer that would respond to downstream demand by pulling from the Kinesis stream. Something along the lines of:

ExAws.Kinesis.get_records(shard_iterator) |> ExAws.start_link

Two side notes (per our conversation on Slack):

  1. It would probably be better handled by a new library that uses ExAws, but it'd be nice if there was an Elixir equivalent to KCL: http://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html#kinesis-record-processor-overview-kcl. In the meantime, I've forked a old library that utilizes KCL's MultiLangDaemon: https://github.com/nealfax/amazon-kinesis-client-elixir
  2. It'd be great to have support for DynamoDB Streams as well. They're very similar to Kinesis streams: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.KCLAdapter.html
benwilson512 commented 7 years ago

Hey, I think this would definitely be useful.

If you want to make a PR that implements this and provides a concrete area for discussion, I welcome it. However I'm not really in a position to take feature requests at the moment. Gonna close this issue, but once again a PR would be welcome.