search

WW-Digital / reactive-kinesis

A Scala & Akka based implementation for working with Amazon Kinesis Streams
Apache License 2.0
34 stars 13 forks source link

Support KCL Version 2.x #68

Open aquamatthias opened 6 years ago

aquamatthias commented 6 years ago

Looks like the AWS guys improved KCL: KCL 2.0 uses async IO with HTTP/2. Would be great to see support in reactive-kinesis for this.

Migration guide from 1.x to 2.x: https://docs.aws.amazon.com/streams/latest/dev/kcl-migration.html

markglh commented 6 years ago

Agreed, I've been waiting for this to be released and stabilise - we've used the underlying sdk 2 in a few projects now and it seems to be just about there. Lot of fixes in kcl 2.0.1: https://github.com/awslabs/amazon-kinesis-client/releases

I'll look into what this entails and figure out an approach. It's definitely gonna be breaking...

lxwbr commented 5 years ago

Hey @markglh just wondering if you had time to make some progress on the migration yet.

agaro1121 commented 5 years ago

We're splitting the consumer and producer into separate modules so we can evolve them separately. I have a PR open for this, once we merge it in, the upgrade to KCL should be a little easier

etspaceman commented 5 years ago

Something to keep in mind with this is that kinesalite does not currently support the 2.x API.

https://github.com/mhart/kinesalite/issues/75

etspaceman commented 5 years ago

You can get the V2 library to work w/ localstack (kinesalite), see comment here:

https://github.com/localstack/localstack/issues/893#issuecomment-486898065