search

WW-Digital / reactive-kinesis

A Scala & Akka based implementation for working with Amazon Kinesis Streams
Apache License 2.0
34 stars 13 forks source link

Kinesis Client Issues? #75

Closed chasdevs closed 4 years ago

chasdevs commented 5 years ago

I noticed that you guys updated the AWS Kinesis Client library today (very recently), and I was wondering if you had been seeing any strange behaviors with workers getting shut-down.

We have a consumer that we wrote using the amazon-kinesis-client library and it stopped processing records yesterday. My coworker swapped in your library to try and tackle the problem, but we saw the same issue (a graceful shutdown is initiated).

Sorry, this is pretty vague but we're at a loss and wanted to reach into the ether to see if others were having unexplained Kinesis issues today.

agaro1121 commented 5 years ago

@chasdevs this could happen for many reasons. Things such as leasing issues and what not around the shards. My team mitigates this issue by shutting down our consumers (Scala Microservice) and rely on our infrastructure (Kubernetes) to restart our service. I know this isn't an ideal solution but it definitely works for us and helps the service get back to a clean state. Just to add to the above, our consumers ONLY consume data and process it. The consumers don't do other things such as serve HTTP endpoints.

agaro1121 commented 5 years ago

Also, @chasdevs just to be clear, this library wraps the Amazon Kinesis KCL library. Any underlying issues with the Amazon KCL would be observed here as well

Chuwiey commented 5 years ago

@agaro1121 (same team as @chasdevs ) - yeah, that has been our solution as well for a while, but somehow specifically this weekend, it's not working anymore and we consistently get shut down for "zombie" workers. Not sure what's going on there, but wondering if you upgraded because you saw something on your end?

chasdevs commented 5 years ago

Thanks for the quick response, @agaro1121. My curiosity was peaked because I saw the note in your release log that says ~~amazon-kinesis-client 1.8.10 -> 1.9.3~~ (broke, reverted) on the same day I happened to be thinking "why does the Kinesis client seem broken. I'm going to try out our code with the updated Kinesis library and see if we see any changes.

chasdevs commented 5 years ago

We're pulling from the maven repo; could you guys push the new release artifact up when you get a chance? Your library also reduces the amount of code in our service, so I have two reasons to want to use it.

agaro1121 commented 5 years ago

@Chuwiey @chasdevs in regards to the issues you're experiencing, I will need more info on your setup. Maybe someone on the team can jump in.

@markglh do you remember why we reverted this: ~~amazon-kinesis-client 1.8.10 -> 1.9.3~~ (broke, reverted) ?

I'll work with @markglh on any missing artifacts