search

waka / kcl-rb

The Amazon Kinesis Client Library implemented by Pure Ruby.
MIT License
11 stars 4 forks source link

KCL issue in consuming the messages #4

Open pmhegde opened 3 years ago

pmhegde commented 3 years ago

I am using this library to produce and consume message from and to kinesis stream. The functionality of this library is working as expected. Initially i tried few sample and it is working fine.

But now I am facing one issue. Whenever i create new stream and start both Producer and consumer and send messages entire flow is working fine. If i ran the consumer continuously also no issues in pipeline.

But when i stop the consumer for some time (Idle for 5 to 6 hrs) and start the consumer again and start pushing the messages again producer is able to successfully send the message and also showing in AWS monitoring. But consumer is not able to receive the record and no error in console also.

i tried below things also.

When i stop the consumer and restart within 5 to 10 mins , no issues. But after 5 to 6 hrs means it will not read.

Only solution working is i need to delete the stream and need to create, then it will work.

elhowm commented 3 years ago

Somehow updating checkpoint in debug mode fix this issue for me for a while. But as you said after few hours with stopped consumer it doesn't work again.

elialbert commented 2 years ago

@elhowm @pmhegde @waka has anyone ever figured anything out here? have we confirmed that this is really on the kcl-rb library? it's hard to tell but I think I'm having the same issue.

elhowm commented 2 years ago

@elhowm @pmhegde @waka has anyone ever figured anything out here? have we confirmed that this is really on the kcl-rb library? it's hard to tell but I think I'm having the same issue.

Unfortunately nope :( I used official java lib with ruby file as event handler. As far as I remember that was this one: https://github.com/awslabs/amazon-kinesis-client-ruby