mhart / kinesalite

An implementation of Amazon's Kinesis built on LevelDB
MIT License
808 stars 86 forks source link

Fix duplicate records returned by GetRecords #36

Closed teamdoug closed 6 years ago

teamdoug commented 8 years ago

Hi @mhart,

Thanks for a great library. It helps a lot in our testing :) However, I noticed one of my tests started failing due to duplicated records being returned, and I think it's related to this recent change: 7384fccf When calling GetRecords on a shard and getting no records less than a second after getting a record, incrementSequence can potentially move the seqTime of the next shard iterator backwards so that the next call gets the previous record again. This happens because when we get a record, it increments the seqTime to be a second after the record's time, but when we call GetRecords again in the same second and get no new records, it defaults to now, which gets truncated to be a second earlier than our current shard iterator. This change makes sure that seqTime doesn't go backwards in this case. This change fixes my test, but I'm not sure it's the right fix or how it will interact with the expected behavior change in #34 I created a gist for a go binary that repros the behavior. Please let me know if there's any more information I can provide.

mhart commented 8 years ago

Ugh, ouch – I was wondering if my "fix" was gonna break things.

It's definitely a tricky problem – I'll have a think for a bit about whether this is the best solution and get back to you.

Thanks!

sevein commented 7 years ago

I've just run into this issue, it was driving me crazy!

@teamdoug, thank you for your report!

rama-nallamilli commented 7 years ago

Great library @mhart, has helped us alot for testing so thanks! We also hit this issue all of a sudden when our docker image pulled the latest, our tests failed due to KCL now receiving the last record in the stream multiple times.

mhart commented 6 years ago

Ok, I'm gonna merge this – I'm not sure what a better solution is at this point

sevein commented 6 years ago

Thanks!