Open benjben opened 4 years ago
I think it's a much bigger workflow. In order to update the KCL lib we'd have to get rid of https://github.com/awslabs/amazon-kinesis-connectors, which is core of the ES (and S3) Loader. I'm considering to switch to something like https://github.com/laserdisc-io/fs2-aws which could make all streaming loaders very source (PubSub/Kinesis/NSQ) and sink (ES, S3) agnostic.
Indeed seems like removing amazon-kinesis-connectors
would be a loss of time as we are already have the intention to merge the loaders and with a better technology. Using fs2
would be great !
Getting this back online involved manually editing the KCL table to ignore the failed shards and moving it to the new shards... very painful recovery.
Hey guys, sorry for being late. I think something similar to it happened for us, too. I think we manually overriden the amazon-kinesis-client
version included in amazon-kinesis-connectors
and it worked. Looks like, something in Kinesis backend has changed. I didn't contributed it back because it was too much mysterious and I didn't have proof to show one-to-one relation.
We started to see weird errors in ES loader logs:
And we can't find a sensible explanation. Version 1.9.1 that we're currently using is 2 years old today. Time to bump!