snowplow / snowplow-elasticsearch-loader

Writes Snowplow enriched events from Kinesis to Elasticsearch
http://snowplowanalytics.com/
11 stars 18 forks source link

Bump amazon-kinesis-client to 2.2.10 #151

Open benjben opened 4 years ago

benjben commented 4 years ago

We started to see weird errors in ES loader logs:

[RecordProcessor-0000] INFO com.amazonaws.services.kinesis.connectors.KinesisConnectorRecordProcessor - Shutting down record processor with shardId: shardId-000000000021 with reason TERMINATE
[RecordProcessor-0000] ERROR com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownTask - Application exception. 
[RecordProcessor-0000] INFO com.amazonaws.services.kinesis.connectors.KinesisConnectorRecordProcessor - Shutting down record processor with shardId: shardId-000000000021 with reason TERMINATE
[RecordProcessor-0000] ERROR com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownTask - Application exception. 

And we can't find a sensible explanation. Version 1.9.1 that we're currently using is 2 years old today. Time to bump!

chuwy commented 4 years ago

I think it's a much bigger workflow. In order to update the KCL lib we'd have to get rid of https://github.com/awslabs/amazon-kinesis-connectors, which is core of the ES (and S3) Loader. I'm considering to switch to something like https://github.com/laserdisc-io/fs2-aws which could make all streaming loaders very source (PubSub/Kinesis/NSQ) and sink (ES, S3) agnostic.

benjben commented 4 years ago

Indeed seems like removing amazon-kinesis-connectors would be a loss of time as we are already have the intention to merge the loaders and with a better technology. Using fs2 would be great !

jbeemster commented 4 years ago

Getting this back online involved manually editing the KCL table to ignore the failed shards and moving it to the new shards... very painful recovery.

szareiangm commented 4 years ago

Hey guys, sorry for being late. I think something similar to it happened for us, too. I think we manually overriden the amazon-kinesis-client version included in amazon-kinesis-connectors and it worked. Looks like, something in Kinesis backend has changed. I didn't contributed it back because it was too much mysterious and I didn't have proof to show one-to-one relation.