logstash-plugins / logstash-input-sqs

Apache License 2.0
16 stars 40 forks source link

SignatureDoesNotMatch and MissingCredentialsError issues #2

Closed mcasado closed 9 years ago

mcasado commented 9 years ago

Sporadically logstash gets the following errors that breaks the pulling loop and at some point all threads stop pulling messages from sqs queue :

logstash.log-20150218.gz:{:timestamp=>"2015-02-17T07:37:33.181000+0000", :message=>"Error reading SQS queue.", :error=>#<AWS::SQS::Errors::SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

logstash.log-20150223.gz:{:timestamp=>"2015-02-22T20:02:04.994000+0000", :message=>"Error reading SQS queue.", :error=>#<AWS::Errors::MissingCredentialsError:

I believe similar problem has been reported in logstash sqs output

ph commented 9 years ago

Thank you for reporting this error, I am currently investigating a similar bug. One question, how many threads are you using in your SQS input?

mcasado commented 9 years ago

I was running 10 threads originally. I changed it to 3 to check but still got some failures. As you now the failure stops the pulling on the thread. If it is a multi-threading issue somewhere I suppose one thread at least is alive but is not fast enough to pull messages from the queue. Probably I need the 10 threads and 2 instances running logstash to keep up consuming messages. Thanks for looking into it.

ph commented 9 years ago

I have encountered this issue when developing #18 and I have changed the logic. There was something wrong with the backoff strategy that was making the input threads stop without warning when we had too many specific SQS errors OR with one RuntimeError error.

RuntimeError should now be raised up to the pipeline and the pipeline will restart the plugin.

ph commented 9 years ago

Fixed in #18 and released as 1.1.0 version. Also the AWS-SDK v2 api have retry logic to deal with checksum error. If it still an issue please reopen it.