search

logstash-plugins / logstash-output-sqs

Apache License 2.0
4 stars 22 forks source link

sqs output won't connect #9

Closed cdenneen closed 8 years ago

cdenneen commented 8 years ago

awscli on same machine with same Access and Secret Key:

aws sqs get-queue-attributes --queue-url https://queue.amazonaws.com/111111/logstash-qa-us-east-1-logstashbrokerqueue --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible | jq '.[]'
{
  "ApproximateNumberOfMessages": "0",
  "ApproximateNumberOfMessagesNotVisible": "0"
}

/opt/logstash/bin/logstash -f /etc/logstash/conf.d --debug

...
Plugin not defined in namespace, checking for plugin file {:type=>"output", :name=>"sqs", :path=>"logstash/outputs/sqs", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"}
config LogStash::Codecs::Plain/@charset = "UTF-8" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@access_key_id = "XXXXXXXXX" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@secret_access_key = "YYYYYYYYY" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@queue = "logstash-qa-us-east-1-logstashbrokerqueue" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@region = "us-east-1" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@codec = <LogStash::Codecs::Plain charset=>"UTF-8"> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@workers = 1 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@use_ssl = true {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@batch = true {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@batch_events = 10 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@batch_timeout = 5 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
Connecting to AWS SQS queue 'logstash-qa-us-east-1-logstashbrokerqueue'... {:level=>:debug, :file=>"logstash/outputs/sqs.rb", :line=>"111", :method=>"register"}
Unable to access SQS queue 'logstash-qa-us-east-1-logstashbrokerqueue': execution expired {:level=>:error, :file=>"logstash/outputs/sqs.rb", :line=>"115", :method=>"register"}
Will start workers for output {:worker_count=>4, :class=>LogStash::Outputs::SQS, :level=>:debug, :file=>"logstash/output_delegator.rb", :line=>"34", :method=>"initialize"}
config LogStash::Codecs::Plain/@charset = "UTF-8" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@access_key_id = "XXXXXXXXXX" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@secret_access_key = "YYYYYYYYY" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@queue = "logstash-qa-us-east-1-logstashbrokerqueue" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@region = "us-east-1" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@codec = <LogStash::Codecs::Plain charset=>"UTF-8"> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@workers = 1 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@use_ssl = true {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@batch = true {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@batch_events = 10 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@batch_timeout = 5 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
Connecting to AWS SQS queue 'logstash-qa-us-east-1-logstashbrokerqueue'... {:level=>:debug, :file=>"logstash/outputs/sqs.rb", :line=>"111", :method=>"register"}
Unable to access SQS queue 'logstash-qa-us-east-1-logstashbrokerqueue': execution expired {:level=>:error, :file=>"logstash/outputs/sqs.rb", :line=>"115", :method=>"register"}
config LogStash::Codecs::Plain/@charset = "UTF-8" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@access_key_id = "XXXXXXXXX" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@secret_access_key = "YYYYYYYYY" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@queue = "logstash-qa-us-east-1-logstashbrokerqueue" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@region = "us-east-1" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@codec = <LogStash::Codecs::Plain charset=>"UTF-8"> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@workers = 1 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@use_ssl = true {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@batch = true {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@batch_events = 10 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
config LogStash::Outputs::SQS/@batch_timeout = 5 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"}
Connecting to AWS SQS queue 'logstash-qa-us-east-1-logstashbrokerqueue'... {:level=>:debug, :file=>"logstash/outputs/sqs.rb", :line=>"111", :method=>"register"}
...
cdenneen commented 8 years ago

I've also tried with use_ssl => false and received Timeout::Error instead of the above execution expired:

Unable to access SQS queue 'logstash-qa-us-east-1-logstashbrokerqueue': Timeout::Error {:level=>:error, :file=>"logstash/outputs/sqs.rb", :line=>"115", :method=>"register"}
ph commented 8 years ago

Which logstash version you are using?
What is the version of the sqs plugins installed?

You can get the version of the plugins by running this command:

bin/plugin list --verbose output-sqs
``
cdenneen commented 8 years ago

logstash 2.2.2 logstash-output-sqs 2.0.2

jrgns commented 8 years ago

@cdenneen Can you also post your config?

jrgns commented 8 years ago

Some googling also suggests that it might be an IPv6 issue: http://stackoverflow.com/questions/16040158/rails-mailer-netopentimeout-execution-expired-exception-on-production-serve

Can you check that?

cdenneen commented 8 years ago

Closing issue. Issue appears to be related to underlying ESX host networking issue. After VMotioning off that node everything started working.