Documentation

[danielkasen]

Hey, this plugin sounds like exactly what I need and would love to try it but can't find any documentation on how to define the input plugin in logstash. Could you please provide any detail around how to actually use this?

[danielkasen]

I dug through the source and looks like queue is the only needed varible so I did:

input { s3-sns-sqs { queue => "stg-elasticsearch-logs-processing" } }

But this gives me the error:

{"level":"ERROR","loggerName":"logstash.agent","timeMillis":1518744439688,"thread":"Ruby-0-Thread-2: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22","logEvent":{"message":"Failed to execute action","action":{"metaClass":{"metaClass":{"metaClass":{"action":"LogStash::PipelineAction::Create/pipeline_id:s3-elb-elasticsearch-logs","exception":"NameError","message":"undefined local variable or methodsns' for #","backtrace":["(eval):8:in <eval>'","org/jruby/RubyKernel.java:994:ineval'","/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:84:in initialize'","/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:ininitialize'","/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in execute'","/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:inblock in converge_state'","/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'","/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:inblock in converge_state'","org/jruby/RubyArray.java:1734:in each'","/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:inconverge_state'","/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in block in converge_state_and_update'","/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'","/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in converge_state_and_update'","/usr/share/logstash/logstash-core/lib/logstash/agent.rb:105:inblock in execute'","/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:18:in interval'","/usr/share/logstash/logstash-core/lib/logstash/agent.rb:94:inexecute'","/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:in block in execute'","/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:inblock in initialize'"]}}}}}`

Any advice on getting this to work?

Version 6.2.0 Logstash

[cherweg]

Hey Daniel, I´ll give you a short Howto...(Documentation comes if all the work is done ;-) ):

s3snssqs {
  region => "eu-central-1"
  queue => "logging-stack"
  queue_owner_aws_account_id => "239710123532"
  type => "test"
  codec => plain
  delete_on_success => true
}

Please see the following lines of code for config: https://github.com/cherweg/logstash-input-s3-sns-sqs/blob/9efbaae9632f995bd7cf99ee6e3899bd152a6135/lib/logstash/inputs/s3snssqs.rb#L91-L95

The Plugin consumes S3 files (even gzip format) by listen on an SQS queue that is subscribed to an SNS. Here is a small structure draft: https://github.com/cherweg/logstash-input-s3-sns-sqs/blob/master/img/aws-components.png

[danielkasen]

Thank you. So is this plugin expected to be working at the moment, just in an alpha phase? Using these variables I was able to advance the plugin, but now receive this error:

error: no implicit conversion of nil into String\n Exception: TypeError\n Stack: json/ext/Parser.java:173:ininitialize'\njson/ext/Parser.java:150:in new'\n/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/json-1.8.6-java/lib/json/common.rb:155:inparse'\n/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.1.6/lib/logstash/inputs/s3snssqs.rb:134:in handle_message'\n/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.1.6/lib/logstash/inputs/s3snssqs.rb:220:inblock in run'\n/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/sqs/queue_poller.rb:413:in block in yield_messages'\norg/jruby/RubyArray.java:1734:ineach'\n/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/sqs/queue_poller.rb:412:in yield_messages'\n/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/sqs/queue_poller.rb:405:inblock in process_messages'\norg/jruby/RubyKernel.java:1114:in catch'\n/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/sqs/queue_poller.rb:404:inprocess_messages'\n/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/sqs/queue_poller.rb:336:in block in poll'\norg/jruby/RubyKernel.java:1292:inloop'\n/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/sqs/queue_poller.rb:331:in block in poll'\norg/jruby/RubyKernel.java:1114:incatch'\n/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/sqs/queue_poller.rb:330:in poll'\n/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.1.6/lib/logstash/inputs/s3snssqs.rb:219:inblock in run'\n/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.1.6/lib/logstash/inputs/s3snssqs.rb:235:in run_with_backoff'\n/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.1.6/lib/logstash/inputs/s3snssqs.rb:218:inrun'\n/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:516:in inputworker'\n/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:509:inblock in start_input'"`

[cherweg]

This Plugin is in an early phase and under active development. (But it is in production at my cluster) Ich you send me some payload examples i will take a look at this error at monday.

This is the "crash line": https://github.com/cherweg/logstash-input-s3-sns-sqs/blob/783a5f5fa5a1e26f4c529daac70971fcb68033ab/lib/logstash/inputs/s3snssqs.rb#L134 It seems that your message body has no Message inside, so the Parser crashes...

You could set logstash to debug:

/opt/logstash/bin/logstash -f yourconf.conf -v --debug This should log the messages ... https://github.com/cherweg/logstash-input-s3-sns-sqs/blob/783a5f5fa5a1e26f4c529daac70971fcb68033ab/lib/logstash/inputs/s3snssqs.rb#L131

Please poste a crashing message and i will fix it next week.

regards christian

[cherweg]

Untested fix is commited...i´ll test it Monday with your data.

[cherweg]

I think i have an idea, why it is crashing in your setup... Is it S3-> SNS -> SQS or S3 -> SQS? My Messages are S3-> SNS -> SQS. This means an additional layer inside the message json.

If so, i could make it optional for you.

regards christian

[danielkasen]

Yea, there's no sns setup since I'm just having the s3 bucket event to sqs. It'd be nice to avoid another piece of software.

Thanks

On Feb 18, 2018 1:26 AM, "cherweg" notifications@github.com wrote:

I think i have an idea, why it is crashing in your setup... Is it S3-> SNS -> SQS or S3 -> SQS? My Messages are S3-> SNS -> SQS. This means an additional layer inside the message json.

If so, i could make it optional for you.

regards christian

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/cherweg/logstash-input-s3-sns-sqs/issues/1#issuecomment-366503023, or mute the thread https://github.com/notifications/unsubscribe-auth/AeFp1H7uDIcetB1erlupte4a8VlIPO0Pks5tV-zVgaJpZM4SHoN6 .

-- *Grab is hiring. Learn more at *https://grab.careers https://grab.careers/

By communicating with Grab Inc and/or its subsidiaries, associate companies and jointly controlled entities (“Grab Group”), you are deemed to have consented to processing of your personal data as set out in the Privacy Notice which can be viewed at https://grab.com/privacy/

This email contains confidential information and is only for the intended recipient(s). If you are not the intended recipient(s), please do not disseminate, distribute or copy this email and notify Grab Group immediately if you have received this by mistake and delete this email from your system. Email transmission cannot be guaranteed to be secure or error-free as any information therein could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do not accept liability for any errors or omissions in the contents of this email arises as a result of email transmission. All intellectual property rights in this email and attachments therein shall remain vested in Grab Group, unless otherwise provided by law.

[danielkasen]

Just updated and disabled the sns option and this works perfectly. Thank you so much!