search

cherweg / logstash-input-s3-sns-sqs

logstash input downloading files from s3 Bucket by OjectKey from SNS/SQS
Other
29 stars 35 forks source link

[2019-11-25T22:02:12,143][WARN ][logstash.inputs.s3snssqs ] Error in poller loop {:error=>#<TypeError: no implicit conversion of nil into String>} #33

Closed joeythelantern closed 4 years ago

joeythelantern commented 4 years ago

Created an s3 bucket, which sends events to SQS whenever a file is placed there.

Here is my config:

region => "us-west-2"
queue => "xxxx"
access_key_id => "xxxxx"
secret_access_key => "xxxxx"
s3_role_session_name => "Dev-Logs"
codec => line

Here is my stacktrace:

[2019-11-25T22:02:12,143][WARN ][logstash.inputs.s3snssqs ] Error in poller loop {:error=>#<TypeError: no implicit conversion of nil into String>}
[2019-11-25T22:02:12,144][WARN ][logstash.inputs.s3snssqs ] Backtrace:
        json/ext/Parser.java:173:in `initialize'
        json/ext/Parser.java:150:in `new'
        /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/json-1.8.6-java/lib/json/common.rb:155:in `parse'
        /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.0.9/lib/logstash/inputs/sqs/poller.rb:143:in `preprocess'
        /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.0.9/lib/logstash/inputs/sqs/poller.rb:109:in `block in run'
        org/jruby/RubyKernel.java:1193:in `catch'
        /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.0.9/lib/logstash/inputs/sqs/poller.rb:108:in `block in run'
        /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.372/lib/aws-sdk-resources/services/sqs/queue_poller.rb:413:in `block in yield_messages'
        org/jruby/RubyArray.java:1800:in `each'
        /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.372/lib/aws-sdk-resources/services/sqs/queue_poller.rb:412:in `yield_messages'
        /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.372/lib/aws-sdk-resources/services/sqs/queue_poller.rb:405:in `block in process_messages'
        org/jruby/RubyKernel.java:1193:in `catch'
        /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.372/lib/aws-sdk-resources/services/sqs/queue_poller.rb:404:in `process_messages'
        /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.372/lib/aws-sdk-resources/services/sqs/queue_poller.rb:336:in `block in poll'
        org/jruby/RubyKernel.java:1425:in `loop'
        /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.372/lib/aws-sdk-resources/services/sqs/queue_poller.rb:331:in `block in poll'
        org/jruby/RubyKernel.java:1193:in `catch'
        /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.372/lib/aws-sdk-resources/services/sqs/queue_poller.rb:330:in `poll'
        /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.0.9/lib/logstash/inputs/sqs/poller.rb:82:in `block in run'
        /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.0.9/lib/logstash/inputs/sqs/poller.rb:176:in `run_with_backoff'
        /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.0.9/lib/logstash/inputs/sqs/poller.rb:80:in `run'
        /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.0.9/lib/logstash/inputs/s3snssqs.rb:320:in `block in run_worker_thread'
cherweg commented 4 years ago

The poller can’t extract the Message part from Sqs Message.

https://github.com/cherweg/logstash-input-s3-sns-sqs/blob/824ad0c653d1baef6126027659524e4147b33955/lib/logstash/inputs/sqs/poller.rb#L143

Is your Setup S3 -> SNS -> SQS ? If not please set

from_sns => false

If this does not resolve your issue, please post one of your sqs Messages.

Regards Christian

joeythelantern commented 4 years ago

from_sns => false seems to do the trick. I thought I tried that configuration yesterday but maybe I had it wrong. I'll leave the ticket open for another day of testing, then I'll close it.

Thank you very much for your fast response!

joeythelantern commented 4 years ago

Also, quick question. How do I get my s3 metadata through your plugin? I originally could access it through %{[@metadata][s3][metadata]}.

EDIT: I accessed the @metadata and only the basic data is shown, none of metadata I've added.

For example, in the s3 plugin, you add include_object_properties => true to get your custom metadata.

Is there a way to get it form your plugin?

christianherweg0807 commented 4 years ago

I think We habe to catch these error and give a suggestion on that sns sqs thing.

About metadata: I´ll take a look in implementation of the s3 plugin. Maybe i could add this in the next release. At this time you only have [@metadata][s3][object_key] [@metadata][s3][bucket_name] [@metadata][s3][object_folder]

regards Christian

joeythelantern commented 4 years ago

That would be fantastic!

Currently, you can add metadata to an s3 object, and with the normal s3 plugin, you can access it through [@metadata][s3][metadata].

joeythelantern commented 4 years ago 
begin
      @factory.get_s3_client(record[:bucket]) do |s3|
        response = s3.get_object(
          bucket: record[:bucket],
          key: record[:key],
          response_target: record[:local_file]
        )
      end

Is there where you grab the metadata?

I feel we could add metadata: record[:metadata] and it should work

christianherweg0807 commented 4 years ago

ok a first try of the S3 metadata expansion is here:

https://github.com/cherweg/logstash-input-s3-sns-sqs/commit/55f4f4531525b8f047612bc3c36e88481936602c

I´ll test next week. (for testing your selfe use: gem build logstash-input-s3-sns-sqs.gemspec )

regards christian

joeythelantern commented 4 years ago

I'm going to test it tomorrow if I can

christianherweg0807 commented 4 years ago

OK. Released 2.1.0: config :include_object_properties, :validate => :array, :default => [:last_modified, :content_type, :metadata]

christianherweg0807 commented 4 years ago

There is an copy n paste bug in the documentation. I will fix this. Sorry.

https://github.com/cherweg/logstash-input-s3-sns-sqs/blob/master/docs/index.asciidoc#plugins-inputs-logstash-input-s3-sns-sqs-include_object_properties

Von: JTL (Saman) notifications@github.com Gesendet: Tuesday, January 21, 2020 6:21:27 PM An: cherweg/logstash-input-s3-sns-sqs logstash-input-s3-sns-sqs@noreply.github.com Cc: Herweg, Christian Christian.Herweg@otto.de; State change state_change@noreply.github.com Betreff: Re: [cherweg/logstash-input-s3-sns-sqs] [2019-11-25T22:02:12,143][WARN ][logstash.inputs.s3snssqs ] Error in poller loop {:error=>#} (#33)

Does this feature work now? I have been unable to test. If so, is it in the docs?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHubhttps://github.com/cherweg/logstash-input-s3-sns-sqs/issues/33?email_source=notifications&email_token=AJOIIALNRGMOL2IU2R5RUM3Q64VJPA5CNFSM4JRPL3K2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJQRJLQ#issuecomment-576787630, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJOIIALVTQS2QVB75WASSBLQ64VJPANCNFSM4JRPL3KQ.