search

cherweg / logstash-input-s3-sns-sqs

logstash input downloading files from s3 Bucket by OjectKey from SNS/SQS
Other
29 stars 35 forks source link

logstash-input-s3-sns-sqs Unknown setting s3_options_by_bucket #63

Closed ruliyeret closed 3 years ago

ruliyeret commented 3 years ago

I tried to run logstash with the input-s3-sns-sqs plugin but encounter with error

[ERROR] 2021-07-12 09:24:20.065 [Converge PipelineAction::Create

] s3snssqs - Unknown setting 's3_options_by_bucket' for s3snssqs [ERROR] 2021-07-12 09:24:20.072 [Converge PipelineAction::Create
] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$VARARGS(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:389)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]} warning: thread "Converge PipelineAction::Create
" terminated with exception (report_on_exception is true): LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main> create at org/logstash/execution/ConvergeResultExt.java:135 add at org/logstash/execution/ConvergeResultExt.java:60 converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:402 [ERROR] 2021-07-12 09:24:20.079 [Agent thread] agent - An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>"} [FATAL] 2021-07-12 09:24:20.088 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in create'", "org/logstash/execution/ConvergeResultExt.java:60:inadd'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:402:in `block in converge_state'"]} [FATAL] 2021-07-12 09:24:20.118 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit org.jruby.exceptions.SystemExit: (SystemExit) exit at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.16.0.jar:?] at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.16.0.jar:?] at usr.share.logstash.lib.bootstrap.environment.
(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]

This is my configuration `input { s3snssqs { region => "us-west-1" s3_default_options => { "endpoint_discovery" => true } queue => "*" access_key_id => "***" secret_access_key => "" type => "sqs-logs" tags => ["client_logs"] sqs_skip_delete => true from_sns => false codec => json s3_options_by_bucket => [ { bucket_name => "" folders => [ { key => ".\/" codec => "json_lines" type => "reports"}] } ] } } filter { grok { break_on_match => false match => { "message" => "%{DATA:timestamp} [%{DATA:severity}] [%{DATA:module}\s{0,10}] [%{DATA:thread_num}:%{DATA:pid}] [%{DATA:function_name} \s{0,35}:\s{0,5}%{NUMBER:line}] [%{DATA:Env}:%{DATA:account_id}:%{DATA:user_id}] %{GREEDYDATA:message}" } overwrite => ["message"] } date { match => ["timestamp", "HH:MM:ss.SSSZZZ"] target => "@timestamp" }

mutate {
  add_field => {
    "file" => "%{[@metadata][s3][object_key]}"
  }
}
grok{
   break_on_match => false
   match => { "file" => "%{WORD}/%{WORD:environment}/%{GREEDYDATA}/%{GREEDYDATA:file_tmp}.log"}
}

mutate {
  split => [ "file_tmp" , "_" ]
}
mutate {
  add_field => {
       "machine_id" => "%{[file_tmp][0]}"
       "os" => "%{[file_tmp][1]}"
       "version" => "%{[file_tmp][2]}"
       "log_type" => "%{[file_tmp][3]}"
  }
}
mutate {
  remove_field => [ "file_tmp", "syslog_severity_code", "syslog_facility"]
}

}

output { elasticsearch { hosts => ["127.0.0.1:9200"] index => "client-demo" } stdout { codec => rubydebug } }

` What I'm missing?

ruliyeret commented 3 years ago

I just upgraded the plugin from 1.6.1 to 2.1.2 and the problem is now solved :)

dubiza commented 3 years ago

@ruliyeret - Mind sharing the process you followed to update the plug-in? I have tried to update it using logstash-plugin and even tried to specifically install version 2.1.2, but I'm having some issues that go beyond my novice logstash and ruby experience.

$ sudo /usr/share/logstash/bin/logstash-plugin install --version 2.1.2 logstash-input-s3-sns-sqs
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Validating logstash-input-s3-sns-sqs-2.1.2
Resolving mixin dependencies
Updating mixin dependencies logstash-mixin-aws
Bundler attempted to update logstash-mixin-aws but its version stayed the same
Installing logstash-input-s3-sns-sqs
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash-mixin-aws":
  In snapshot (Gemfile.lock):
    logstash-mixin-aws (= 5.0.0)

  In Gemfile:
    logstash-input-s3 was resolved to 3.8.1, which depends on
      logstash-mixin-aws (>= 4.3.0)

    logstash-input-s3-sns-sqs (= 2.1.2) was resolved to 2.1.2, which depends on
      logstash-mixin-aws (~> 4.3)

    logstash-output-cloudwatch was resolved to 3.0.9, which depends on
      logstash-mixin-aws (>= 1.0.0)

Running bundle update will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.

The above is what I get when trying to update. I'm running Logstash 7.15.1 installed on Ubuntu 20.04 with basically everything as default. I have changed no configuration files and have no pipelines configured yet. Just been trying to figure out how to get the the s3snssqs input configuration working. That's when I found your issue.

ykram commented 2 years ago

@ruliyeret - Mind sharing the process you followed to update the plug-in? I have tried to update it using logstash-plugin and even tried to specifically install version 2.1.2, but I'm having some issues that go beyond my novice logstash and ruby experience.

$ sudo /usr/share/logstash/bin/logstash-plugin install --version 2.1.2 logstash-input-s3-sns-sqs
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Validating logstash-input-s3-sns-sqs-2.1.2
Resolving mixin dependencies
Updating mixin dependencies logstash-mixin-aws
Bundler attempted to update logstash-mixin-aws but its version stayed the same
Installing logstash-input-s3-sns-sqs
Plugin version conflict, aborting
ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash-mixin-aws":
  In snapshot (Gemfile.lock):
    logstash-mixin-aws (= 5.0.0)

  In Gemfile:
    logstash-input-s3 was resolved to 3.8.1, which depends on
      logstash-mixin-aws (>= 4.3.0)

    logstash-input-s3-sns-sqs (= 2.1.2) was resolved to 2.1.2, which depends on
      logstash-mixin-aws (~> 4.3)

    logstash-output-cloudwatch was resolved to 3.0.9, which depends on
      logstash-mixin-aws (>= 1.0.0)

Running bundle update will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.

The above is what I get when trying to update. I'm running Logstash 7.15.1 installed on Ubuntu 20.04 with basically everything as default. I have changed no configuration files and have no pipelines configured yet. Just been trying to figure out how to get the the s3snssqs input configuration working. That's when I found your issue.

hey @dubiZA , not sure if you figured it out or not but what worked for me was to use --no-verify to install 2.1.2:

sudo /usr/share/logstash/bin/logstash-plugin install --no-verify --version 2.1.2 logstash-input-s3-sns-sqs

dubiza commented 2 years ago

@ykram - thanks for the feedback. I can't remember if I tried --no-verify. In the end I just used Filebeat's AWS module which at least has Elastic support and various filesets out the box to help with some of the typical log types. It was much easier to work with, but I guess the trade-off is increases system resource usage if running Filebeat and Logstash.

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-aws.html