logstash-plugins / logstash-integration-aws

Apache License 2.0
7 stars 17 forks source link

CloudWatch input plugin for EC2 metrics fails with Name error #41

Open 498749 opened 1 year ago

498749 commented 1 year ago

Logstash information:

Please include the following information:

  1. Logstash version (e.g. bin/logstash --version) : 8.4 and above
  2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker) docker hub elasticsearch logstash, logstash-oss version and one with opensearch plugin
  3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes): EKS fargate
  4. How was the Logstash Plugin installed: Part of the logstash docker image

JVM (e.g. java -version):

If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:

  1. JVM version (java -version)
  2. JVM installation source (e.g. from the Operating System's package manager, from source, etc).
  3. Value of the JAVA_HOME environment variable if set.

OS version (uname -a if on a Unix-like system):

Description of the problem including expected versus actual behavior:

Steps to reproduce:

Setup the logstash in the EKS fargate pod Use the below input plugin configuration input { cloudwatch { namespace => "AWS/EC2" metrics => [ "CPUUtilization" ] filters => { "tag:AutoStop"=> "false"} region => "ap-northeast-1" } }

[INFO ] 2023-10-26 03:27:23.784 [[main]<beats] Server - Starting server on port: 5044 [INFO ] 2023-10-26 03:27:28.575 [[main]<cloudwatch] cloudwatch - [Aws::CloudWatch::Client 200 2.583544 0 retries] list_metrics(namespace:"AWS/EC2")

[ERROR] 2023-10-26 03:27:28.589 [[main]<cloudwatch] javapipeline - A plugin had an unrecoverable error. Will restart this plugin. Pipeline_id:main Plugin: <LogStash::Inputs::CloudWatch namespace=>"AWS/EC2", id=>"cloudwatch-ec2-dev", metrics=>["CPUUtilization"], region=>"us-east-1", statistics=>["Average", "Maximum", "Minimum", "Sum"], enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_b272d550-00cc-4fa6-adaa-11d1acb1ec4b", enable_metric=>true, charset=>"UTF-8">, role_session_name=>"logstash", interval=>900, period=>300, combined=>false> Error: uninitialized constant Aws::EC2 Exception: NameError Stack: org/jruby/RubyModule.java:3948:in const_missing' org/jruby/RubyModule.java:3890:inconst_get' /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.0.0/lib/logstash/inputs/cloudwatch.rb:247:in block in clients' org/jruby/RubyHash.java:794:indefault' org/jruby/RubyHash.java:1265:in []' /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.0.0/lib/logstash/inputs/cloudwatch.rb:316:inresources' /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.0.0/lib/logstash/inputs/cloudwatch.rb:179:in from_resources' /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.0.0/lib/logstash/inputs/cloudwatch.rb:160:inblock in run' org/jruby/RubyArray.java:1865:in each' /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.0.0/lib/logstash/inputs/cloudwatch.rb:157:inrun' /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:410:in inputworker' /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:401:inblock in start_input' [INFO ] 2023-10-26 03:27:29.590 [[main]<cloudwatch] cloudwatch - Polling CloudWatch API [ERROR] 2023-10-26 03:27:29.594 [[main]<cloudwatch] javapipeline - A plugin had an unrecoverable error. Will restart this plugin. Pipeline_id:main Plugin: <LogStash::Inputs::CloudWatch namespace=>"AWS/EC2", id=>"cloudwatch-ec2-dev", metrics=>["CPUUtilization"], region=>"us-east-1", statistics=>["Average", "Maximum", "Minimum", "Sum"], enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_b272d550-00cc-4fa6-adaa-11d1acb1ec4b", enable_metric=>true, charset=>"UTF-8">, role_session_name=>"logstash", interval=>900, period=>300, combined=>false> Error: uninitialized constant Aws::EC2 Exception: NameError Stack: org/jruby/RubyModule.java:3948:in const_missing' org/jruby/RubyModule.java:3890:inconst_get' /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.0.0/lib/logstash/inputs/cloudwatch.rb:247:in block in clients' org/jruby/RubyHash.java:794:indefault' org/jruby/RubyHash.java:1265:in []' /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.0.0/lib/logstash/inputs/cloudwatch.rb:316:inresources' /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.0.0/lib/logstash/inputs/cloudwatch.rb:179:in from_resources' /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.0.0/lib/logstash/inputs/cloudwatch.rb:160:inblock in run' org/jruby/RubyArray.java:1865:in each' /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.0.0/lib/logstash/inputs/cloudwatch.rb:157:inrun' /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:410:in inputworker' /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:401:inblock in start_input' [INFO ] 2023-10-26 03:27:30.595 [[main]<cloudwatch] cloudwatch - Polling CloudWatch API [ERROR] 2023-10-26 03:27:30.598 [[main]<cloudwatch] javapipeline - A plugin had an unrecoverable error. Will restart this plugin. Pipeline_id:main Plugin: <LogStash::Inputs::CloudWatch namespace=>"AWS/EC2", id=>"cloudwatch-ec2-dev", metrics=>["CPUUtilization"], region=>"us-east-1", statistics=>["Average", "Maximum", "Minimum", "Sum"], enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_b272d550-00cc-4fa6-adaa-11d1acb1ec4b", enable_metric=>true, charset=>"UTF-8">, role_session_name=>"logstash", interval=>900, period=>300, combined=>false> Error: uninitialized constant Aws::EC2 Exception: NameError Stack: org/jruby/RubyModule.java:3948:in const_missing' org/jruby/RubyModule.java:3890:inconst_get' /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.0.0/lib/logstash/inputs/cloudwatch.rb:247:in block in clients' org/jruby/RubyHash.java:794:indefault' org/jruby/RubyHash.java:1265:in []' /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.0.0/lib/logstash/inputs/cloudwatch.rb:316:inresources' /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.0.0/lib/logstash/inputs/cloudwatch.rb:179:in from_resources' /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.0.0/lib/logstash/inputs/cloudwatch.rb:160:inblock in run' org/jruby/RubyArray.java:1865:in each' /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-aws-7.0.0/lib/logstash/inputs/cloudwatch.rb:157:inrun' /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:410:in inputworker' /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:401:inblock in start_input'

Provide logs (if relevant):