search

opensearch-project / logstash-output-opensearch

A Logstash plugin that sends event data to a OpenSearch clusters and stores as an index.
https://opensearch.org/docs/latest/clients/logstash/index/
Apache License 2.0
105 stars 80 forks source link

[BUG] aws-core-sdk 3.207.0 #258

Open nastiazaran opened 1 week ago

nastiazaran commented 1 week ago

Describe the bug There is a new minor version of aws-core-sdk gem -> https://rubygems.org/gems/aws-sdk-core/versions/3.207.0 This version requires the 'account_id' parameter -> https://github.com/aws/aws-sdk-ruby/blob/version-3/gems/aws-sdk-core/CHANGELOG.md?plain=1#L13 My current config, which was working fine with the gem version 3.205.0, uses the aws_iam auth_type to authenticate with the iam profile of the instance:

        auth_type => {
            type => 'aws_iam'
            region => 'us-east-1'
        }

Since the new version release I'm not able to use this output plugin with my current config, I've tried to add the 'account_id' parameter but I get the same error:

Pipeline error {:pipeline_id=>"main", :exception=>#<NoMethodError: undefined method account_id' for #>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-3.207.0/lib/aws-sdk-core/credential_provider_chain.rb:49:in static_credentials'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-3.207.0/lib/aws-sdk-core/credential_provider_chain.rb:13:in block in resolve'", "org/jruby/RubyArray.java:1821:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-3.207.0/lib/aws-sdk-core/credential_provider_chain.rb:12:in resolve'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:85:in aws_iam_auth_initialization'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:55:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client.rb:333:in build_adapter'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client.rb:337:in build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client.rb:58:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client_builder.rb:119:in create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client_builder.rb:112:in build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/plugin_mixins/opensearch/common.rb:38:in build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch.rb:227:in register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233:in block in register_plugins'", "org/jruby/RubyArray.java:1821:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:598:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:245:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:190:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:142:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/logs.conf"], :thread=>"#"}

To Reproduce Steps to reproduce the behavior:

  1. Install this plugin --version '2.0.2' logstash-output-opensearch
  2. Add an output block with this auth type: 
        auth_type => {
        type => 'aws_iam'
        region => 'us-east-1'
    }
  3. See error

Expected behavior Logstash process should start and index data to the aws opensearch cluster using the IAM profile of the instance where logstash is running. Plugins

Host/Environment (please complete the following information):

dblock commented 1 week ago

I'd dig a little deeper to find out the change in the SDK that caused this, what does that account ID need to be set to?

Did you try adding the account id here: https://github.com/opensearch-project/logstash-output-opensearch/blob/ccd571fc912cb0efcb2bca81d91ff3020be2de68/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb#L29

nastiazaran commented 1 week ago

Hi,

I don't know what is the purpose for the account id parameter in this case, I just want to use the instance IAM profile. I'm sorry but I've not so much knowledge about ruby/packages, I'm just a 'final user'

I've tried what you suggested (adding account id to manticore_adapter.rb but then I have the following error:

[2024-09-24T13:01:20,688][ERROR][logstash.javapipeline ] Pipeline error {:pipeline_id=>"main", :exception=>#, :backtrace=>["org/jruby/RubyModule.java:3766:in const_missing'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:205:insign_aws_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:167:in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:272:inperform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:226:in health_check_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:233:inblock in healthcheck!'", "org/jruby/RubyHash.java:1415:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:231:inhealthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:320:in update_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:92:inupdate_initial_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:86:in start'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client.rb:354:inbuild_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client.rb:58:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client_builder.rb:119:increate_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client_builder.rb:112:in build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/plugin_mixins/opensearch/common.rb:38:inbuild_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch.rb:227:in register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:inregister'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233:inblock in register_plugins'", "org/jruby/RubyArray.java:1821:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:inregister_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:598:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:245:instart_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:190:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:142:inblock in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/logs.conf"], :thread=>"#"}

Thanks for your help

dblock commented 1 week ago

Looks like it's more complicated. Just to set expectations I won't have cycles to debug this, hopefully someone reading this can help.