IMDS v2 not supported - Githubissues

HarshadRanganathan commented 3 years ago

Logstash aws es output plugin is not working when IMDS v1 is disabled.

Logstash version: 7.6.1 Plugin version: 6.4.1

Error:


[2021-05-03T08:47:04,071][ERROR][logstash.javapipeline ][main] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<NoMethodError: undefined method `credentials' for nil:NilClass>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/signers/v4.rb:45:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb:108:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:291:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:245:in `block in healthcheck!'", "org/jruby/RubyHash.java:1428:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:241:in `healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:341:in `update_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:71:in `start'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client.rb:304:in `build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client.rb:54:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client_builder.rb:110:in `create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client_builder.rb:98:in `build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es.rb:256:in `build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/common.rb:25:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:106:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:48:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:200:in `block in register_plugins'", "org/jruby/RubyArray.java:1814:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:199:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:501:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:212:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:154:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:109:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x4dbdc20d run>"}

[2021-05-03T08:47:04,146][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}

AustinTag commented 3 years ago

This appears to be an issue with the plugin pulling the credentials from the instance. 2 questions:

What type of security are you using for your Amazon Elasticsearch cluster? VPC/Internet, username/password or IAM auth?
Can you post your logstash configuration file here? (With all identifying information or credentials removed of course)

HarshadRanganathan commented 3 years ago

This appears to be an issue with the plugin pulling the credentials from the instance. 2 questions:

What type of security are you using for your Amazon Elasticsearch cluster? VPC/Internet, username/password or IAM auth?

Can you post your logstash configuration file here? (With all identifying information or credentials removed of course)

We are using VPC security. Our Domain access policy is pretty basic at the moment -

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": [
        "arn:aws:es:us-east-1:<account-id>:domain/<es-domain>/*",
        "arn:aws:es:us-east-1:<account-id>:domain/<es-domain>"
      ]
    }
  ]
}

Logstash configuration (removed identifying information)

input {
  kafka {
    auto_offset_reset => "latest"
    bootstrap_servers => ""
    security_protocol => "SSL"
    consumer_threads => 2
    topics => [""]
    decorate_events => true
    max_poll_interval_ms => "2147483647"
    max_poll_records => "100"
    request_timeout_ms => "50000"
    session_timeout_ms => "40000"
    group_id => ""
    codec => "json"
  }
}

output {
  amazon_es {
    hosts => ["vpc-:443"]
    region => "us-east-1"
    ssl => true
    index => ""
    document_id =>""
  }
}

HarshadRanganathan commented 3 years ago

Fine-grained access control -> Disabled SAML authentication -> Disabled Amazon Cognito for authentication -> Disabled

AustinTag commented 3 years ago

Gotcha. So if I am understanding correctly, you can actually use the regular logstash output plugin for Elasticsearch:

https://github.com/logstash-plugins/logstash-output-elasticsearch

This amazon-es plugin is needed if you are using IAM auth for access to your cluster, because it will add the credentials needed for IAM auth with an Amazon Elasticsearch Cluster.

HarshadRanganathan commented 3 years ago

Gotcha. So if I am understanding correctly, you can actually use the regular logstash output plugin for Elasticsearch:

https://github.com/logstash-plugins/logstash-output-elasticsearch

This amazon-es plugin is needed if you are using IAM auth for access to your cluster, because it will add the credentials needed for IAM auth with an Amazon Elasticsearch Cluster.

Yes, we could use it. But, we might want to use fine-grained access control with IAM master user for RBAC/Security/Kibana auth in the future.

Is the suggestion to go with logstash-output-elasticsearch because we have ES in a VPC and that we don't necessarily need RBAC/Security/Kibana auth?

Also, all the requests need to be signed even if we use an open domain access policy (not v4 signature). I hope that's supported in the logstash-output-elasticsearch plugin?

HarshadRanganathan commented 3 years ago

We've switched to logstash-output-elasticsearch plugin. Will keep this ticket open as the plugin needs a fix for anyone planning to use IAM auth with IMDS v2.

HarshadRanganathan commented 3 years ago

Since logstash-output-elasticsearch has added license compatibility checks, we're switching to Opensearch logstash plugin - https://opensearch.org/docs/latest/clients/logstash/index/

awslabs / logstash-output-amazon_es

IMDS v2 not supported #169