auth_type 'aws_iam' with IAM Roles For Service Account (IRSA)

[oridool]

The documentation states that we can use 'aws_iam' with ACCESS_KEY+SECRET_KEY.

output {        
   opensearch {     
          hosts => ["hostname:port"]              
          auth_type => {    
              type => 'aws_iam'     
              aws_access_key_id => 'ACCESS_KEY'     
              aws_secret_access_key => 'SECRET_KEY'     
              region => 'us-west-2'         
          }         
          index  => "logstash-logs-%{+YYYY.MM.dd}"      
   }            
}

But if I'm running on EKS, I'd like to take advantage of it and use IRSA: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html This means, that I don't need to specify the SECRET_KEY in my configuration and potentially expose it. Any chance this is already being supporterd, but not documented?

Describe the solution you'd like I'd like to remove the aws_access_key_id / aws_secret_access_key , and have a more secure system by using IRSA. I can set an IAM role to my logstash POD on EKS. Such a solution would:

1. Add an option to helm chart (preferred) to configure IRSA for logstash deployment.
2. Provide the ability to remove the access+secret key from plugin configuration, and use the AWS Default Credential Provider Chain (eventually resolved to WebIdentityTokenCredentialsProvider)
3. Add to documentation exact instructions of how this can be done.

Describe alternatives you've considered Unfortunately, there are no current alternatives.

Additional context Security wise, it is not advisable to store secrets in configuration files. This is a big no-no in building a secured system.

[vpulagarwal]

+1 A similar issue is discussed here (https://github.com/opensearch-project/logstash-output-opensearch/issues/96)

[vrajannagari]

+1

[joannewei]

+1 It seems that topkiachu's solution is not merged into master branch. What is the status of this request? Hopefully the IAM Role support will be available soon, also included in the release of opensearchproject/logstash-oss-with-opensearch-output-plugin. Thanks!

[aruandre]

using IAM roles is possible, for some reason this isn't documented at all. the following example worked for me (used empty string for aws creds)

output {
 opensearch {
     index => "myindex-%{+YYYY.MM.dd}"
     hosts => ["https://opensearch:443"]
     auth_type => {
         type => 'aws_iam'
         aws_access_key_id => ''
         aws_secret_access_key => ''
         region => 'eu-central-1'
     }
   }
}

Your role should look like this

{
    "Statement": [
        {
            "Action": [
                "es:ESHttpHead",
                "es:ESHttpPost",
                "es:ESHttpGet",
                "es:ESHttpPut"
            ],
            "Resource": [
                "arn:aws:es:us-east-1:aws_accountid:domain/opensearch"
            ],
            "Effect": "Allow"
        }
    ]
}

Using the http_compression => true does not work with IAM roles though and returns 403 error when enabled.

[alexsanderp]

@aruandre I tried using as empty credentials and got the error below:

NoMethodError: undefined method `credentials' for nil:NilClass

Is there any way to use IAM Role? The documentation says it supports it, but I can't use it!

[trunet]

I have it working from an instance, but not from the service account role.

I can see on my opensearch audit logs the worker role being used to try to authenticate leading to a 403.

Definitely this plugin is using the instance metadata to get the instance profile iam role instead of AWS_WEB_IDENTITY_TOKEN_FILE.

I guess @topikachu https://github.com/opensearch-project/logstash-output-opensearch/pull/138 would have fixed as newer versions of aws sdk fix this.

[trunet]

this is duplicate of https://github.com/opensearch-project/logstash-output-opensearch/issues/96 BTW...

[alencar]

IAM Instance/Task roles works just fine. Your output section will look like below, and there will be an IAM Instance/Task Role attached.

output {
  opensearch {
     ...
     auth_type => {
       type => 'aws_iam'
       region => 'us-east-1'
     }
    ...
 }

However, the role must be in the same account as the cluster. Any news on supporting a cross-account role?

Scenario

AWS Account 1 AWS Account 2 Logstash OpenSearch Domain

Account 1 IAM Instance Role with AssumeRole to Account 2

Account 2 IAM Instance Role with es:ESHttpHead, es:ESHttpGet, es:ESHttpPut, es:ESHttpPost permissions to the relevant domain

[alencar]

On the above comment, I have tried this:

Account 1: Allow Logstash role to assume a cross-account role in Account 2 Account 2: Create a role that allows account one to assume, give the role Elasticsearch/OpenSearch access.

Within Logstash instance, in logstash's ~/.aws/config. I have also tried with another variation where credential_source was directly in the desired profile.

[default]
region = <region>
credential_source = Ec2InstanceMetadata

[profile research]
source_profile = default
region = <region>
role_arn = arn:aws:iam::XXXXXXXXXXXX:role/ec2-sysadmin-Logstash-Test

AWS CLI and SDK works just fine if trying to use the cross-account role, thus no issues from there.

Logstash output section goes by

output {
  opensearch {
    hosts => ["https://<domain>.<region>.es.amazonaws.com:443"]
    ssl => true
    ssl_certificate_verification => true
    auth_type => {
      type => "aws_iam"
      region => "<region>"
      profile => "research"
    }
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM}"
    ecs_compatibility => "disabled"
    manage_template => false
  }
}

Results in

ERROR] 2022-10-27 00:16:13.646 [[beats]-pipeline-manager] javapipeline - Pipeline error {:pipeline_id=>"beats", :exception=>#<Aws::Errors::NoSourceProfileError: Profile research has a role_arn, and source_profile, but the source_profile does not have credentials.>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/shared_config.rb:187:in `assume_role_from_profile'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/shared_config.rb:111:in `assume_role_credentials_from_config'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/credential_provider_chain.rb:95:in `assume_role_with_profile'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/credential_provider_chain.rb:78:in `assume_role_credentials'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/credential_provider_chain.rb:12:in `block in resolve'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/credential_provider_chain.rb:11:in `resolve'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:81:in `aws_iam_auth_initialization'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:52:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/http_client.rb:318:in `build_adapter'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/http_client.rb:322:in `build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/http_client.rb:58:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/http_client_builder.rb:117:in `create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/http_client_builder.rb:110:in `build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/plugin_mixins/opensearch/common.rb:38:in `build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch.rb:220:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in `block in register_plugins'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:231:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:589:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:244:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:189:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/beats/beats.conf"], :thread=>"#<Thread:0x53b447d4 run>"}

[andrejvanderzee]

We also require web identity. The AWS SDK 3.x supports it ( (https://docs.aws.amazon.com/sdkref/latest/guide/feature-assume-role-credentials.html), but is not implemented in this plugin.

For example:

[default]
role_arn = arn:aws:iam::111111111111111:role/opensearch-bulk
credential_source = Environment

Gives error:

Aws::Errors::InvalidCredentialSourceError: Unsupported credential_source: Environment 

Or:

[default]
role_arn = arn:aws:iam::111111111111111:role/opensearch-bulk
source_profile = web-identity

[web-identity]
role_arn = arn:aws:iam::111111111111111:role/logstash-opensearch
web_identity_token_file = /var/run/secrets/eks.amazonaws.com/serviceaccount/token

Giver error:

Aws::Errors::NoSourceProfileError: Profile default has a role_arn, and source_profile, but the source_profile does not have credentials.

Or:

[opensearch]
credential_process = /usr/local/bin/aws sts assume-role --role-arn arn:aws:iam::388814315208:role/opensearch-bulk --no-cli-pager --duration-seconds 900 --role-session-name opensearch --query "Credentials.{Version:to_number('1'),AccessKeyId:AccessKeyId,SecretAccessKey:SecretAccessKey,SessionToken:SessionToken}"

This works but credentials are not refreshed properly. So also useless.

Very frustrating.

[Rwwn]

I think the reason IAM roles work for some and not for others is that the plugin currently relies on v1 of the Instance Metadata Service (IMDSv1) to get the credentials, which doesn't enforce the need for API tokens when hitting the metadata. I've tried enforcing metadata v2 in the launch template for my EC2, which requires an API token for accessing the metadata. This has broken my OpenSearch output, with this in the error message:

"exception":"undefined method `credentials' for nil:NilClass"

My assumption is that the people in this thread who it isn't working for have also enforced metadata v2.

As mentioned above, you could set up something which pulls the access and secret keys from the metadata using the token and use them into your Logstash config, but these will need regularly refreshing. I guess you could set up a cron job or something to run this script and refresh the credentials on a schedule and insert them into the Logstash config, and restart Logstash to pick them up, but that's starting to get a bit too hacky and it'd much better if this was natively supported by the plugin. The only other way around it I can see for now is to go back to v1, which is unfortunate since tfsec complains about it.

Reference for IMDS v1 / v2: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html

Update: It turns out the plugin does actually support this, the problem was that Logstash was running in a Docker container and I needed set the metadata hop limit on the launch template to greater than 1 for it to be able to reach the metadata when v2 is enabled: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/launch_template#http_put_response_hop_limit It's now working for me, I hope this helps someone else if they stumble across this issue.

[zentavr]

@Rwwn I run Logstash in the pod inside K8S cluster (EC2 based nodes). The cluster had been created using AWS CDK. How I can see, the instance has

IMDSv2
Optional     EC2 recommends setting IMDSv2 to required

At the launch template it has Metadata response hop limit: 2.

The pod uses the certain serviceAccount with attached EC2 role to it. And the auth not works.

[zentavr]

Is there any solution to run the plugin in K8S/EKS with IAM attached to the service role?