search

opensearch-project / logstash-output-opensearch

A Logstash plugin that sends event data to a OpenSearch clusters and stores as an index.
https://opensearch.org/docs/latest/clients/logstash/index/
Apache License 2.0
106 stars 80 forks source link

Support for IAM Roles (Instance Profile/IRSA) for Authentication #96

Open davivcgarcia opened 2 years ago

davivcgarcia commented 2 years ago

Is your feature request related to a problem? Please describe. The current best practices for security in AWS recommends using short-lived credentials through IAM Role (STS) instead of static IAM Credentials (Access Key/Secret). That capacity is critically important when using Amazon EKS, since we can leverage IAM Roles for Service Accounts (IRSA) and improve security posture with Logstash on Kubernetes.

Describe the solution you'd like Logstash should be able to consume an IAM Role, from an Instance Profile (EC2) or from IRSA (EKS), and use it to authenticate against the OpenSearch environment.

Describe alternatives you've considered Use the legacy output plugin from awslabs/logstash-output-amazon_es.

Additional context The current aws-sdk gem used in this plugin (>= 2.11.632) already supports IRSA capability.

jigar-bhalodia1 commented 2 years ago

The legacy plugin doesn't support this either. https://github.com/awslabs/logstash-output-amazon_es/issues/133

asyschikov commented 2 years ago

Looks like according to docs the instance profiles are supported: https://opensearch.org/docs/latest/clients/logstash/ship-to-opensearch/#opensearch-output-plugin Look for optional parameters section:

Instance profile credentials delivered through the Amazon EC2 metadata service

vpulagarwal commented 2 years ago

Apparently, the latest plugin does not have a feature to read the Web Identity Token file (used for IRSA) mounted on the pod.

If we pass the token stored in /var/run/secrets/eks.amazonaws.com/serviceaccount/token file on the pod to variable session_token defined here, then it does not authorize the request but basic auth to the domain works just fine.

Logstassh output plugin config for IAM: 

output {
  opensearch {
     hosts => "${LOGSTASH_OS_HOST}:${LOGSTASH_OS_PORT}"
     index => "${LOGSTASH_OS_INDEX}"
     auth_type => {    
          type => 'aws_iam'     
          session_token => "${LOGSTASH_OS_TOKEN}"     
          region => 'us-east-1'         
      }
      ssl => true
     document_id => "%{id}"
  }
}
VijayanB commented 2 years ago

@vpulagarwal Thanks for reporting. Will check why session token is not accepted by config

oridool commented 2 years ago

Please check https://github.com/awsdocs/amazon-eks-user-guide/issues/281 It means the aws sdk first needs to upgraded to 3.58+

vpulagarwal commented 2 years ago

@VijayanB Any plans to upgrade this plugin on Ruby 3.58+ and start supporting IRSA?

VijayanB commented 2 years ago

@vpulagarwal is it trivial? If so will you be able to contribute?

amine250 commented 2 years ago

I'm sorry I don't understand why upgrading the aws-sdk to 3.58.0 would take this much time. Maybe some non-regression testing is all what's needed to be done. Can we at least get an ETA ?

Thank you.

vpulagarwal commented 2 years ago

I am no ruby expert, but I feel to support IRSA, there should be an option to provide the path of the service account token file located at /var/run/secrets/eks.amazonaws.com/serviceaccount/token in this function

dblock commented 2 years ago

I see @topikachu closed https://github.com/opensearch-project/logstash-output-opensearch/pull/138, it wasn't merged.

Looking for someone to contribute here. Happy to help review and merge PRs. I would want/start with integration test CI, https://github.com/opensearch-project/logstash-output-opensearch/issues/76 and then it will make it easier to add features.

topikachu commented 2 years ago

I see @topikachu closed #138, it wasn't merged.

Looking for someone to contribute here. Happy to help review and merge PRs. I would want/start with integration test CI, #76 and then it will make it easier to add featurebreak othe aws plugins.

I see @topikachu closed #138, it wasn't merged.

Looking for someone to contribute here. Happy to help review and merge PRs. I would want/start with integration test CI, #76 and then it will make it easier to add features.

I realize it breaks all other aws plugins because they are all using the old aws sdk. :(

trunet commented 2 years ago

I can see on my opensearch audit logs the EKS worker role being used to try to authenticate leading to a 403.

Definitely this plugin is using the instance metadata to get the instance profile iam role instead of AWS_WEB_IDENTITY_TOKEN_FILE.

dblock commented 2 years ago

@topikachu What should we do about this? What else needs to be upgraded at the same time? Care to help?

trunet commented 2 years ago

@dblock My guess is that aws-sdk ruby gem needs to be update to v3. There's an upgrade guide for this, we need to be sure that all the requires are updated across this plugin.

mikef-nl commented 2 years ago

Is there any further progress on this? I currently have it setup using an IAM user but would prefer to use a role if possible. Has anyone tested this using a container execution role in AWS Fargate?

keymon commented 2 years ago

Any update on logstash-output-opensearch supporting the Web Identity Token file (used for IRSA), ie by reading the file in AWS_WEB_IDENTITY_TOKEN_FILE?

the PR #171 has been merged and the aws-sdk v3 is meant to support this. I think it was added here: https://github.com/aws/aws-sdk-ruby/pull/2075

From what I read, it seems that the library might just work and pull the credentials out of the box. It is one of the default authentication providers but we need to test it.

andrejvanderzee commented 1 year ago

We require this too.

Thamizhvanan-R commented 1 year ago

Any update on logstash-output-opensearch supporting the Web Identity Token file (used for IRSA), ie by reading the file in AWS_WEB_IDENTITY_TOKEN_FILE?

the PR #171 has been merged and the aws-sdk v3 is meant to support this. I think it was added here: aws/aws-sdk-ruby#2075

From what I read, it seems that the library might just work and pull the credentials out of the box. It is one of the default authentication providers but we need to test it.

I am able to get it working by below configuration, the logstash has a default value for the region input, if you pass the correct region then it works

 auth_type => {
      type => 'aws_iam'
      region => <region>
    }
zentavr commented 1 year ago

should work in EKS just with:

output {
            opensearch {
                hosts => [ "https://search-kibana-stage-xxx.us-east-1.es.amazonaws.com:443" ]
                http_compression => true
                index => "clb-logs-test-%{+YYYY.MM.dd}"
                ecs_compatibility => disabled
                auth_type => {
                    type => "aws_iam"
                    region => "us-east-1"
                    service_name => "es"
                }
                legacy_template => true
                template => "/opt/bitnami/logstash/templates/alb-access-logs.json"
                template_name => "alb-access-logs"
            }
            stdout { codec => rubydebug }
        }
joewragg commented 7 months ago

+1 on this, tried a lot of the "workarounds" on both these issues and none work. I will have to revert to basic auth for now unfortunately

dblock commented 7 months ago

@Thamizhvanan-R do you think you can maybe describe in detail how you made it work? @joewragg can you please check that you're running the latest version and describe where and how it doesn't work?

Thamizhvanan-R commented 7 months ago

@Thamizhvanan-R do you think you can maybe describe in detail how you made it work? @joewragg can you please check that you're running the latest version and describe where and how it doesn't work?

@dblock

The Opensearch project uses aws-sdk-ruby for aws authentication. https://github.com/opensearch-project/logstash-output-opensearch/blob/d60a58f382ac3c772cd70ad52c79aecb0d085d25/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb#L85 Here we can see the call to the credentialproviderchain the credential_config is created with options we configure in the auth_type block in opensearch output section. If the AWS_REGION is not passed in the auth_type the default value us-east-1 is passed to the credentialproviderchain.

https://github.com/aws/aws-sdk-ruby/blob/b95f3f045032ac9c79917a65beaf4111f3228f94/gems/aws-sdk-core/lib/aws-sdk-core/credential_provider_chain.rb#L143 Here the underlying credentialproviderchain uses this options object to authenticate to aws, we can see the ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE are read from the Env variables but the AWS_REGION is not read from env instead it uses the values passed by the logstash which is "us-east-1" by default.

I made it work by explicitly setting the aws_region in the output section but the proper fix is to read the AWS_REGION from env and fall back to default if no such enviroment variable is set. These Env variables are set already by the EKS cluster when we enable service account with aws iam role assumption.

@joewragg I would suggest you check if these env variables are set in your case.

dblock commented 7 months ago

the proper fix is to read the AWS_REGION from env and fall back to default if no such environment variable is set.

In the code you pointed out does setting the region to nil work? If so then the fact that we pass through the us-east-1 default is incorrect, the default should be nil everywhere.

Whichever the fix is here, do you think you could write a test for it that shows that the regions configured by the AWS_REGION environment variable produce a different value here? This makes an easy fix.

joewragg commented 7 months ago

@Thamizhvanan-R do you think you can maybe describe in detail how you made it work? @joewragg can you please check that you're running the latest version and describe where and how it doesn't work?

@dblock

The Opensearch project uses aws-sdk-ruby for aws authentication.

https://github.com/opensearch-project/logstash-output-opensearch/blob/d60a58f382ac3c772cd70ad52c79aecb0d085d25/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb#L85

Here we can see the call to the credentialproviderchain the credential_config is created with options we configure in the auth_type block in opensearch output section. If the AWS_REGION is not passed in the auth_type the default value us-east-1 is passed to the credentialproviderchain. https://github.com/aws/aws-sdk-ruby/blob/b95f3f045032ac9c79917a65beaf4111f3228f94/gems/aws-sdk-core/lib/aws-sdk-core/credential_provider_chain.rb#L143 Here the underlying credentialproviderchain uses this options object to authenticate to aws, we can see the ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE are read from the Env variables but the AWS_REGION is not read from env instead it uses the values passed by the logstash which is "us-east-1" by default.

I made it work by explicitly setting the aws_region in the output section but the proper fix is to read the AWS_REGION from env and fall back to default if no such enviroment variable is set. These Env variables are set already by the EKS cluster when we enable service account with aws iam role assumption.

@joewragg I would suggest you check if these env variables are set in your case.

I have set the correct ENV vars as mentioned (redacted):

AWS_REGION=eu-west-1
AWS_ROLE_ARN=arn:aws:iam::ACCOUNTID:role/MYROLE
AWS_ACCOUNT_ID=ACCOUNTID
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_STS_REGIONAL_ENDPOINTS=regional

My config (redacted):

output {
  if [type] == "cloudfront" {
    opensearch {
      hosts => ["https://MYURL:443"]
      index => "cloudfront-logs-${ENV}-%{+YYYY-MM-dd}"
      auth_type => {
        type => 'aws_iam'     
        region => "${AWS_REGION}"
      }         
      ecs_compatibility => disabled
      }
  }
}

My IAM role is mapped to all_access in opensearch for testing

I am seeing 403s coming from opensearch in the logs

Thamizhvanan-R commented 7 months ago

@Thamizhvanan-R do you think you can maybe describe in detail how you made it work? @joewragg can you please check that you're running the latest version and describe where and how it doesn't work?

@dblock

The Opensearch project uses aws-sdk-ruby for aws authentication.

https://github.com/opensearch-project/logstash-output-opensearch/blob/d60a58f382ac3c772cd70ad52c79aecb0d085d25/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb#L85

Here we can see the call to the credentialproviderchain the credential_config is created with options we configure in the auth_type block in opensearch output section. If the AWS_REGION is not passed in the auth_type the default value us-east-1 is passed to the credentialproviderchain. https://github.com/aws/aws-sdk-ruby/blob/b95f3f045032ac9c79917a65beaf4111f3228f94/gems/aws-sdk-core/lib/aws-sdk-core/credential_provider_chain.rb#L143 Here the underlying credentialproviderchain uses this options object to authenticate to aws, we can see the ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE are read from the Env variables but the AWS_REGION is not read from env instead it uses the values passed by the logstash which is "us-east-1" by default.

I made it work by explicitly setting the aws_region in the output section but the proper fix is to read the AWS_REGION from env and fall back to default if no such enviroment variable is set. These Env variables are set already by the EKS cluster when we enable service account with aws iam role assumption.

@joewragg I would suggest you check if these env variables are set in your case.

I have set the correct ENV vars as mentioned (redacted):

AWS_REGION=eu-west-1
AWS_ROLE_ARN=arn:aws:iam::ACCOUNTID:role/MYROLE
AWS_ACCOUNT_ID=ACCOUNTID
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_STS_REGIONAL_ENDPOINTS=regional

My config (redacted):

output {
  if [type] == "cloudfront" {
    opensearch {
      hosts => ["https://MYURL:443"]
      index => "cloudfront-logs-${ENV}-%{+YYYY-MM-dd}"
      auth_type => {
        type => 'aws_iam'     
        region => "${AWS_REGION}"
      }         
      ecs_compatibility => disabled
      }
  }
}

My IAM role is mapped to all_access in opensearch for testing

I am seeing 403s coming from opensearch in the logs

One more thing we can check is if your role has assumerolepolicy and it allows your eks service account

joewragg commented 7 months ago

@Thamizhvanan-R do you think you can maybe describe in detail how you made it work? @joewragg can you please check that you're running the latest version and describe where and how it doesn't work?

@dblock The Opensearch project uses aws-sdk-ruby for aws authentication. https://github.com/opensearch-project/logstash-output-opensearch/blob/d60a58f382ac3c772cd70ad52c79aecb0d085d25/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb#L85

Here we can see the call to the credentialproviderchain the credential_config is created with options we configure in the auth_type block in opensearch output section. If the AWS_REGION is not passed in the auth_type the default value us-east-1 is passed to the credentialproviderchain. https://github.com/aws/aws-sdk-ruby/blob/b95f3f045032ac9c79917a65beaf4111f3228f94/gems/aws-sdk-core/lib/aws-sdk-core/credential_provider_chain.rb#L143 Here the underlying credentialproviderchain uses this options object to authenticate to aws, we can see the ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE are read from the Env variables but the AWS_REGION is not read from env instead it uses the values passed by the logstash which is "us-east-1" by default. I made it work by explicitly setting the aws_region in the output section but the proper fix is to read the AWS_REGION from env and fall back to default if no such enviroment variable is set. These Env variables are set already by the EKS cluster when we enable service account with aws iam role assumption. @joewragg I would suggest you check if these env variables are set in your case.

I have set the correct ENV vars as mentioned (redacted):

AWS_REGION=eu-west-1
AWS_ROLE_ARN=arn:aws:iam::ACCOUNTID:role/MYROLE
AWS_ACCOUNT_ID=ACCOUNTID
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_STS_REGIONAL_ENDPOINTS=regional

My config (redacted):

output {
  if [type] == "cloudfront" {
    opensearch {
      hosts => ["https://MYURL:443"]
      index => "cloudfront-logs-${ENV}-%{+YYYY-MM-dd}"
      auth_type => {
        type => 'aws_iam'     
        region => "${AWS_REGION}"
      }         
      ecs_compatibility => disabled
      }
  }
}

My IAM role is mapped to all_access in opensearch for testing I am seeing 403s coming from opensearch in the logs

One more thing we can check is if your role has assumerolepolicy and it allows your eks service account

Yeah I definitely have the assumerolepolicy working as it's already using this role to S3 GET in the pod

dblock commented 7 months ago

@joewragg Is it 100% of requests? Can you paste a (redacted) log line here?

joewragg commented 7 months ago

@joewragg Is it 100% of requests? Can you paste a (redacted) log line here?

Pretty sure it's 100% of requests.

If I use username and password it works.

I see this log line over and over that's it

[2024-04-09T07:06:02,353][WARN ][logstash.outputs.opensearch][main] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"https://MYURL:443/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '403' contacting OpenSearch at URL 'https://MYURL:443/'"}
dblock commented 7 months ago

@joewragg Turn on debug level logging, let's see the complete error?

I think https://forum.opensearch.org/t/enable-debug-logging-on-logstash/12194 or passing --debug to logstash will give you that.

joewragg commented 7 months ago

@joewragg Turn on debug level logging, let's see the complete error?

I think https://forum.opensearch.org/t/enable-debug-logging-on-logstash/12194 or passing --debug to logstash will give you that.

Does this have something to do with these settings in AWS: image

Relevant logs I see repeating:

[2024-04-10T08:04:53,137][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:53,138][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"G1 Young Generation"}
[2024-04-10T08:04:53,138][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:53,138][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"G1 Old Generation"}
[2024-04-10T08:04:53,139][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:53,139][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:53,140][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:53,143][DEBUG][logstash.agent           ] API HTTP Request {:status=>200, :request_method=>"GET", :path_info=>"/", :query_string=>"", :http_version=>"HTTP/1.1", :http_accept=>"*/*"}
[2024-04-10T08:04:54,138][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:54,139][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:54,140][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:54,141][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:54,141][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:54,145][DEBUG][logstash.agent           ] API HTTP Request {:status=>200, :request_method=>"GET", :path_info=>"/", :query_string=>"", :http_version=>"HTTP/1.1", :http_accept=>"*/*"}
[2024-04-10T08:04:55,137][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:55,138][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:55,139][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:55,139][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:55,140][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:55,143][DEBUG][logstash.agent           ] API HTTP Request {:status=>200, :request_method=>"GET", :path_info=>"/", :query_string=>"", :http_version=>"HTTP/1.1", :http_accept=>"*/*"}
[2024-04-10T08:04:56,139][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:56,139][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:56,140][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:56,141][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:56,141][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:56,145][DEBUG][logstash.agent           ] API HTTP Request {:status=>200, :request_method=>"GET", :path_info=>"/", :query_string=>"", :http_version=>"HTTP/1.1", :http_accept=>"*/*"}
[2024-04-10T08:04:56,923][DEBUG][logstash.outputs.opensearch][main] Waiting for connectivity to OpenSearch cluster, retrying in 32s
[2024-04-10T08:04:57,138][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:57,139][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:57,140][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:57,141][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:57,141][DEBUG][logstash.api.service     ] [api-service] start
[2024-04-10T08:04:57,145][DEBUG][logstash.agent           ] API HTTP Request {:status=>200, :request_method=>"GET", :path_info=>"/", :query_string=>"", :http_version=>"HTTP/1.1", :http_accept=>"*/*"}
[2024-04-10T08:04:57,187][DEBUG][logstash.outputs.opensearch][main] Running health check to see if an OpenSearch connection is working {:url=>"https://MYURL:443/", :path=>"/"}
[2024-04-10T08:04:57,191][DEBUG][org.apache.http.client.protocol.RequestAuthCache][main] Auth cache not set in the context
[2024-04-10T08:04:57,191][DEBUG][org.apache.http.impl.conn.PoolingHttpClientConnectionManager][main] Connection request: [route: {s}->https://MYURL:443][total available: 1; route allocated: 1 of 100; total allocated: 1 of 1000]
[2024-04-10T08:04:57,191][DEBUG][org.apache.http.impl.conn.PoolingHttpClientConnectionManager][main] Connection leased: [id: 0][route: {s}->https://MYURL:443][total available: 0; route allocated: 1 of 100; total allocated: 1 of 1000]
[2024-04-10T08:04:57,191][DEBUG][org.apache.http.impl.conn.DefaultManagedHttpClientConnection][main] http-outgoing-0: set socket timeout to 60000
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.impl.conn.DefaultManagedHttpClientConnection][main] http-outgoing-0: set socket timeout to 60000
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.impl.execchain.MainClientExec][main] Executing request HEAD / HTTP/1.1
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.impl.execchain.MainClientExec][main] Proxy auth state: UNCHALLENGED
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 >> HEAD / HTTP/1.1
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 >> Connection: Keep-Alive
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 >> content-type: application/json
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 >> host: MYURL
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 >> x-amz-date: 20240410T080457Z
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 >> x-amz-security-token: REDACTED
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 >> x-amz-content-sha256: REDACTED
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 >> authorization: AWS4-HMAC-SHA256 Credential=REDACTED/20240410/eu-west-1/es/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=REDACTED
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 >> User-Agent: Manticore 0.9.1
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 >> Accept-Encoding: gzip,deflate
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "HEAD / HTTP/1.1[\r][\n]"
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "content-type: application/json[\r][\n]"
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "host: MYURL[\r][\n]"
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "x-amz-date: 20240410T080457Z[\r][\n]"
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "x-amz-security-token: REDACTED
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "x-amz-content-sha256: REDACTED
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "authorization: AWS4-HMAC-SHA256 Credential=REDACTED/20240410/eu-west-1/es/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=REDACTED
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "User-Agent: Manticore 0.9.1[\r][\n]"
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
[2024-04-10T08:04:57,192][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 >> "[\r][\n]"
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 << "HTTP/1.1 403 Forbidden[\r][\n]"
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 << "Date: Wed, 10 Apr 2024 08:04:57 GMT[\r][\n]"
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 << "Content-Type: application/json[\r][\n]"
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 << "Content-Length: 251[\r][\n]"
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 << "Connection: keep-alive[\r][\n]"
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 << "x-amzn-requestid: REDACTED
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 << "access-control-allow-origin: *[\r][\n]"
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.wire     ][main] http-outgoing-0 << "[\r][\n]"
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 << HTTP/1.1 403 Forbidden
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 << Date: Wed, 10 Apr 2024 08:04:57 GMT
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 << Content-Type: application/json
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 << Content-Length: 251
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 << Connection: keep-alive
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 << x-amzn-requestid: REDACTED
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.headers  ][main] http-outgoing-0 << access-control-allow-origin: *
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.impl.execchain.MainClientExec][main] Connection can be kept alive indefinitely
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.impl.conn.PoolingHttpClientConnectionManager][main] Connection [id: 0][route: {s}->https://MYURL:443] can be kept alive indefinitely
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.impl.conn.DefaultManagedHttpClientConnection][main] http-outgoing-0: set socket timeout to 0
[2024-04-10T08:04:57,210][DEBUG][org.apache.http.impl.conn.PoolingHttpClientConnectionManager][main] Connection released: [id: 0][route: {s}->https://MYURL:443][total available: 1; route allocated: 1 of 100; total allocated: 1 of 1000]
[2024-04-10T08:04:57,212][WARN ][logstash.outputs.opensearch][main] Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"https://MYURL:443/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '403' contacting OpenSearch at URL 'https://MYURL:443/'"}
dblock commented 7 months ago

@joewragg Can you curl or awscurl your server with these credentials (https://code.dblock.org/2022/07/11/making-sigv4-authenticated-requests-to-managed-opensearch.html) and if so with what exact parameters/settings? I'm sure you're reading https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac.html already, but it's worth another pass in case something is incorrect in your setup.

joewragg commented 6 months ago

@joewragg Can you curl or awscurl your server with these credentials (https://code.dblock.org/2022/07/11/making-sigv4-authenticated-requests-to-managed-opensearch.html) and if so with what exact parameters/settings? I'm sure you're reading https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac.html already, but it's worth another pass in case something is incorrect in your setup.

Yeah awscurl works

dblock commented 6 months ago

@joewragg Can you curl or awscurl your server with these credentials (https://code.dblock.org/2022/07/11/making-sigv4-authenticated-requests-to-managed-opensearch.html) and if so with what exact parameters/settings? I'm sure you're reading https://docs.aws.amazon.com/opensearch-service/latest/developerguide/fgac.html already, but it's worth another pass in case something is incorrect in your setup.

Yeah awscurl works

Good. Knowing that the credentials are valid clears that. The next step is to debug the code in this library. It's doing its own signing here, I'd ensure that the parameters passed in match what's in the environment (maybe add some log lines?).

I'm happy to try to help if this can be reproduced easily.

zentavr commented 6 months ago

@dblock it's like it tries to use instance profile instead of STS...

dblock commented 6 months ago

I think it was mentioned above:

Definitely this plugin is using the instance metadata to get the instance profile iam role instead of AWS_WEB_IDENTITY_TOKEN_FILE.

I am hoping someone can implement this, I doubt I'll get to it any time soon.