search

awslabs / logstash-output-amazon_es

Logstash output plugin to sign and export logstash events to Amazon Elasticsearch Service
Apache License 2.0
375 stars 103 forks source link

Temporary credentials rotation after the start #167

Open ado2p opened 3 years ago

ado2p commented 3 years ago

It looks like logstash is unable to rotate the credentials without the restart of the service. This implies that it is mandatory to use long term credentials in the configuration. It would be nice to have the possibility to configure temporary and rotating credentials coming from an assume role to satisfy the security best practices. https://github.com/awslabs/logstash-output-amazon_es/issues/161

ajnelapu commented 3 years ago

Goal: Using Logstash to build a cross account pipeline.

Observation/Error/Issue: The logstash-output-amazon_es plugin is unable to fetch credentials from the instance profile and is unable to write to the AES domain in another account.

Steps to replicate:

  1. Setup Logstash on an EC2 instance in Account A.
  2. Steup an AES domain in Account B.
  3. Create a role in Account B with policies to access the ES domain and whitelist the same role in the access policy.
  4. Create an instance profile role in Account A and add permissions to assume the role in Account B. Allow role in Account B to be assumed by instance profile role in Account A.
  5. Attach the instance profile to the Logstash instance and run it.

Requirement: The logstash-output-amazon_es plugin to fetch credentials from the instance profile. Although, this is said to be supported in README, I wasn't able to figure out a way and couldn't find a documentation for the same.

giambasten commented 2 years ago

It seems that "amazon_es" output lugin accepts only "aws_access_key_id" and "aws_secret_access_key" , but not session_token.

Thus, temporary credentials rotation is not feasible while logstash is running. The only possible solution seems to restart logstash periodically, before the expiration of the session token.

dlvenable commented 2 years ago

@giambasten ,

The session token is a temporary credential and can last no more than 36 hours. It may even last less. Because it is temporary and needs to be refreshed, it shouldn't be configured in the file.

This plugin supports using the profile defined in the ~/.aws/credentials file. I believe it only supports the default profile. Also, it can use EC2 instance profiles if that helps. Both of these approach are more dynamic and allow the amazon_es plugin to refresh the credentials.