search

aws / amazon-ssm-agent

An agent to enable remote management of your EC2 instances, on-premises servers, or virtual machines (VMs).
https://aws.amazon.com/systems-manager/
Apache License 2.0
1.06k stars 326 forks source link

Support for SecureString parameters #429

Open zendern opened 2 years ago

zendern commented 2 years ago

Use-case

I have a distributor document that I need to pass api keys into to be able to install the software included in the package. These keys should be stored in an encrypted fashion so putting them into a SecureString in Parameter Store and passing them as part of the additionalArguments would help get around additional IAM permissions that are now needed for the package to run.

Example

  1. Go to AWS Console SSM Distributor and create a new package
  2. Create some SSM Parameter Store secrets (secureString and other)
  3. Run the distributor package with some additional parameters that point to the new SSM parameters

Example commands to kick off distributor: Working when no SecureString:

aws ssm send-command --document-name "AWS-ConfigureAWSPackage" --document-version "1" --targets '[{"Key":"InstanceIds","Values":["i-1111111111"]}]' --parameters '{"action":["Install"],"installationType":["Uninstall and reinstall"],"name":["distributor-test"],"version":[""],"additionalArguments":["{\"SSM_cdk\":\"{{ssm:/cdk-bootstrap/hnb659fds/version}}\"}"]}' --timeout-seconds 600 --max-concurrency "50" --max-errors "0" --region us-east-2

Fails with Secure String: 

aws ssm send-command --document-name "AWS-ConfigureAWSPackage" --document-version "1" --targets '[{"Key":"InstanceIds","Values":["i-111111111111"]}]' --parameters '{"action":["Install"],"installationType":["Uninstall and reinstall"],"name":["distributor-test"],"version":[""],"additionalArguments":["{\"SSM_cdk\":\"{{ssm:/cdk-bootstrap/hnb659fds/version}}\", \"SSM_asecret\":\"{{ssm:asecret}}\", \"SSM_custom_kms_key\":\"{{asecret-custom-kms}}\"}"]}' --timeout-seconds 600 --max-concurrency "50" --max-errors "0" --region us-east-2

EC2 Details

Access to parameters from EC2 box directly

Out of the box if using SSMManagedInstanceCore you have all the access right to access both String and SecureString (if using default encryption). A command like this will work with no issues from that EC2 when requesting both encrypted and unencrypted parameters. 

sh-4.2$ aws ssm get-parameters --region us-east-2 --with-decryption --names /cdk-bootstrap/hnb659fds/version asecret
{
    "Parameters": [
        {
            "Name": "/cdk-bootstrap/hnb659fds/version",
            "Type": "String",
            "Value": "10",
            "Version": 1,
            "LastModifiedDate": "2022-02-16T04:25:43.950000+00:00",
            "ARN": "arn:aws:ssm:us-east-2:021785982708:parameter/cdk-bootstrap/hnb659fds/version",
            "DataType": "text"
        },
        {
            "Name": "asecret",
            "Type": "SecureString",
            "Value": "YOU GOT IT",
            "Version": 1,
            "LastModifiedDate": "2022-03-03T12:56:35.775000+00:00",
            "ARN": "arn:aws:ssm:us-east-2:021785982708:parameter/asecret",
            "DataType": "text"
        }
    ],
    "InvalidParameters": []
}

And as long as the EC2 instance profile has been granted access to the customer provided KMS key i can now pull back all 3 secrets with no issues. 

sh-4.2$ aws ssm get-parameters --region us-east-2 --with-decryption --names /cdk-bootstrap/hnb659fds/version asecret asecret-custom-kms
{
    "Parameters": [
        {
            "Name": "/cdk-bootstrap/hnb659fds/version",
            "Type": "String",
            "Value": "10",
            "Version": 1,
            "LastModifiedDate": "2022-02-16T04:25:43.950000+00:00",
            "ARN": "arn:aws:ssm:us-east-2:021785982708:parameter/cdk-bootstrap/hnb659fds/version",
            "DataType": "text"
        },
        {
            "Name": "asecret",
            "Type": "SecureString",
            "Value": "YOU GOT IT",
            "Version": 1,
            "LastModifiedDate": "2022-03-03T12:56:35.775000+00:00",
            "ARN": "arn:aws:ssm:us-east-2:021785982708:parameter/asecret",
            "DataType": "text"
        },
        {
            "Name": "asecret-custom-kms",
            "Type": "SecureString",
            "Value": "CUSTOM KMS SECRET",
            "Version": 1,
            "LastModifiedDate": "2022-03-03T14:01:22.027000+00:00",
            "ARN": "arn:aws:ssm:us-east-2:021785982708:parameter/asecret-custom-kms",
            "DataType": "text"
        }
    ],
    "InvalidParameters": []
}

SSM Agent log details

If you run the distributor document with any SecureString additional parameters it fails and provides no output in the console as to why it failed.

Looking into var/log/amazon/ssm/errors.log I can see the following error : 

2022-03-03 13:10:50 ERROR [processMessage @ mdsinteractor.go.324] [ssm-agent-worker] [MessageService] [MDSInteractor] [messageID=aws.ssm.78e76e24-c49e-4ae4-b7d6-068f95799364.i-025eecc61388da492] Parameters [asecret] of type SecureString are not supported

Ask

Add support for using SecureString in parameter store. I'm not currently seeing any blockers but would love some feedback on whether or not this can be supported.

Digging a little bit I do see it was maybe supported before but removed? https://github.com/aws/amazon-ssm-agent/commit/2e921da2ff3815fece0454e57459c0f25fa7c023

I cant see the CR that is linked in that commit so any context would be helpful.

fitzoh commented 2 years ago

It looks like the code changes for this could potentially be very simple?

Screen Shot 2022-03-04 at 9 05 41 PM

I'm definitely curious what the logic is for not allowing secureString parameters at the moment. Is there a risk of the values being logged/exposed downstream? If there is a blocker/rationale getting that documented might be a reasonable alternative outcome

bgeesaman commented 2 years ago

Came here after reading https://github.com/awsdocs/aws-systems-manager-user-guide/issues/104 and I'll share that the recommendation to shell out to the AWS CLI (or similar) from the target system absolutely works if the target system has the AWS CLI already installed. But in my use case (updating a config file with a marginally sensitive key in it) across a wide range of OSes and configurations, it's tricky to manage all the possible combinations of dependencies to check for and install the AWS CLI first. If the SSM agent handled this natively, it would make the more secure approach the easier/more native approach, and I'm all for that.

takesson commented 2 years ago

This would be extremely useful for various provisioning tasks. Especially if the Secrets Manager reference capability in SSM Parameter Store would also be supported.

We would like to configure the secret used by an OpenID Connect client. The oidc secret is managed/rotated by Secrets Manager.