search

aws / secrets-store-csi-driver-provider-aws

The AWS provider for the Secrets Store CSI Driver allows you to fetch secrets from AWS Secrets Manager and AWS Systems Manager Parameter Store, and mount them into Kubernetes pods.
Apache License 2.0
474 stars 134 forks source link

readme

AWS Secrets Manager and Config Provider for Secret Store CSI Driver

badge codecov

AWS offers two services to manage secrets and parameters conveniently in your code. AWS Secrets Manager allows you to easily rotate, manage, and retrieve database credentials, API keys, certificates, and other secrets throughout their lifecycle. AWS Systems Manager Parameter Store provides hierarchical storage for configuration data. The AWS provider for the Secrets Store CSI Driver allows you to make secrets stored in Secrets Manager and parameters stored in Parameter Store appear as files mounted in Kubernetes pods.

Installation

Requirements

[^1]: The CSI Secret Store driver runs as a DaemonSet, and as described in the AWS documentation, DaemonSet is not supported on Fargate.

Installing the AWS Provider

To install the Secrets Manager and Config Provider use the YAML file in the deployment directory:

kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

Usage

Set the region name and name of your cluster to use in the bash commands that follow:

REGION=<REGION>
CLUSTERNAME=<CLUSTERNAME>

Where <REGION> is the region in which your Kubernetes cluster is running and <CLUSTERNAME> is the name of your cluster.

Now create a test secret:

aws --region "$REGION" secretsmanager  create-secret --name MySecret --secret-string '{"username":"memeuser", "password":"hunter2"}'

Create an access policy for the pod scoped down to just the secrets it should have and save the policy ARN in a shell variable:

POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn --output text iam create-policy --policy-name nginx-deployment-policy --policy-document '{
    "Version": "2012-10-17",
    "Statement": [ {
        "Effect": "Allow",
        "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
        "Resource": ["arn:*:secretsmanager:*:*:secret:MySecret-??????"]
    } ]
}')

Note, when using SSM parameters the permission "ssm:GetParameters" is needed in the policy. To simplify this example we use wild card matches above but you could lock this down further using the full ARN from the output of create-secret above.

Create the IAM OIDC provider for the cluster if you have not already done so:

eksctl utils associate-iam-oidc-provider --region="$REGION" --cluster="$CLUSTERNAME" --approve # Only run this once

Next create the service account to be used by the pod and associate the above IAM policy with that service account. For this example we use nginx-deployment-sa for the service account name:

eksctl create iamserviceaccount --name nginx-deployment-sa --region="$REGION" --cluster "$CLUSTERNAME" --attach-policy-arn "$POLICY_ARN" --approve --override-existing-serviceaccounts

For a private cluster, ensure that the VPC the cluster is in has an AWS STS endpoint. For more information, see Interface VPC endpoints in the AWS IAM User Guide.

Now create the SecretProviderClass which tells the AWS provider which secrets are to be mounted in the pod. The ExampleSecretProviderClass.yaml in the examples directory will mount "MySecret" created above:

kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/examples/ExampleSecretProviderClass.yaml

Finally we can deploy our pod. The ExampleDeployment.yaml in the examples directory contains a sample nginx deployment that mounts the secrets under /mnt/secrets-store in the pod:

kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/examples/ExampleDeployment.yaml

To verify the secret has been mounted properly, See the example below:

kubectl exec -it $(kubectl get pods | awk '/nginx-deployment/{print $1}' | head -1) cat /mnt/secrets-store/MySecret; echo

Troubleshooting

Most errors can be viewed by describing the pod deployment. For the deployment, find the pod names using get pods (use -n <NAMESPACE> if you are not using the default namespace):

kubectl get pods

Then describe the pod (substitute the pod ID from above for <PODID>, as before use -n if you are not using the default namespace):

kubectl describe pod/<PODID>

Additional information may be available in the provider logs:

kubectl -n kube-system get pods
kubectl -n kube-system logs pod/<PODID>

Where <PODID> in this case is the id of the csi-secrets-store-provider-aws pod.

SecretProviderClass options

The SecretProviderClass has the following format:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: <NAME>
spec:
  provider: aws
  parameters:

The parameters section contains the details of the mount request and contain one of the three fields:

The primary objects field of the SecretProviderClass can contain the following sub-fields:

Additional Considerations

Rotation

When using the optional alpha rotation reconciler feature of the Secrets Store CSI driver the driver will periodically remount the secrets in the SecretProviderClass. This will cause additional API calls which results in additional charges. Applications should use a reasonable poll interval that works with their rotation strategy. A one hour poll interval is recommended as a default to reduce excessive API costs.

Anyone wishing to test out the rotation reconciler feature can enable it using helm:

helm upgrade -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --set enableSecretRotation=true --set rotationPollInterval=3600s

Automated Failover Regions

In order to provide availability during connectivity outages or for disaster recovery configurations, this provider supports an automated failover feature to fetch secrets or parameters from a secondary region. To define an automated failover region, define the failoverRegion in the SecretProviderClass.yaml file:

spec:
  provider: aws
  parameters:
    region: us-east-1
    failoverRegion: us-east-2

When the failoverRegion is defined, the driver will attempt to get the secret value from both regions.

Private Builds

You can pull down this git repository and build and install this plugin into your account's AWS ECR registry using the following steps. First clone the repository:

git clone https://github.com/aws/secrets-store-csi-driver-provider-aws
cd secrets-store-csi-driver-provider-aws

Next, set your region and repository name in bash shell variables to be used later:

export REGION=<REGION>
export PRIVREPO=<ACCOUNT>.dkr.ecr.$REGION.amazonaws.com/secrets-store-csi-driver-provider-aws

Where <REGION> is the AWS region in which your Kubernetes cluster is running, and <ACCOUNT> is your AWS account Id. Next create your ECR repository if you have not already done so:

aws --region $REGION ecr create-repository --repository-name secrets-store-csi-driver-provider-aws # Only do this once

Now run make to build the plugin and push it into your account's repo:

make

Once the image is in your repo you can install it into your cluster from your repo rather than the public repo:

envsubst < deployment/private-installer.yaml | kubectl apply -f -

Configure the Underlying Secrets Manager Client to Use FIPS Endpoint

If you use Helm chart to install the provider, append the --set useFipsEndpoint=true flag on the install step. Your install command would be something like

helm repo add aws-secrets-manager https://aws.github.io/secrets-store-csi-driver-provider-aws
helm install -n kube-system secrets-provider-aws aws-secrets-manager/secrets-store-csi-driver-provider-aws --set useFipsEndpoint=true

Client-Side Rate-Limitting to Kubernetes API server

To mount each secret on each pod, the AWS CSI provider lookups the region of the pod and the role ARN associated with the service account by calling the Kubernetes APIs. You can increase the value of qps and burst if you notice the provider is throttled by client-side limit to the API server.

If you use Helm chart to install the provider, append the --set-json 'k8sThrottlingParams={"qps": "<custom qps>", "burst": "<custom qps>"}' flag in the install step.

Security Considerations

The AWS Secrets Manager and Config Provider provides compatibility for legacy applications that access secrets as mounted files in the pod. Security conscious applications should use the native AWS APIs to fetch secrets and optionally cache them in memory rather than storing them in the file system.

Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.