WARNING :warning: This solution has been deprecated. Continue to use this for kubernetes version 1.15. For 1.16+, please use Azure Key Vault Provider for Secret Store CSI Driver
This solution is no longer supported as 1.15 is out of support in AKS
Seamlessly integrate your key management systems with Kubernetes.
Secrets, keys, and certificates in a key management system become a volume accessible to pods. Once the volume is mounted, its data is available directly in the container filesystem for your application.
Azure Key Vault
💡 NOTE: To enable encryption at rest of Kubernetes data in
etcd
, use the Kubernetes KMS plugin for Azure Key Vault.
AKS Engine creates customized Kubernetes clusters on Azure.
Follow the AKS Engine add-on documentation to create a new Kubernetes cluster with Key Vault FlexVolume already deployed.
Azure Kubernetes Service (AKS) creates managed, supported Kubernetes clusters on Azure.
Deploy Key Vault FlexVolume to your AKS cluster with this command:
kubectl create -f https://raw.githubusercontent.com/Azure/kubernetes-keyvault-flexvol/master/deployment/kv-flexvol-installer.yaml
To validate Key Vault FlexVolume is running as expected, run the following command:
kubectl get pods -n kv
The output should show keyvault-flexvolume
pods running on each agent node:
NAME READY STATUS RESTARTS AGE
keyvault-flexvolume-f7bx8 1/1 Running 0 3m
keyvault-flexvolume-rcxbl 1/1 Running 0 3m
keyvault-flexvolume-z6jm6 1/1 Running 0 3m
If using keyvault-flexvolume in a cluster with pod security policy enabled, create the following policy that enables the spec required for keyvault-flexvolume to work -
kubectl apply -f https://raw.githubusercontent.com/Azure/kubernetes-keyvault-flexvol/master/deployment/kv-flexvol-psp.yaml
Key Vault FlexVolume offers four modes for accessing a Key Vault instance: Service Principal, Pod Identity, [VMSS User Assigned Managed Identity], [VMSS System Assigned Managed Identity].
Add your service principal credentials as Kubernetes secrets accessible by the Key Vault FlexVolume driver.
kubectl create secret generic kvcreds --from-literal clientid=<CLIENTID> --from-literal clientsecret=<CLIENTSECRET> --type=azure/kv
Ensure this service principal has all the required permissions to access content in your Key Vault instance. If not, run the following Azure CLI commands:
# [Required for version < v0.0.13] Assign Reader Role to the service principal for your keyvault
az role assignment create --role Reader --assignee <principalid> --scope /subscriptions/<subscriptionid>/resourcegroups/<resourcegroup>/providers/Microsoft.KeyVault/vaults/<keyvaultname>
# Assign key vault permissions to your service principal
az keyvault set-policy -n $KV_NAME --key-permissions get --spn <YOUR SPN CLIENT ID>
az keyvault set-policy -n $KV_NAME --secret-permissions get --spn <YOUR SPN CLIENT ID>
az keyvault set-policy -n $KV_NAME --certificate-permissions get --spn <YOUR SPN CLIENT ID>
Fill in the missing pieces in this deployment for your own deployment. Make sure to:
Reference the service principal Kubernetes secret created in the previous step
secretRef:
name: kvcreds
Pass in properties for the Key Vault instance to the FlexVolume driver.
Name | Required | Description | Default Value |
---|---|---|---|
usepodidentity | no | specify access mode: use a service principal or pod identity or vm managed identity | "false" |
usevmmanagedidentity | not required, available for version >= v0.0.15 | specify access mode: use a service principal or pod identity or vm managed identity | "false" |
vmmanagedidentityclientid | not required, available for version >= v0.0.15 | If using a user assigned identity as the VM's managed identity, then specify the identity's client id. If empty, then defaults to use the system assigned identity on the VM | "" |
keyvaultname | yes | name of Key Vault instance | "" |
keyvaultobjectnames | yes | names of Key Vault objects to access | "" |
keyvaultobjectaliases | no | filenames to use when writing the objects | keyvaultobjectnames |
keyvaultobjecttypes | yes | types of Key Vault objects: secret, key or cert | "" |
keyvaultobjectversions | no | versions of Key Vault objects, if not provided, will use latest | "" |
resourcegroup | required for version < v0.0.14 | name of resource group containing Key Vault instance | "" |
subscriptionid | required for version < v0.0.14 | name of subscription containing Key Vault instance | "" |
tenantid | yes | name of tenant containing Key Vault instance | "" |
cloudname | no | Name of the cloud environment, e.g. something like AzureChinaCloud, AzureGermanCloud. If not provided, the default public Azure cloud will be used | "" |
nmiport | not required, available for version >= v0.0.17 | Port number of the NMI daemonset. If not provided, the default NMI port is used | "2579" |
Multiple values in the keyvaultobjectnames
, keyvaultobjecttypes
and keyvaultobjectversions
properties should be separated with semicolons (;
).
Specify mount path of flexvolume to mount key vault objects
volumeMounts:
- name: test
mountPath: /kvmnt
readOnly: true
Example of an nginx pod accessing a secret from a Key Vault instance:
apiVersion: v1
kind: Pod
metadata:
name: nginx-flex-kv
spec:
containers:
- name: nginx-flex-kv
image: nginx
volumeMounts:
- name: test
mountPath: /kvmnt
readOnly: true
volumes:
- name: test
flexVolume:
driver: "azure/kv"
secretRef:
name: kvcreds # [OPTIONAL] not required if using Pod Identity
options:
usepodidentity: "false" # [OPTIONAL] if not provided, will default to "false"
usevmmanagedidentity: "false" # [OPTIONAL new in version >= v0.0.15] if not provided, will default to "false"
vmmanagedidentityclientid: "clientid" # [OPTIONAL new in version >= v0.0.15] use the client id to specify which user assigned managed identity to use, leave empty to use system assigned managed identity
keyvaultname: "testkeyvault" # [REQUIRED] the name of the KeyVault
keyvaultobjectnames: "testsecret" # [REQUIRED] list of KeyVault object names (semi-colon separated)
keyvaultobjectaliases: "secret.json" # [OPTIONAL] list of KeyVault object aliases
keyvaultobjecttypes: secret # [REQUIRED] list of KeyVault object types: secret, key, cert (semi-colon separated)
keyvaultobjectversions: "testversion" # [OPTIONAL] list of KeyVault object versions (semi-colon separated), will get latest if empty
resourcegroup: "testresourcegroup" # [REQUIRED for version < v0.0.14] the resource group of the KeyVault
subscriptionid: "testsub" # [REQUIRED for version < v0.0.14] the subscription ID of the KeyVault
tenantid: "testtenant" # [REQUIRED] the tenant ID of the KeyVault
nmiport: "nmiportnumber" # [OPTIONAL new in version >= v0.0.17] port number of the NMI daemonset, will default to "2579"
Deploy your app
kubectl create -f deployment/nginx-flex-kv.yaml
Validate the pod has access to the secret from key vault:
kubectl exec -it nginx-flex-kv cat /kvmnt/testsecret
testvalue
💡 The basic steps to configure AAD Pod Identity are reproduced here, but please refer to that project's README for more detail.
Install AAD Pod Identity
Run this command to create the aad-pod-identity
deployment on an RBAC-enabled cluster:
kubectl apply -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml
Create an Azure Identity
Run this Azure CLI command, and take note of the clientId
and id
values it returns:
az identity create -g <resourcegroup> -n <name> -o json
Assign Cluster SPN Role
If the Service Principal used for the cluster was created separately (not automatically, as part of an AKS cluster's MC_
resource group), assign it the "Managed Identity Operator" role:
az role assignment create --role "Managed Identity Operator" --assignee <sp id> --scope <full id of the managed identity>
Assign Azure Identity Roles
Ensure that your Azure Identity has the role assignments required to see your Key Vault instance and to access its content. Run the following Azure CLI commands to assign these roles if needed:
# Assign Reader Role to new Identity for your Key Vault
az role assignment create --role Reader --assignee <principalid> --scope /subscriptions/<subscriptionid>/resourcegroups/<resourcegroup>/providers/Microsoft.KeyVault/vaults/<keyvaultname>
# set policy to access keys in your Key Vault
az keyvault set-policy -n $KV_NAME --key-permissions get --spn <YOUR AZURE USER IDENTITY CLIENT ID>
# set policy to access secrets in your Key Vault
az keyvault set-policy -n $KV_NAME --secret-permissions get --spn <YOUR AZURE USER IDENTITY CLIENT ID>
# set policy to access certs in your Key Vault
az keyvault set-policy -n $KV_NAME --certificate-permissions get --spn <YOUR AZURE USER IDENTITY CLIENT ID>
Install the Azure Identity
Save this Kubernetes manifest to a file named aadpodidentity.yaml
:
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentity
metadata:
name: <a-idname>
spec:
type: 0
resourceID: /subscriptions/<subid>/resourcegroups/<resourcegroup>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<name>
clientID: <clientId>
Replace the placeholders with your user identity values. Set type: 0
for user-assigned MSI.
Finally, save your changes to the file, then create the AzureIdentity
resource in your cluster:
kubectl apply -f aadpodidentity.yaml
Install the Azure Identity Binding
Save this Kubernetes manifest to a file named aadpodidentitybinding.yaml
:
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
name: demo1-azure-identity-binding
spec:
azureIdentity: <a-idname>
selector: <label value to match>
Replace the placeholders with your values. Ensure that the AzureIdentity
name matches the one in aadpodidentity.yaml
.
Finally, save your changes to the file, then create the AzureIdentityBinding
resource in your cluster:
kubectl apply -f aadpodidentitybinding.yaml
Update the Application Deployment
Your application manifest needs a couple of changes. Refer to the nginx-flex-kv-podid deployment as an example.
a. Include the aadpodidbinding
label to match the selector
value from the previous step:
metadata:
labels:
aadpodidbinding: "NAME OF the AzureIdentityBinding SELECTOR"
b. Set usepodidentity
to true
:
usepodidentity: "true"
Deploy your app
kubectl create -f deployment/nginx-flex-kv-podidentity.yaml
Validate the pod can access the secret from Key Vault:
kubectl exec -it nginx-flex-kv-podid cat /kvmnt/testsecret
testvalue
NOTE: When using the
Pod Identity
option mode, there may be some delay in obtaining the objects from Key Vault. During pod creation time, AAD Pod Identity needs to create theAzureAssignedIdentity
for the pod based on theAzureIdentity
andAzureIdentityBinding
and retrieve the token for Key Vault. It is possible for the pod volume mount to fail during this time. If it does, the kubelet will keep retrying until after the token retrieval is complete and the mount succeeds.
This option allows flexvol to use the user assigned managed identity on the k8s cluster VMSS directly.
Warning: As of today (2019/09), AKS does not preserve the user assigned identity on VMSS during upgrade. You will need to re-assign the managed identities to VMSS after an upgrade. The improved experience is planned.
az identity create -g <RESOURCE GROUP> -n <IDENTITY NAME>
Grant Azure Managed Identity KeyVault permissions
Ensure that your Azure Identity has the role assignments required to see your Key Vault instance and to access its content. Run the following Azure CLI commands to assign these roles if needed:
# set policy to access keys in your Key Vault
az keyvault set-policy -n $KV_NAME --key-permissions get --spn <YOUR AZURE MANAGED IDENTITY CLIENT ID>
# set policy to access secrets in your Key Vault
az keyvault set-policy -n $KV_NAME --secret-permissions get --spn <YOUR AZURE MANAGED IDENTITY CLIENT ID>
# set policy to access certs in your Key Vault
az keyvault set-policy -n $KV_NAME --certificate-permissions get --spn <YOUR AZURE MANAGED IDENTITY CLIENT ID>
Assign Azure Managed Identity to VMSS
az vmss identity assign -g <RESOURCE GROUP> -n <K8S-AGENT-POOL-VMSS> --identities <USER ASSIGNED IDENTITY RESOURCE ID>
usevmmanagedidentity
to true
and provide vmmanagedidentityclientid
.
usevmmanagedidentity: "true" # [OPTIONAL] if not provided, will default to "false"
vmmanagedidentityclientid: "clientid" # [OPTIONAL] use the client id to specify which user assigned managed identity to use, leave empty to use system assigned managed identity
This option allows flexvol to use the system assigned managed identity on the k8s cluster VMSS directly.
az vmss identity show -g <resource group> -n <vmss scalset name> -o yaml
The output should contain type: SystemAssigned
.
Grant Azure Managed Identity KeyVault permissions
Ensure that the Azure system assigned Identity has the role assignments required to see your Key Vault instance and to access its content. Run the following Azure CLI commands to assign these roles if needed:
# set policy to access keys in your Key Vault
az keyvault set-policy -n $KV_NAME --key-permissions get --object-id <YOUR AZURE MANAGED IDENTITY OBJECT ID>
# set policy to access secrets in your Key Vault
az keyvault set-policy -n $KV_NAME --secret-permissions get --object-id <YOUR AZURE MANAGED IDENTITY OBJECT ID>
# set policy to access certs in your Key Vault
az keyvault set-policy -n $KV_NAME --certificate-permissions get --object-id <YOUR AZURE MANAGED IDENTITY OBJECT ID>
Deploy your application. Specify usevmmanagedidentity
to true
.
usevmmanagedidentity: "true" # [OPTIONAL] if not provided, will default to "false"
To learn more about the design of Key Vault FlexVolume, see Concept.
Key Vault FlexVolume interacts with Key Vault objects by using the Key Vault API.
Azure Key Vault has thorough documentation available to help clarify the difference between keys, secrets, and certificates.
It is important to understand how a certificate is structured in Key Vault.
As mentioned in Certificates are complex objects and Composition of a Certificate, Azure Key Vault (AKV) represents an X.509 certificate as three related resources:
All three will share the same name and the same version and can be fetched independently.
cert
in keyvaultobjecttypes
will fetch the public key and certificate metadata.key
in keyvaultobjecttypes
will fetch the private key of the certificate if its policy allows for private key exporting.secret
in keyvaultobjecttypes
will fetch the base64-encoded certificate bundle.The Key Vault FlexVolume project welcomes contributions and suggestions. Please see CONTRIBUTING for details.
This project has adopted the Microsoft Open Source Code of Conduct. For more information, see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.