:construction: :warning: This project was an exploration for Azure integration with the Kubernetes HPA and has been in Alpha status. It was a success and helped inspire other projects in this solution space. It is now in maintaince mode and will not be getting any new updates given that KEDA (CNCF Sandbox Project) has all the features in this solution. Please checkout KEDA's Scalers for Service Bus Subscriptions and Queues, Azure Monitor, and more. Thanks for all your support and contributions :tada:
An implementation of the Kubernetes Custom Metrics API and External Metrics API for Azure Services.
This adapter enables you to scale your application deployment pods running on AKS using the Horizontal Pod Autoscaler (HPA) with External Metrics from Azure Resources (such as Service Bus Queues) and Custom Metrics stored in Application Insights.
Try it out:
This was build using the Custom Metric Adapter Server Boilerplate project. Learn more about using an HPA to autoscale with external and custom metrics.
Try out scaling with External Metrics using the a Azure Service Bus Queue in this walkthrough.
Try out scaling with Custom Metrics using Requests per Second and Application Insights in this walkthrough
This describes the basic steps for deploying the metric adapter. For full deployment details see how to set up on your AKS Cluster and checkout the samples. Make sure the Metric Server is already deployed to your cluster.
Create a Service Principle and Secret:
az ad sp create-for-rbac -n "azure-k8s-metric-adapter-sp" --role "Monitoring Reader" --scopes /subscriptions/{SubID}/resourceGroups/{ResourceGroup1}
#use values from service principle created above to create secret
kubectl create secret generic azure-k8s-metrics-adapter -n custom-metrics \
--from-literal=azure-tenant-id=<tenantid> \
--from-literal=azure-client-id=<clientid> \
--from-literal=azure-client-secret=<secret>
Deploy the adapter:
kubectl apply -f https://raw.githubusercontent.com/Azure/azure-k8s-metrics-adapter/master/deploy/adapter.yaml
Deploy a metric configuration (requires you to configure the file below with your settings to a Service Bus Queue):
kubectl apply -f https://raw.githubusercontent.com/Azure/azure-k8s-metrics-adapter/master/samples/resources/externalmetric-example.yaml
There is also a Helm chart available for deployment for those using Helm in their cluster.
Deploy a Horizontal Pod Auto Scaler (HPA) to scale of your external metric of choice:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: consumer-scaler
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: consumer
minReplicas: 1
maxReplicas: 10
metrics:
- type: External
external:
metricName: queuemessages
targetValue: 30
Checkout the samples for more examples and details.
You can also can query the api to verify it is installed:
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq .
To Query for a specific custom metric:
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/test/pods/*/custom-metric" | jq .
To query for a specific external metric:
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/test/queuemessages" | jq .
Requires k8s 1.10+
See a full list of hundreds of available azure external metrics that can be used.
Common external metrics to use for autoscaling are:
Custom metrics are currently retrieved from Application Insights. View a list of basic metrics that come out of the box and see sample values at the AI api explorer.
Common Custom Metrics are:
Authenticating with Azure Monitor can be achieved via a variety of authentication mechanisms. (full list)
Use one of the following options:
The Azure AD entity needs to have Monitoring Reader
permission on the resource group that will be queried. More information can be found here.
aad-pod-identity is currently in beta and allows to bind a user managed identity or a service principal to a pod. That means that instead of using the same managed identity for all the pod running on a node like explained above, you are able to get a specific identity with specific RBAC for a specific pod.
Using this project requires to deploy a bit of infrastructure first. You can do it following the Get started page of the project.
Once the aad-pod-identity infrastructure is running, you need to create an Azure identity scoped to the resource group you are monitoring:
az identity create -g {ResourceGroup1} -n custom-metrics-identity
Assign Monitoring Reader
to it
az role assignment create --role "Monitoring Reader" --assignee <principalId> --scope /subscriptions/{SubID}/resourceGroups/{ResourceGroup1}
Note: you need to assign this role to all resources groups that you want the identity to be able to read Azure Monitor data.
As documented here aad-pod-identity uses the service principal of your Kubernetes cluster to access the Azure resources. You need to give this service principal the rights to use the managed identity created before:
az role assignment create --role "Managed Identity Operator" --assignee <servicePrincipalId> --scope /subscriptions/{SubID}/resourceGroups/{ResourceGroup1}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/custom-metrics-identity
Install the Azure Identity to your Kubernetes cluster:
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentity
metadata:
name: custom-metrics-identity
spec:
type: 0
ResourceID: /subscriptions/{SubID}/resourceGroups/{ResourceGroup1}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/custom-metrics-identity
ClientID: <clientid>
Install the Azure Identity Binding on your Kubernetes cluster:
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
name: custom-metrics-identity-binding
spec:
AzureIdentity: custom-metrics-identity
Selector: custom-metrics-identity
Note: pay attention to the name of the selector above. You will need to use it to bind the identity to your pod.
If you use the Helm Chart to deploy the custom metrics adapter to your Kubernetes cluster, you can configure Azure AD Pod Identity directly in the config values:
azureAuthentication:
method: aadPodIdentity
# if you use aadPodIdentity authentication
azureIdentityName: "custom-metrics-identity"
azureIdentityBindingName: "custom-metrics-identity-binding"
# The full Azure resource id of the managed identity (/subscriptions/{SubID}/resourceGroups/{ResourceGroup1}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{IdentityName})
azureIdentityResourceId: ""
# The Client Id of the managed identity
azureIdentityClientId: ""
Switch method
to aadPodIdentity
a give the value for the Azure Identity resource id and client id, for example:
helm install ./charts/azure-k8s-metrics-adapter --set azureAuthentication.method="aadPodIdentity" \
--set azureAuthentication.azureIdentityResourceId="/subscriptions/{SubID}/resourceGroups/{ResourceGroup1}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{IdentityName}" \
--set azureAuthentication.azureIdentityClientId="{ClientId}" \
--name "custom-metrics-adapter"
See how to create an example deployment.
Create a service principal scoped to the resource group the resource you monitoring and assign Monitoring Reader
to it:
az ad sp create-for-rbac -n "adapter-sp" --role "Monitoring Reader" --scopes /subscriptions/{SubID}/resourceGroups/{ResourceGroup1}
Required environment variables:
AZURE_TENANT_ID
: Specifies the Tenant to which to authenticate.AZURE_CLIENT_ID
: Specifies the app client ID to use.AZURE_CLIENT_SECRET
: Specifies the app secret to use.Deploy the environment variables via secret:
kubectl create secret generic azure-k8s-metrics-adapter -n custom-metrics \
--from-literal=azure-tenant-id=<tenantid> \
--from-literal=azure-client-id=<clientid> \
--from-literal=azure-client-secret=<secret>
Required environment variables:
AZURE_TENANT_ID
: Specifies the Tenant to which to authenticate.AZURE_CLIENT_ID
: Specifies the app client ID to use.AZURE_CERTIFICATE_PATH
: Specifies the certificate Path to use.AZURE_CERTIFICATE_PASSWORD
: Specifies the certificate password to use.The use the adapter your Azure Subscription must be provided. There are a few ways to provide this information:
SUBSCRIPTION_ID
on the adapter deployment. This takes precedence over the Azure Instance Metadata.subscriptionID
on each HPA. This overrides Environment variables and Azure Instance Metadata settings.See Contributing for more information.
Report any issues in the Github issues.
See the Projects tab for current roadmap.
Security issues and bugs should be reported privately, via email, to the Microsoft Security Response Center (MSRC) at secure@microsoft.com. You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Further information, including the MSRC PGP key, can be found in the Security TechCenter.