Open aelmanaa opened 4 years ago
@TomGeske @palma21 WDYT?
@azooinmyluggage ^
@aelmanaa Could you please explain the scenario where you want to use kubelogin as a service connection? I think the three ways of creating a kubernetes service connection in Azure Devops already cover all types of clusters.
@shigupt202 we arleady use "service account option". however this option has the following disadvantages:
during the creation of a cluster, one has to foresee to manually create the service account and the role bindings (manual step)
service accounts are outside of AAD. generally within an organization, we prefer to centrally manage an identity within AAD. There are already "service principals" or "managed identities" in AAD ; hence better to use these ones
Moreover, it seems that kubelogin was created as a response to this feature request https://feedback.azure.com/forums/914020-azure-kubernetes-service-aks/suggestions/35146387-support-non-interactive-login-for-aad-integrated-c?tracking_code=e9effc5721de49e6c717ca62967a20d2
hence it would be nice if in AZure devops, we could directly use a service principal or a managed identity as service connection. Behind the scenes, Azure devops uses kubelogin to authenticate (non-interactively) with the cluster
@aelmanaa Thanks for the feedback! I'll get in touch with the PM's to discuss this feature request. cc: @azooinmyluggage @anraghun
Ack. We will work on adding this support post Ignite. Adding @anraghun to take this forward.
thanks @azooinmyluggage
In case someone else runs into this problem, this is how I ended up doing it.
It works but I agree with @aelmanaa and would prefer if there were a built-in ADO Task to handle this.
-l azurecli
to avoid having to continue to set AAD_SERVICE_PRINCIPAL_CLIENT_ID
for every subsequent stepFor more about templates, see https://docs.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops
Note you will also need to set the following variables:
variables:
appNamespace: my-namespace
aksClusterName: my-aks-cluster
aksClusterResourceGroup: cluster-rg-not-the-nodes-one
which are used below
# steps/setup-kubelogin.yaml
steps:
- bash: |
# Download and install
curl -LO "https://github.com/Azure/kubelogin/releases/download/$(kubeloginVersion)/kubelogin-linux-amd64.zip"
sudo unzip -j "kubelogin-linux-amd64.zip" -d /usr/local/bin
rm -f "kubelogin-linux-amd64.zip"
kubelogin --version
displayName: kubelogin - install
# For details, see https://stackoverflow.com/questions/54004007/azure-devop-pipelines-authentication-to-aks-with-azure-ad-rbac-configured
- bash: |
# First login to Azure
az login \
--service-principal \
--username $AAD_SERVICE_PRINCIPAL_CLIENT_ID \
--password $AAD_SERVICE_PRINCIPAL_CLIENT_SECRET \
--tenant $AZ_TENANT_ID
# Fails if it doesn't exist
touch .kubeconfig-$(aksClusterName)
chmod 600 .kubeconfig-$(aksClusterName)
# Populate kubeconfig
az aks get-credentials \
--resource-group $(aksClusterResourceGroup) \
--name $(aksClusterName) \
--overwrite-existing \
--file .kubeconfig-$(aksClusterName)
# Pass kubeconfig to kubelogin to access k8s API
kubelogin convert-kubeconfig -l azurecli
# confirm it works
kubectl get pods --namespace $(appNamespace)
displayName: kubelogin - aks-credentials to kubecontext
env:
AZ_TENANT_ID: $(tenant-id)
AAD_SERVICE_PRINCIPAL_CLIENT_ID: $(aks-architect-ci-dev-sp-client-id)
AAD_SERVICE_PRINCIPAL_CLIENT_SECRET: $(aks-architect-ci-dev-sp-client-secret)
KUBECONFIG: $(Build.SourcesDirectory)/.kubeconfig-$(aksClusterName)
# …
jobs:
- job: Deploy
displayName: Deploy
variables:
- group: mask-subscription-ids # hack to mask my subscription ids
- group: aks-credentials-kv-group # the service principal credentials
steps:
- template: ../steps/setup-kubelogin.yaml # what we created above
- bash: |
# confirm it works
kubectl get pods --namespace $(appNamespace)
displayName: "kubectl apply"
env:
KUBECONFIG: $(Build.SourcesDirectory)/.kubeconfig-$(aksClusterName)
- bash: |
kubelogin remove-tokens
displayName: kubelogin - clear cache
In this way I only need to repeat the env: KUBECONFIG…
part for subsequent steps.
Special thanks to https://stackoverflow.com/questions/54004007/azure-devop-pipelines-authentication-to-aks-with-azure-ad-rbac-configured
@aelmanaa Could you please explain the scenario where you want to use kubelogin as a service connection? I think the three ways of creating a kubernetes service connection in Azure Devops already cover all types of clusters.
Use case - I disabled local accounts per https://docs.microsoft.com/en-us/azure/aks/managed-aad#disable-local-accounts-preview
To be honest this doesn't need a new service connection, there is already an Azure Resource Manager service connection that gets you most of the way, just need to configure kubelogin correctly.
As things stand right now I have no idea what the use case is for the Azure DevOps KubernetesV1 task configured with Azure Resource Manager connection type and useClusterAdmin set to false (the default). I don't understand how the Azure Resource Manager examples in the documentation could possibly work.
@julie-ng Thank you for the very useful write-up.
When I use an ARM Service Connection with MSI on an agent with kubelogin
preinstalled the following just works:
- task: AzureCLI@2
displayName: aks non-admin login using kubelogin
inputs:
azureSubscription: ${{parameters.serviceConnection}}
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
az aks get-credentials -n <aks cluster> -g <resource group>
kubelogin convert-kubeconfig -l azurecli
- task: AzureCLI@2
displayName: test kubectl and helm from bash
continueOnError: true
inputs:
azureSubscription: ${{parameters.serviceConnection}}
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
kubectl get pods -n <some namespace>
helm list --namespace <some namespace>
However, if I try to use the HelmDeploy@0
and Kubernetes@1
tasks that effectively do the same as second task with kubectl
and helm
commands, it does not work and I get (for the helm case):
2022-01-19T20:33:52.7909008Z ##[debug]set helmExitCode=1
2022-01-19T20:33:52.7910461Z Error: Kubernetes cluster unreachable: Get "https://<aks cluster>-a15872ed.hcp.westeurope.azmk8s.io:443/version?timeout=32s": getting credentials: exec: executable kubelogin failed with exit code 1
2022-01-19T20:33:52.7912489Z helm.go:88: [debug] Get "https://<aks cluster>-a15872ed.hcp.westeurope.azmk8s.io:443/version?timeout=32s": getting credentials: exec: executable kubelogin failed with exit code 1
2022-01-19T20:33:52.7915456Z ##[debug]Processed: ##vso[task.setvariable variable=helmExitCode;issecret=false;]1
2022-01-19T20:33:52.7925848Z ##[debug]publishPipelineMetadata=true
2022-01-19T20:33:52.7926838Z Kubernetes cluster unreachable
2022-01-19T20:33:52.7930444Z ##[debug]execResult: {"code":1,"stdout":"","stderr":"Error: failed to get token: expected an empty error but received: Azure CLI Credential: ERROR: Please run 'az login' to setup account.\n\nError: Kubernetes cluster unreachable: Get \"https://<aks cluster>-a15872ed.hcp.westeurope.azmk8s.io:***@v1.2.1/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.2.1/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.2.1/command.go:902\nmain.main\n\thelm.sh/helm/v3/cmd/helm/helm.go:87\nruntime.main\n\truntime/proc.go:225\nruntime.goexit\n\truntime/asm_amd64.s:1371\n"}
2022-01-19T20:33:52.7933131Z helm.sh/helm/v3/pkg/kube.(*Client).IsReachable
2022-01-19T20:33:52.7933638Z ##[debug]task result: Failed
2022-01-19T20:33:52.7934668Z helm.sh/helm/v3/pkg/kube/client.go:121
2022-01-19T20:33:52.7969750Z ##[error]Error: failed to get token: expected an empty error but received: Azure CLI Credential: ERROR: Please run 'az login' to setup account.
- task: HelmDeploy@0
displayName: test HelmDeploy task
continueOnError: true
inputs:
connectionType: None
namespace: <some namespace>
command: list
- task: Kubernetes@1
displayName: test Kubernetes task
continueOnError: true
inputs:
connectionType: None
command: get
arguments: pods -n <some namespace>
I created issue microsoft/azure-pipelines-tasks#15802 but I am not sure if they can solve this.
How does kubelogin
retrieve the MSI credentials when using azurecli
as login mode? Does it need access to certain environment variables or a token file that can explain the difference between the two scenario's?
@nlighten connectionType: None
does not appear to be a valid connection type in the documentation https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/kubernetes?view=azure-devops
I dived into the code a few months back and the Kubernetes and Helm tasks appear to always create their own custom kubeconfig file based upon the defined service connection https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/KubernetesV1/src/clusterconnection.ts#L71 which means that it will ignore the one you created with aks get-credentials
Browsing through the code again, what might work would be taking the kubeconfig that's generated by kubelogin and using that to create a Kubeconfig based service connection https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#kubeconfig-option with the caveat that you'll need to update the service connection any time you rotate the cluster certificates (can probably be automated if you're creating the service connections with terraform).
@sharebear I agree that None
is not a documented connection type but it actually works for both tasks. The kubeconfig file is not created in that case (see https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/HelmDeployV0/src/helm.ts#L41) and a pre-existing kubeconfig file is used.
So if I do an kubectl aks get-credentials
with --admin
(no kubelogin) using connectionType: None
works just fine. Additionally from the logs I can see that it actually uses the kubeconfig created by kubelogin convert-kubeconfig -l azurecli
because it tries to use kubelogin for the authentication. It only fails on the authentication step. The puzzle is: why does it fail when helm/kubectl is invoked from the Task but not from the command line.
@nlighten
I agree that None is not a documented connection type but it actually works for both tasks. The kubeconfig file is not created in that case (see https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/HelmDeployV0/src/helm.ts#L41) and a pre-existing kubeconfig file is used.
TIL ... althought on a quick skim I don't see similar code in the Kubernetes task, the code isn't that easy to follow.
Additionally from the logs I can see that it actually uses the kubeconfig created by kubelogin convert-kubeconfig -l azurecli because it tries to use kubelogin for the authentication. It only fails on the authentication step.
In that case, what's your error message? I had a problem where I had forgotten to actually enable AzureRBAC on the AKS cluster, and it's not very visible in the portal, so might be worth verifying that.
@sharebear Thank you for thinking with me.
TIL ... althought on a quick skim I don't see similar code in the Kubernetes task, the code isn't that easy to follow.
In the Kubernetes task the switch is here: https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/KubernetesV1/src/clusterconnection.ts#L59
In that case, what's your error message?
The error I get is:
2022-01-19T20:33:52.7910461Z Error: Kubernetes cluster unreachable: Get "https://<aks cluster>-a15872ed.hcp.westeurope.azmk8s.io:443/version?timeout=32s": getting credentials: exec: executable kubelogin failed with exit code 1
So kubelogin apparently fails with exit code 1. Full task logs can be found at: https://github.com/microsoft/azure-pipelines-tasks/files/7900774/logs.zip
RBAC and AKS-managed AAD is definitely enabled on the cluster. In the pipeline steps listed above I can successfully use the managed identity credentials when using helm and kubectl directly from bash. As far as I know the DevOps Tasks are only a wrapper around the same binaries, but somehow when helm or kubectl are called from the Tasks kubelogin fails. My impression is that cannot get a hold of the managed identity credentials but logging is too limited to pinpoint.
Perhaps the helm and kubernetes tasks do not have access to the ~/.azure/
directory? I will do some testing to see if this directory is removed after the AzureCLI@2 is finished (which would make sense from a security perspective).
It was indeed caused by AzureCLI@2 doing az account clear
at the end. If I use plain bash and an explicit login/logout everything works as expected in combination with a MSI:
- task: Bash@3
displayName: az and aks login
inputs:
targetType: 'inline'
script: |
az login --identity
az aks get-credentials -n <aks cluster> -g <resource group>
kubelogin convert-kubeconfig -l azurecli
- task: HelmDeploy@0
displayName: test HelmDeploy task
inputs:
connectionType: None
namespace: <namespace>
command: list
- task: Kubernetes@1
displayName: test Kubernetes task
inputs:
connectionType: None
command: get
arguments: pods -n <namespace>
- task: Bash@3
condition: always()
displayName: az and aks logout
inputs:
targetType: 'inline'
script: |
az account clear
rm -rf ~/.kube
Still, I would prefer explicit support for kubelogin as part an ARM Service Connection on the Kubernetes and HelmDeploy tasks.
@sharebear Thanks for your time.
@nlighten Sorry I stopped replying there, got caught up in the day-to-day work.
Great that you were able to work it out, it's helped me develop my understanding too... I went with a "service account" based solution when I disabled the local admin account, but still keeping my eyes open for better solutions.
I do wonder though, as you are using msi, what's the motivation for using azurecli auth in kubelogin instead of using the msi support there? If you just ran kubelogin convert-kubeconfig -l msi
would that also have solved the problem?
@sharebear I actually missed the -l msi
option (rtfm). It works and results in a slightly more elegant:
- task: AzureCLI@2
displayName: aks non-admin login using kubelogin
inputs:
azureSubscription: ${{parameters.serviceConnection}}
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
az aks get-credentials -n <aks cluster> -g <resource group>
kubelogin convert-kubeconfig -l msi
- task: HelmDeploy@0
displayName: test HelmDeploy task
inputs:
connectionType: None
namespace: <namespace>
command: list
- task: Kubernetes@1
displayName: test Kubernetes task
inputs:
connectionType: None
command: get
arguments: pods -n <namespace>
@nlighten Great... now, as far as I understand it, the contents of the kubeconfig after you've run kubelogin convert-kubeconfig -l msi
are pretty static and will only change if/when you rotate the cluster certificates.
Does it then make sense to move it into a service connection with kubeconfig so you don't need to generate it every pipeline run? Or is kubelogin only installed when you run get-credentials?
@sharebear
Does it then make sense to move it into a service connection with kubeconfig so you don't need to generate it every pipeline run?
I would prefer not. We rotate cluster certificates every 3 months and also rebuild clusters quite frequent. With moving it into the Kubernetes Service Connection I would still need to update the Service Connection after each certificate rotation or rebuild.
In my mind something like kubelogin: msi
option as part of the HelmDeploy and Kubernetes Task when using connectionType: Azure Resource Manager
would be a better option.
@sharebear
Does it then make sense to move it into a service connection with kubeconfig so you don't need to generate it every pipeline run?
I would prefer not. We rotate cluster certificates every 3 months and also rebuild clusters quite frequent. With moving it into the Kubernetes Service Connection I would still need to update the Service Connection after each certificate rotation or rebuild.
Fair enough. We manage the service connections through terraform https://registry.terraform.io/providers/microsoft/azuredevops/latest/docs/resources/serviceendpoint_kubernetes so I think I could automate this.
In my mind something like
kubelogin: msi
option as part of the HelmDeploy and Kubernetes Task when usingconnectionType: Azure Resource Manager
would be a better option.
I completely agree on this point, what we're discussing is still a workaround for something that should work out of the box. Especially when it's recommended to disable local accounts which the current implementations appear to depend upon being enabled.
Thanks for the good discussion!
@sharebear @nlighten have you considered using gitops? Because you're walking a slippery path and depending on your requirements, from a security perspective, it would be more straight-forward with less gaps to just have the cluster pull in the changes.
That being said, let me explain what concerns me about this discussion. Keep in mind I work for MSFT but am not part of the AKS product group.
In the future, you must use kubelogin to align with k8s upstream. Old methods will be deprecated soon starting with v1.23
re using msi, what's the motivation for using azurecli auth in kubelogin instead of using the msi support there? If you just ran kubelogin convert-kubeconfig -l msi
See AKS public roadmap item here
TL;DR; is that it has to do with upstream Kubernetes moving cloud provider specific code out of core k8s announced in 2019. So simple azure cli
will not work anymore. You will need an explicit kubeconfig.
Well not exactly. AKS is an offering where Azure manages your Kubernetes control plane (some of which you do not see in your Azure account). The Kubernetes components that host your workload, e.g. VMs, Load Balancers, etc. are Kubernetes (not Azure). Azure will provide interfaces that make interaction easier, including identity integration.
Managed Identity is an Azure offering and designed for accessing Azure resources. See MSFT Blog - Demystifying Service Principals – Managed Identities.
Technically you're trying to access a Kubernetes resource, not an Azure resource. Because they both speak and understand OIDC/OAuth, it works.
Perhaps the helm and kubernetes tasks do not have access to the ~/.azure/ directory? I will do some testing to see if this directory is removed after the AzureCLI@2 is finished (which would make sense from a security perspective).
K8s and Helm are not Azure and should not have access to ~/.azure
. Instead Azure must speak k8s and use kubeconfig. See roadmap link above.
IMO Managed identities are often misunderstood and people think it's magic without understanding what's happening under the hood - and thus how Shared Responsibility Principle still applies. Keep this in mind
KUBECONFIG
environment variable set to $(Build.SourcesDirectory)/.kubeconfig-$(aksClusterName)
are designed to ensure these files don't persist beyond their lifecyclesSo… @sharebear
Does it then make sense to move it into a service connection with kubeconfig so you don't need to generate it every pipeline run? Or is kubelogin only installed when you run get-credentials?
No, please do not move the kubeconfig into a service connection because it shouldn't exist beyond lifecycles, e.g. pipeline runs. Even though OAuth tokens can be long lived, that's a slippery slope. IMO, in the spirit of OAuth, build agents should re-authenticate and fetch a new access token every time they start a new job.
They're just wrappers around credentials that allow you to apply some authorization mechanisms, e.g. which pipeline can use this? Deployments should only happen during business hours, etc.
@nlighten
In my mind something like kubelogin: msi option as part of the HelmDeploy and Kubernetes Task when using connectionType: Azure Resource Manager would be a better option.
That's an interesting idea. I'd be curious to know why you use the tasks and not just the bash? Personally I find the tasks (except ARM deployment) limiting and abstractions not useful. If I'm using Terraform, kubectl, etc. I prefer scripting. Also lets me test/code locally and thus make pipelines faster.
@sharebear @nlighten have you considered using gitops? Because you're walking a slippery path and depending on your requirements, from a security perspective, it would be more straight-forward with less gaps to just have the cluster pull in the changes.
That being said, let me explain what concerns me about this discussion. Keep in mind I work for MSFT but am not part of the AKS product group.
kubelogin
In the future, you must use kubelogin to align with k8s upstream. Old methods will be deprecated soon starting with v1.23
Exactly, this strengthens the argument that it should be supported out of the box by the DevOps Pipeline tasks provided for interacting with k8s.
re using msi, what's the motivation for using azurecli auth in kubelogin instead of using the msi support there? If you just ran kubelogin convert-kubeconfig -l msi
See AKS public roadmap item here
TL;DR; is that it has to do with upstream Kubernetes moving cloud provider specific code out of core k8s announced in 2019. So simple
azure cli
will not work anymore. You will need an explicit kubeconfig.
Correct, an this is exactly what we are supporting with the proposed config here.
Identity…AKS != Kubernetes
Well not exactly. AKS is an offering where Azure manages your Kubernetes control plane (some of which you do not see in your Azure account). The Kubernetes components that host your workload, e.g. VMs, Load Balancers, etc. are Kubernetes (not Azure). Azure will provide interfaces that make interaction easier, including identity integration.
Managed Identity is an Azure offering and designed for accessing Azure resources. See MSFT Blog - Demystifying Service Principals – Managed Identities.
Technically you're trying to access a Kubernetes resource, not an Azure resource. Because they both speak and understand OIDC/OAuth, it works.
I really don't see how this is any different to using managed identities for data plane access for Azure Storage, Azure SQL, Azure Cache for Redis, etc. With the AzureRBAC support we can manage all access with one tool instead of constantly wondering whether access is granted in AzureRBAC or K8sRBAC, with potential confusion being a security risk.
Perhaps the helm and kubernetes tasks do not have access to the ~/.azure/ directory? I will do some testing to see if this directory is removed after the AzureCLI@2 is finished (which would make sense from a security perspective).
K8s and Helm are not Azure and should not have access to
~/.azure
. Instead Azure must speak k8s and use kubeconfig. See roadmap link above.Managed Identities - the slippery part
IMO Managed identities are often misunderstood and people think it's magic without understanding what's happening under the hood - and thus how Shared Responsibility Principle still applies. Keep this in mind
- Managed Identities are still service principals under the hood. It saves you from managing secrets, but secrets still exist, in the form of tokens - although short lived and frequently rotated, they exist and are maybe "lying around" somewhere.
- The pipeline tasks - and my
KUBECONFIG
environment variable set to$(Build.SourcesDirectory)/.kubeconfig-$(aksClusterName)
are designed to ensure these files don't persist beyond their lifecyclesSo… @sharebear
Does it then make sense to move it into a service connection with kubeconfig so you don't need to generate it every pipeline run? Or is kubelogin only installed when you run get-credentials?
No, please do not move the kubeconfig into a service connection because it shouldn't exist beyond lifecycles, e.g. pipeline runs. Even though OAuth tokens can be long lived, that's a slippery slope. IMO, in the spirit of OAuth, build agents should re-authenticate and fetch a new access token every time they start a new job.
Here I think you've misunderstood the contents of the kubeconfig file. There are no tokens stored in the file, just the public cluster root CA certificate and the necessary yaml to tell kubectl which arguments to pass to kubelogin. Nothing secret or sensitive here at all.
Service Connections - are just wrappers
They're just wrappers around credentials that allow you to apply some authorization mechanisms, e.g. which pipeline can use this? Deployments should only happen during business hours, etc.
@nlighten
In my mind something like kubelogin: msi option as part of the HelmDeploy and Kubernetes Task when using connectionType: Azure Resource Manager would be a better option.
That's an interesting idea. I'd be curious to know why you use the tasks and not just the bash? Personally I find the tasks (except ARM deployment) limiting and abstractions not useful. If I'm using Terraform, kubectl, etc. I prefer scripting. Also lets me test/code locally and thus make pipelines faster.
I can't anser for @nlighten but my answer would be;
@julie-ng Thank you for sharing your thoughts. To add to @sharebear feedback:
have you considered using gitops? Yes, but AFAIK this currently only covers K8S resources and not yet Azure resources (not considering Azure Service Operator which does not seem to be production ready yet).
Perhaps to explain a little bit the background of my experiments with kubelogin: we are looking at providing individual subscriptions to product teams. We are considering using a dedicated agent pool per subscription for deployment (no build). This pool would have an MSI associated with it that would allow the the team to deploy to resources within that subscription. AKS is one of the targets, but they (for example) could also do database migrations (CosmosDB/SQL Server), access keyvault etc, all using the same MSI.
K8s and Helm are not Azure and should not have access to ~/.azure. Instead Azure must speak k8s and use kubeconfig. See roadmap link above.
Agreed. This was my mind trying to understand how this worked and doing it wrong ...
I'd be curious to know why you use the tasks and not just the bash?
Personally I prefer the command line, but there is a large developer community and existing pipelines that already use the Tasks and this is also how Microsoft shows them how to do it in the documentation.
In the future, you must use kubelogin to align with k8s upstream. Old methods will be deprecated soon starting with v1.23
Exactly, this strengthens the argument that it should be supported out of the box by the DevOps Pipeline tasks provided for interacting with k8s.
I agree with @sharebear . Perhaps we should open a feature request in the azure-pipelines-tasks project with this as motivation.
@sharebear - being pragmatic about your options is practical. I think that way too. But on this part I disagree:
Here I think you've misunderstood the contents of the kubeconfig file. There are no tokens stored in the file, just the public cluster root CA certificate and the necessary yaml to tell kubectl which arguments to pass to kubelogin. Nothing secret or sensitive here at all.
Are you sure? Check your ~/.kube/config
and see also https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens
I haven't used all the different methods, for example certs for authentication. But my understanding and my personal kubeconfig is full of secrets.
Re: pipeline tasks - yep all the docs use it and the vast majority of my colleagues use them. I find more verbose than just using the command line. And often they're not stable or documentation of the tasks themselves are weird, i.e. break. Pipelines with bash are cleaner. I tend to hide them in tasks in sub templates. Personal preference. We'll probably just agree to disagree here… :)
@nlighten re: MSI
~/.kube/config
, well hello security challenge for future jobs. That's why the task and I create and send an explicit kubeconfig, where the lifecycle matches that of the build job. Maybe you're designing your node pools now to be product team specific - but when your company does an re-org, you'll need to update your tech concept.I used to be an engineer and enterprise architect at Allianz Germany, a multibillion dollar company. Back in ~2019 (not sure what they use now), our team designed and created a Jenkins infrastructure that leveraged k8s pods for build agents. If you can kill a pod after a job is done, you have to worry less about housekeeping after a job.
Using a VM with a longer lifecycle? Be very careful. Housekeeping is not easy. And you then need to understand every toolchain you have on that machine and how it works, including kubectl
Anyhow, I'll probably sign off this thread now as I could rant about this forever. I don't expect or even want you to agree with me. But I'd love for everyone to dig deeper and understand how things work under the hood - esp. for people who have any architecture responsibility.
I'll probably make a YouTube video about this soon. Same discussion came up at work this week.
It's been a fun discussion :)
Are you sure? Check your ~/.kube/config and see also https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens
@sharebear is correct. The only thing that is in the kubeconfig is the cluster public cert and the config to delegate the token retrieval to kubelogin. For an MSI this looks like:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: ******
server: https://mycluster-a15872ed.hcp.westeurope.azmk8s.io:443
name: mycluster
contexts:
- context:
cluster: mycluster
user: clusterUser_myresourcegroup_mycluster
name: mycluster
current-context: mycluster
kind: Config
preferences: {}
users:
- name: clusterUser_myresourcegroup_mycluster
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- get-token
- --server-id
- 6dae42f8-****-****-****-3960e28e3630
- --login
- msi
command: kubelogin
env: null
provideClusterInfo: false
be careful with managed identities and build agents
Agreed. That is we are considering dedicated 'deployment' agents that cannot be used for builds. Deployment agents are restricted to a single team/subscription. It also gives us more control over the whole 'application supply chain'.
our team designed and created a Jenkins infrastructure that leveraged k8s pods for build agents
That is nowadays fairly simple to setup using AKS + KEDA which has an Azure Pipelines scaler. If you combine it with Pod Identity you can create very fine grained agents pools each with their own MSI (if you want). But in our scenario, an AKS cluster per subscription just for deployment agents is a little bit overkill.
Anyhow, I'll probably sign off this thread now as I could rant about this forever. I don't expect or even want you to agree with me. But I'd love for everyone to dig deeper and understand how things work under the hood - esp. for people who have any architecture responsibility.
You know what happened when the Dwarves dug too greedily and too deep ;-). Good luck with your YouTube video.
@nlighten cool, thanks for sharing the example config. I've never used kubectl with msi. I'm overly cautious and just assume certs and tokens are in there :P
Haha, simple? Not really. You'd need a team to maintain it. A large insurance company has those resources, i.e. dedicated teams with dedicated people for maintaining patches, people's feature/software requests, etc. If not, it's overrated in my opinion.
It's a fun ops problem if you're just into ops. If you're like me and prioritize creating business value, I'd question whether if there's another solution that's simpler and just as secure.
Btw, because no one mentioned it… most people don't realize if you use managed identities, you have a ginormous single point of failure…AAD 😬 unlikely, but in my current non pre-sales role, I can tell customers that sorry, it's happened. So make an informed decision.
Ack. We will work on adding this support post Ignite. Adding @anraghun to take this forward.
@azooinmyluggage @anraghun where did we land with this?
Adding @vijayma
I've been able to build out a proof of concept to this problem of the ADO kubernetes tasks not supporting kubelogin. Going to put it here and describe how it works. It's based on info gathered from earlier in this discussion.
Prereqs:
Azure Kubernetes Service RBAC Writer
to it.myakstestsp
)And here's the pipeline. I'll describe the components of it below.
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
#trigger:
#- master
pool:
vmImage: ubuntu-latest
variables:
azureResourceGroup: myrg
kubernetesCluster: myakscluster
useClusterAdmin: false
steps:
- task: AzureCLI@2
displayName: azcli login and save creds
inputs:
azureSubscription: 'myakstestsp'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
#Unfortunately azurecli task does a logout at the end, so we need to grab creds and then login from bash
echo "export servicePrincipalId=$servicePrincipalId" >.env
echo "export servicePrincipalKey=$servicePrincipalKey" >>.env
echo "export tenantId=$tenantId" >>.env
addSpnToEnvironment: true
- bash: |
source .env
az login --service-principal -u $servicePrincipalId -p $servicePrincipalKey -t $tenantId
mkdir -p .bin
az aks install-cli --install-location .bin/kubectl # install kubelogin
az aks get-credentials -n $(kubernetesCluster) -g $(azureResourceGroup)
kubelogin convert-kubeconfig -l azurecli
kubectl get all -A
- task: Kubernetes@1
inputs:
# connectionType:None is not documented, but it's in the code https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/KubernetesV1/src/clusterconnection.ts#L59
# with this setting, the task will just use the kubeconfig context
connectionType: 'None'
command: 'apply'
useConfigurationFile: true
configuration: 'myapp.yaml'
secretType: 'generic'
forceUpdate: false
- task: HelmDeploy@0
displayName: Helm deploy
inputs:
connectionType: 'None'
command: 'install'
chartType: 'FilePath'
chartPath: 'mydemo'
waitForExecution: false
- task: KubernetesManifest@0
inputs:
action: 'deploy'
kubernetesServiceConnection: 'myaksserviceconnection'
namespace: 'default'
manifests: 'demo2.yaml'
Let's look at each of the tasks in turn:
- task: AzureCLI@2
displayName: azcli login and save creds
inputs:
azureSubscription: 'myakstestsp'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
# azurecli task does a logout at the end, so we need to grab creds and then login via az cli in bash in subsequent step
echo "export servicePrincipalId=$servicePrincipalId" >.env
echo "export servicePrincipalKey=$servicePrincipalKey" >>.env
echo "export tenantId=$tenantId" >>.env
addSpnToEnvironment: true
In the above task, I'm using the azurecli task to leverage the service connection to log into azure. The nuance is that this task executes a logout when it completes, so it's not useful if you want to stay logged in for subsequent tasks. So what I'm doing is grabbing the SP id & key and storing them into a file to be used later. There are certainly other ways of achieving this; this was just a quick and dirty approach.
- bash: |
source .env
az login --service-principal -u $servicePrincipalId -p $servicePrincipalKey -t $tenantId
mkdir -p .bin
az aks install-cli --install-location .bin/kubectl # install kubelogin
az aks get-credentials -n $(kubernetesCluster) -g $(azureResourceGroup)
kubelogin convert-kubeconfig -l azurecli
kubectl get all -A
The next block does a number of things (and could have been split into multiple tasks.)
az aks install-cli
in order to install the kubelogin tool.az aks get-credentials
, and gets an AAD token using kubeloginkubetctl get all -A
simply to show that the login worked. (obviously this would be removed in a real scenario)Next we'll prove out the three Kubernetes-specific ADO tasks:
- task: Kubernetes@1
inputs:
# connectionType:None is not documented, but it's in the code https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/KubernetesV1/src/clusterconnection.ts#L59
# with this setting, the task will just use the kubeconfig context
connectionType: 'None'
command: 'apply'
useConfigurationFile: true
configuration: 'myapp.yaml'
secretType: 'generic'
forceUpdate: false
This task Kubernetes@1
acts like a wrapper around kubectl. Normally it wants to leverage either an Azure Service Connection or a Kubernetes Service Connection, but in this case, we make use of the undocumented option connectionType:None
to force it to use the kubeconfig context.
- task: HelmDeploy@0
displayName: Helm deploy
inputs:
connectionType: 'None'
command: 'install'
chartType: 'FilePath'
chartPath: 'mydemo'
waitForExecution: false
I take a similar approach for the HelmDeploy@0
task.
- task: KubernetesManifest@0
inputs:
action: 'deploy'
kubernetesServiceConnection: 'myaksserviceconnection'
namespace: 'default'
manifests: 'demo2.yaml'
Last is the KubernetesManifest@0
task. This was more of a challenge to get working, as there was no option for connectionType: 'None'
. Instead, to get this to work, I needed to create another Service Connection in Azure DevOps, but this new connection needs to be a Kubernetes Service Connection, and specifically, it needs to be of type Kubeconfig. You will need to paste in the appropriate kubeconfig for this cluster that you previously pulled via az aks get-credentials
. (Note that this kubeconfig won't work without the associated kubelogin setup). In the example above, you'll see my kuberentesServiceConnection is named myaksserviceconnection
I'm interested in any feedback on this solution.
Hi @larryclaman,
Essentially, there is no need to export SPN credentials as you can retrieve aks credentials and convert cubeconfig with kubelogin in one go:
- task: AzureCLI@2
inputs:
azureSubscription: '$(serviceConnection)'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
az aks get-credentials --resource-group $(resourceGroup) --name $(kubernetesCluster)
kubelogin convert-kubeconfig -l azurecli
displayName: 'Get cluster credentials'
Could you elaborate on the last task? Does it imply that cubeconfig has to be converted with kubelogin first and then its content to be pasted to the service connection? In this case, the last task is completely independent of the first task, because it already has the same kubeconfig content as the first task produces.
Thanks.
@sergeidavydov As mentioned by several people in this thread, AzureCLI@2 runs az account clear
at the end of the task, which invalidates the login session that was exported by kubelogin.
https://github.com/Azure/kubelogin/pull/142 has blocked some workarounds discussed here.
Following on from the existing AzureCLI@2 task logic mentioned above by @carlin-q-scott, you can no longer run a separate kubelogin task, followed by one or more generic tasks using running kubectl commands or tasks like HelmDeploy@0 with connectionType: 'None'
I made and published a template based on the information contained in this issue: https://github.com/carlin-q-scott/azure-devops-pipeline-templates/blob/main/steps/kubectl.yml
My readme includes instructions on using it.
AKS used to be easy to integrate into the pipeline, and easy to access. It is no longer so. reading all the heavy work-arounds and less than optimal implementations makes me wonder if anybody is really watching the ecosystem for such intrinsic changes like this that has cascaded all sorts of problems.
Thank you to the people working tirelessly that aren't microsoft and sharing their work-arounds, but still, All I see are work-arounds here.
For somebody who DOESN'T have Azure AAD enabled on his cluster and relied on service connections which used to work seamlessly with AKS for integration, what are my options? The service connection holds the spn / secret....
@darthmolen what's not working for you? az aks install-cli
?
I have been playing around with this in Azure Pipelines. Wanted to throw my 2 cents into the ring for some possible solutions. The following solutions will work if your Azure DevOps Service Connection is a Service Principal
- task: AzureCLI@2
inputs:
azureSubscription: $(serviceConnection)
addSpnToEnvironment: true
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
# download kubectl and kubelogin
sudo az aks install-cli --client-version $(kubernetesVersion) --kubelogin-version latest --only-show-errors
# download ~/.kube/config file
az aks get-credentials --resource-group $(aksResourceGroup) --name $(aksClusterName) --only-show-errors
# convert ~/.kube/config to a format compatible with kubelogin
# this stores the SPN's secret directly in the kubeconfig file
kubelogin convert-kubeconfig --login spn --client-id $servicePrincipalId --client-secret $servicePrincipalKey
- task: Bash@3
inputs:
targetType: 'inline'
script: |
kubectl get nodes
- task: AzureCLI@2
inputs:
azureSubscription: $(serviceConnection)
addSpnToEnvironment: true
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
# download kubectl and kubelogin
sudo az aks install-cli --client-version $(kubernetesVersion) --kubelogin-version latest --only-show-errors
# download ~/.kube/config file
az aks get-credentials --resource-group $(aksResourceGroup) --name $(aksClusterName) --only-show-errors
# convert ~/.kube/config to a format compatible with kubelogin
kubelogin convert-kubeconfig --login spn
# create secure azure devops variables for later steps
echo "##vso[task.setvariable variable=spnId;isSecret=true]$servicePrincipalId"
echo "##vso[task.setvariable variable=spnSecret;isSecret=true]$servicePrincipalKey"
# each subsequent task that uses kubelogin will need to set 2 environment variables and populate them with the secure variables we created in the previous task
- task: Bash@3
env:
AAD_SERVICE_PRINCIPAL_CLIENT_ID: $(spnId)
AAD_SERVICE_PRINCIPAL_CLIENT_SECRET: $(spnSecret)
inputs:
targetType: 'inline'
script: |
kubectl get nodes
We are attempting to disable local accounts in our private AKS clusters and have run into this same issue where we are unable to authenticate to our clusters using any of the built-in Azure DevOps pipeline tasks (e.g. HelmDeploy). I read this blog post:
and it made it seem like this should be possible based on this:
AKS can be accessed even when local accounts are disabled.
however later in the Q&A it states:
A: Accessing Kubernetes when AAD RBAC is enabled is unrelated to token creation. To prevent an interactive prompt, we will support kubelogin in a future update and blog post in June.
So I think this is saying that it is not currently possible to authenticate to an AKS cluster with local accounts disabled but there will be an update in June that will support this. Is anybody able to confirm?
I am linking this to some issues filed in the azure-pipelines-tasks repo:
/microsoft/azure-pipelines-tasks#17486 /microsoft/azure-pipelines-tasks#15802 /microsoft/azure-pipelines-tasks#10022
We are attempting to disable local accounts in our private AKS clusters and have run into this same issue where we are unable to authenticate to our clusters using any of the built-in Azure DevOps pipeline tasks (e.g. HelmDeploy). I read this blog post:
and it made it seem like this should be possible based on this:
AKS can be accessed even when local accounts are disabled.
however later in the Q&A it states:
A: Accessing Kubernetes when AAD RBAC is enabled is unrelated to token creation. To prevent an interactive prompt, we will support kubelogin in a future update and blog post in June.
So I think this is saying that it is not currently possible to authenticate to an AKS cluster with local accounts disabled but there will be an update in June that will support this. Is anybody able to confirm?
I can only confirm the work is ongoing, so lets wait together, see whether we can get it in June :=)... I am looking forward to it.
Will it also be available for ADO environments of type Kubernetes? https://learn.microsoft.com/en-us/azure/devops/pipelines/process/environments-kubernetes?view=azure-devops
Or is there an existing way to integrate environments without using Kubernetes service accounts? We're looking for an integrated AAD AKS connection method to use in ADO environments, without the "static" service accounts.
Will it also be available for ADO environments of type Kubernetes? https://learn.microsoft.com/en-us/azure/devops/pipelines/process/environments-kubernetes?view=azure-devops
Or is there an existing way to integrate environments without using Kubernetes service accounts? We're looking for an integrated AAD AKS connection method to use in ADO environments, without the "static" service accounts.
Sorry, I don't know, I am not from AKS team. Lets patiently wait for the update,
Hello,
thanks for this project. Is it foreseen to integration with "service connections" in Azure devops? Our pipelines use kubectl tasks: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/kubernetes?view=azure-devops . These tasks use a "service connection". It would be nice if kubelogin becomes one of the "service connection" choices for kubernetes.