search

Azure / kubelogin

A Kubernetes credential (exec) plugin implementing azure authentication
https://azure.github.io/kubelogin/
MIT License
486 stars 91 forks source link

Azure devops integration #20

Open aelmanaa opened 4 years ago

aelmanaa commented 4 years ago

Hello,

thanks for this project. Is it foreseen to integration with "service connections" in Azure devops? Our pipelines use kubectl tasks: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/kubernetes?view=azure-devops . These tasks use a "service connection". It would be nice if kubelogin becomes one of the "service connection" choices for kubernetes.

weinong commented 4 years ago

@TomGeske @palma21 WDYT?

palma21 commented 4 years ago

@azooinmyluggage ^

shigupt202 commented 4 years ago

@aelmanaa Could you please explain the scenario where you want to use kubelogin as a service connection? I think the three ways of creating a kubernetes service connection in Azure Devops already cover all types of clusters.

aelmanaa commented 4 years ago

@shigupt202 we arleady use "service account option". however this option has the following disadvantages:

Moreover, it seems that kubelogin was created as a response to this feature request https://feedback.azure.com/forums/914020-azure-kubernetes-service-aks/suggestions/35146387-support-non-interactive-login-for-aad-integrated-c?tracking_code=e9effc5721de49e6c717ca62967a20d2

hence it would be nice if in AZure devops, we could directly use a service principal or a managed identity as service connection. Behind the scenes, Azure devops uses kubelogin to authenticate (non-interactively) with the cluster

shigupt202 commented 4 years ago

@aelmanaa Thanks for the feedback! I'll get in touch with the PM's to discuss this feature request. cc: @azooinmyluggage @anraghun

azooinmyluggage commented 4 years ago

Ack. We will work on adding this support post Ignite. Adding @anraghun to take this forward.

aelmanaa commented 4 years ago

thanks @azooinmyluggage

julie-ng commented 3 years ago

In case someone else runs into this problem, this is how I ended up doing it.

It works but I agree with @aelmanaa and would prefer if there were a built-in ADO Task to handle this.

Summary

sub-template just for kubelogin

For more about templates, see https://docs.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops

Note you will also need to set the following variables:

variables:
  appNamespace:            my-namespace
  aksClusterName:          my-aks-cluster
  aksClusterResourceGroup: cluster-rg-not-the-nodes-one

which are used below

# steps/setup-kubelogin.yaml
steps:
- bash: |

    # Download and install
    curl -LO "https://github.com/Azure/kubelogin/releases/download/$(kubeloginVersion)/kubelogin-linux-amd64.zip"
    sudo unzip -j "kubelogin-linux-amd64.zip" -d /usr/local/bin
    rm -f "kubelogin-linux-amd64.zip"
    kubelogin --version
  displayName: kubelogin - install

  # For details, see https://stackoverflow.com/questions/54004007/azure-devop-pipelines-authentication-to-aks-with-azure-ad-rbac-configured
- bash: |

    # First login to Azure
    az login \
      --service-principal \
      --username $AAD_SERVICE_PRINCIPAL_CLIENT_ID \
      --password $AAD_SERVICE_PRINCIPAL_CLIENT_SECRET \
      --tenant $AZ_TENANT_ID

    # Fails if it doesn't exist
    touch .kubeconfig-$(aksClusterName)
    chmod 600 .kubeconfig-$(aksClusterName)

    # Populate kubeconfig
    az aks get-credentials \
      --resource-group $(aksClusterResourceGroup) \
      --name $(aksClusterName) \
      --overwrite-existing \
      --file .kubeconfig-$(aksClusterName)

    # Pass kubeconfig to kubelogin to access k8s API
    kubelogin convert-kubeconfig -l azurecli

    # confirm it works
    kubectl get pods --namespace $(appNamespace)
  displayName: kubelogin - aks-credentials to kubecontext
  env:
    AZ_TENANT_ID:                        $(tenant-id)
    AAD_SERVICE_PRINCIPAL_CLIENT_ID:     $(aks-architect-ci-dev-sp-client-id)
    AAD_SERVICE_PRINCIPAL_CLIENT_SECRET: $(aks-architect-ci-dev-sp-client-secret)
    KUBECONFIG:                          $(Build.SourcesDirectory)/.kubeconfig-$(aksClusterName)

Pipeline excerpt

# …
jobs:
- job: Deploy
  displayName: Deploy
  variables:
  - group: mask-subscription-ids # hack to mask my subscription ids
  - group: aks-credentials-kv-group # the service principal credentials

  steps:
  - template: ../steps/setup-kubelogin.yaml # what we created above

  - bash: |

      # confirm it works
      kubectl get pods --namespace $(appNamespace)

    displayName: "kubectl apply"
    env:
      KUBECONFIG: $(Build.SourcesDirectory)/.kubeconfig-$(aksClusterName)

  - bash: |
      kubelogin remove-tokens
    displayName: kubelogin - clear cache

In this way I only need to repeat the env: KUBECONFIG… part for subsequent steps.

Special thanks to https://stackoverflow.com/questions/54004007/azure-devop-pipelines-authentication-to-aks-with-azure-ad-rbac-configured

julie-ng commented 3 years ago

@aelmanaa Could you please explain the scenario where you want to use kubelogin as a service connection? I think the three ways of creating a kubernetes service connection in Azure Devops already cover all types of clusters.

Use case - I disabled local accounts per https://docs.microsoft.com/en-us/azure/aks/managed-aad#disable-local-accounts-preview

sharebear commented 2 years ago

To be honest this doesn't need a new service connection, there is already an Azure Resource Manager service connection that gets you most of the way, just need to configure kubelogin correctly.

As things stand right now I have no idea what the use case is for the Azure DevOps KubernetesV1 task configured with Azure Resource Manager connection type and useClusterAdmin set to false (the default). I don't understand how the Azure Resource Manager examples in the documentation could possibly work.

nlighten commented 2 years ago

@julie-ng Thank you for the very useful write-up.

When I use an ARM Service Connection with MSI on an agent with kubelogin preinstalled the following just works:

      - task: AzureCLI@2
        displayName: aks non-admin login using kubelogin
        inputs:
          azureSubscription: ${{parameters.serviceConnection}}
          scriptType: bash
          scriptLocation: inlineScript
          inlineScript: |
            az aks get-credentials -n <aks cluster> -g <resource group>
            kubelogin convert-kubeconfig -l azurecli

      - task: AzureCLI@2
        displayName: test kubectl and helm from bash
        continueOnError: true 
        inputs:
          azureSubscription: ${{parameters.serviceConnection}}
          scriptType: bash
          scriptLocation: inlineScript
          inlineScript: |
            kubectl get pods -n <some namespace>
            helm list --namespace <some namespace>

However, if I try to use the HelmDeploy@0 and Kubernetes@1 tasks that effectively do the same as second task with kubectl and helm commands, it does not work and I get (for the helm case):

2022-01-19T20:33:52.7909008Z ##[debug]set helmExitCode=1
2022-01-19T20:33:52.7910461Z Error: Kubernetes cluster unreachable: Get "https://<aks cluster>-a15872ed.hcp.westeurope.azmk8s.io:443/version?timeout=32s": getting credentials: exec: executable kubelogin failed with exit code 1
2022-01-19T20:33:52.7912489Z helm.go:88: [debug] Get "https://<aks cluster>-a15872ed.hcp.westeurope.azmk8s.io:443/version?timeout=32s": getting credentials: exec: executable kubelogin failed with exit code 1
2022-01-19T20:33:52.7915456Z ##[debug]Processed: ##vso[task.setvariable variable=helmExitCode;issecret=false;]1
2022-01-19T20:33:52.7925848Z ##[debug]publishPipelineMetadata=true
2022-01-19T20:33:52.7926838Z Kubernetes cluster unreachable
2022-01-19T20:33:52.7930444Z ##[debug]execResult: {"code":1,"stdout":"","stderr":"Error: failed to get token: expected an empty error but received: Azure CLI Credential: ERROR: Please run 'az login' to setup account.\n\nError: Kubernetes cluster unreachable: Get \"https://<aks cluster>-a15872ed.hcp.westeurope.azmk8s.io:***@v1.2.1/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.2.1/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.2.1/command.go:902\nmain.main\n\thelm.sh/helm/v3/cmd/helm/helm.go:87\nruntime.main\n\truntime/proc.go:225\nruntime.goexit\n\truntime/asm_amd64.s:1371\n"}
2022-01-19T20:33:52.7933131Z helm.sh/helm/v3/pkg/kube.(*Client).IsReachable
2022-01-19T20:33:52.7933638Z ##[debug]task result: Failed
2022-01-19T20:33:52.7934668Z    helm.sh/helm/v3/pkg/kube/client.go:121
2022-01-19T20:33:52.7969750Z ##[error]Error: failed to get token: expected an empty error but received: Azure CLI Credential: ERROR: Please run 'az login' to setup account.
      - task: HelmDeploy@0
        displayName: test HelmDeploy task
        continueOnError: true 
        inputs:
          connectionType: None
          namespace: <some namespace>
          command: list

      - task: Kubernetes@1
        displayName: test Kubernetes task
        continueOnError: true 
        inputs:
          connectionType: None
          command: get
          arguments: pods -n <some namespace>

I created issue microsoft/azure-pipelines-tasks#15802 but I am not sure if they can solve this.

How does kubelogin retrieve the MSI credentials when using azurecli as login mode? Does it need access to certain environment variables or a token file that can explain the difference between the two scenario's?

sharebear commented 2 years ago

@nlighten connectionType: None does not appear to be a valid connection type in the documentation https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/kubernetes?view=azure-devops

I dived into the code a few months back and the Kubernetes and Helm tasks appear to always create their own custom kubeconfig file based upon the defined service connection https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/KubernetesV1/src/clusterconnection.ts#L71 which means that it will ignore the one you created with aks get-credentials

Browsing through the code again, what might work would be taking the kubeconfig that's generated by kubelogin and using that to create a Kubeconfig based service connection https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#kubeconfig-option with the caveat that you'll need to update the service connection any time you rotate the cluster certificates (can probably be automated if you're creating the service connections with terraform).

nlighten commented 2 years ago

@sharebear I agree that None is not a documented connection type but it actually works for both tasks. The kubeconfig file is not created in that case (see https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/HelmDeployV0/src/helm.ts#L41) and a pre-existing kubeconfig file is used.

So if I do an kubectl aks get-credentials with --admin (no kubelogin) using connectionType: None works just fine. Additionally from the logs I can see that it actually uses the kubeconfig created by kubelogin convert-kubeconfig -l azurecli because it tries to use kubelogin for the authentication. It only fails on the authentication step. The puzzle is: why does it fail when helm/kubectl is invoked from the Task but not from the command line.

sharebear commented 2 years ago

@nlighten

I agree that None is not a documented connection type but it actually works for both tasks. The kubeconfig file is not created in that case (see https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/HelmDeployV0/src/helm.ts#L41) and a pre-existing kubeconfig file is used.

TIL ... althought on a quick skim I don't see similar code in the Kubernetes task, the code isn't that easy to follow.

Additionally from the logs I can see that it actually uses the kubeconfig created by kubelogin convert-kubeconfig -l azurecli because it tries to use kubelogin for the authentication. It only fails on the authentication step.

In that case, what's your error message? I had a problem where I had forgotten to actually enable AzureRBAC on the AKS cluster, and it's not very visible in the portal, so might be worth verifying that.

nlighten commented 2 years ago

@sharebear Thank you for thinking with me.

TIL ... althought on a quick skim I don't see similar code in the Kubernetes task, the code isn't that easy to follow.

In the Kubernetes task the switch is here: https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/KubernetesV1/src/clusterconnection.ts#L59

In that case, what's your error message?

The error I get is: 

2022-01-19T20:33:52.7910461Z Error: Kubernetes cluster unreachable: Get "https://<aks cluster>-a15872ed.hcp.westeurope.azmk8s.io:443/version?timeout=32s": getting credentials: exec: executable kubelogin failed with exit code 1

So kubelogin apparently fails with exit code 1. Full task logs can be found at: https://github.com/microsoft/azure-pipelines-tasks/files/7900774/logs.zip

RBAC and AKS-managed AAD is definitely enabled on the cluster. In the pipeline steps listed above I can successfully use the managed identity credentials when using helm and kubectl directly from bash. As far as I know the DevOps Tasks are only a wrapper around the same binaries, but somehow when helm or kubectl are called from the Tasks kubelogin fails. My impression is that cannot get a hold of the managed identity credentials but logging is too limited to pinpoint.

Perhaps the helm and kubernetes tasks do not have access to the ~/.azure/ directory? I will do some testing to see if this directory is removed after the AzureCLI@2 is finished (which would make sense from a security perspective).

nlighten commented 2 years ago

It was indeed caused by AzureCLI@2 doing az account clear at the end. If I use plain bash and an explicit login/logout everything works as expected in combination with a MSI: 

      - task: Bash@3
        displayName: az and aks login
        inputs:
          targetType: 'inline'
          script: |
            az login --identity 
            az aks get-credentials -n <aks cluster> -g <resource group>
            kubelogin convert-kubeconfig -l azurecli

      - task: HelmDeploy@0
        displayName: test HelmDeploy task
        inputs:
          connectionType: None
          namespace: <namespace>
          command: list

      - task: Kubernetes@1
        displayName: test Kubernetes task
        inputs:
          connectionType: None
          command: get
          arguments: pods -n <namespace>

      - task: Bash@3
        condition: always()
        displayName: az and aks logout
        inputs:
          targetType: 'inline'
          script: |
            az account clear
            rm -rf ~/.kube

Still, I would prefer explicit support for kubelogin as part an ARM Service Connection on the Kubernetes and HelmDeploy tasks.

@sharebear Thanks for your time.

sharebear commented 2 years ago

@nlighten Sorry I stopped replying there, got caught up in the day-to-day work.

Great that you were able to work it out, it's helped me develop my understanding too... I went with a "service account" based solution when I disabled the local admin account, but still keeping my eyes open for better solutions.

I do wonder though, as you are using msi, what's the motivation for using azurecli auth in kubelogin instead of using the msi support there? If you just ran kubelogin convert-kubeconfig -l msi would that also have solved the problem?

nlighten commented 2 years ago

@sharebear I actually missed the -l msi option (rtfm). It works and results in a slightly more elegant:

      - task: AzureCLI@2
        displayName: aks non-admin login using kubelogin
        inputs:
          azureSubscription: ${{parameters.serviceConnection}}
          scriptType: bash
          scriptLocation: inlineScript
          inlineScript: |
            az aks get-credentials -n <aks cluster> -g <resource group>
            kubelogin convert-kubeconfig -l msi

      - task: HelmDeploy@0
        displayName: test HelmDeploy task
        inputs:
          connectionType: None
          namespace: <namespace>
          command: list

      - task: Kubernetes@1
        displayName: test Kubernetes task
        inputs:
          connectionType: None
          command: get
          arguments: pods -n <namespace>
sharebear commented 2 years ago

@nlighten Great... now, as far as I understand it, the contents of the kubeconfig after you've run kubelogin convert-kubeconfig -l msi are pretty static and will only change if/when you rotate the cluster certificates.

Does it then make sense to move it into a service connection with kubeconfig so you don't need to generate it every pipeline run? Or is kubelogin only installed when you run get-credentials?

nlighten commented 2 years ago

@sharebear

Does it then make sense to move it into a service connection with kubeconfig so you don't need to generate it every pipeline run?

I would prefer not. We rotate cluster certificates every 3 months and also rebuild clusters quite frequent. With moving it into the Kubernetes Service Connection I would still need to update the Service Connection after each certificate rotation or rebuild.

In my mind something like kubelogin: msi option as part of the HelmDeploy and Kubernetes Task when using connectionType: Azure Resource Manager would be a better option.

sharebear commented 2 years ago

@sharebear

Does it then make sense to move it into a service connection with kubeconfig so you don't need to generate it every pipeline run?

I would prefer not. We rotate cluster certificates every 3 months and also rebuild clusters quite frequent. With moving it into the Kubernetes Service Connection I would still need to update the Service Connection after each certificate rotation or rebuild.

Fair enough. We manage the service connections through terraform https://registry.terraform.io/providers/microsoft/azuredevops/latest/docs/resources/serviceendpoint_kubernetes so I think I could automate this.

In my mind something like kubelogin: msi option as part of the HelmDeploy and Kubernetes Task when using connectionType: Azure Resource Manager would be a better option.

I completely agree on this point, what we're discussing is still a workaround for something that should work out of the box. Especially when it's recommended to disable local accounts which the current implementations appear to depend upon being enabled.

Thanks for the good discussion!

julie-ng commented 2 years ago

@sharebear @nlighten have you considered using gitops? Because you're walking a slippery path and depending on your requirements, from a security perspective, it would be more straight-forward with less gaps to just have the cluster pull in the changes.

That being said, let me explain what concerns me about this discussion. Keep in mind I work for MSFT but am not part of the AKS product group.

kubelogin

In the future, you must use kubelogin to align with k8s upstream. Old methods will be deprecated soon starting with v1.23

re using msi, what's the motivation for using azurecli auth in kubelogin instead of using the msi support there? If you just ran kubelogin convert-kubeconfig -l msi

See AKS public roadmap item here

TL;DR; is that it has to do with upstream Kubernetes moving cloud provider specific code out of core k8s announced in 2019. So simple azure cli will not work anymore. You will need an explicit kubeconfig.

Identity…AKS != Kubernetes

Well not exactly. AKS is an offering where Azure manages your Kubernetes control plane (some of which you do not see in your Azure account). The Kubernetes components that host your workload, e.g. VMs, Load Balancers, etc. are Kubernetes (not Azure). Azure will provide interfaces that make interaction easier, including identity integration.

Managed Identity is an Azure offering and designed for accessing Azure resources. See MSFT Blog - Demystifying Service Principals – Managed Identities.

Technically you're trying to access a Kubernetes resource, not an Azure resource. Because they both speak and understand OIDC/OAuth, it works.

Perhaps the helm and kubernetes tasks do not have access to the ~/.azure/ directory? I will do some testing to see if this directory is removed after the AzureCLI@2 is finished (which would make sense from a security perspective).

K8s and Helm are not Azure and should not have access to ~/.azure. Instead Azure must speak k8s and use kubeconfig. See roadmap link above.

Managed Identities - the slippery part

IMO Managed identities are often misunderstood and people think it's magic without understanding what's happening under the hood - and thus how Shared Responsibility Principle still applies. Keep this in mind

So… @sharebear

Does it then make sense to move it into a service connection with kubeconfig so you don't need to generate it every pipeline run? Or is kubelogin only installed when you run get-credentials?

No, please do not move the kubeconfig into a service connection because it shouldn't exist beyond lifecycles, e.g. pipeline runs. Even though OAuth tokens can be long lived, that's a slippery slope. IMO, in the spirit of OAuth, build agents should re-authenticate and fetch a new access token every time they start a new job.

Service Connections - are just wrappers

They're just wrappers around credentials that allow you to apply some authorization mechanisms, e.g. which pipeline can use this? Deployments should only happen during business hours, etc.

@nlighten

In my mind something like kubelogin: msi option as part of the HelmDeploy and Kubernetes Task when using connectionType: Azure Resource Manager would be a better option.

That's an interesting idea. I'd be curious to know why you use the tasks and not just the bash? Personally I find the tasks (except ARM deployment) limiting and abstractions not useful. If I'm using Terraform, kubectl, etc. I prefer scripting. Also lets me test/code locally and thus make pipelines faster.

sharebear commented 2 years ago

@sharebear @nlighten have you considered using gitops? Because you're walking a slippery path and depending on your requirements, from a security perspective, it would be more straight-forward with less gaps to just have the cluster pull in the changes.

  1. I've inherited a shared pipeline that is used by ~60 developers practicing continuous deployment, it's been stable and working well for them for a long while. While I do want to change this, as there's one big DX feature I can't implement on DevOps Pipelines, I've got lower hanging fruit to pick first. Such as removing local accounts from the clusters.
  2. Assuming practicing Infrastructure as Code for everything (we do) this just gives you a chicken and egg problem. How do you deploy the GitOps tooling? Sure AKS now provides Flux 2 as a plugin, but that's only preview stage and there are multiple stronger competitors.

That being said, let me explain what concerns me about this discussion. Keep in mind I work for MSFT but am not part of the AKS product group.

kubelogin

In the future, you must use kubelogin to align with k8s upstream. Old methods will be deprecated soon starting with v1.23

Exactly, this strengthens the argument that it should be supported out of the box by the DevOps Pipeline tasks provided for interacting with k8s.

re using msi, what's the motivation for using azurecli auth in kubelogin instead of using the msi support there? If you just ran kubelogin convert-kubeconfig -l msi

See AKS public roadmap item here

TL;DR; is that it has to do with upstream Kubernetes moving cloud provider specific code out of core k8s announced in 2019. So simple azure cli will not work anymore. You will need an explicit kubeconfig.

Correct, an this is exactly what we are supporting with the proposed config here.

Identity…AKS != Kubernetes

Well not exactly. AKS is an offering where Azure manages your Kubernetes control plane (some of which you do not see in your Azure account). The Kubernetes components that host your workload, e.g. VMs, Load Balancers, etc. are Kubernetes (not Azure). Azure will provide interfaces that make interaction easier, including identity integration.

Managed Identity is an Azure offering and designed for accessing Azure resources. See MSFT Blog - Demystifying Service Principals – Managed Identities.

Technically you're trying to access a Kubernetes resource, not an Azure resource. Because they both speak and understand OIDC/OAuth, it works.

I really don't see how this is any different to using managed identities for data plane access for Azure Storage, Azure SQL, Azure Cache for Redis, etc. With the AzureRBAC support we can manage all access with one tool instead of constantly wondering whether access is granted in AzureRBAC or K8sRBAC, with potential confusion being a security risk.

Perhaps the helm and kubernetes tasks do not have access to the ~/.azure/ directory? I will do some testing to see if this directory is removed after the AzureCLI@2 is finished (which would make sense from a security perspective).

K8s and Helm are not Azure and should not have access to ~/.azure. Instead Azure must speak k8s and use kubeconfig. See roadmap link above.

Managed Identities - the slippery part

IMO Managed identities are often misunderstood and people think it's magic without understanding what's happening under the hood - and thus how Shared Responsibility Principle still applies. Keep this in mind

  • Managed Identities are still service principals under the hood. It saves you from managing secrets, but secrets still exist, in the form of tokens - although short lived and frequently rotated, they exist and are maybe "lying around" somewhere.
  • The pipeline tasks - and my KUBECONFIG environment variable set to $(Build.SourcesDirectory)/.kubeconfig-$(aksClusterName) are designed to ensure these files don't persist beyond their lifecycles

So… @sharebear

Does it then make sense to move it into a service connection with kubeconfig so you don't need to generate it every pipeline run? Or is kubelogin only installed when you run get-credentials?

No, please do not move the kubeconfig into a service connection because it shouldn't exist beyond lifecycles, e.g. pipeline runs. Even though OAuth tokens can be long lived, that's a slippery slope. IMO, in the spirit of OAuth, build agents should re-authenticate and fetch a new access token every time they start a new job.

Here I think you've misunderstood the contents of the kubeconfig file. There are no tokens stored in the file, just the public cluster root CA certificate and the necessary yaml to tell kubectl which arguments to pass to kubelogin. Nothing secret or sensitive here at all.

Service Connections - are just wrappers

They're just wrappers around credentials that allow you to apply some authorization mechanisms, e.g. which pipeline can use this? Deployments should only happen during business hours, etc.

@nlighten

In my mind something like kubelogin: msi option as part of the HelmDeploy and Kubernetes Task when using connectionType: Azure Resource Manager would be a better option.

That's an interesting idea. I'd be curious to know why you use the tasks and not just the bash? Personally I find the tasks (except ARM deployment) limiting and abstractions not useful. If I'm using Terraform, kubectl, etc. I prefer scripting. Also lets me test/code locally and thus make pipelines faster.

I can't anser for @nlighten but my answer would be;

  1. It feels more idiomatic to use the provided tasks. All MS provided documentation I've seen points to this as being the recommended way to do things. This goes a long way to transferring skils from one workplace to another, these tasks work the same everywhere, you don't need to dig into the details of someone's home-grown bash script.
  2. By using bash you lose all the advantages you just mentioned of providing additional controls per service connection.
  3. While the current discussion has focussed on MSI, this requires self-hosted runners. Not everyone uses them, a solution that properly used the AzureRM service connection would allow usage from MS hosted runners too.
nlighten commented 2 years ago

@julie-ng Thank you for sharing your thoughts. To add to @sharebear feedback:

have you considered using gitops? Yes, but AFAIK this currently only covers K8S resources and not yet Azure resources (not considering Azure Service Operator which does not seem to be production ready yet).

Perhaps to explain a little bit the background of my experiments with kubelogin: we are looking at providing individual subscriptions to product teams. We are considering using a dedicated agent pool per subscription for deployment (no build). This pool would have an MSI associated with it that would allow the the team to deploy to resources within that subscription. AKS is one of the targets, but they (for example) could also do database migrations (CosmosDB/SQL Server), access keyvault etc, all using the same MSI.

K8s and Helm are not Azure and should not have access to ~/.azure. Instead Azure must speak k8s and use kubeconfig. See roadmap link above.

Agreed. This was my mind trying to understand how this worked and doing it wrong ...

I'd be curious to know why you use the tasks and not just the bash?

Personally I prefer the command line, but there is a large developer community and existing pipelines that already use the Tasks and this is also how Microsoft shows them how to do it in the documentation.

In the future, you must use kubelogin to align with k8s upstream. Old methods will be deprecated soon starting with v1.23

Exactly, this strengthens the argument that it should be supported out of the box by the DevOps Pipeline tasks provided for interacting with k8s.

I agree with @sharebear . Perhaps we should open a feature request in the azure-pipelines-tasks project with this as motivation.

julie-ng commented 2 years ago

@sharebear - being pragmatic about your options is practical. I think that way too. But on this part I disagree:

Here I think you've misunderstood the contents of the kubeconfig file. There are no tokens stored in the file, just the public cluster root CA certificate and the necessary yaml to tell kubectl which arguments to pass to kubelogin. Nothing secret or sensitive here at all.

Are you sure? Check your ~/.kube/config and see also https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens

I haven't used all the different methods, for example certs for authentication. But my understanding and my personal kubeconfig is full of secrets.

Re: pipeline tasks - yep all the docs use it and the vast majority of my colleagues use them. I find more verbose than just using the command line. And often they're not stable or documentation of the tasks themselves are weird, i.e. break. Pipelines with bash are cleaner. I tend to hide them in tasks in sub templates. Personal preference. We'll probably just agree to disagree here… :)

@nlighten re: MSI

I used to be an engineer and enterprise architect at Allianz Germany, a multibillion dollar company. Back in ~2019 (not sure what they use now), our team designed and created a Jenkins infrastructure that leveraged k8s pods for build agents. If you can kill a pod after a job is done, you have to worry less about housekeeping after a job.

Using a VM with a longer lifecycle? Be very careful. Housekeeping is not easy. And you then need to understand every toolchain you have on that machine and how it works, including kubectl

Anyhow, I'll probably sign off this thread now as I could rant about this forever. I don't expect or even want you to agree with me. But I'd love for everyone to dig deeper and understand how things work under the hood - esp. for people who have any architecture responsibility.

I'll probably make a YouTube video about this soon. Same discussion came up at work this week.

It's been a fun discussion :)

nlighten commented 2 years ago

Are you sure? Check your ~/.kube/config and see also https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens

@sharebear is correct. The only thing that is in the kubeconfig is the cluster public cert and the config to delegate the token retrieval to kubelogin. For an MSI this looks like:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ******
    server: https://mycluster-a15872ed.hcp.westeurope.azmk8s.io:443
  name: mycluster
contexts:
- context:
    cluster: mycluster
    user: clusterUser_myresourcegroup_mycluster
  name: mycluster
current-context: mycluster
kind: Config
preferences: {}
users:
- name: clusterUser_myresourcegroup_mycluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - get-token
      - --server-id
      - 6dae42f8-****-****-****-3960e28e3630
      - --login
      - msi
      command: kubelogin
      env: null
      provideClusterInfo: false

be careful with managed identities and build agents

Agreed. That is we are considering dedicated 'deployment' agents that cannot be used for builds. Deployment agents are restricted to a single team/subscription. It also gives us more control over the whole 'application supply chain'.

our team designed and created a Jenkins infrastructure that leveraged k8s pods for build agents

That is nowadays fairly simple to setup using AKS + KEDA which has an Azure Pipelines scaler. If you combine it with Pod Identity you can create very fine grained agents pools each with their own MSI (if you want). But in our scenario, an AKS cluster per subscription just for deployment agents is a little bit overkill.

Anyhow, I'll probably sign off this thread now as I could rant about this forever. I don't expect or even want you to agree with me. But I'd love for everyone to dig deeper and understand how things work under the hood - esp. for people who have any architecture responsibility.

You know what happened when the Dwarves dug too greedily and too deep ;-). Good luck with your YouTube video.

julie-ng commented 2 years ago

@nlighten cool, thanks for sharing the example config. I've never used kubectl with msi. I'm overly cautious and just assume certs and tokens are in there :P

Haha, simple? Not really. You'd need a team to maintain it. A large insurance company has those resources, i.e. dedicated teams with dedicated people for maintaining patches, people's feature/software requests, etc. If not, it's overrated in my opinion.

It's a fun ops problem if you're just into ops. If you're like me and prioritize creating business value, I'd question whether if there's another solution that's simpler and just as secure.

Btw, because no one mentioned it… most people don't realize if you use managed identities, you have a ginormous single point of failure…AAD 😬 unlikely, but in my current non pre-sales role, I can tell customers that sorry, it's happened. So make an informed decision.

scottstout commented 2 years ago

Ack. We will work on adding this support post Ignite. Adding @anraghun to take this forward.

@azooinmyluggage @anraghun where did we land with this?

azooinmyluggage commented 2 years ago

Adding @vijayma

larryclaman commented 2 years ago

I've been able to build out a proof of concept to this problem of the ADO kubernetes tasks not supporting kubelogin. Going to put it here and describe how it works. It's based on info gathered from earlier in this discussion.

Prereqs:

And here's the pipeline. I'll describe the components of it below.

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

#trigger:
#- master

pool:
  vmImage: ubuntu-latest

variables:
  azureResourceGroup: myrg
  kubernetesCluster: myakscluster
  useClusterAdmin: false

steps:
- task: AzureCLI@2
  displayName: azcli login and save creds
  inputs:
    azureSubscription: 'myakstestsp'
    scriptType: 'bash'
    scriptLocation: 'inlineScript'
    inlineScript: |
      #Unfortunately azurecli task does a logout at the end, so we need to grab creds and then login from bash
      echo "export servicePrincipalId=$servicePrincipalId" >.env
      echo "export servicePrincipalKey=$servicePrincipalKey" >>.env
      echo "export tenantId=$tenantId" >>.env
    addSpnToEnvironment: true

- bash: |
    source .env
    az login --service-principal -u $servicePrincipalId -p $servicePrincipalKey -t $tenantId
    mkdir -p .bin
    az aks install-cli --install-location .bin/kubectl  # install kubelogin
    az aks get-credentials -n $(kubernetesCluster) -g $(azureResourceGroup)
    kubelogin convert-kubeconfig -l azurecli 
    kubectl get all -A

- task: Kubernetes@1
  inputs:
    # connectionType:None is not documented, but it's in the code https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/KubernetesV1/src/clusterconnection.ts#L59
    # with this setting, the task will just use the kubeconfig context
    connectionType: 'None' 
    command: 'apply'
    useConfigurationFile: true
    configuration: 'myapp.yaml'
    secretType: 'generic'
    forceUpdate: false

- task: HelmDeploy@0
  displayName: Helm deploy
  inputs:
    connectionType: 'None'
    command: 'install'
    chartType: 'FilePath'
    chartPath: 'mydemo'
    waitForExecution: false

- task: KubernetesManifest@0
  inputs:
    action: 'deploy'
    kubernetesServiceConnection: 'myaksserviceconnection'
    namespace: 'default'
    manifests: 'demo2.yaml'

Let's look at each of the tasks in turn:

- task: AzureCLI@2
  displayName: azcli login and save creds
  inputs:
    azureSubscription: 'myakstestsp'
    scriptType: 'bash'
    scriptLocation: 'inlineScript'
    inlineScript: |
      # azurecli task does a logout at the end, so we need to grab creds and then login via az cli in bash in subsequent step
      echo "export servicePrincipalId=$servicePrincipalId" >.env
      echo "export servicePrincipalKey=$servicePrincipalKey" >>.env
      echo "export tenantId=$tenantId" >>.env
    addSpnToEnvironment: true

In the above task, I'm using the azurecli task to leverage the service connection to log into azure. The nuance is that this task executes a logout when it completes, so it's not useful if you want to stay logged in for subsequent tasks. So what I'm doing is grabbing the SP id & key and storing them into a file to be used later. There are certainly other ways of achieving this; this was just a quick and dirty approach.

- bash: |
    source .env
    az login --service-principal -u $servicePrincipalId -p $servicePrincipalKey -t $tenantId
    mkdir -p .bin
    az aks install-cli --install-location .bin/kubectl  # install kubelogin
    az aks get-credentials -n $(kubernetesCluster) -g $(azureResourceGroup)
    kubelogin convert-kubeconfig -l azurecli 
    kubectl get all -A

The next block does a number of things (and could have been split into multiple tasks.)

  1. First it grabs the Id and Key from the prior step, and then uses these to log into Azure.
  2. Next it runs az aks install-cli in order to install the kubelogin tool.
  3. Next it downloads the kubeconfig using az aks get-credentials, and gets an AAD token using kubelogin
  4. Lastly it runs a kubetctl get all -A simply to show that the login worked. (obviously this would be removed in a real scenario)

Next we'll prove out the three Kubernetes-specific ADO tasks:

- task: Kubernetes@1
  inputs:
    # connectionType:None is not documented, but it's in the code https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/KubernetesV1/src/clusterconnection.ts#L59
    # with this setting, the task will just use the kubeconfig context
    connectionType: 'None' 
    command: 'apply'
    useConfigurationFile: true
    configuration: 'myapp.yaml'
    secretType: 'generic'
    forceUpdate: false

This task Kubernetes@1 acts like a wrapper around kubectl. Normally it wants to leverage either an Azure Service Connection or a Kubernetes Service Connection, but in this case, we make use of the undocumented option connectionType:None to force it to use the kubeconfig context.

- task: HelmDeploy@0
  displayName: Helm deploy
  inputs:
    connectionType: 'None'
    command: 'install'
    chartType: 'FilePath'
    chartPath: 'mydemo'
    waitForExecution: false

I take a similar approach for the HelmDeploy@0 task.

- task: KubernetesManifest@0
  inputs:
    action: 'deploy'
    kubernetesServiceConnection: 'myaksserviceconnection'
    namespace: 'default'
    manifests: 'demo2.yaml'

Last is the KubernetesManifest@0 task. This was more of a challenge to get working, as there was no option for connectionType: 'None'. Instead, to get this to work, I needed to create another Service Connection in Azure DevOps, but this new connection needs to be a Kubernetes Service Connection, and specifically, it needs to be of type Kubeconfig. You will need to paste in the appropriate kubeconfig for this cluster that you previously pulled via az aks get-credentials. (Note that this kubeconfig won't work without the associated kubelogin setup). In the example above, you'll see my kuberentesServiceConnection is named myaksserviceconnection

I'm interested in any feedback on this solution.

sergeidavydov commented 2 years ago

Hi @larryclaman,

Essentially, there is no need to export SPN credentials as you can retrieve aks credentials and convert cubeconfig with kubelogin in one go:

- task: AzureCLI@2
  inputs:
    azureSubscription: '$(serviceConnection)'
    scriptType: 'bash'
    scriptLocation: 'inlineScript'
    inlineScript: |
      az aks get-credentials --resource-group $(resourceGroup) --name $(kubernetesCluster)
      kubelogin convert-kubeconfig -l azurecli
  displayName: 'Get cluster credentials'

Could you elaborate on the last task? Does it imply that cubeconfig has to be converted with kubelogin first and then its content to be pasted to the service connection? In this case, the last task is completely independent of the first task, because it already has the same kubeconfig content as the first task produces.

Thanks.

carlin-q-scott commented 1 year ago

@sergeidavydov As mentioned by several people in this thread, AzureCLI@2 runs az account clear at the end of the task, which invalidates the login session that was exported by kubelogin.

davidharkis commented 1 year ago

https://github.com/Azure/kubelogin/pull/142 has blocked some workarounds discussed here.

Following on from the existing AzureCLI@2 task logic mentioned above by @carlin-q-scott, you can no longer run a separate kubelogin task, followed by one or more generic tasks using running kubectl commands or tasks like HelmDeploy@0 with connectionType: 'None'

carlin-q-scott commented 1 year ago

I made and published a template based on the information contained in this issue: https://github.com/carlin-q-scott/azure-devops-pipeline-templates/blob/main/steps/kubectl.yml

My readme includes instructions on using it.

darthmolen commented 1 year ago

AKS used to be easy to integrate into the pipeline, and easy to access. It is no longer so. reading all the heavy work-arounds and less than optimal implementations makes me wonder if anybody is really watching the ecosystem for such intrinsic changes like this that has cascaded all sorts of problems.

Thank you to the people working tirelessly that aren't microsoft and sharing their work-arounds, but still, All I see are work-arounds here.

darthmolen commented 1 year ago

For somebody who DOESN'T have Azure AAD enabled on his cluster and relied on service connections which used to work seamlessly with AKS for integration, what are my options? The service connection holds the spn / secret....

weinong commented 1 year ago

@darthmolen what's not working for you? az aks install-cli?

nnellanspdl commented 1 year ago

I have been playing around with this in Azure Pipelines. Wanted to throw my 2 cents into the ring for some possible solutions. The following solutions will work if your Azure DevOps Service Connection is a Service Principal

Solution 1 - If you don't mind writing your SPN's secret permanently into the the kubeconfig

- task: AzureCLI@2
  inputs:
    azureSubscription: $(serviceConnection)
    addSpnToEnvironment: true
    scriptType: bash
    scriptLocation: inlineScript
    inlineScript: |
      # download kubectl and kubelogin
      sudo az aks install-cli --client-version $(kubernetesVersion) --kubelogin-version latest --only-show-errors

      # download ~/.kube/config file
      az aks get-credentials --resource-group $(aksResourceGroup) --name $(aksClusterName) --only-show-errors

      # convert ~/.kube/config to a format compatible with kubelogin
      # this stores the SPN's secret directly in the kubeconfig file
      kubelogin convert-kubeconfig --login spn --client-id $servicePrincipalId --client-secret $servicePrincipalKey

- task: Bash@3
  inputs:
    targetType: 'inline'
    script: |
      kubectl get nodes

Solution 2 - Do not store secrets in the kubeconfig file

- task: AzureCLI@2
  inputs:
    azureSubscription: $(serviceConnection)
    addSpnToEnvironment: true
    scriptType: bash
    scriptLocation: inlineScript
    inlineScript: |
      # download kubectl and kubelogin
      sudo az aks install-cli --client-version $(kubernetesVersion) --kubelogin-version latest --only-show-errors

      # download ~/.kube/config file
      az aks get-credentials --resource-group $(aksResourceGroup) --name $(aksClusterName) --only-show-errors

      # convert ~/.kube/config to a format compatible with kubelogin
      kubelogin convert-kubeconfig --login spn

      # create secure azure devops variables for later steps
      echo "##vso[task.setvariable variable=spnId;isSecret=true]$servicePrincipalId"
      echo "##vso[task.setvariable variable=spnSecret;isSecret=true]$servicePrincipalKey"

# each subsequent task that uses kubelogin will need to set 2 environment variables and populate them with the secure variables we created in the previous task
- task: Bash@3
  env:
    AAD_SERVICE_PRINCIPAL_CLIENT_ID: $(spnId)
    AAD_SERVICE_PRINCIPAL_CLIENT_SECRET: $(spnSecret)
  inputs:
    targetType: 'inline'
    script: |
      kubectl get nodes
chandlerkent commented 1 year ago

We are attempting to disable local accounts in our private AKS clusters and have run into this same issue where we are unable to authenticate to our clusters using any of the built-in Azure DevOps pipeline tasks (e.g. HelmDeploy). I read this blog post:

https://devblogs.microsoft.com/devops/service-connection-guidance-for-aks-customers-using-kubernetes-tasks/

and it made it seem like this should be possible based on this:

AKS can be accessed even when local accounts are disabled.

however later in the Q&A it states:

A: Accessing Kubernetes when AAD RBAC is enabled is unrelated to token creation. To prevent an interactive prompt, we will support kubelogin in a future update and blog post in June.

So I think this is saying that it is not currently possible to authenticate to an AKS cluster with local accounts disabled but there will be an update in June that will support this. Is anybody able to confirm?

chandlerkent commented 1 year ago

I am linking this to some issues filed in the azure-pipelines-tasks repo:

/microsoft/azure-pipelines-tasks#17486 /microsoft/azure-pipelines-tasks#15802 /microsoft/azure-pipelines-tasks#10022

BenoityipMSFT commented 1 year ago

We are attempting to disable local accounts in our private AKS clusters and have run into this same issue where we are unable to authenticate to our clusters using any of the built-in Azure DevOps pipeline tasks (e.g. HelmDeploy). I read this blog post:

https://devblogs.microsoft.com/devops/service-connection-guidance-for-aks-customers-using-kubernetes-tasks/

and it made it seem like this should be possible based on this:

AKS can be accessed even when local accounts are disabled.

however later in the Q&A it states:

A: Accessing Kubernetes when AAD RBAC is enabled is unrelated to token creation. To prevent an interactive prompt, we will support kubelogin in a future update and blog post in June.

So I think this is saying that it is not currently possible to authenticate to an AKS cluster with local accounts disabled but there will be an update in June that will support this. Is anybody able to confirm?

I can only confirm the work is ongoing, so lets wait together, see whether we can get it in June :=)... I am looking forward to it.

lopf commented 1 year ago

Will it also be available for ADO environments of type Kubernetes? https://learn.microsoft.com/en-us/azure/devops/pipelines/process/environments-kubernetes?view=azure-devops

Or is there an existing way to integrate environments without using Kubernetes service accounts? We're looking for an integrated AAD AKS connection method to use in ADO environments, without the "static" service accounts.

BenoityipMSFT commented 1 year ago

Will it also be available for ADO environments of type Kubernetes? https://learn.microsoft.com/en-us/azure/devops/pipelines/process/environments-kubernetes?view=azure-devops

Or is there an existing way to integrate environments without using Kubernetes service accounts? We're looking for an integrated AAD AKS connection method to use in ADO environments, without the "static" service accounts.

Sorry, I don't know, I am not from AKS team. Lets patiently wait for the update,