Run kubectl command encountered failed get token issue by Azure CLI

[iop45636]

Describe the bug

We are using AKS, which is also integrated with the Azure Private DNS Zone. We can confirm there is no network blocking, as we tested the network connection between the AKS Cluster and the Private DNS Zone.

When we accessed the cluster using the AKS connect command and tried to perform some operations on AKS with kubectl, we encountered an error indicating that kubectl could not retrieve the Azure CLI credentials.

Currently, this issue is blocking our team from performing some operational tasks, although occasionally we can run them without any issues.

We also raised a ticket with Microsoft support, and they suggested that the issue might be related to the Azure CLI. Therefore, I raised the question on this public GitHub repository.

Related command

What actions I performed:

No error for theses commands: az login az account set --subscription az aks get-credentials --resource-group --name --overwrite-existing kubelogin convert-kubeconfig -l azurecli

Not work commands: kubectl get pod -A kubectl get ns

Errors

Error: failed to get token: expected an empty error but received: AzureCLICredential: exit status 1 Unable to connect to the server: getting credentials: exec: executable kubelogin failed with exit code 1

Issue script & Debug output

Error: failed to get token: expected an empty error but received: AzureCLICredential: exit status 1 Unable to connect to the server: getting credentials: exec: executable kubelogin failed with exit code 1

Expected behavior

Can output information by kubectl

Environment Summary

azure-cli 2.63.0

core 2.63.0 telemetry 1.1.0

Dependencies: msal 1.30.0 azure-mgmt-resource 23.1.1

Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe' Extensions directory 'C:\Users\UserNameA.azure\cliextensions'

Python (Windows) 3.11.8 (tags/v3.11.8:db85d51, Feb 6 2024, 22:03:32) [MSC v.1937 64 bit (AMD64)]

Legal docs and information: aka.ms/AzureCliLegal

PS C:\Users\UserbNameA> kubectl version Client Version: v1.28.7 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.3

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[github-actions[bot]]

Here are some similar issues that might help you. Please check if they can solve your problem.

• 26908

[microsoft-github-policy-service[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @dyu1208, @FumingZhang, @andyliuliming.

[microsoft-github-policy-service[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @josephkwchan, @jennyhunter-msft.

[FumingZhang]

Hey @iop45636, based on the information you provided, it's hard to tell why the command cannot obtain the token. You may need to provide the kubeconfig files before and after the conversion, and the logs of the az aks get-credentials/kubelogin/kubectl commands. These are sensitive information, so please do not provide them in this issue, contact support instead.

[JoeyC-Dev]

I faced this issue before, not sure why conversion is failed, but my workaround is:

1. DO NOT execute kubelogin convert-kubeconfig -l azurecli
2. Execute kubelogin remove-tokens
3. Use az aks get-credentials
4. Try any kubectl command, it will ask you to log in with Azure account (if AKS configured with AAD method)
5. Good to go.

If this works, you probably want to open issue here, cuz it looks like conversion issue: https://github.com/Azure/kubelogin