Azure / azure-sdk-for-python

This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/python/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-python.
MIT License
4.53k stars 2.76k forks source link

Azure Credentials seem to be extremely flaky #37167

Open felixnext opened 2 weeks ago

felixnext commented 2 weeks ago

Packages:

azure-appconfiguration==1.1.1
    # via azure-cli
azure-batch==14.2.0
    # via azure-cli
azure-cli==2.63.0
    # via -r third-party/req_azure.txt
azure-cli-core==2.63.0
    # via azure-cli
azure-cli-telemetry==1.1.0
    # via azure-cli-core
azure-common==1.1.28
    # via
    #   azure-batch
    #   azure-graphrbac
    #   azure-keyvault-administration
    #   azure-keyvault-certificates
    #   azure-keyvault-keys
    #   azure-keyvault-secrets
    #   azure-mgmt-advisor
    #   azure-mgmt-apimanagement
    #   azure-mgmt-appconfiguration
    #   azure-mgmt-appcontainers
    #   azure-mgmt-applicationinsights
    #   azure-mgmt-authorization
    #   azure-mgmt-batch
    #   azure-mgmt-batchai
    #   azure-mgmt-billing
    #   azure-mgmt-botservice
    #   azure-mgmt-cdn
    #   azure-mgmt-cognitiveservices
    #   azure-mgmt-compute
    #   azure-mgmt-containerinstance
    #   azure-mgmt-containerregistry
    #   azure-mgmt-containerservice
    #   azure-mgmt-cosmosdb
    #   azure-mgmt-databoxedge
    #   azure-mgmt-datamigration
    #   azure-mgmt-devtestlabs
    #   azure-mgmt-dns
    #   azure-mgmt-eventgrid
    #   azure-mgmt-eventhub
    #   azure-mgmt-extendedlocation
    #   azure-mgmt-hdinsight
    #   azure-mgmt-imagebuilder
    #   azure-mgmt-iotcentral
    #   azure-mgmt-iothub
    #   azure-mgmt-iothubprovisioningservices
    #   azure-mgmt-keyvault
    #   azure-mgmt-kusto
    #   azure-mgmt-loganalytics
    #   azure-mgmt-managedservices
    #   azure-mgmt-managementgroups
    #   azure-mgmt-maps
    #   azure-mgmt-marketplaceordering
    #   azure-mgmt-media
    #   azure-mgmt-monitor
    #   azure-mgmt-msi
    #   azure-mgmt-netapp
    #   azure-mgmt-policyinsights
    #   azure-mgmt-privatedns
    #   azure-mgmt-rdbms
    #   azure-mgmt-recoveryservices
    #   azure-mgmt-recoveryservicesbackup
    #   azure-mgmt-redhatopenshift
    #   azure-mgmt-redis
    #   azure-mgmt-resource
    #   azure-mgmt-search
    #   azure-mgmt-security
    #   azure-mgmt-servicebus
    #   azure-mgmt-servicefabric
    #   azure-mgmt-servicefabricmanagedclusters
    #   azure-mgmt-servicelinker
    #   azure-mgmt-signalr
    #   azure-mgmt-sql
    #   azure-mgmt-sqlvirtualmachine
    #   azure-mgmt-storage
    #   azure-mgmt-synapse
    #   azure-mgmt-trafficmanager
    #   azure-mgmt-web
    #   azure-multiapi-storage
    #   azure-search-documents
    #   azure-storage-common
    #   azure-synapse-accesscontrol
    #   azure-synapse-artifacts
    #   azure-synapse-managedprivateendpoints
    #   azure-synapse-spark
azure-core==1.30.2
    # via
    #   -r third-party/req_azure.txt
    #   azure-appconfiguration
    #   azure-data-tables
    #   azure-eventgrid
    #   azure-eventhub
    #   azure-identity
    #   azure-keyvault-administration
    #   azure-keyvault-certificates
    #   azure-keyvault-keys
    #   azure-keyvault-secrets
    #   azure-mgmt-core
    #   azure-monitor-query
    #   azure-multiapi-storage
    #   azure-search-documents
    #   azure-storage-blob
    #   azure-storage-file-datalake
    #   azure-synapse-accesscontrol
    #   azure-synapse-managedprivateendpoints
    #   azure-synapse-spark
    #   msrest
    #   opencensus-ext-azure
azure-cosmos==3.2.0
    # via
    #   -r third-party/req_azure.txt
    #   azure-cli
azure-data-tables==12.4.0
    # via
    #   -r third-party/req_azure.txt
    #   azure-cli
azure-datalake-store==0.0.53
    # via azure-cli
azure-eventgrid==4.20.0
    # via -r third-party/req_azure.txt
azure-eventhub==5.12.1
    # via -r third-party/req_azure.txt
azure-functions==1.20.0
    # via
    #   -r third-party/req_azure.txt
    #   functown
azure-graphrbac==0.60.0
    # via azure-cli
azure-identity==1.17.1
    # via
    #   -r third-party/req_azure.txt
    #   opencensus-ext-azure
azure-keyvault-administration==4.4.0b2
    # via azure-cli
azure-keyvault-certificates==4.7.0
    # via azure-cli
azure-keyvault-keys==4.9.0b3
    # via azure-cli
azure-keyvault-secrets==4.7.0
    # via
    #   -r third-party/req_azure.txt
    #   azure-cli
azure-mgmt-advisor==9.0.0
    # via azure-cli
azure-mgmt-apimanagement==4.0.0
    # via azure-cli
azure-mgmt-appconfiguration==3.0.0
    # via azure-cli
azure-mgmt-appcontainers==2.0.0
    # via azure-cli
azure-mgmt-applicationinsights==1.0.0
    # via azure-cli
azure-mgmt-authorization==4.0.0
    # via azure-cli
azure-mgmt-batch==17.3.0
    # via azure-cli
azure-mgmt-batchai==7.0.0b1
    # via azure-cli
azure-mgmt-billing==6.0.0
    # via azure-cli
azure-mgmt-botservice==2.0.0
    # via azure-cli
azure-mgmt-cdn==12.0.0
    # via azure-cli
azure-mgmt-cognitiveservices==13.5.0
    # via azure-cli
azure-mgmt-compute==31.0.0
    # via azure-cli
azure-mgmt-containerinstance==10.1.0
    # via azure-cli
azure-mgmt-containerregistry==10.3.0
    # via azure-cli
azure-mgmt-containerservice==31.0.0
    # via azure-cli
azure-mgmt-core==1.4.0
    # via
    #   azure-cli-core
    #   azure-mgmt-advisor
    #   azure-mgmt-apimanagement
    #   azure-mgmt-appconfiguration
    #   azure-mgmt-appcontainers
    #   azure-mgmt-applicationinsights
    #   azure-mgmt-authorization
    #   azure-mgmt-batch
    #   azure-mgmt-batchai
    #   azure-mgmt-billing
    #   azure-mgmt-botservice
    #   azure-mgmt-cdn
    #   azure-mgmt-cognitiveservices
    #   azure-mgmt-compute
    #   azure-mgmt-containerinstance
    #   azure-mgmt-containerregistry
    #   azure-mgmt-containerservice
    #   azure-mgmt-cosmosdb
    #   azure-mgmt-databoxedge
    #   azure-mgmt-datamigration
    #   azure-mgmt-dns
    #   azure-mgmt-eventgrid
    #   azure-mgmt-eventhub
    #   azure-mgmt-extendedlocation
    #   azure-mgmt-hdinsight
    #   azure-mgmt-imagebuilder
    #   azure-mgmt-iotcentral
    #   azure-mgmt-iothub
    #   azure-mgmt-iothubprovisioningservices
    #   azure-mgmt-keyvault
    #   azure-mgmt-loganalytics
    #   azure-mgmt-managementgroups
    #   azure-mgmt-maps
    #   azure-mgmt-marketplaceordering
    #   azure-mgmt-media
    #   azure-mgmt-monitor
    #   azure-mgmt-msi
    #   azure-mgmt-netapp
    #   azure-mgmt-policyinsights
    #   azure-mgmt-privatedns
    #   azure-mgmt-rdbms
    #   azure-mgmt-recoveryservices
    #   azure-mgmt-recoveryservicesbackup
    #   azure-mgmt-redhatopenshift
    #   azure-mgmt-redis
    #   azure-mgmt-resource
    #   azure-mgmt-search
    #   azure-mgmt-security
    #   azure-mgmt-servicebus
    #   azure-mgmt-servicefabric
    #   azure-mgmt-servicefabricmanagedclusters
    #   azure-mgmt-servicelinker
    #   azure-mgmt-signalr
    #   azure-mgmt-sql
    #   azure-mgmt-sqlvirtualmachine
    #   azure-mgmt-storage
    #   azure-mgmt-synapse
    #   azure-mgmt-trafficmanager
    #   azure-mgmt-web
    #   azure-synapse-artifacts
azure-mgmt-cosmosdb==9.5.1
    # via
    #   -r third-party/req_azure.txt
    #   azure-cli
azure-mgmt-databoxedge==1.0.0
    # via azure-cli
azure-mgmt-datamigration==10.0.0
    # via azure-cli
azure-mgmt-devtestlabs==4.0.0
    # via azure-cli
azure-mgmt-dns==8.0.0
    # via azure-cli
azure-mgmt-eventgrid==10.2.0b2
    # via azure-cli
azure-mgmt-eventhub==10.1.0
    # via azure-cli
azure-mgmt-extendedlocation==1.0.0b2
    # via azure-cli
azure-mgmt-hdinsight==9.0.0
    # via azure-cli
azure-mgmt-imagebuilder==1.3.0
    # via azure-cli
azure-mgmt-iotcentral==10.0.0b2
    # via azure-cli
azure-mgmt-iothub==3.0.0
    # via azure-cli
azure-mgmt-iothubprovisioningservices==1.1.0
    # via azure-cli
azure-mgmt-keyvault==10.3.0
    # via azure-cli
azure-mgmt-kusto==0.3.0
    # via azure-cli
azure-mgmt-loganalytics==13.0.0b4
    # via azure-cli
azure-mgmt-managedservices==1.0.0
    # via azure-cli
azure-mgmt-managementgroups==1.0.0
    # via azure-cli
azure-mgmt-maps==2.0.0
    # via azure-cli
azure-mgmt-marketplaceordering==1.1.0
    # via azure-cli
azure-mgmt-media==9.0.0
    # via azure-cli
azure-mgmt-monitor==5.0.1
    # via azure-cli
azure-mgmt-msi==7.0.0
    # via azure-cli
azure-mgmt-netapp==10.1.0
    # via azure-cli
azure-mgmt-policyinsights==1.1.0b4
    # via azure-cli
azure-mgmt-privatedns==1.0.0
    # via azure-cli
azure-mgmt-rdbms==10.2.0b17
    # via azure-cli
azure-mgmt-recoveryservices==3.0.0
    # via azure-cli
azure-mgmt-recoveryservicesbackup==9.1.0
    # via azure-cli
azure-mgmt-redhatopenshift==1.4.0
    # via azure-cli
azure-mgmt-redis==14.3.0
    # via azure-cli
azure-mgmt-resource==23.1.1
    # via azure-cli
azure-mgmt-search==9.2.0b2
    # via azure-cli
azure-mgmt-security==6.0.0
    # via azure-cli
azure-mgmt-servicebus==8.2.0
    # via azure-cli
azure-mgmt-servicefabric==2.1.0
    # via azure-cli
azure-mgmt-servicefabricmanagedclusters==2.0.0b6
    # via azure-cli
azure-mgmt-servicelinker==1.2.0b2
    # via azure-cli
azure-mgmt-signalr==2.0.0b1
    # via azure-cli
azure-mgmt-sql==4.0.0b17
    # via azure-cli
azure-mgmt-sqlvirtualmachine==1.0.0b5
    # via azure-cli
azure-mgmt-storage==21.2.0
    # via azure-cli
azure-mgmt-synapse==2.1.0b5
    # via azure-cli
azure-mgmt-trafficmanager==1.0.0
    # via azure-cli
azure-mgmt-web==7.2.0
    # via azure-cli
azure-monitor-query==1.2.0
    # via azure-cli
azure-multiapi-storage==1.2.0
    # via azure-cli
azure-search-documents==11.6.0b4
    # via -r third-party/req_azure.txt
azure-storage-blob==12.21.0
    # via azure-storage-file-datalake
azure-storage-common==1.4.2
    # via azure-cli
azure-storage-file-datalake==12.16.0
    # via -r third-party/req_azure.txt
azure-synapse-accesscontrol==0.5.0
    # via azure-cli
azure-synapse-artifacts==0.19.0
    # via azure-cli
azure-synapse-managedprivateendpoints==0.4.0
    # via azure-cli
azure-synapse-spark==0.2.0

Describe the bug I am trying to retrieve an Azure credential for connecting to key-vault. I have tried DefaultCredential, AzureCliCredential, etc. But always get the following error:

WARNING:azure.identity._credentials.chained:DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
    EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot this issue.
    ManagedIdentityCredential: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint.
    SharedTokenCacheCredential: SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache.
    AzureCliCredential: Failed to invoke the Azure CLI
    AzurePowerShellCredential: Failed to invoke PowerShell
    AzureDeveloperCliCredential: Azure Developer CLI could not be found. Please visit https://aka.ms/azure-dev for installation instructions and then,once installed, authenticate to your Azure account using 'azd auth login'.
To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
ERROR:root:Failed to setup config: DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
    EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot this issue.
    ManagedIdentityCredential: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint.
    SharedTokenCacheCredential: SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache.
    AzureCliCredential: Failed to invoke the Azure CLI
    AzurePowerShellCredential: Failed to invoke PowerShell
    AzureDeveloperCliCredential: Azure Developer CLI could not be found. Please visit https://aka.ms/azure-dev for installation instructions and then,once installed, authenticate to your Azure account using 'azd auth login'.
To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.

Strange thing is it works in 2% of the cases (roughly every 50 times I execute the code). Also InteractiveBrowser is working, but annoying for this usecase. It worked 2 days ago, so it seems like a regression (with no changes to version, etc). Since then updated (to Azure CLI 2.63.0) with no effect. Running az cli directly works without problems and I can interact with the keyvault. So the problem here really seems that for some reason the Python SDK cannot interact with the installed version of azure cli.

To Reproduce Steps to reproduce the behavior:

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Additional context Add any other context about the problem here.

felixnext commented 2 weeks ago

Also tried to reinstall the Azure CLI with no effect

xiangyan99 commented 1 week ago

Thanks for reaching out.

DefaultAzureCredential attempts a chain of credentials.

Could you share the information which credential did you expect to work?

github-actions[bot] commented 1 week ago

Hi @felixnext. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

felixnext commented 1 week ago

Hi @xiangyan99 as mentioned above I would expect the CLI Credential to work. I also tried it separately. But it only works every once in a while. Meanwhile az cli is updated to latest version and works when accessing the keyvault directly with it.

xiangyan99 commented 1 week ago

Could you share your code snippet?

github-actions[bot] commented 1 week ago

Hi @felixnext. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

felixnext commented 1 week ago

Sure. I have a function that triggeres the credentials:

from azure.identity import (
    DefaultAzureCredential,
    InteractiveBrowserCredential,
)

def get_credential(allow_browser: bool = True) -> DefaultAzureCredential:
    return DefaultAzureCredential(
        additionally_allowed_tenants=["*"],
        # NOTE: this is due to bug in azure-sdk-for-python
        # see: https://github.com/Azure/azure-sdk-for-python/issues/37167
        exclude_interactive_browser_credential=not allow_browser,
    )

Then they are used, for example with secret handler like this:

from azure.keyvault.secrets import SecretClient
from azure.core.exceptions import ResourceNotFoundError

class FooBar
  def __init__(self, vault_name):
    # ....
    self.credential = get_credential()
    self.client = SecretClient(
                vault_url=f"https://{vault_name}.vault.azure.net",
                credential=self.credential,
            )

  def has_secret(self, name: str) -> bool:
      try:
          return self.client.get_secret(name) is not None
      except ResourceNotFoundError:
          return False
xiangyan99 commented 1 week ago

Can you try run "az account get-access-token --output json --resource https://vault.azure.net" in the command windows and share the output? (please sanitize sensitive data)

github-actions[bot] commented 1 week ago

Hi @felixnext. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

felixnext commented 1 week ago

Here:

{
  "accessToken": "ey*************",
  "expiresOn": "2024-09-05 22:19:56.000000",
  "expires_on": 1725567596,
  "subscription": "************************************",
  "tenant": "************************************",
  "tokenType": "Bearer"
}
xiangyan99 commented 1 week ago

Thank you.

Is it possible because the process needs more time?

Can you try AzureCliCredential(process_timeout=60)

github-actions[bot] commented 1 week ago

Hi @felixnext. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

felixnext commented 1 week ago

Thanks @xiangyan99 that seems to solve the problem. May I raise a feature request of making the error around the CLI more descriptive?

The proposal would be that it should highlight the timeout and suggest using the process_timeout parameter. So instead of:

AzureCliCredential: Failed to invoke the Azure CLI

It would be:

AzureCliCredential: Timeout while invoking Azure CLI. (Use `process_timeout` to increase timeout duration)
felixnext commented 5 days ago

@xiangyan99 As an update to this: I set the timer up to 30sec. It now works roughly 50% of the time. Setting it to 60sec improves that (but still some misses). Is there a way to diagnose underlying problems here? (My internet connection is at 250mbps, so do not think this is the issue). It would be great to fix this speed bump (as it takes a long time now to startup services for local debugging) as an agility enabler.

xiangyan99 commented 5 days ago

Thank you for the update. Based on what you're describing, the issue likely isn’t related to your network connection. The Python code starts a subprocess, so the underlying problem is more likely due to your machine's performance. If your machine is slow or there are multiple resource-intensive processes running, it could lead to delays. This is especially true if you're running the code in a VM that shares resources with other processes. Checking your CPU usage and system load when the issue occurs might help identify the cause.

github-actions[bot] commented 5 days ago

Hi @felixnext. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

felixnext commented 2 days ago

@xiangyan99 Hm. Okay. That lowers dev-agility on the CLI a bit. Is there ongoing work to make CLI less resource hungry? And loop back on the feature request for a more expressive error message, that would help a lot here I think.

Another thing worse considering is dynamic timeouts. Not sure how much engineering effort it would be to retrieve current system load and make the timeout adaptive to that (and allowing users to set a flag for that)? (psutil might be a good lib for that)