ImdsCredential.get_token failed: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint.

[vanlanchoy]

• Package Name: azure-identity
• Package Version: 1.7.1
• Operating System: linux
• Python Version: 3.10.16

Describe the bug ImdsCredential.get_token failed: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint.

To Reproduce Steps to reproduce the behavior:

1. Called ManagedIdentitfyCredential get_token() to Dynamics-365, and it will says IMDS endpoint not available.

Expected behavior a token string returned by the service endpoint.

Screenshots

Additional context

---------------------------------------------------------------------------
ServiceRequestError                       Traceback (most recent call last)
File ~/cluster-env/clonedenv/lib/python3.10/site-packages/azure/identity/_credentials/imds.py:80, in ImdsCredential._request_token(self, *scopes, **kwargs)
     79 try:
---> 80     self._client.request_token(*scopes, connection_timeout=0.3, retry_total=0)
     81     self._endpoint_available = True

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/azure/identity/_internal/managed_identity_client.py:123, in ManagedIdentityClient.request_token(self, *scopes, **kwargs)
    122 request_time = int(time.time())
--> 123 response = self._pipeline.run(request, retry_on_methods=[request.method], **kwargs)
    124 token = self._process_response(response, request_time)

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/azure/core/pipeline/_base.py:211, in Pipeline.run(self, request, **kwargs)
    206 first_node = (
    207     self._impl_policies[0]
    208     if self._impl_policies
    209     else _TransportRunner(self._transport)
    210 )
--> 211 return first_node.send(pipeline_request)

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/azure/core/pipeline/_base.py:71, in _SansIOHTTPPolicyRunner.send(self, request)
     70 try:
---> 71     response = self.next.send(request)
     72 except Exception:  # pylint: disable=broad-except

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/azure/core/pipeline/_base.py:71, in _SansIOHTTPPolicyRunner.send(self, request)
     70 try:
---> 71     response = self.next.send(request)
     72 except Exception:  # pylint: disable=broad-except

    [... skipping similar frames: _SansIOHTTPPolicyRunner.send at line 71 (1 times)]

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/azure/core/pipeline/_base.py:71, in _SansIOHTTPPolicyRunner.send(self, request)
     70 try:
---> 71     response = self.next.send(request)
     72 except Exception:  # pylint: disable=broad-except

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/azure/core/pipeline/policies/_retry.py:467, in RetryPolicy.send(self, request)
    466             continue
--> 467     raise err
    468 finally:

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/azure/core/pipeline/policies/_retry.py:445, in RetryPolicy.send(self, request)
    444 self._configure_timeout(request, absolute_timeout, is_response_error)
--> 445 response = self.next.send(request)
    446 if self.is_retry(retry_settings, response):

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/azure/core/pipeline/_base.py:71, in _SansIOHTTPPolicyRunner.send(self, request)
     70 try:
---> 71     response = self.next.send(request)
     72 except Exception:  # pylint: disable=broad-except

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/azure/core/pipeline/_base.py:71, in _SansIOHTTPPolicyRunner.send(self, request)
     70 try:
---> 71     response = self.next.send(request)
     72 except Exception:  # pylint: disable=broad-except

    [... skipping similar frames: _SansIOHTTPPolicyRunner.send at line 71 (1 times)]

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/azure/core/pipeline/_base.py:71, in _SansIOHTTPPolicyRunner.send(self, request)
     70 try:
---> 71     response = self.next.send(request)
     72 except Exception:  # pylint: disable=broad-except

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/azure/core/pipeline/_base.py:103, in _TransportRunner.send(self, request)
     94 """HTTP transport send method.
     95 
     96 :param request: The PipelineRequest object.
   (...)
     99 :rtype: ~azure.core.pipeline.PipelineResponse
    100 """
    101 return PipelineResponse(
    102     request.http_request,
--> 103     self._sender.send(request.http_request, **request.context.options),
    104     context=request.context,
    105 )

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/azure/core/pipeline/transport/_requests_basic.py:360, in RequestsTransport.send(self, request, **kwargs)
    359 if error:
--> 360     raise error
    361 if _is_rest(request):

ServiceRequestError: (<urllib3.connection.HTTPConnection object at 0x7f902dfc0c10>, 'Connection to 169.254.169.254 timed out. (connect timeout=0.3)')

The above exception was the direct cause of the following exception:

CredentialUnavailableError                Traceback (most recent call last)
Cell In [11], line 26
     23 credential = ManagedIdentityCredential(client_id = uami_client_id, client_credential = None, authority = authority_2019_update)    
     24 #scope = "https://graph.microsoft.com/.default"
     25 #credential = DefaultAzureCredential()
---> 26 token = credential.get_token(scope)
     27 print(token)

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/azure/identity/_internal/decorators.py:30, in log_get_token.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
     27 @functools.wraps(fn)
     28 def wrapper(*args, **kwargs):
     29     try:
---> 30         token = fn(*args, **kwargs)
     31         _LOGGER.log(
     32             logging.DEBUG if within_credential_chain.get() else logging.INFO, "%s succeeded", qualified_name
     33         )
     34         return token

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/azure/identity/_credentials/managed_identity.py:119, in ManagedIdentityCredential.get_token(self, *scopes, **kwargs)
    112 if not self._credential:
    113     raise CredentialUnavailableError(
    114         message="No managed identity endpoint found. \n"
    115                 "The Target Azure platform could not be determined from environment variables. \n"
    116                 "Visit https://aka.ms/azsdk/python/identity/managedidentitycredential/troubleshoot to "
    117                 "troubleshoot this issue."
    118     )
--> 119 return self._credential.get_token(*scopes, **kwargs)

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/azure/identity/_internal/get_token_mixin.py:76, in GetTokenMixin.get_token(self, *scopes, **kwargs)
     74 if not token:
     75     self._last_request_time = int(time.time())
---> 76     token = self._request_token(*scopes, **kwargs)
     77 elif self._should_refresh(token):
     78     try:

File ~/cluster-env/clonedenv/lib/python3.10/site-packages/azure/identity/_credentials/imds.py:91, in ImdsCredential._request_token(self, *scopes, **kwargs)
     87         self._endpoint_available = False
     88         self._error_message = (
     89             "ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint."
     90         )
---> 91         six.raise_from(CredentialUnavailableError(self._error_message), ex)
     93 if not self._endpoint_available:
     94     raise CredentialUnavailableError(self._error_message)

File <string>:3, in raise_from(value, from_value)

CredentialUnavailableError: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint.

[github-actions[bot]]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[pvaneck]

Hi, @vanlanchoy. It appears you are using a fairly old version of azure-identity. Is it possible for you to upgrade to the latest with pip install -U azure-identity and then retry?

Also, what Azure hosting environment/service are you running this code on? ManagedIdentityCredential only works in certain Azure hosts with Managed Identity enabled.

[github-actions[bot]]

Hi @vanlanchoy. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[vanlanchoy]

Hi @pvaneck , I've tried upgraded my Azure.Identity module to 1.16.0 msal-1.28.0, and I'm running my code in Azure Synapse Workspace and still getting the same error. Does the module currently supporting UAMI/SAMI?

[pvaneck]

Based on the docs here: https://learn.microsoft.com/azure/synapse-analytics/synapse-service-identity#user-assigned-managed-identity, "User-assigned Managed Identity is not currently supported in Synapse notebooks and Spark job definitions.". I'm not sure why this is the case, but this is the likely cause of the UAMI Credential being unable to work. Try using system-assigned MI, and see if you encounter the same problem.

[vanlanchoy]

I am seeing the same error for SAMI scenario as well.

[pvaneck]

Hmm, @xiangyan99 do you know if Managed Identity authentication is supported through our SDKs in Synapse Notebooks? I feel like it might not be as it doesn't seem like the IMDS endpoint is accessible from within the notebook/Spark pool, and we don't do any specific checks for Synapse specific environment variable within ManagedIdentityCredential...

You'd probably have to use service principals (ClientSecretCredential/EnvironmentCredential), or perhaps some of the msspark util functions listed here: https://learn.microsoft.com/azure/synapse-analytics/spark/microsoft-spark-utilities?pivots=programming-language-python

[xiangyan99]

No. Azure identity library does not work in Synapse workspace.

In order to use managed identity in Synapse workspace, you need to use mssparkutils.

For more information: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-secure-credentials-with-tokenlibrary?pivots=programming-language-python

[github-actions[bot]]

Hi @vanlanchoy. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

[github-actions[bot]]

Hi @vanlanchoy, since you haven’t asked that we /unresolve the issue, we’ll close this out. If you believe further discussion is needed, please add a comment /unresolve to reopen the issue.

[github-actions[bot]]

Hi $abhbhatt, only the original author of the issue can ask that it be unresolved. Please open a new issue with your scenario and details if you would like to discuss this topic with the team.

[bluEEil]

No. Azure identity library does not work in Synapse workspace.

In order to use managed identity in Synapse workspace, you need to use mssparkutils.

For more information: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-secure-credentials-with-tokenlibrary?pivots=programming-language-python

Hey @xiangyan99 I'm trying to also use managed identity to connect to a service from a workspace (to Azure Open AI in my case). I don't understand how using mssparkutils can help with that. Is there any way to get the workspace or any managed identity from the notebook code in Synapse?

[aayushsin]

@xiangyan99 , I cannot use SPN due to security issues. I have to switch to MI. After switching to system assigned MI and following the steps mentioned at https://learn.microsoft.com/en-us/azure/machine-learning/how-to-authenticate-batch-endpoint?view=azureml-api-2&tabs=sdk#running-jobs-using-a-managed-identity, I am still facing the ImdsCredential.get_token failed

[github-actions[bot]]

Hi $aayushsin, only the original author of the issue can ask that it be unresolved. Please open a new issue with your scenario and details if you would like to discuss this topic with the team.

[MohammedSarfarazuddin]

No. Azure identity library does not work in Synapse workspace. In order to use managed identity in Synapse workspace, you need to use mssparkutils. For more information: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-secure-credentials-with-tokenlibrary?pivots=programming-language-python

Hey @xiangyan99 I'm trying to also use managed identity to connect to a service from a workspace (to Azure Open AI in my case). I don't understand how using mssparkutils can help with that. Is there any way to get the workspace or any managed identity from the notebook code in Synapse?

In my case I am trying to connect to an Azure ML workspace

I managed to get it working by creating a linkedservice to Azure ML workspace from Synapse after making sure that the Synapse's Managed Identity has at least a role of a Data Scientist (it can be perhaps "Reader" in your case?) to the Azure ML workspace.

I used the following code with mssparkutils library:

from notebookutils import mssparkutils

aml_token = mssparkutils.credentials.getConnectionStringOrCreds("ls_azureml")

where "ls_azureml" is the linkedservice name.

I also had to use a class to get the credential working through an access token from the linkedservice (thanks chatGPT):

# Assuming aml_token contains the access token
class AmlTokenCredential(TokenCredential):
    def __init__(self, token):
        self.token = token

    def get_token(self, *scopes, **kwargs):
        # Wrap the token in an AccessToken object with a valid expiry date
        return AccessToken(self.token, expires_on=int(time.time()) + 3600)

# Use the aml_token retrieved earlier
aml_token = mssparkutils.credentials.getConnectionStringOrCreds("ls_azureml")

# Create an instance of AmlTokenCredential with the token
credential = AmlTokenCredential(aml_token)

You can then use credential variable in MLClient class (my specific case) to connect to the ML Workspace.

Hope this helps!