"ManagedIdentityCredential: Unexpected content type "text/html"" error during AzureMachineLearningFileSystem function call

[Wor0001]

Operating System

Linux

Version Information

Python version 3.8.17 azure-ai-ml package version 1.8.0 azureml-fsspec version 1.1.1

Steps to reproduce

1. Initialize a AzureMachineLearningFileSystem object with an AML datastore URI, no other arguments.
2. Call a function of the filesystem object, such as .ls().

Expected behavior

The AMLFS object uses DefaultAzureCredential to authenticate to AML successfully with the AzureCLICredential, after making attempts with other credentials in its list of credentials to try, including ManagedIdentityCredential.

Actual behavior

The AMLFS object fails to authenticate with an EnvironmentCredential, then it attempts to authenticate with a ManagedIdentityCredential, and fails with the error: Unexpected content type "text/html". Authentication attempts cease and execution fails.

Addition information

With this same setup, any time I use DefaultAzureCredential, I have to include the arg "exclude_managed_identity_credential=True", otherwise this same failure with the ManagedIdentityCredential will cause an error. This works for things like MLClient, where one can give it a credential to use when instantiating. To use an MLClient, for example, I could create a DefaultAzureCredential with "exclude_managed_identity_credential=True", then pass that credential into MLClient when initializing it. If I don't do this, I will get the same error when trying to do anything with the MLClient that required authentication. I'm also on USGovCloud.

It would be great if I could pass a credential to AzureMachineLearningFileSystem to use, or if I could tell it to skip the ManagedIdentityCredential attempt while authenticating. Is there a way to exert control over its authentication this way?

I submitted at ticket to the wrong place earlier today: https://github.com/Azure/azure-sdk-for-python/issues/34936

[Carl132]

I am also getting this issue and am interested in a resolution - just to note though that if I'm in an AzureML Compute Instance, simply running az login --identity will throw this same error for me (albeit with a different stacktrace as it's not routing through the DefaultAzureCredential object.

I still arrive at the following:

RuntimeError: MSI: Failed to retrieve a token from 'http://127.0.0.1:46808/MSI/auth/?resource=https://management.core.windows.net/&api-version=2017-09-01' with an error of 'InternalError - :{
Info: Request failure status code: 404
}

Which, when using the DefaultAzureCredential (with debug enabled) shows

[WARNING azure.identity._credentials.chained] DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
    EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot this issue.
    ManagedIdentityCredential: Unexpected content type "text/plain; charset=utf-8"
Content: InternalError - :{
Info: Request failure status code: 404

}
To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.

which is identical to above.