[FEATURE REQ] DefaultAzureCredential for local docker testing

[et1975]

Azure.Identity Testing code that uses DefaultAzureCredential in a container locally seems to require a lot of effort, unless one is willing to supply username/password into the environment. Creating a service principal and supplying the clientID + Secret is not much better, but also requires a whole lot of additional effort - like setting up the SP, granting the permissions that the developer account already has, etc.

There should be a way to use VS/VSCode/CLI tokens simply by mounting ~/.azure into /root/.azure of the container, unfortunately this does not work today. #12749 mentions installation of the CLI as a working solution, but I just tried this on Alpine and a) it's a hassle - installing all that stuff on Alpine is error-prone experience and takes a long time (on each build!) b) it doesn't work, as I still get the exception

SharedTokenCacheCredential authentication failed: Persistence check failed. Inspect inner exception for details ---> Azure.Identity.AuthenticationFailedException: SharedTokenCacheCredential authentication failed: Persistence check failed. Inspect inner exception for details ---> Microsoft.Identity.Client.Extensions.Msal.MsalCachePersistenceException: Persistence check failed. Inspect inner exception for details ---> System.DllNotFoundException: Unable to load shared library 'libsecret-1.so.0' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: Error loading shared library liblibsecret-1.so.0: No such file or directory at Microsoft.Identity.Client.Extensions.Msal.Libsecret.secret_schema_new(String name, Int32 flags, String attribute1, Int32 attribute1Type, String attribute2, Int32 attribute2Type, IntPtr end) at Microsoft.Identity.Client.Extensions.Msal.LinuxKeyringAccessor.GetLibsecretSchema() at Microsoft.Identity.Client.Extensions.Msal.LinuxKeyringAccessor.Write(Byte[] data) at Microsoft.Identity.Client.Extensions.Msal.MsalCacheStorage.VerifyPersistence() --- End of inner exception stack trace --- at Microsoft.Identity.Client.Extensions.Msal.MsalCacheStorage.VerifyPersistence() at Microsoft.Identity.Client.Extensions.Msal.MsalCacheHelper.VerifyPersistence() at Azure.Identity.MsalClientBase1.GetClientAsync(Boolean async, CancellationToken cancellationToken) at Azure.Identity.MsalClientBase1.GetClientAsync(Boolean async, CancellationToken cancellationToken) at Azure.Identity.MsalPublicClient.GetAccountsAsync(Boolean async, CancellationToken cancellationToken) at Azure.Identity.SharedTokenCacheCredential.GetAccountAsync(Boolean async, CancellationToken cancellationToken) at Azure.Identity.SharedTokenCacheCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)

[et1975]

I figured a workaround just now:

• have a Dockerfile just for running stuff locally (not a great start, but easier than the alternatives)
• that uses mcr.microsoft.com/azure-cli as the base image and
• in docker-compose has:

  volumes:
  - ~/.azure:/root/.azure 

  And in the code sets up the auth chain:

ChainedTokenCredential(ManagedIdentityCredential() or EnvironmentCredential(), AzureCliCredential())

This works, but would be great if we didn't need az cli in the first place.

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[jongio]

See here for how I do it, which is the same as you, but checkout the CLI install script in my dev container, it's a one liner.

RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash

VIDEO: https://youtu.be/oDNGs7B2g1A CODE: https://github.com/jongio/azureclicredentialcontainer

The only thing better than this would be local ManagedIdentity, but that isn't available right now. We have discussed it, but it opens issues that need to be fleshed out.

[nhart12]

Agreed, to be able use/mount IDE azure credentials when local testing would be awesome. Azure CLI bloats images by almost a gig

[jdthorpe]

See here for how I do it, which is the same as you, but checkout the CLI install script in my dev container, it's a one liner.

RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash

VIDEO: https://youtu.be/oDNGs7B2g1A CODE: https://github.com/jongio/azureclicredentialcontainer

The only thing better than this would be local ManagedIdentity, but that isn't available right now. We have discussed it, but it opens issues that need to be fleshed out.

@jongio, This worked for me up until I upgraded my Azure CLI to 2.33. Now it seems the windows host machine encrypts the tokens in a .bin file, but the linux azure CLI inside the container expects the unencrypted .json file, so I get a message inside the container stating Please run 'az login' from a command prompt to authenticate before using this credential. inside the container, but the same code running on the windows host fetches an access token without issue.

[philipwolfe]

@et1975 @jdthorpe @jongio @christothes I am running into this too. Here is what I came up with. We too need ways for a container running on a QA engineer machine to authenticate to Azure without checking credentials into SCC in a YAML file. Would love some feedback. My goal is to take the access token from the engineer and use it for this session...doesn't need to be long term like the EnvironmentCredential. https://github.com/philipwolfe/azure-sdk-for-net/commit/5dff08d75bdb41a1bef4618d2478adbfb6f02338 based on ideas from: https://stackoverflow.com/a/61498506/13122820

[jdthorpe]

@philipwolfe this solution may work for you for now. It essentially requires installing a previous version of the Azure CLI onto both the host machine and in the container, logging into Azure (az login) on the host machine, mapping the ~/.azrue directory into the container.

[goenning]

We're also using the CLI solution, but the az cli on developer machines is auto updating to the 2.33 version, so that means every day developers have to downgrade to 2.29.

Based on az cli docs, it's not meant to auto-upgrade by default, but apparently it is...

By default, auto-upgrade for Azure CLI is disabled. If you would like to keep up with the latest version, you can enable auto-upgrade through [configuration](https://docs.microsoft.com/en-us/cli/azure/config).

[tpanum]

Surreal to read that no progress has been made on such a fundamental problem for over a year. Much like the Python counter part (azure-identities), this package simply seems to be poorly designed, as it relies on some unversioned binary to function. Thus this binary dependency has to be baked in to the container images, despite serving no use in production.

The least destructive hack I have come up with is simply to retrieve secrets (e.g. access token) from my host machine (using Azure CLI) and pass it into my docker container using environment variables, and overrule the azure-identity clients, like so: docker run -e TOKEN=$(az account get-access-token --resource <resource-id> | jq -r .accessToken) my/fantastic-image.

[paupal1]

Also running into this issue... Is there a recommended workaround other than downgrading AzCli version?

[dmitriyse]

Was forced to write a tool that proxies the local tokens for local user (obtained from the DefaultAzureCredential) to the container through the same protocol as MSI are delivered to the ARC enabled servers. https://github.com/ClrCoder/ClrPro.AzureFX/releases/tag/v0.1.0

This tool should be executed from a developer account on port 40342

$env:ASPNETCORE_URLS="http://+:40342"
.\ClrPro.Azure.LocalCredentialBridge.exe

Then container should have the next env, volumes:

docker ....
      -e IDENTITY_ENDPOINT=http://host.docker.internal:40342/metadata/identity/oauth2/token
      -e IMDS_ENDPOINT=http://host.docker.internal:40342 
      -v %USERPROFILE%/.LocalCredentialBridgeTokens:/var/opt/azcmagent/tokens:ro
      ...

And the DefaultAzureCredential will work inside the container.

[dmitriyse]

Ideally such functionality should be inside Visual Studio out of the box. 1) Docker containers development is a first-class feature of the Visual Studio 2) Azure secret-less resource access is a first-class feature of the Azure SDK 3) Azure connectivity from Visual-Studio again is a first class feature

Why developers should do the IDE enhancement job for the first class features to make them works together ?

Lack of support of zero secrets connectivity is appearing here and there. For example here there was also a problem https://github.com/dotnet/efcore/issues/26491

Please increase the priority of this feature request. It's spanning a year already.

[esimkowitz]

Hi @jongio, any updates here? While we would like to get all our developers working in Docker containers to improve compatibility with our production environments, requiring a complicated login process versus just running in VS is too much of a burden.

[asanjabi]

@esimkowitz one workaround is to mount a volume that's shared between all containers, you'd have to connect to one and login once, but the rest will be fine after that. You would need to install the CLI on all the images, so there is that. Not ideal, but workable sample

[esimkowitz]

Frankly that seems like more work to explain to my devs and write troubleshooting docs for than to just tell them to test their changes separately against our Linux environments.

[stefanolsenn]

We fixed it by injecting the environment variables into the containers: in our docker-compose file and using InTune to set the environment variables on all developer pc's.

      [...]
        environment:
         - AZURE_TENANT_ID
         - AZURE_CLIENT_ID
         - AZURE_CLIENT_SECRET
      [...]

[esimkowitz]

That kind of fix won't work for us. We do not store client credentials on local dev boxes, we need to have RBAC set up to someone's own account for any dev resources. We are able to use DefaultAzureCredential in Visual Studio with no issue, ideally this should pipe automatically into Docker when running locally.

[dmitriyse]

https://github.com/Azure/azure-sdk-for-net/issues/19167#issuecomment-1127081646

Please try this approach. Works good enough in our team. Visual Studio Credential get passed into containers.

~ 1/2 Year, all good, we forgot about this problem.

[mykola-yakovliev]

Just to add another argument to this problem: for someone (like me), who is new to development of cloud solutions using Azure and wants to try things out, it is a little bit frustrating experience to get an exception after you generate the project from a template and just want it to run with zero-configuration needed. Of course, it is not really much critical in my case, but from my point of view, people would expect it to work locally out-of-box equally with or without Docker.

[esimkowitz]

@jongio @jsquire any updates here on prioritizing this ask? I think this is a very popular scenario given the increasing focus on Dockerizing development and debug environments.

[jsquire]

Hi @esimkowitz. I have no insight; my role for this issue was only initial triage. Any updates would need to be provided by the assigned engineers, @christothes and @schaabs).

[dmitriyse]

Updated the LocalCredentialBridge utility: https://github.com/ClrCoder/ClrPro.AzureFX/releases/tag/v0.1.1

[jongio]

@dmitriyse - Do you have more info on that?

[jonms90]

I am running into the same issue for local development with docker containers in Visual Studio 2022 that relies on Azure services. In production/test I use Managed Identities without any issue, but that is not an option locally. The code uses the chained DefaultAzureCredential to support multiple credential providers. To summarize;

• EnvironmentalCredential: This works fine for User accounts, but not when MFA is enabled (which should always be enabled). The other option here is to use a Service Principal and pass in the client credentials using a .env file that is not checked in to source control. This works, but it is a hassle to manage with a lot of management overhead when your development teams starts to grow.
• ManagedIdentityCredential: As mentioned: works great for test/prod, but not available for local development.
• SharedTokenCacheCredential: There is little to no documentation on how this is supposed to work with a container?
• VisualStudioCredential: This is what I would expect to be the default developer experience in 2022, but it does not seem to be integrated with docker container support in VisualStudio. Ideally, logging into VS should be enough to authenticate regardless of running in a container or not. Unfortunately this is not how it works.
• Using Azure CLI. Another option that works with some hacks including mounting azure folders onto the running container, but the largest downside is that we have to include the Azure CLI in our container images. This dramaticly bloats our images and really is not an option considering the amount of images we create.
• InteractiveBrowserCredential does not seem to do anything when running in a container context

Using Visual Studio 2022, Azure and Docker in combination should not be this complicated. This seems like a very basic setup that will hit everyone trying to containerize their cloud-native applications. I must be missing something obvious.

[jonms90]

I guess the lesser evil is to use a Service Principal for each user, but that really does not seem to be the correct way of solving this issue.

[NCarlsonMSFT]

Update on this: I am a dev on the Container Tools team in VS and we are actively working on solving this issue; but unfortunately, I can't give you an exact timeline for when support will ship. Until then I have two samples to try and make the current experience more bearable: EnvironmentCredentialExample and AzureCliCredentialExample. Both use a combination of PowerShell scripts and debugging customizations to make the process of authenticating in development containers as straight forward as possible.

[Augustukas]

one more workaround described here https://endjin.com/blog/2022/09/using-azcli-authentication-within-local-containers

[BC89]

Incredibly frustrating. MS pushing Dockerized approach in all the VS2002 marketing BS and something as fundamental as this breaks down.

[BC89]

Looks like 1.9.0-beta.2 just hit and this still hasn't been addressed.

[NCarlsonMSFT]

Update: Using the new Azure.Identity 1.9.0-beta.2 and Visual Studio 2022 17.6 Preview 1 the VisualStudioCredential should now work when using Visual Studio to Launch a .NET Core project in a Windows or Linux container.

[BC89]

Can confirm that Nathan is correct and this issue appears to be addressed with that combination out of the box. Thanks!

[saravanakumarvelayutham]

I ran into the same problem to allow running docker-compose with mounted volume of az token location to the container from the windows host. Using the beta identity also did not work with az cli included in docker image

Since window az cli uses credentials manager to encrypt, it generates the token cache in ".bin" format. While Linux cli generates ".json" token cache. To make the mount work from windows host to docker container , I disabled the encryption when logging into az cli from windows.

az config set core.encrypt_token_cache=false

Then do az login, it will generate the token json which can be mounted to docker :)

https://github.com/Azure/azure-cli/issues/21010#issuecomment-1104844455

I hope this helps someone :)

Still looking for way without disabling encryption

[karpikpl]

hey @NCarlsonMSFT is there planned support for VS Code solution that uses VisualStudioCredential, where Docker Desktop is not needed?

[NCarlsonMSFT]

@karpikpl that would be a good question to ask at: https://github.com/microsoft/vscode-docker.

[IisAnh]

Update: Using the new Azure.Identity 1.9.0-beta.2 and Visual Studio 2022 17.6 Preview 1 the VisualStudioCredential should now work when using Visual Studio to Launch a .NET Core project in a Windows or Linux container.

Hey @NCarlsonMSFT , is there an example of the VisualStudioCredential working with these packages that I could look at just like your other examples?

[NoamTD]

@NCarlsonMSFT When trying the setup you described I get this error: Visual Studio Token provider can't be accessed at /root/.IdentityService/AzureServiceAuth/tokenprovider.json. Do I need to do anything other than Using Azure.Identity 1.9.0-beta.2 and Visual Studio 2022 17.6 Preview 1 to make it work?

[karpikpl]

@NCarlsonMSFT When trying the setup you described I get this error: Visual Studio Token provider can't be accessed at /root/.IdentityService/AzureServiceAuth/tokenprovider.json. Do I need to do anything other than Using Azure.Identity 1.9.0-beta.2 and Visual Studio 2022 17.6 Preview 1 to make it work?

I got the same thing when I was trying to run it in this setup.

[NCarlsonMSFT]

@IisAnh There is now: https://github.com/NCarlsonMSFT/VisualStudioCredentialExample

[NCarlsonMSFT]

@NoamTD, @karpikpl Probably you need to update Microsoft.VisualStudio.Azure.Containers.Tools.Targets to 1.18.1 (my bad didn't mention it earlier).

[flashQarl]

@IisAnh There is now: https://github.com/NCarlsonMSFT/VisualStudioCredentialExample

Hi! This example does not work for me. I get this error:

DefaultAzureCredential failed to retrieve a token from the included credentials. See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/defaultazurecredential/troubleshoot
- EnvironmentCredential authentication unavailable. Environment variables are not fully configured. See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/environmentcredential/troubleshoot
- WorkloadIdentityCredential authentication unavailable. The workload options are not fully configured. See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/workloadidentitycredential/troubleshoot
- ManagedIdentityCredential authentication unavailable. Multiple attempts failed to obtain a token from the managed identity endpoint.
- Azure Developer CLI could not be found.
- Visual Studio Token provider can't be accessed at /root/.IdentityService/AzureServiceAuth/tokenprovider.json
- Azure CLI not installed
- PowerShell is not installed.

[NCarlsonMSFT]

@flashQarl Looking through Azure.Identity, that seems to happen when there is a problem reading the configuration file. Could you try launching a second time after seeing this failure to see if it works?

[NoamTD]

@NCarlsonMSFT Thank you, it's working now! When can we expect the official release of 17.6?

[KalyanChanumolu]

@NCarlsonMSFT The project you uploaded didnt work for me

Fails with the below error

Exception thrown: 'Azure.Identity.CredentialUnavailableException' in System.Private.CoreLib.dll Exception thrown: 'Azure.Identity.CredentialUnavailableException' in System.Private.CoreLib.dll Exception thrown: 'Azure.Identity.CredentialUnavailableException' in System.Private.CoreLib.dll Exception thrown: 'Azure.Identity.CredentialUnavailableException' in Azure.Identity.dll Exception thrown: 'Azure.Identity.CredentialUnavailableException' in System.Private.CoreLib.dll Exception thrown: 'Azure.Identity.CredentialUnavailableException' in System.Private.CoreLib.dll Exception thrown: 'Azure.Identity.CredentialUnavailableException' in System.Private.CoreLib.dll

[NCarlsonMSFT]

@KalyanChanumolu could you please open an issue there with details from the exceptions?

[asimmon]

👋 At GSoft, we use Azure resources in almost every service we develop, and we access them with Azure credentials (DefaultAzureCredential):

• In cloud environments, we use managed identities (ManagedIdentityCredential).
• In local development/testing environments, such as IDEs or command-line tools (dotnet run), we use developer credentials issued by az login (AzureCliCredential).

Since we have several containerized services as dependencies, we tried running them locally using Docker compose. However, the developer credentials authentication failed because the Azure CLI was not included in the services' Docker images. Modifying the Docker images to include Azure CLI was not an option, as we wanted to use our production-ready Docker images.

After reading this GitHub issue thread, we created a local Docker sidecar/companion/proxy to allow developers to use service Docker images with their developer credentials (az login) without installing the Azure CLI on those images:

https://github.com/gsoft-inc/azure-cli-credentials-proxy

It is quite similar to this this solution, but it is actually simpler and distributed as a Docker image, making it very easy to consume. Works for both Windows & Linux with WSL:

version: "3"

services:
  azclicredsproxy:
    image: ghcr.io/gsoft-inc/azure-cli-credentials-proxy:latest
    volumes:
      - "\\\\wsl$\\<DISTRONAME>\\home\\<USERNAME>\\.azure\\:/app/.azure/" # <-- On Windows with WSL
      - "/home/<USERNAME>/.azure:/app/.azure/"                            # <-- On Linux

  myservice: # Your service that requires your local Azure credentials without installing Azure CLI inside its image
    build: .
    depends_on:
      - azclicredsproxy
    environment:
      - "IDENTITY_ENDPOINT=http://azclicredsproxy/token"
      - "IMDS_ENDPOINT=dummy_required_value"

[et1975]

@asimmon Doesn't solve cross-plat issues, but very elegant solution for linux-on-linux, thank you!

[asimmon]

@et1975 Thanks! Could you be more specific about "cross-plat issues"? Because we actually use it on Windows, like:

• I'm on Windows and I want to run one of my Linux-based ASP.NET Docker containers that requires Azure credentials with Docker Desktop
• I run az login on WSL to retrieve unencrypted azure developer credentials in WSL distro's ~/.azure dir
• I run a new container of our gsoft-inc/azure-cli-credentials-proxy proxy using this mount:

version: "3"

services:
  azclicredsproxy:
    image: ghcr.io/gsoft-inc/azure-cli-credentials-proxy:latest
    volumes:
      - "\\\\wsl$\\<DISTRONAME>\\home\\<USERNAME>\\.azure\\:/app/.azure/"

When I develop on Linux only, I use another mount: /home/<USERNAME>/.azure:/app/.azure/

[et1975]

@asimmon it's mentioned in the comments here, but essentially cli token is encoded differently on windows (not WSL!).

[asimmon]

Yep I understand. The workaround is to install Azure CLI on WSL and use az login on WSL. This will give you the same cli token (your developer identity) than on Windows, but unencrypted. Then from Windows you can access this unencrypted cli token with this mount: \\\\wsl$\\<DISTRONAME>\\home\\<USERNAME>\\.azure\\:/app/.azure/ (path escaped for Docker compose). Use this mount with our proxy and you now have DefaultAzureCredentialworking for Docker on Window-to-Linux.

[saravanakumarvelayutham]

@asimmon our work around was a pre-build powershell to login by disabling the encryption on windows az cli using experimental flag -> "az config set core.encrypt_token_cache=false;"

with this setup, the WSL login is not needed, the mount from windows to container will work by default