[FEATURE REQ] DefaultAzureCredential for local docker testing

[et1975]

Azure.Identity Testing code that uses DefaultAzureCredential in a container locally seems to require a lot of effort, unless one is willing to supply username/password into the environment. Creating a service principal and supplying the clientID + Secret is not much better, but also requires a whole lot of additional effort - like setting up the SP, granting the permissions that the developer account already has, etc.

There should be a way to use VS/VSCode/CLI tokens simply by mounting ~/.azure into /root/.azure of the container, unfortunately this does not work today. #12749 mentions installation of the CLI as a working solution, but I just tried this on Alpine and a) it's a hassle - installing all that stuff on Alpine is error-prone experience and takes a long time (on each build!) b) it doesn't work, as I still get the exception

SharedTokenCacheCredential authentication failed: Persistence check failed. Inspect inner exception for details ---> Azure.Identity.AuthenticationFailedException: SharedTokenCacheCredential authentication failed: Persistence check failed. Inspect inner exception for details ---> Microsoft.Identity.Client.Extensions.Msal.MsalCachePersistenceException: Persistence check failed. Inspect inner exception for details ---> System.DllNotFoundException: Unable to load shared library 'libsecret-1.so.0' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: Error loading shared library liblibsecret-1.so.0: No such file or directory at Microsoft.Identity.Client.Extensions.Msal.Libsecret.secret_schema_new(String name, Int32 flags, String attribute1, Int32 attribute1Type, String attribute2, Int32 attribute2Type, IntPtr end) at Microsoft.Identity.Client.Extensions.Msal.LinuxKeyringAccessor.GetLibsecretSchema() at Microsoft.Identity.Client.Extensions.Msal.LinuxKeyringAccessor.Write(Byte[] data) at Microsoft.Identity.Client.Extensions.Msal.MsalCacheStorage.VerifyPersistence() --- End of inner exception stack trace --- at Microsoft.Identity.Client.Extensions.Msal.MsalCacheStorage.VerifyPersistence() at Microsoft.Identity.Client.Extensions.Msal.MsalCacheHelper.VerifyPersistence() at Azure.Identity.MsalClientBase1.GetClientAsync(Boolean async, CancellationToken cancellationToken) at Azure.Identity.MsalClientBase1.GetClientAsync(Boolean async, CancellationToken cancellationToken) at Azure.Identity.MsalPublicClient.GetAccountsAsync(Boolean async, CancellationToken cancellationToken) at Azure.Identity.SharedTokenCacheCredential.GetAccountAsync(Boolean async, CancellationToken cancellationToken) at Azure.Identity.SharedTokenCacheCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)

[jsheetzmt]

VisualStudioCredential is not working for us with isolated functions + Docker. I believe this is the issue we are running into.

https://github.com/Azure/azure-functions-dotnet-worker/issues/1318

[BC89]

Greetings,

Not sure if there was a regression but this was working fine after upgrading to 1.9.0+ beta and 17.6 p but just today I pulled down 17.7.0 p1 and 1.9.0 Azure.Identity and now the following:

• Visual Studio Token provider can't be accessed at .IdentityService/AzureServiceAuth/tokenprovider.json ---> System.AggregateException: Multiple exceptions were encountered while attempting to authenticate. (ManagedIdentityCredential authentication unavailable. The requested identity has not been assigned to this resource. Status: 400 (Bad Request)

Content: {"error":"invalid_resource","error_description":"AADSTS500011: The resource principal named https://.vault.azure.net/ was not found in the tenant named kcidev. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You might have sent your authentication request to the wrong tenant.\r\nTrace ID: ...

I'm using Azure Batch to run a container and mount a storage share etc. This was working perfectly fine until the upgrade and now the container no longer starts as expected and throws the error above indicating the associated identity is wrong.

Thanks! -BC

[NCarlsonMSFT]

@BC89 for issues specific to the VS Container Tools implementation could you please open an issue at https://github.com/microsoft/DockerTools

For this specific issue, the first thing to check would be the identity used for Azure Service Authentication in VS:

[BC89]

Sure thing. The account looks correct and I've scrubbed accounts, cleared cache and done an az login. It's not only happening in my local development env. but also in the deployed version on Azure container app services.

---

From: Nathan Carlson @.> Sent: Wednesday, May 17, 2023 2:09 PM To: Azure/azure-sdk-for-net @.> Cc: Brendan Carroll @.>; Mention @.> Subject: [External Email] Re: [Azure/azure-sdk-for-net] [FEATURE REQ] DefaultAzureCredential for local docker testing (#19167)

https://github.com/BC89

From @. 410-316-7820 This is an External Email from outside of KCI. https://github.com/BC89

---

@BC89https://github.com/BC89 for issues specific to the VS Container Tools implementation could you please open an issue at https://github.com/microsoft/DockerToolshttps://github.com/microsoft/DockerTools

For this specific issue, the first thing to check would be the identity used for Azure Service Authentication in VS: [image]https://user-images.githubusercontent.com/14239288/239040601-37bcbfc1-9c9b-4a3f-ac3c-fd03a23449e3.png

— Reply to this email directly, view it on GitHubhttps://github.com/Azure/azure-sdk-for-net/issues/19167#issuecomment-1551846309, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAZFIM6MOZ3XEV5EMJVKEXDXGUH6DANCNFSM4YRRE2EQ. You are receiving this because you were mentioned.Message ID: @.***>

[BC89]

Created -Here and also had a tread in Azure Batch. Update - Figured out my issue. dumb-dumb mistake on my part.

[vitor-baptista-vfx]

is there any plan for an official solution for people relying on AzureCliCredential?

[tpanum]

@vitor-baptista-vfx Judging by the age of this issue, and the lack of responsiveness from the Azure team. I wouldn't expect it.

If you're just getting started, I would highly recommend use other measures of authentication.

[asimmon]

@vitor-baptista-vfx I agree with @tpanum. Check out our non-intrusive solution which uses a "sidecar-ish" approach to keep supporting AzureCliCredential without modifying our existing Docker images: https://github.com/Azure/azure-sdk-for-net/issues/19167#issuecomment-1505493978

[vmcbaptista]

@asimmon is your organisation publishing that image publicly? I'm not being able to pull it, I keep getting "Error response from daemon: denied", even though I have done docker login into ghcr.io

[dferri]

@tpanum

The least destructive hack I have come up with is simply to retrieve secrets (e.g. access token) from my host machine (using Azure CLI) and pass it into my docker container using environment variables, and overrule the azure-identity clients

This seems a good workaround, but how do you use that token?

[vitor-baptista-vfx]

For anyone using Jetbrains Rider, there's an issue currently gathering votes to add support for this: https://youtrack.jetbrains.com/issue/RIDER-94485

[gamontal]

Yep I understand. The workaround is to install Azure CLI on WSL and use az login on WSL. This will give you the same cli token (your developer identity) than on Windows, but unencrypted. Then from Windows you can access this unencrypted cli token with this mount: \\\\wsl$\\<DISTRONAME>\\home\\<USERNAME>\\.azure\\:/app/.azure/ (path escaped for Docker compose). Use this mount with our proxy and you now have DefaultAzureCredentialworking for Docker on Window-to-Linux.

@asimmon @et1975 Workaround using Docker: docker run -it --rm -v "$($HOME)/.azure-docker:/root/.azure" mcr.microsoft.com/azure-cli az login

docker-compose volume config: <path>/.azure-docker:/app/.azure"

[jaredpar]

Now that the Azure token cache is encrypted by default the traditional approach of volume mapping ${USERPROFILE}\.azure:/root/.azure no longer works. This thread seems to have settled on two approaches to working around that.

1. Use az config set core.encrypt_token_cache=false. This effectively undoes the encryption and restores the ability to directly volume map from Windows to Docker.
2. Use az login in WSL then volume map from WSL to docker \\\\wsl$\\<DISTRONAME>\\home\\<USERNAME>\\.azure\\:/app/.azure

I'm trying to understand the downsides of (1) compared to (2). The setup for (1) is much easier: a one time command. Yes it leaves the tokens un-encrpyted on the hard drive but so does (2) as Linux does not encrypt the tokens (to my understanding at least). (2) is a bit more awkward to steup and requires that you keep WSL logins active which is another item to maintain.

Is there something I'm missing in my analysis here that would make me want to prefer (2)?

[karpikpl]

I have a multi-layer docker image. When I want to run in "local" mode - I include layers with azure CLI that inflate the image size. I first run it with custom entry point, log in to Azure, commit layer, run image with another entrypoint that starts the app (at this point token is present).

[krukowskid]

A summary for everyone with a standard use case. In order to use your Visual Studio identity in a local docker container when debugging in VS you will need the following:

• Visual Studio 17.6 or later.
• azure.identity package version 1.9.0 or later.
• Microsoft.VisualStudio.Azure.Containers.Tools.Targets package version 1.18.1 or later.
• Docker Desktop with WSL (for Windows machines).
• A Dockerfile for your project.

If you use DefaultAzureCredential() will seamlessly work when running the container from VS, as it maps a volume with VS identity token to the container. For more detailed information, refer to my blog post.

[jaredpar]

@krukowskid read through your post and still having trouble understanding how this is working. The current problem started happening because the Azure CLI is encrypting the tokens on disk. This means previous solutions that mapped Windows drives to Linux fail because Linux OS isn't able to decrypt the tokens.

The blog post indicates that the solution is using drive mapping to solve the problem. For that to work, based on what I've learned thus far, the token must be un-encrypted. Or the targets / tasks are doing some magic in the middle to decrypt the token before volume mounting occurs. Can you elaborate on the mechanism at work here? Particularly if it's something else I'm not considering 😄

[krukowskid]

Hey @jaredpar, this is a built-in mechanism introduced in the latest updates that doesn't use Azure CLI but relies on Visual Studio's logged-in identity. The mapping is handled by Visual Studio itself, so all you need are the correct versions of packages. However, it's worth noting that this mechanism works with the mcr.microsoft.com/dotnet/sdk:6.0 image but won't work with the alpine image :(

@NCarlsonMSFT, could you share with us how this works underneath and what's required to use it with the alpine image?

[NCarlsonMSFT]

@jaredpar the volume mount is not for tokens, it is for a Proxy that VS uses to handle Token Requests: https://learn.microsoft.com/en-us/visualstudio/containers/container-tools-configure?view=vs-2022#configure-azure-authentication

[NCarlsonMSFT]

@krukowskid Can you try with the latest preview: https://visualstudio.microsoft.com/vs/preview/ I believe the Alpine issue has been fixed there.

[krukowskid]

@NCarlsonMSFT installed preview version 17.7.0 Preview 5.0 and problem with alpine image still persist. its working! :)

[NCarlsonMSFT]

@krukowskid could you open an issue over at https://github.com/microsoft/DockerTools with a link to a repro project? I just did a quick test retargeting my sample (https://github.com/NCarlsonMSFT/VisualStudioCredentialExample) to mcr.microsoft.com/dotnet/runtime:6.0-alpine and was able to get a token.

[krukowskid]

@NCarlsonMSFT I've missed that after installing preview version, VS asked me to "Re-enter my credentials". I can confirm that alpine images are working with preview version. Can't wait for the stable release!

[henriksen]

Any immediate plans of making the VS proxy work with AzureCliCredential outside of VS? So I don't have to run the container(s) though Visual Studio?

[kjreuter]

@krukowskid @NCarlsonMSFT Does this only work with WSL? Will it work with the mcr.microsoft.com/dotnet/aspnet:6.0 image?

I followed @krukowskid's blog post's instructions using Docker Desktop (minus WSL) with mcr.microsoft.com/dotnet/aspnet:6.0 and the latest versions of the required packages and Visual Studio (17.7.1), but am getting a 403 on a ManagedIdentityCredential requests to 169.254.169.254. I understand that IMDS uses that IP in Azure, but it is not routable and should not be being used locally, and based on the blog post, I would expect that VisualStudioIdentity would be used locally.

One other item potentially of note, if I run Fiddler, the request succeeds.

Here is the full error: Azure.Identity.AuthenticationFailedException: 'ManagedIdentityCredential authentication failed: Service request failed. Status: 403 (connecting to 169.254.169.254:80: connecting to 169.254.169.254:80: dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network.)

[christothes]

@kjreuter This 403 response from Docker is a known issue - we should have a fix soon for it.

[kjreuter]

@christothes Your workaround and downgrading to version 1.9.0 of Azure.Identity both worked. Thank you! I will keep an eye out for a new version with a fix...

[jonnekleijer]

Some teammates are on mac's and since VS will be retired next year for mac, it would be great to see the VS proxy feature land in VSCode as well.

[svrooij]

Drumroll 🥁 ..... I've managed to use DefaultAzureCredentials in docker with the use of an open-source proxy (build by myself). https://svrooij.io/2023/08/03/emulate-managed-identities/

I know this is not a real solution but at least you can start testing your apps running in docker.

[flaxh]

I tried solution proposed by @krukowskid via example from https://github.com/NCarlsonMSFT/VisualStudioCredentialExample. I successfully authenticate but only in isolated Azure function app. Do you have any idea why I'm getting "Visual Studio Token provider can't be accessed at /home/.IdentityService/AzureServiceAuth/tokenprovider.json" for in-process function app ?

[NCarlsonMSFT]

@flaxh the error message is that the VisualStudioCredential isn't configured properly. The underlying issue is that the Integrated Azure Functions images don’t have a dotnet runtime so the proxy isn’t able to run. I’ve updated the sample with a new project FunctionAppIntegrated that demonstrates a work-around to unblock debugging.

[rhythmnewt]

@NCarlsonMSFT I tried your visual studio credential and it worked properly for me with the isolated function project, I was able to run the function both locally and in docker using your solution. However, I hit a snag when trying this approach in my solution.

In my solution I have a service bus queue trigger that authenticates using managedidentity (configured in the local.settings.json) "MyServiceBusConnection__fullyQualifiedNamespace": "[myservicebus].servicebus.windows.net"

In this case when debugging or running the project within docker, the authentication to servicebus fails with this exception in console (it works with VisualStudioCredential when executing natively in VS2022). It just can't authenticate the listener, even though it should be using same VisualStudioCredential under the hood.

Azure.Identity: ManagedIdentityCredential authentication failed: Managed Identity response was not in the expected format. See the inner exception for details.

I was able to create a basic replication using your solution. In the FunctionAppIsolated project add nuget package: <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.ServiceBus" Version="5.14.0" /> Add new function definition in Function1.cs

        [Function("ProcessMessage")]
        public async Task ProcessMessageAsync([ServiceBusTrigger("%QueueName%", Connection = "ServiceBusConnection")] ServiceBusReceivedMessage receivedMessage, CancellationToken cancellationToken, ServiceBusMessageActions messageActions, FunctionContext context)
        {

            IDictionary<string, string> messageProperties = new Dictionary<string, string>();

            var message = receivedMessage.Body.ToString();
            Console.WriteLine(message);
        }

update local.settings.json

   "QueueName": "[your sample queue]",
   "ServiceBusConnection__fullyQualifiedNamespace": "[sample service bus instance].servicebus.windows.net"

[NCarlsonMSFT]

@rhythmnewt If you're only getting an error from ManagedIdentityCredential then the other identity providers aren't being used. I've found some docs on how this is supposed to work locally: Guidance for developing Azure Functions. If those docs don't help, you're best bet is to contact the Azure Functions team : Azure Functions questions

[github-actions[bot]]

Hi @et1975, we deeply appreciate your input into this project. Regrettably, this issue has remained inactive for over 2 years, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.

[verydarkmagic]

Hmm, closed by bot after 3 years :D I think this issue is a really accurate picture of development for Azure as a whole. Rich on marketing promises, confusing on pricing, and downright abysmal on actual features.

[LL-SRN]

If the az client could retrieve the relevant env variables - or read them out from my local ~/.azure - I could easily write a makefile or similar to pass them to docker run and automate the whole thing without having to commit any secrets to a repo.

As is, it is very strange that I have, on my PC, everything I need to do dotnet run and have everything work, but I don't have any way to pass those necessary things into my container.

[jsheetzmt]

@NCarlsonMSFT Are you aware of any issues with TokenService.Proxy not working when using 17.10.0 Preview 5.0? The TokenService.Proxy bind mount isn't appearing when using latest preview version of VS. Works fine with 17.9

[NCarlsonMSFT]

@jsheetzmt in 17.10 the various helper CLIs for supporting the container tools got moved into one mount /VSTools. Does that appear? Are you having an issue with getting Tokens in 17.10? If so, please open an issue in microsoft/DockerTools.

[svrooij]

I never saw this working, so I checked my settings and it's turned on, I never turned it on, so I guess its on by default. Does not work, never did.

Windows 10 Visual Studio Enterprise 2022 17.9

Back to using my open-source implementation.... https://svrooij.io/2023/08/03/emulate-managed-identities/ Which just works everywhere, without any complicated mounts, just setting a single environment variable and poof it works.

[github-actions[bot]]

Hi @et1975, we deeply appreciate your input into this project. Regrettably, this issue has remained unresolved for over 2 years and inactive for 30 days, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.

[jongio]

Reopening as we still need to figure this out