Azure Keyvault secret reference not resolving within Azure Function connection string

[adetayorabiu]

I'm having issues integrating key vault into an existing function app. Function app, key vault and other resources are deployed via ARM template.

I replaced the password segment the function connection string entry with key vault secret url but I keep getting an error because the reference doesn't seem to resolve the secret at runtime. It turns out the reference @Microsoft.KeyVault(SecretUri=https://mykeyvault.vault.azure.net/secrets/mysecret/12345678901234567890) ends up as the db server password at runtime.

System Manage Identity and access policy (GET, LIST) are in place so I can't seem to figure out the problem. I have restarted function multiple times and recreated managed identity and access policy too as suggested here and here

I'm wondering if this was suppose to work at all since the connection string value containing the key vault reference is as below

Server=tcp:servername.windows.net,1433;Initial Catalog=dbname;Persist Security Info=False;User ID=username;Password=@Microsoft.KeyVault(SecretUri=https://myvault.vault.azure.net/secrets/usernamesecret/ec96f02080254f109c51a1f14cdb1931);MultipleActiveResultSets=True;Encrypt=True;TrustServerCertificate=False;Connection Timeout=5; I can't see to figure out what may be missing here. See access policy page below showing get and list right to function app. Please help with some insight @mattchenderson @jeffhollan

[Ohmniox]

Same issue here. KeyVault secrets are not getting resolved.

[adetayorabiu]

@Ohmniox something I figured out is the key vault secret reference doesn't get resolved when used in Connection string configuration and retrieved as IConfiguration.GetConnectionString("conn").

All examples I found has it in AppSettings and retrieved with Environment.GetEnvironmentVariable and that sure works.

It will be nice to document its limitation with connection strings

[kummerer94]

This is still happening to me for any Function App on Linux that I try to run (based on Python). I already tried restarting and switched managed identity off and back on.

Is there an update on this issue?

[SidBarker]

I have the same issue, Azure Function is not resolving reference to Secrets in KV. Do you have any updates about this one?

I'am using Linux and Python based functions, i did tests on HttpTriggered Functions and EventGridTriggers.

Related discussion on stackoverflow: https://stackoverflow.com/questions/58384094/access-azure-key-vault-secret-in-azure-function

[SidBarker]

After hours of tests I found proper way to resolve this one.

In case of problems with Key Vault Reference make sure that App Function used for Azure Function is based on proper Hosting Plan.

Functions on 'Consumption Plan' are unaable to use Key Vault Reference. Same code on 'App Service Plan' works correctly.

@adetayorabiu can you confirm that it solves your case?

[jeffhollan]

To clarify its only Linux Consumption this isn’t yet supported in.

[kummerer94]

Thanks, @jeffhollan, for clarifying this. I might have overlooked in the docs. Is there a roadmap or any kind of indication when we can expect this feature? A hosting plan at the moment defeats the purpose of using Azure Functions at least for us.

[SidBarker]

@jeffhollan it would be great to find this info in documentation regarding Key Vault Reference :)

[jeffhollan]

That purple note at the top of the docs:

https://docs.microsoft.com/en-us/azure/app-service/app-service-key-vault-references

Is on roadmap - I don't have an ETA though. Totally understand the frustration and is a very useful feature. For now you'd have to connect and resolve to key vault in code which works in the interim but understand it's not ideal

[kummerer94]

It's just that you get used to all the awesome features of key vault... but thanks a lot for the information!

[horeich]

I have similar issues here: Yesterday I set up a Key Vault and I'm now desperately trying to retrieve secrets from my (currently locally running) .Net Core App without any luck so far. What I've done:

• Setup AAD and an App Registration
• Added a client secret to the registration
• Created new key vault The connection to the Key Vault is established withouht problems but the GetSecretAsync-method won't resolve it.

Even an old version of my code that used to work with a different subscription stopped getting secrets from my new onmircosoft account. Must be some different config?.

Edit: I found another troubling red box:

Key Vault references are not currently available in Linux consumption plans.

I want to access the key vault from my ubuntu VM? Will that be possible?

Edit: After hours of figuring out how to setup up everything manually, I just followed these steps: https://docs.microsoft.com/en-us/azure/key-vault/quick-create-net

The managed identity created in this example finally granted me access to my key vault. It's working as expected now.

[dariopad]

This is usually happening when you change app service plan and the system MI may have changed the GUID. Two ways to resolve:

1. Just remove the system MI from the KV Access policies and add it back again. (GET is enough)
2. Recreate the System MI from scratch from the App Service/Settings/Identity. Switch off the status and back to on to recreate the GUID. Then, go to AKV and add the access policies for the new GUID. Remove the old one that won't be recognized anymore. Hope it helps!

[ClientIco]

I really think it is unfair to prioritize Linux lower than Windows machines. Honestly, not even having an ETA ? Surely you jest ? Please make this a priority, these gaps cause people to loose days of productive work! @jeffhollan couldn't you atleast speak with the team and hear whether this will be in a years time ? or 6 months ? Also maybe refer us to the roadmap, maybe we can upvote the need for this. After all we are talking about our applications security as well as IP we want protected.

[Agazoth]

I have just experienced the exact same issue (not on Linux, though).

I have a function running on Y1, that cannot resolve the @Microsoft.KeyVaul. This is working for me on multiple instances. This time I deployed the environment with a TerraForm template as opposed to the ARM template, VS Code and portal deployments I have done earlier.

I got that annoying red cross on all my secret references. After a few hours of reading documentation and issue reports, I decided to compare it to one of the working setups bit by bit.

After a while I came to the network on the key vault. Apparently I had decided to up the security on this to only allow access to the vault from my current location (uploading keys with PowerShell Secret Management) and AzureServices.

This ought to be all good, as my understanding of the AzureServices bypass switch is, that it allows Azure Services to access the vault in case of network restrictions and default action set to Deny.

 (Get-AzKeyVault -VaultName "MyKeyVault").NetworkAcls

DefaultAction                 : Deny
Bypass                        : AzureServices
IpAddressRanges               : {85.184.122.12/32}
IpAddressRangesText           : 85.184.122.12/32
VirtualNetworkResourceIds     :
VirtualNetworkResourceIdsText :

Just to make sure, that this was nat ohat was causing my issue, I flipped the switch to allow All networks.

DefaultAction                 : Allow
Bypass                        : AzureServices
IpAddressRanges               : {85.184.122.12/32}
IpAddressRangesText           : 85.184.122.12/32
VirtualNetworkResourceIds     :
VirtualNetworkResourceIdsText :

And suddenly my function was able to resolve the keys.

The question now is, why does my deployment not allow the function to access the key vault, even though bypass for Azure Services is enabled? According to the documentation here Azure App Service is is a trusted Azure Service.

I'm not sure if these observations are relevant i this issue, but under all circumstances I would like to know, why I experience this behavior.

[DarinMacRae]

I ran into this issue today. In our case we use an Azure AD Group to include our various function apps that need access the our Key Vault. I set this all up as it should work and it does work if you access the key vault via code but using this configuration feature it didn't.

On a hunch I created a new KV access policy that referenced the function app identity directly and then it worked. Is there some reason that AAD groups aren't supported? @jeffhollan

The reason that using a group is important to us is that early on we created KV access policies for several dozen function apps and we hit an internal limit on access policies. We were told by MS support to use an AAD group as a work-around. This has been working for us for years to allow accessing our KV from code but today I wanted to try this neat configuration feature and was disappointed.

[Cdoherty95]

I am having the same issue as described by @DarinMacRae. Assigning the managed identity direct permission in the key vault's access policy works, however, using an AD group does not work. The result description I found in the diagnostics setting log reads "The user, group or application 'appid=00******-****-****-****-**********00;oid=ce******-****-****-****-**********28;iss=https://sts.windows.net/10******-****-****-****-**********95/' does not have secrets get permission on key vault 'keyvaultname;location=centralus'. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287"

[mattchenderson]

• Networking is something we are still working on. It's a problem with the outbound resolution path and how the traffic there gets routed, basically. But it's something we're actively trying to address
• Group resolution should work. Mechanically, this is the same as use of any other managed identity or principal. Group checks from the access policy side should perform a transitive isMemberOf check. It sounds like that is the case for a manual interaction, but it's not working for the references feature, correct? My quick theory is that this would be due to caching of the token without the group membership claims from the perspective of the referencing component, but that would go away quickly enough and shouldn't present an ongoing issue. We'll look into that.

[erwelch]

@mattchenderson - with regards to the networking issue - does this mean we can't use key vault references for a key vault with a private endpoint? I created a private endpoint for my key vault and the Azure Function is integrated into the VNET. It's able to resolve other private endpoints within the same VNET, but it's unable to resolve the key vault reference once I added the private endpoint.

[mattchenderson]

Correct. The problem is that the resolution is occurring before the join to the VNET that your code is able to leverage.

[adiun]

@mattchenderson thanks for the update. Do you have an ETA on when the networking problem would be addressed? My team has a VNet infra in place and we'd like to continue using KeyVault references.

[mattchenderson]

Unfortunately, I do not have an ETA I can share at this time.

[oliverwhk]

this issue has been around for a year, is it just not high priority enough to have it resolved?

[jeffhollan]

Was a twitter exchange on this today. It’s high priority and in flight, but often these networking features require many dependent pieces to all fall into place and then a larger upgrade of our fleet so do take a fair bit more time than a bug fix on the runtime for example. But in coming months should allow you to resolve key vaults using this pattern behind networking constraints. https://twitter.com/ccompy/status/1309883379228131329?s=21

[thdotnet]

Thanks for the update @jeffhollan . You point the purple bar in the official doc (https://docs.microsoft.com/en-us/azure/app-service/app-service-key-vault-references) however, it doesn't contain a workaround. What should we do while this feature is not released? Allow network traffic as @Agazoth did? Use code to retrieve the secrets rather than Key Vault reference?

[jeffhollan]

@thdotnet yes either of those would work. Trigger secrets would be the exception for the piece that you wouldn’t have a viable workaround for - but rest should - though obviously not ideal

[ghost]

The only thing that I was able to do to get around this issue is to white list the virtual IP address of the Azure Function in the firewall of the Key Vault. It's unfortunate but the VNET and private endpoint by itself is not working. I have the same issue as everyone else.

[MarkAtAgilliance]

Grrr I wish I had read this article instead of the docs before changing 15 services to use KV references instead of code. We have app services plans and use groups for permissions and vnets. Failed every test, now rolling back, re-deploying and re-testing. What a waste of time. IMHO KeyVault related access schemes should be first class citizens of security. A work around that says "open up the networking" I don't think is helpful.

[dlandi]

This is frustrating. Can someone post the exact conditions in which this does work?

Is there a happy path?

[balag0]

@dlandi Could you share your appname. https://github.com/Azure/azure-functions-host/wiki/Sharing-Your-Function-App-name-privately

[schaijkw]

Was a twitter exchange on this today. It’s high priority and in flight, but often these networking features require many dependent pieces to all fall into place and then a larger upgrade of our fleet so do take a fair bit more time than a bug fix on the runtime for example. But in coming months should allow you to resolve key vaults using this pattern behind networking constraints. https://twitter.com/ccompy/status/1309883379228131329?s=21

Hi @jeffhollan , a few months have past. Is there an ETA? I have to store my SendGrid ApiKey for a FunctionApp and am willing to use AzureWebJobsSendGridApiKey to pass the ApiKey for the SendGrid output binding. Due to sensitive data we process we cannot open up our network. Is there another way to workaround this issue so I can use a secret in the FunctionApp output binding?

[Nlouis00100]

Hi, ETA on this feature. Very high priority for us. Thank you.

[jeffhollan]

Updating here:

• key vault reference should now work in Linux consumption
• Key vault references no longer need to be pinned to a version, but can point to a key and will be updated on new versions (not instant, check the docs)
• Key vault references won’t work when the key vault is behind a private network

Support for private network references is rolling out globally in the coming weeks. Keep an eye out on azure updates for that.

If having an issue with something that should be supported - I’d recommend opening a support ticket. This issue is not actively being monitored by support

[Nlouis00100]

Thank you for the feedback. Are you saying the keyvault inside a vnet will soon support reference call (app settings) from webapp/funtion? I haven’t seen any update on this limitation anywhere (other than it does not work). Nicolas

On Sat, Feb 20, 2021 at 00:19 Jeff Hollan notifications@github.com wrote:

Updating here:

• key vault reference should now work in Linux consumption
• Key vault references no longer need to be pinned to a version, but can point to a key and will be updated on new versions (not instant, check the docs)
• Key vault references won’t work when the key vault is behind a private network

Support for private network references is rolling out globally in the coming weeks. Keep an eye out on azure updates for that.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Azure/Azure-Functions/issues/1291#issuecomment-782563107, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM6IN5XDALBPMTL7CCKHYR3S75A7JANCNFSM4IIWU22A .

[jeffhollan]

Correct. The >IMPORTANT callout on the doc calls out this current limitation. No concrete ETA but work / deployments are in flight to remove this restriction https://docs.microsoft.com/en-us/azure/app-service/app-service-key-vault-references

[leonK-DI]

Hello

I stiill cannot reference secrets in my keyvault. Is this reference feature as well rolloed out to Europe West?

[balag0]

Yes. It is rolled out in West Europe. Could you share your app name if you are having issues with this - https://github.com/Azure/azure-functions-host/wiki/Sharing-Your-Function-App-name-privately

[leonK-DI]

Yes. It is rolled out in West Europe. Could you share your app name if you are having issues with this - https://github.com/Azure/azure-functions-host/wiki/Sharing-Your-Function-App-name-privately

Here we go: 2021-02-22T04:08:28Z [Information] (Id=f4c91f58-cb63-406c-95f5-cfe76188ea49)

[balag0]

Looks like there is some issue with the configured permissions. (I have removed the actual appid and keyvault names in the below exception) - https://docs.microsoft.com/en-us/azure/key-vault/general/assign-access-policy-portal

WebServiceError.Code:Forbidden WebServiceError.Message:The user, group or application 'appid=b0ca..' does not have secrets get permission on key vault '[Keyvaultname];location=westeurope'. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287

[leonK-DI]

Looks like there is some issue with the configured permissions. (I have removed the actual appid and keyvault names in the below exception) - https://docs.microsoft.com/en-us/azure/key-vault/general/assign-access-policy-portal

WebServiceError.Code:Forbidden WebServiceError.Message:The user, group or application 'appid=b0ca..' does not have secrets get permission on key vault '[Keyvaultname];location=westeurope'. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287

I turned on the system managed identity for my function & an access policy is as well added. But I will try to delete the access policy and add it again. Lets hope that helps...

----UPDATE---- This same error

[mzhukovs]

@jeffhollan Apologies still have trouble confirming what is coming out imminently (or should already be working). I have a Key Vault that is behind a vnet in West Europe, and a Windows Consumption-based .NET Azure Functions App - I added the Function App's MSI to the Key Vault with Key Reader role with Get/List permissions on secrets etc. - am still getting this error:

AccessToKeyVaultDenied: Key Vault reference was not able to be resolved because site was denied access to Key Vault reference's vault.

Is the fix that's coming applicable to this scenario, or is it only addressing some other issue for non-consumption-based plans?

Appreciate the support.

[leonK-DI]

@mzhukovs you have to add an "Access Policy" to your vault and not ad a role to the managed identity of your app. At least that solved for me the issue

[mzhukovs]

you have to add an "Access Policy" to your vault and not ad a role to the managed identity of your app. At least that solved for me the issue

Yes I did both (Access policy for the function app's managed identity AND Azure key vault reader role) trying to get it to work, knowing this doesn't make sense as you can specify on your KeyVault which one to adhere to, but I got desperate.... and of course still no luck.

[mzhukovs]

Unless the fix is imminent, which may not be the case (at least note based on the status of the below user voice) - if you can afford it, the workaround below works, just tested to confirm.

the whitelist approach does work to allow serverless apps to reference keyvault values. In the portal, in your functions app sidebar -> custom domains, grab that IP and whitelist it. Better than disabling the firewall!

https://feedback.azure.com/forums/355860-azure-functions/suggestions/38817385-allow-key-vault-references-to-access-secrets-behin

[konjp]

@oliverwhk Have you resolved your issue? I still have the same thing with AzureWebJobsStorage put it in key vault, and fails when deployed using AzureFunctionApp@1 with error "Unable to find the storage account associated with the function app". No firewall rules, no VNET configured. Strange that there is small number of people asking for this, because should not this be recommended way to store access key in key vault?

[oliverwhk]

@konjp i haven't tried since then 😅

[konjp]

I think problem in this case that you @oliverwhk referenced, and I confirmed that still exists. is between Azure DevOps and Azure Cloud. And "nobody's" problem is hard to resolve.

Only I can you do for now is not use key vault for AzureWebJobsStorage property.

[JV-HCA]

Was having this same issue, Function-App configured with VNET and KeyVault configured with VNET and Firewall. All built with Terraform

Function app configuration was complaining of not being able to find the KeyVault.

Ran the troubleshooting to review the vnet config and most everything came back clean.

Still no dice on the keyvault connection. Was reading about adding the "Custom Domain IP" for the function app to the keyvault firewall. I added that and it worked. (where y.y.y.y/32 is the function app ip)

network_acls {
    bypass                     = "AzureServices"
    default_action             = "Deny"
    ip_rules                   = ["x.x.x.x/x", "x.x.x.x/x", "x.x.x.x/x","y.y.y.y/32"]
    virtual_network_subnet_ids = [data.azurerm_subnet.subnet.id]
  }

I got to thinking and talking with my team, it should work without it because of the vnet integration and wanted to troubleshoot more. Removed the Function App IP from the firewall and It's still working?!

network_acls {
    bypass                     = "AzureServices"
    default_action             = "Deny"
    ip_rules                   = ["x.x.x.x/x", "x.x.x.x/x", "x.x.x.x/x"]
    virtual_network_subnet_ids = [data.azurerm_subnet.subnet.id]
  }

Its almost as if you had to give it a nudge and it worked itself out. It's really strange.

Anyone else seen similar behavior? If so, is my fix short lived and will it break again soon?

[balag0]

@JV-HCA Could you please move your question to a separate issue since this current issue is unrelated to vnet scenario and we will close this issue to avoid confusion.

Also, if I understand your issue correctly, you had some transient issues at some point and they are resolved now and you are looking to get more details on why those failures happened earlier. Correct? In that case please include the time range when the app was broken and the app name using https://github.com/Azure/azure-functions-host/wiki/Sharing-Your-Function-App-name-privately

[jamyc]

We're currently facing an issue where one of our Azure Functions doesn't appear to be resolving the KeyVault Reference correctly. Not sure if this is the correct function id, but couldn't find it using the link posted above (perhaps because we're running a dotnet-isolated function?): Id=3474fdc1-a852-4f65-b4b3-df00e6188f0c

It's a dotnet-isolated Azure Function which has an EventHubTrigger. I get an exception in Application Insights like: Error indexing method 'XXX' Illegal connection string parameter name '@Microsoft.KeyVault(SecretUri' (Parameter 'connectionString')

This appears to be thrown from this line in the EventHubsConnectionStringBuilder.cs which makes me believe that the KeyVault reference is not properly being resolved.

This is especially interesting since the KeyVault reference is giving us a green checkmark on the Application settings page:

@balag0 Could you perhaps have a look at this? Happy to PM/mail you the instance details if the id posted above doesn't provide you with any results.

[balag0]

@jamyc Sure, could you please open a new issue with these details and tag me since this is a closed issue.