Deploying a Keyvault with Bicep in Azure Devops prevents it from being listed in AZ CLI

[nubgamerz]

Bicep version 0.4.1008

Describe the bug Deploying a Standard Keyvault in Bicep using Azure Devops. Then trying to reference that keyvault in a later deployment with 'existing' keyword shows the error

'code': 'KeyVaultParameterReferenceNotFound', 'message': "The specified KeyVault '/subscriptions/[subId]/resourceGroups/[rgName]/providers/Microsoft.KeyVault/vaults/[kvName]' could not be found

To Reproduce Steps to reproduce the behavior:

Create an Azure Devops Pipeline to create a keyvault using Bicep.

resource keyvault 'Microsoft.KeyVault/vaults@2021-06-01-preview' = {
}

Have a second deploying reference that keyvault by using:

resource keyVault 'Microsoft.KeyVault/vaults@2021-06-01-preview' existing = {
  name: kvName
}

During the pipeline, it will error out saying that the keyvault cannot be found, but it exists in Azure.

Additional context

I ran multiple tests here.

1. I created a keyvault manually in the portal, and used "az keyvault list" to retrieve all keyvaults. The new keyvault was listed.
2. I deployed via new-azresourcegroupdeployment locally, with just the keyvault. Then used the cli 'az keyvault list' to retrieve all keyvaults. The new keyvault was listed.
3. Deploying the same script but through Azure Devops (Microsoft hosted Agent, Ubuntu-latest), then running the same command again locally, the new keyvault is NOT listed.

I check resources.azure.com - the keyvault is listed there. I tried also using powershell:

get-azkeyvault -name 'kvName'

It does not show.

But, if I use get-azresource -resourceId 'kvResourceId' - the key vault shows.

The same issues does not apply when using ARM templates. The keyvault lists through CLI when deploying in ARM.

So this is an issue with Bicep and ADO together.

[alex-frankel]

Sometimes a newly created resource in region A may not have it's info replicated to all regions of Azure, so depending on what region you are querying, it is possible for a new resource not to be returned by a particular client at a particular time. I don't think in this case the issue is specific to any particular tool/client, but rather a general replication delay issue.

Unfortunately, we don't have a great solution to this problem as this behavior is not deterministic.

Can you share the full bicep code repro? Since the KeyVault is created and reference as existing, I'm assuming this is being called across multiple modules. Want to make sure that at least all the right dependencies are in place.

[nubgamerz]

It seems your right about the replication thing... After waiting 24 hours, it works. This is incredibly frustrating, but seems to be more of an Azure issue than anything else. I just hope it gets fixed (though if you deploy the KV through the portal, it's immediately available).

I was able to confirm these findings using AZ CLI. The resource is not showing when trying to list in CLI, but if you directly reference it with a resource ID, it does list (which I guess Bicep is not doing, since you need to call the resource in order to get it's ID).

As for sharing the repo, that's not something I'm willing to do, as there's some proprietary information and code there which I'm not authorised to make public. The dependencies are correct. It's just that the resource cannot be found because Azure is not reporting it correctly.

I've also noticed the same if you go to the portal and click on Delete Resource Group, Normally, all resources in the resource group would show in the list of items that would be deleted. But those that are not reporting correctly are not shown.

After 24 hours or so, these resources are eventually made available to be read by CLI. Which then triggers the entire script to work.

I will report these findings to Microsoft directly, I'm currently working with a few engineers from MS on another project and ask them to raise this issue internally.

[alex-frankel]

I will report these findings to Microsoft directly, I'm currently working with a few engineers from MS on another project and ask them to raise this issue internally.

That's perfect - thank you. Since this is not directly related to Bicep, I'm going to close this one, but feel free to keep us updated on where this leads.

[dracan]

@nubgamerz - Did you report this? I've just hit the same issue. Can see the Bicep-generated KeyVault in the portal, but not in az keyvault list :(