Private Container registries with anonymous access

[mpr555]

Description of the issue

We are using pipeline deployments from our tenant to client tenant. We have made the container registry open for anonymous access and this is working well. but PSRule does not like it and will not authenticate.

Error output

   at Azure.Containers.ContainerRegistry.ContainerRegistryRefreshTokenCache.GetRefreshTokenFromCredentialAsync(TokenRequestContext context, String service, Boolean async, CancellationToken cancellationToken)
   at Azure.Containers.ContainerRegistry.ContainerRegistryRefreshTokenCache.GetAcrRefreshTokenAsync(HttpMessage message, TokenRequestContext context, String service, Boolean async)
   at Azure.Containers.ContainerRegistry.ContainerRegistryRefreshTokenCache.GetAcrRefreshTokenAsync(HttpMessage message, TokenRequestContext context, String service, Boolean async)
   at Azure.Containers.ContainerRegistry.ContainerRegistryChallengeAuthenticationPolicy.AuthorizeRequestOnChallengeAsyncInternal(HttpMessage message, Boolean async)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Containers.ContainerRegistry.ContainerRegistryRestClient.GetManifestAsync(String name, String reference, String accept, CancellationToken cancellationToken)
   at Azure.Containers.ContainerRegistry.Specialized.ContainerRegistryBlobClient.DownloadManifestAsync(DownloadManifestOptions options, CancellationToken cancellationToken)
   at Bicep.Core.Registry.AzureContainerRegistryManager.DownloadManifestAsync(OciArtifactModuleReference moduleReference, ContainerRegistryBlobClient client)
   at Bicep.Core.Registry.AzureContainerRegistryManager.PullArtifactAsync(RootConfiguration configuration, OciArtifactModuleReference moduleReference)
   at Bicep.Core.Registry.OciModuleRegistry.TryPullArtifactAsync(RootConfiguration configuration, OciArtifactModuleReference reference)

Module in use and version:

Using PSRule v2.7.0 Using PSRule.Rules.Azure v1.23.0

[BernieWhite]

@mpr555 Thanks for reporting the issue. What version of the Bicep are you using? It should be reported in output.

Bicep v0.14.6 should automatically try authenticated then if an unauthenticated error occurs retry as anonymous. Bicep v0.13.1 or earlier will try to authenticate and then just fail.

[mpr555]

@BernieWhite - Looks like I need a version upgrade of Bicep. I will let you know how I get on. Thanks again for the quick response.

[mpr555]

I have upgraded and still getting the following

Bicep (0.14.6) compilation of 'C:\a\1\s.devops\bicep\avd\avd.bicep' failed with: C:\a\1\s.devops\bicep\avd\avd.bicep(66,11) : Error BCP192: Unable to restore the module with reference "br:exgprdukacr001.azurecr.io/bicep/modules/microsoft.keyvault.vaults:0.5.1533": Unhandled exception: Azure.Identity.CredentialUnavailableException: Please run 'az login' to set up account

This does work when you are deploying to the same tentant etc.

Is there a different approach needed for the bicepconfig.json for PSRule? I have it set to "AzureCLI"

[BernieWhite]

@mpr555 Thanks for the additional information. Let me do some digging to work out what is going on and try to reproduce.

A few more questions:

What's ACR SKU are you using? You are using anonymous access to ACR, however do you have any private endpoints or firewall restrictions in place? Are you using any customisations in bicepconfig.json for cloud profile? Is the credentialPrecedence setting within bicepconfig.json set to: [ "AzureCLI" ]?

[mpr555]

@BernieWhite We are using Standard SKU and as per MS docs it supports anon access. We have not configure private endpoints or network restrictions as yet.

This is the bicepconfig.json snippet

{
  "cloud": {
      "credentialPrecedence": [
          "AzureCLI"
      ],
      "currentProfile": "AzureCloud"
  }
}

[BernieWhite]

@mpr555 I see what you mean. I reproduced the issue and haven't found a work around for this. I also tried it directly via the Bicep CLI outside of PSRule and couldn't make it work when completely logged out of all Azure tools.

Bicep should try to restore a module 1. authenticated, then try 2. anonymous. However it never tries the anonymous path because the credential provider called in the authenticated path errors with an exception that you are not logged in and stops the process.

Bicep only tries the anonymous path if you are authenticated to some Azure AD tenant but just don't have the ACR Pull permission (unauthorized).

So the end result is currently you need to still be authenticated to Azure, however the anonymous option will allow any authenticated user pull from the Bicep registry regardless of permissions.

---

We're going to need to get the Bicep team/ community involved on this one because I feel it will require a bug fix to the CLI.

[BernieWhite]

@mpr555 Are you able to work around this issue by setting the environment variable shown in Restoring modules from a private registry to target your multi-tenanted app?

[maxricketts]

@BernieWhite It seems that it is not working and working all in the same deployement. I have attached raw log. new log.txt

This might be to do with the pipeline is only deploying the avd.bicep in this pipeline. So it is initiating the module download locally. but with the others it is not.

[BernieWhite]

@maxricketts Ok can you try with:

{
  "credentialPrecedence": [
    "Environment",
    "AzureCLI",
  ]
}

[mpr555]

This looks the same env cred new log.txt

[mpr555]

@BernieWhite - It is pointing more towards working with the defined bicep file that is being deployed in the pipeline. PSRule is picking up other bicep files that are not being used for that pipeline, but are in the same repo for other pipelines to do with the same project, and they are not pulling down the module from the registry. Is there away of having multiple PS-rule files, and in the pipeline define which one to use? Then we could set it to only use the bicep file that is being used with the pipeline.

[BernieWhite]

@maxricketts @mpr555 I think there is a few issues from the log:

1. Restoring modules is erroring on the tenant id. Double check you have set the AZURE_TENANT_ID environment variable on the task, that it is a GUID, and that it matches your Azure AD tenant GUID (no missing characters) where the service principal is. See https://azure.github.io/PSRule.Rules.Azure/using-bicep/#set-pipeline-environment-variables
2. The officeVm.bicep module is failing with a missing appSecret parameter. Since hard coding a secret is a bad idea you can use the AZURE_PARAMETER_DEFAULTS option to set a placeholder value for testing. See https://azure.github.io/PSRule.Rules.Azure/setup/configuring-expansion/#required-parameter-defaults

Notes:

If you are using a self-hosted agent, then some of the modules could be cached which is why these are not failing to download. the Bicep CLI only attempts to download them locally if they are not already in the cache.

---

In regards to your question. If you are deploying from a single .bicep file then you could exclude other Bicep modules. Since all the other Bicep modules are included in Bicep deployment when it is compiled, it is all tested by the root Bicep deployment.

You could do this either:

• Set the inputPath parameter on the task. Which will accept a path or specific file. See: https://azure.github.io/PSRule.Rules.Azure/creating-your-pipeline/#limiting-input-to-a-specific-path
• You could configure a path exclusion with the Input.PathIgnore option, see: https://azure.github.io/PSRule.Rules.Azure/using-bicep/#configuring-path-exclusions

[maxricketts]

@BernieWhite - It seems to be working on the whole. But I am seeing some issues. Example. I have 3 folders in my Azure DevOps repo, inside the folders I have various amounts of bicep files, usually 1-3. only one of those files will be linked to a pipeline. When the pipeline runs, PSRule is trying to scan all of them, and it fails as the bicep build process is not pulling down that particular module for the bicep files that are not linked to the pipeline.

I would be good if we could per pipeline exclude those files, or have multiples ps-rule.yaml files that we can somehow set with inthe psrule task in the pipeline.

If I have missed something and there is away of doing this, then please can you link me to the docs that state this.

Kind regards,

Max

[BernieWhite]

@mpr555 @maxricketts You can by setting the Input.PathIgnore option. There is an example here: https://azure.github.io/PSRule.Rules.Azure/using-bicep/#configuring-path-exclusions

Also in the Bicep team gave an update that the anonymous bug should be fixed in the next release v0.15.x which is imminent.

[BernieWhite]

@maxricketts The v0.15.31 release of Bicep is live. https://github.com/Azure/bicep/releases/tag/v0.15.31

---

If you need further assistance with the Input.PathIgnore option? or are we fine to close the issue?

[maxricketts]

@BernieWhite - I will test and let you know.

[maxricketts]

@BernieWhite - I dont think the input path is going to work in this instance. Here is my folder structure for a little AVD project I am working on. We are using this template to deploy to all of our clients.

As you see we have a few pipeline files that run various bicep files for this project.

Ideally I would like to include just the bicep file that I am running in the pipeline and pass that into the PSRule step, but I am unsure how. I know you git it a path, but being able to define just a file would be good.

Let me know your thoughts on this.

Kind regards,

Max

[BernieWhite]

@maxricketts Yes, that can be done as mentioned above.

---

Set the inputPath parameter on the task. Which will accept a path or specific file. See: https://azure.github.io/PSRule.Rules.Azure/creating-your-pipeline/#limiting-input-to-a-specific-path

[maxricketts]

@BernieWhite - I had looked at this, but it only states for path, and it didn't state that you can specify a file. I will try this and let you know.

[maxricketts]

@BernieWhite - This has worked an absolute treat!