Support OCI-compliant alternative container registry for bicep publish

[dirien]

Hi everyone,

with the lastest version of bicep, we can now publish bicep modules in the ACR. This works very fine:

bicep publish secure.bicep --target 'br:dirien.azurecr.io/bicep/modules/app-service-plan:v1.0'

If i want to use, for example Artifactory its not working.

➜  flw-controlplane-azure git:(initial) ✗ bicep publish secure.bicep --target 'br:schwarzit-xx-sit-aebi-playground-docker-local.jfrog.io/bicep/modules/app-service-plan:v1.0'
Unhandled exception. Azure.RequestFailedException: Service request failed.
Status: 404 (Not Found)

Content:
404 page not found

Headers:
Date: Sat, 16 Oct 2021 16:39:28 GMT
Connection: keep-alive
X-Content-Type-Options: nosniff
Strict-Transport-Security: REDACTED
Content-Type: text/plain; charset=utf-8
Content-Length: 19

   at Bicep.Core.RegistryClient.AuthenticationRestClient.ExchangeAadAccessTokenForAcrRefreshTokenAsync(String service, String accessToken, CancellationToken cancellationToken)
   at Bicep.Core.RegistryClient.ContainerRegistryRefreshTokenCache.GetRefreshTokenFromCredentialAsync(TokenRequestContext context, String service, Boolean async, CancellationToken cancellationToken)
   at Bicep.Core.RegistryClient.ContainerRegistryRefreshTokenCache.GetAcrRefreshTokenAsync(HttpMessage message, TokenRequestContext context, String service, Boolean async)
   at Bicep.Core.RegistryClient.ContainerRegistryRefreshTokenCache.GetAcrRefreshTokenAsync(HttpMessage message, TokenRequestContext context, String service, Boolean async)
   at Bicep.Core.RegistryClient.ContainerRegistryChallengeAuthenticationPolicy.AuthorizeRequestOnChallengeAsyncInternal(HttpMessage message, Boolean async)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Bicep.Core.RegistryClient.ContainerRegistryBlobRestClient.StartUploadAsync(String name, CancellationToken cancellationToken)
   at Bicep.Core.RegistryClient.BicepRegistryBlobClient.UploadBlobAsync(Stream stream, CancellationToken cancellationToken)
   at Bicep.Core.Registry.AzureContainerRegistryManager.PushArtifactAsync(RootConfiguration configuration, OciArtifactModuleReference moduleReference, StreamDescriptor config, StreamDescriptor[] layers)
   at Bicep.Core.Registry.OciModuleRegistry.PublishModule(RootConfiguration configuration, OciArtifactModuleReference moduleReference, Stream compiled)
   at Bicep.Core.Registry.ModuleDispatcher.PublishModule(RootConfiguration configuration, ModuleReference moduleReference, Stream compiled)
   at Bicep.Cli.Commands.PublishCommand.RunAsync(PublishArguments args)
   at Bicep.Cli.Program.RunAsync(String[] args)
   at Bicep.Cli.Program.Main(String[] args)
   at Bicep.Cli.Program.<Main>(String[] args)
[1]    11253 abort      bicep publish secure.bicep --target 

It would be cool, if we had support for alternative OCP compliant registries. Most of the enterprises, have already a binary repository in place. So it could be difficult to use a different one for "just" modul sharing.

[rouke-broersma]

This was also mentioned as an open question on the original research issue #3283

[majastrz]

We feel that this is worth considering but difficult to prioritize right now. Upvoting this issue or adding comments would help us prioritize this.

[rouke-broersma]

We feel that this is worth considering but difficult to prioritize right now. Upvoting this issue or adding comments would help us prioritize this.

Outside of supporting different oci registry my company has a need for non-az cli authentication options even if using azure container registry.

Our scenario is this:

We have a platform team (I'm in this team) that would be responsible for curating Bicep modules and making them available through our azure container registry.

We are also a csp partner and we manage many tenants for many customers. We want to make these modules available to our customers so they have a standard certified (by us) way of deploying azure components.

We would provide our customers with service principal credentials with acr pull permission on our acr. We will also provide them with service principals with azure deployment permissions on their subscriptions.

Because bicep uses az cli as the authentication mechanism we now have an issue. How can our customers be logged into both the sp in our tenant with acr pull permission and the sp in their tenant with subscription deployment permission at the same time?

We are very excited to use this feature but the current authentication options make this feature dead on arrival for our use case unfortunately.

[majastrz]

Thanks for that example. I agree that this scenario would be difficult to achieve today. (There may be some workarounds by using different credentialPrecedence in config but they're not great.)

I think we have two separate issues captured here:

1. We need to support authentication with tokens available with ACR premium (https://docs.microsoft.com/en-us/azure/container-registry/container-registry-repository-scoped-permissions). This is something we are planning to do, but don't have an issue created for it yet. I will create that shortly.
2. Support for non-ACR OCI-compliant registries. More on that below...

Bicep is written in .net where as most OCI libraries that we can find are written in Go. We haven't been able to find a generic .net library to interact with OCI registries. We're currently relying on the official ACR SDK that is being built as we speak. (There's a fork of an early version checked in this repo but due to be removed soon.) This is why the current implementation is only working with ACR.

To make number 2 happen, we need to implement number 1 first and then investigate what else remains. Then, we'll either modify the ACR SDK to work with non-ACR registries or create a new one for that purpose.

[majastrz]

Created #4947 to track number 1.

[rouke-broersma]

Number 1 makes sense to me but I urge you to really consider not 'reinventing the wheel' here in terms of authentication with OCI registries. All tooling I know use some form of <tool-binary> login <loginServer> --username --password format to login to the registry. Docker uses this, helm uses this, oras uses this. Even the az cli supports az acr login which then persists your acr credentials in the docker config. I strongly feel that Bicep must support some form of bicep login or bicep registry login. This might even take the credentials automatically from your current az cli session if you don't specify separate credentials, but this command should at a minimum cache the credentials separate from az cli and it is imo not acceptable that bicep automagically logs you in to the container registry on publish simply because you happen to have access to that registry. This should be a conscious action where at a minimum you specify the registry to use on the login command.

If the registry credentials would be cached separate from the az cli we would already be unblocked on this, because in that case we could simply do a az login again after login to the registry.

[dirien]

Working much with cosign i know they support many registries to upload the signiature to. https://github.com/sigstore/cosign#registry-support

Granted it is on Go but they to like @rouke-broersma suggest with the login part. So I highly support his comment regarding the authentication with OCI registries.

[alex-frankel]

Our goal will be to reinvent as little as possible :) My understanding is the token auth is about supporting an existing feature of ACR.

When we look into supporting non-ACR registries, our goal will be to do things in as standard a way as possible to get the broadest coverage. Ideally, we get an ORAS interface in .NET, otherwise we will look into options like calling the ORAS CLI from our codebase.

[majastrz]

+1 to what @alex-frankel said. We will explore supporting Docker auth as well.

[JamesDawson]

Just to add our scenario to this thread. We were looking to use GitHub Packages to host some modules that we wanted to be publicly available without the need to authenticate.

In this case, the az publish command appears to fail whilst trying to validate the capabilities of the registry.

az bicep publish --file mymodule.bicep --target br:ghcr.io/<org-name>/<repo-name>/mymodule:v1      
Unhandled exception. Azure.RequestFailedException: Service request failed.
Status: 405 (Method Not Allowed)

Content:
{"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}

Headers:
docker-distribution-api-version: registry/2.0
Date: Mon, 31 Jan 2022 15:03:11 GMT
X-GitHub-Request-Id: REDACTED
Content-Type: application/json
Content-Length: 78

   at Azure.Containers.ContainerRegistry.AuthenticationRestClient.ExchangeAadAccessTokenForAcrRefreshTokenAsync(String service, String accessToken, CancellationToken cancellationToken)
   at Azure.Containers.ContainerRegistry.ContainerRegistryRefreshTokenCache.GetRefreshTokenFromCredentialAsync(TokenRequestContext context, String service, Boolean async, CancellationToken cancellationToken)
   at Azure.Containers.ContainerRegistry.ContainerRegistryRefreshTokenCache.GetAcrRefreshTokenAsync(HttpMessage message, TokenRequestContext context, String service, Boolean async)
   at Azure.Containers.ContainerRegistry.ContainerRegistryRefreshTokenCache.GetAcrRefreshTokenAsync(HttpMessage message, TokenRequestContext context, String service, Boolean async)
   at Azure.Containers.ContainerRegistry.ContainerRegistryChallengeAuthenticationPolicy.AuthorizeRequestOnChallengeAsyncInternal(HttpMessage message, Boolean async)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)   at Azure.Containers.ContainerRegistry.ContainerRegistryBlobRestClient.StartUploadAsync(String name, CancellationToken cancellationToken)
   at Azure.Containers.ContainerRegistry.Specialized.ContainerRegistryBlobClient.UploadBlobAsync(Stream stream, CancellationToken cancellationToken)
   at Bicep.Core.Registry.AzureContainerRegistryManager.PushArtifactAsync(RootConfiguration configuration, OciArtifactModuleReference moduleReference, StreamDescriptor config, StreamDescriptor[] layers)
   at Bicep.Core.Registry.OciModuleRegistry.PublishModule(RootConfiguration configuration, OciArtifactModuleReference moduleReference, Stream compiled)
   at Bicep.Core.Registry.ModuleDispatcher.PublishModule(RootConfiguration configuration, ModuleReference moduleReference, Stream compiled)
   at Bicep.Cli.Commands.PublishCommand.RunAsync(PublishArguments args)
   at Bicep.Cli.Program.RunAsync(String[] args)
   at Bicep.Cli.Program.Main(String[] args)
   at Bicep.Cli.Program.<Main>(String[] args)

Whilst the public registry will tick the box for the unauthenticated access requirement, I can still imagine the benefits of being able to push to an intermediate registry ahead of publishing to the official registry. For example, to support some automated testing that consumes those modules, as published.

[ghost]

Hi dirien, this issue has been marked as stale because it was labeled as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. Thanks for contributing to bicep! :smile: :mechanical_arm:

[rouke-broersma]

Not stale

[AaronCrawfis]

ORAS has a .NET client that's now in development: https://github.com/oras-project/oras-dotnet

Once it's reached a suitable stability level this could be used to interact with non-ACR registries

[Manbearpiet]

We'd love to use GitHub Packages too for Bicep 😄

[garoyeri]

We are still interested in having bicep modules published to GitHub Packages or other OCI compliant registries.

[microsoft-github-policy-service[bot]]

Hi @dirien, this issue has been marked as stale because it was labeled as requiring author feedback but has not had any activity for 7 days. It will be closed if no further activity occurs within 3 days of this comment. Thanks for contributing to bicep! :smile: :mechanical_arm:

[jpeinola]

I'd also like to see this implemented.

In enterprise context companies usually have a private artifact repository for all types of artifacts & dependencies (Docker images, Maven packages, npm modules etc.). This is usually implemented by using JFrog Artifactory, Nexus or similar. I don't want to make an exception for Bicep modules in a way that "We fetch all the packages from our private artifact repository but Bicep modules are fetched from Azure Container Registry). This causes unnecessary architectural drift in development tooling and company platform team has to spend some effort to explain this to development teams.

So as a Platform Team DevOps Engineer I want to be able to authentiate towards private OCI compliant artifact registry and pull and push bicep templates from private registry.

[andy-mcm-nov]

Also would like to see this implimented, we are just jumping on the Bicep train, but use Jfrog Artifactory as our Registry platform. Could be a bit of a hard sell as we will need to impliment ACR just for Bicep Modules.

[MicrosoftAzureAaron]

Honestly, I am surprised there is not a native way to reference Bicep modules stored in public GitHub Repos. Having to standup and then support ACR to share modules that are already stored on a public github repo seems like a step backwards.

I am just a network engineer learning bicep, but if I write and store my deployments in github, why can't a reference another repo's bicep module without cloning the external modules folder to my own repo.

I would like to be able to reference it using a URL ' https://github.com/Azure/bicep-registry-modules/blob/main/modules/network/virtual-network/main.bicep'

(yes, I know that the above is in the public Bicep module repo, but why can't I use the above URL?)

Rather than 'module hw 'br/public:samples/hello-world:1.0.2''

[alex-frankel]

@microsoftazureaaron -- We made an intentional decision to not support GH as a source for modules as it would make it easy to accidentally take a dependency on a non-versioned asset. We agree that standing up an ACR instance is a meaningful amount of friction, which is why we'd like to support other OCI-compliant registries like the GitHub Container Registry.

That being said, feel free to file this ask in a separate issue and we can think about revisiting our decision if there is enough interest.