Ability to reference "external" modules

alex-frankel commented 4 years ago

As we finish up support for "local" modules (modules that exist in the same repo as the current .bicep file), we need to follow it closely with support for "external" modules (modules that do not exist in the same repo). We expect the mainline case to be referencing public github repos, but other locations that are publicly accessible should work as well. For now, we are taking pretty heavy inspiration from the GitHub actions implementation of the uses property.

Goals

with a terse syntax, I can reference bicep modules that live on github (e.g. bicep/vm)
external modules must be (should be?) versioned. I should get intellisense for available versions.
consuming external modules should be "just as good" as consuming local modules
- intellisense and validation should work just as well

Non-goals:

support private repos. We will support private repos in some capacity in the future, and the design should take it into consideration, but we don't need to close on this right now
"first-class" support for non-github repos. We should allow users to provide the full .git url, but we don't need an equally terse syntax for this

Open questions:

what org should we use for "microsoft provided" modules? GitHub created a net-new org for actions called actions, which helps them keep the declaration shorter and they won't have conflicts with anything in the MS or Azure orgs
what types of versioning should we support? GitHub actions supports branch, tag, and SHA.

slavizh commented 4 years ago

I would like to ask if with this work you will able to call a template spec or that is separate work/path where we just need to use the template spec resource with linked deployment to call it?

alex-frankel commented 4 years ago

@slavih - great question. I think we should support it as part of the modules spec, as we also want to allow you easily reference a publicly accessible or local template JSON file. The syntax could be something like this:

var tsId = resourceId('Microsoft.Resources/templateSpecs', 'networking', '1.0')

module networking tsId = { ... }

But I think that introduces it's own complexities as we don't yet support expressions for types. Also not sure how we would do validation & intellisense since it would require a roundtrip to Azure to figure out what the template spec looks like.

Stijnc commented 4 years ago

As previously discussed, would a Open Container Initiative (OCI) registry also be an option? my opiniated view:

supports versions
possibility to offer a public bundle, but keeping the repo private
package invocation and bundles for easy execution

As an example, ACR already supports OCI artifacts.

Just to be clear, I do think repositories should be an external module store, but it should not be limited to just repositories.

maybe @SteveLasker also has an opinion?

SteveLasker commented 4 years ago

Hi folks, We'd love to see and support bicep as an OCI Artifact. We've had this similar discussion for ARM templates prior. The thing to also realize is GIT is NOT a production asset. It's not geo-replicated to assure reliability, close to your deployments, typically not in a VNet, doesn't support customer managed keys, etc. While GIT is an important source of the workflow, I'd suggest teams should NOT depend on GIT for production deployments, scaling, or fault tolerance.

We've started conversations with the Weave folks about elevating GitOps to RegOps. Where Git pushes content to a registry, which is geo-replicated, secured, reliable, scales, and the production deployments are done from the registry. GitOps would still be the term, but there could be a higher dependency on the registry to provide the scalability, reliability, and security required for critical production workloads.

Now that all major clouds support OCI Artifacts, this isn't just an Azure thing, but a cloud-native thing.

However, something else to consider. As more Azure Services require registries to boot-up new regions, ACR is becoming a dial-tone service. Having your ARM or Bicep template available within ACR as an OCI artifact will enable Azure Regions to boot-up.

There's lots of content under OCI Artifact Authors Guidance

Happy to help with anything that's not covered as well. Steve

miqm commented 3 years ago

In my humble opinion, we might not need this feature, if we implement linking external arm templates.

If we treat bicep as source code language (like c#) then we should not link external code -but the output of compilation - ARM Template.

ARM Templates do have now established way of versioning, public repos, etc.

If we treat bicep as source code language then we could enable things like compilation directives ( ifdef) or including other files to build one bugger template deplyment rather than series of small sub-deployments.

peter-bertok commented 3 years ago

But I think that introduces it's own complexities as we don't yet support expressions for types. Also not sure how we would do validation & intellisense since it would require a roundtrip to Azure to figure out what the template spec looks like.

My concern is that it looks like Template Specs is reinventing source control, but in Azure and worse.

This means that Templates in general (Bicep or not) will have "versions" in both Git and ARM. What does it even mean to branch the code in this scenario?

I was hoping Bicep would eliminate the need to use URLs for modules in templates, and that proper Git-based development would become the norm...

miqm commented 3 years ago

But I think that introduces it's own complexities as we don't yet support expressions for types. Also not sure how we would do validation & intellisense since it would require a roundtrip to Azure to figure out what the template spec looks like.

My concern is that it looks like Template Specs is reinventing source control, but in Azure and worse.

This means that Templates in general (Bicep or not) will have "versions" in both Git and ARM. What does it even mean to branch the code in this scenario?

I was hoping Bicep would eliminate the need to use URLs for modules in templates, and that proper Git-based development would become the norm...

I think of Template Specs not as source code but as organisation-wide package manager. You do version your code and then during build you version yours application packages that you want to share. Same would be here.

alex-frankel commented 3 years ago

Just want to +1 what @miqm said. Our goal is not to replicate/replace source control with template specs. A template spec is more like an npm module or nuget package that is stamped with a version at distinct points in its lifecycle.

So the expected workflow is:

author bicep code
check-in/manage with source control
Validate/test code locally and/or with CI
publish template spec with az cli/ps directly with .bicep file (this is not supported yet, but will be)

peter-bertok commented 3 years ago

A template spec is more like an npm module or nuget package that is stamped with a version at distinct points in its lifecycle.

From what I have seen (in my admittedly limited experience), Template Specs use relatively fine-grained relative references to other specs. I.e.: Down to the level of "output only" templates for establishing naming conventions and the like.

Meanwhile Bicep seems to prefer merging multiple modules into a single template instead of preserving the references.

It's these kinds of impedance mismatches between the two systems that leave me scratching my head a little bit...

SteveLasker commented 3 years ago

I'm no expert on the details of bicep. What I am seeing is a question for when source becomes a published "binary" that is needed for distribution. The binary is versioned and may have dependencies on other binaries, or environmental configuration values. While gitops is great, a conversation we've been having is how much should git be part of the production deployment. Meaning, we still keep the git elements, but farther to the "left". The output of your deployment is packaged, as a "binary", and optimized for distribution. This gives you all the benefits of git for who/what/when changes were made, while leveraging deployment optimizations, including performance, reliability and security. In this case, at what point is the "compiled" output of a bicep template staged for deployment? When it is, the proposal above allows publishing bicep as an artifact, with the ability to reference other artifacts. I may be way off in the intentions here.

JustinGrote commented 3 years ago

I think the way GoLang Modules work is a good approach to take, it's a simple syntax to consume that works with third party repositories and you just have some sort of basic manifest to replace go.mod, and versioning is done via git tags. terraform modules are similar but go modules are more intuitive and less unnecessary boilerplate and "compiling" to get them into their marketplace. A tag is a release, and they should follow semver but are not required to.

I'm currently using a similar approach by importing my "building block" bicep modules to my "stack" repository as git submodules with pinned tags, then referencing them with just relative file paths.

miqm commented 3 years ago

Another idea (as we're in ideation phase I think :) ) - https://www.terraform.io/docs/language/modules/sources.html#generic-git-repository

realrubberduckdev commented 3 years ago

Another idea (as we're in ideation phase I think :) ) - https://www.terraform.io/docs/language/modules/sources.html#generic-git-repository

Supporting generic git repo as module source (with ref) will be amazing. It would make easier to use external modules as well as allow to lock to specific versions.

I'm considering moving away from state file based IaC code base and adopt bicep. Currently the lack of sharing and consuming bicep modules is a blocker.

alex-frankel commented 3 years ago

@realrubberduckdev - out of curiosity, how much does git submodules solve some of this for you? As an end user consuming external modules would go something like:

create new repo
add submodule which points to external bicep modules repo
pull regularly to get any module updates

submodules have always felt a bit foreign to me, but I am not sure how people find them generally.

Either way, we definitely recognize the importance of both a public and private module registry backed by source control in git. I'm hoping we have something available in June or July as part of an 0.5 release (0.4 is mostly a quality/bugfixes release).

miqm commented 3 years ago

we use submodules for our 'external' bicep code and although this mechanism is pretty powerful, it's biggest disadvantage is using hash as reference, which makes hard to resolve merge conflicts and you need external tools to check which version is 'newer'. on the other side, with submodules it's faster to develop simultaneously main infra and external module together.

AlexanderSehr commented 3 years ago

To bring yet another related request to the table:

On a platform I'm working on we host individual modules for Azure services (that is, a feature-rich ARM template). We then leverage these modules in multiple different ways, but most notably to orchestrate infrastructure deployments. This orchestration template does not define any resources on its own, but only links our modules together in the order we want them deployed. This linkage happens either via a storage account reference, or template specs.

"templateLink": {
   "uri": "[concat(variables('modulesPath'), 'deploy.json', concat('?', listAccountSas(parameters('componentStorageAccountId'), '2019-04-01', variables('accountSasProperties')).accountSasToken)))]",
   "contentVersion": "1.0.0.0"
}

My feeling is, that 'modules' are somewhat like linked templates. However, as there is no way to reference bicep files from any remote location, I see no way to orchestrate them if I don't have local references handy.

Though the bicep registry seems like a good alternative (=> like a template spec counterpart for ARM), it would be awesome if there'd be support for template specs or urls in general.

sajayantony commented 3 years ago

There are multiple aspects to leveraging OCI here.

Content addressable storage provides an accepted content validation strategy of the imported modules that can bound to a digest.
Most cloud providers have an OCI registry and there is a CNCF version of a self hostable registry.
The OCI work is also evolving to support artifact linking and signing which will work itself into the secure supply chain initiative.

dkirrane commented 3 years ago

+1 for modules via generic git repo. We currently use Terraform modules, each in their own repo (not using submodules). Each PR into a module repo runs a Terratest to verify the module works as expected. On merge of PR to main we increment the version of the module via git tag.

Something similar for bicep would be great. e.g. main.bicep pulling in a module like:

module aks 'git::https://github.com/mycompany/bicep-module-aks.git?ref=v1.2.0' = {
}

SteveLasker commented 3 years ago

Attempting to create Yet Another Storage Solution is a huge challenge.

How do you synchronize content across multiple regions?
How do you promote content into VNet, air-gapped networks?
How do you sign content?
How do you enable double encryption at rest

How can we help the team leverage registries, across all cloud providers and on-prem instances, to store bicep modules?

majastrz commented 3 years ago

+1 for modules via generic git repo. We currently use Terraform modules, each in their own repo (not using submodules). Each PR into a module repo runs a Terratest to verify the module works as expected. On merge of PR to main we increment the version of the module via git tag.

Something similar for bicep would be great. e.g. main.bicep pulling in a module like:
module aks 'git::https://github.com/mycompany/bicep-module-aks.git?ref=v1.2.0' = {
}

JSON is the intermediate language for Bicep, so module content will be consumed via JSON rather than Bicep files directly. (In short, this allows us to decouple compiler versions and linter settings between the producer and consumer of a module. See #3266 for more details.)

With a Git-based workflow, this forces an extra compilation step that has to be done either after edits to .bicep files are done or be done by a CI job after a check-in or PR gets merged. The mixing of source and build artifacts within the same repo causes the workflows to be unnatural with Git. TF doesn't require a compilation, so they didn't have to solve the problem.

majastrz commented 3 years ago

Attempting to create Yet Another Storage Solution is a huge challenge.

How do you synchronize content across multiple regions?

How do you promote content into VNet, air-gapped networks?

How do you sign content?

How do you enable double encryption at rest

How can we help the team leverage registries, across all cloud providers and on-prem instances, to store bicep modules?

We want to leverage an existing package management or storage solution. We have no intention of inventing a new one. NuGet, OCI, and Git-based approaches are currently being considered (see #2128, #3188, #3283, #3556).

SteveLasker commented 3 years ago

@majastrz, thanks. This is great to hear. I imagine there may be multiple solutions. I'll track #3283 to see how we can help, and what gaps may exist.

JRRN commented 3 years ago

we use submodules for our 'external' bicep code and although this mechanism is pretty powerful, it's biggest disadvantage is using hash as reference, which makes hard to resolve merge conflicts and you need external tools to check which version is 'newer'. on the other side, with submodules it's faster to develop simultaneously main infra and external module together.

Do you think it's better to use versioning in modules instead external tools?

alex-frankel commented 2 years ago

Closing this. Private registry work is done, and the remaining work is tracked with #2128

Azure / bicep