Support wildcards in defining versions used for bicep registry modules and template spec modules

[slavizh]

Is your feature request related to a problem? Please describe. We use semantic versioning. This means that when you release minor or patch version you are not introducing a breaking change and end users can use your solution without having to modify input parameters. With the current implementation of Bicep registry you have to define static version value if it is template spec or bicep registry module. This means that if you want to use new version of the referenced module you have two options:

• modify your solution to use another version. This means that you need to increase the version of that solution also. This also results in more development hours (code, test, review, release) and if you want to fix the reference of that module in multiple versions of your solution you will have to code and release multiple versions of your solutions as well
• overwrite the referenced bicep registry / template spec module. In this case the semantic versioning is lost when you use bicep registry / template spec. You are loosing following some best practices for coding and releasing.

Describe the solution you'd like as you fetch the modules upon bicep build or when developing via VS code nothing stops you if you have version defined like 1. to fetch all 1.x versions and take the latest. It is simple sorting. For example in our pipelines when we publish solutions as template spec our end users can define which version of the solution they want to publish by not only specifying version like 1.8.4 but also like 1.8., 1. and .*. That way when a solution is published as template spec they fetch the latest version available according to the value they have specified from our artifacts and publish that version as template spec.

[majastrz]

There some interesting complexities if we supported version ranges rather than single versions:

• Every module restore that uses a version range would require an additional LIST call to enumerate tags in an ACR repository before downloading the module content.
• Local caching benefits would be reduced a bit as well because we always have to query to check if there's a newer version that falls within the requested range.
• bicep build becomes non-deterministic. Running it once and then another time could produce two different results if new module was published. NPM and NuGet solve this problem via lock files that get committed side by side with the source.
• Not all versions/tags follow the x.y.z format.
• The compiled JSON that gets published to the registry contains all the dependencies as they were resolved during bicep publish. Even if you rev'd one of the dependencies, nothing will change until the module is re-published.

I am also wondering if your problem can't be solved by tagging a module with multiple tags. For example if you have a module, you could publish it with the following tags: 1.2.3, 1.2-latest, and 1-latest. The OCI registries are content-addressable, so only one copy of the module woudl exist in the registry.

@SteveLasker I'm curious what your thoughts are on this as well.

I would also argue that if the contents of the dependencies change, you should retest things regardless of what the actual version number says 😊.

[slavizh]

Some comments from me below.

There some interesting complexities if we supported version ranges rather than single versions:

* Every module restore that uses a version range would require an additional LIST call to enumerate tags in an ACR repository before downloading the module content. - If this feature is documented and resolution will be slower than having the full version defined I will be ok with that experience.

* Local caching benefits would be reduced a bit as well because we always have to query to check if there's a newer version that falls within the requested range. - Could be setting in VSC that you can set to check every 5 minutes for example in that specific scenario. It will be perfectly fine

* `bicep build` becomes non-deterministic. Running it once and then another time could produce two different results if new module was published. NPM and NuGet solve this problem via lock files that get committed side by side with the source. - Perfectly fine with that. Overall that would be the point. Better do bicep build than full development for just patch version increase in referenced module.

* Not all versions/tags follow the `x.y.z` format. - Fine if support is only for semantic versioning. Obviously it is scenario for organization that use Bicep in more advanced way.

* The compiled JSON that gets published to the registry contains all the dependencies as they were resolved during `bicep publish`. Even if you rev'd one of the dependencies, nothing will change until the module is re-published. - That is the intent. someone could just execute pipeline to just re-publish.

I am also wondering if your problem can't be solved by tagging a module with multiple tags. For example if you have a module, you could publish it with the following tags: 1.2.3, 1.2-latest, and 1-latest. The OCI registries are content-addressable, so only one copy of the module woudl exist in the registry. - Yes that is option close to the option of overwriting versions. Same problems arise. That probably will be the nasty workaround if wildcards are not supported.

@SteveLasker I'm curious what your thoughts are on this as well.

I would also argue that if the contents of the dependencies change, you should retest things regardless of what the actual version number says 😊. - The referenced module will be tested as standalone deployment. If we have made some breaking change that appears only when the module is used from another module we can just remove the published version. That will force people to use the previous stable version.

[SteveLasker]

Versioning is always an interesting challenge and often based on what interpretation, promise, and expectations are set, met, and observed. The range testing is a good question. For instance, minor versions and patches SHOULD not introduce breaking changes, rather fixing behaviors, or possibly adding some capability to fix an issue. The challenge is any code change, even a fix of a bug can be a breaking change to a consumer as the consumer's code may have some dependency on the malformed behavior. Back in the Silverlight days, we implemented quirks-mode to allow clients to pin to specific versions. But, I digress.

I agree with @slavizh that all changes should be tested and verified. But, there could be a balance, if the promise is clearly set.

• If a user binds to :1, you could argue they want all updates to :1.*
• If a user binds to :1.0, you could argue they want all updates to :1.0.*
• If a user binds to 1.0.0.0, they don't want any floating updates, and should explicitly either change their dependency to 1.0.1.0
• When 2 is released, the users bound to :1 do not float forwards, as they chose the 1.* band.
• If a user binds to :2, they want all updates to :2.* which supercedes any changes in :1.*

While I believe friends shouldn't let friends build against :latest, I get the desired concept. I want the most recent version of a thing. The most recent version can use floating tags.

For some background, I wrote about the stable and unique tags here.

The :1 tag is effectively the :latest version of anything that has :1.* The :2 tag is effectively the :latest version of anything that has :2.* The :1.1 tag is effectively the :latest version of anything that has :1.1.*

If the consumer doesn't want the float, they would bind to :1.0.0.0, or any specific 4 digit tag. By double tagging all subsequent releases with the :latest of that version, you don't actually need to query the registry, searching for newer versions of a given tag.

@majastrz, Is that what you were suggesting?

[majastrz]

@SteveLasker Yeah, pretty much.

@slavizh What you think about the multi-tagging approach?

[slavizh]

@majastrz yes, that is the approach that I understood in the beginning. If wildcarding is not supported I would go with that approach although for me it is still workaround as it is not native and the same approach have to be done on template spec as well. When you have module with version 1 you will not know if it is 1.1.0 or 1.2.0 in case someone forgot to overwrite the latest version to 1 as well. I am not fan of the workaround but if there is no native support I will be forced to use it :)

[tyconsulting]

hi @alex-frankel @majastrz any news on this feature that you can share at this stage?

[alex-frankel]

No updates on this one. The recommendation is still to overwrite the minor and major versions according to semver expectations to create a pseudo-version of the ask described here.

[Jaykul]

This is very broken right now in Bicep @alex-frankel

We've been manually publishing each "major.minor.patch" release as "major" and "major.minor" (and even "latest") with tags --the same way we do with docker container images ...

1. When we publish 1.0.0 we also set 1.0 and 1 and latest ...
2. When we publish 1.0.1 we move 1.0 and 1 and latest ...
3. When we publish 1.1.0 we move 1 and latest and set 1.1
4. When we publish 2.0.0 we move "latest" and set 2.0 and 2 ...

But the bicep cache doesn't care. If you use VSCode, or even run bicep build ... it doesn't pull the new "1" or "latest" -- it uses the one it pulled before, and you get the wrong intellisense in VSCode and your build fails.

[anthony-c-martin]

My preference would be to build semantic versioning support into the language so that it can be reasoned about and handled safely, rather than having users mutating existing tags (which IMO isn't something we designed for, and I don't feel good about recommending as a pattern).

For example, in NuGet - you can either request a specific version, or a range: https://learn.microsoft.com/en-us/nuget/concepts/package-versioning#version-ranges Similar functionality in NPM: https://gist.github.com/jonlabelle/706b28d50ba75bf81d40782aa3c84b3e#npm-version-symbols

Some mockups:

// give me exactly 1.2.3
module foo 'br:mock-registry-two.invalid/demo/site:1.2.3' = { ... }

// give me the latest matching 1.2.*
module foo 'br:mock-registry-two.invalid/demo/site:1.2.*' = { ... }

// give me the latest matching 1.*.*
module foo 'br:mock-registry-two.invalid/demo/site:1.*.*' = { ... }

This would probably also necessitate a lockfile equivalent to give users the ability to reason about implicit upgrades and handle them safely.

This also feels relevant to the discussion we're having about Bicep providers.

[slavizh]

yep, that was my initial suggestion :)