Feature Request: Allow for Simplified Version Tagging

David-Duong commented 1 year ago

Hello!

Our internal versioning scheme is set up to use a simplified format, such as "0.2", it would be incredibly useful if we could have the flexibility to use a simplified version rather than SemVer.

Proposed solution: Introduce the option to use a two-digit version tagging format (major.minor), alongside the existing three-digit format. As i understand the current version format is enforced through RegEx - If needed, we can certainly provide the specific RegEx string that would accommodate this proposed change. By adjusting the RegEx, we can ensure that the two-digit format remains valid while preserving the integrity of our versioning system.

Thank you for your time and consideration.

MatthewJohn commented 1 year ago

Hey @David-Duong,

Sure, can certainly take a look at this - I assume the only change is around how tags are handled? (As opposed to how it's handled afterwards, as Terraform itself will require a full semantic version). If so, as you say, if we can allow for a regex or (or formatted string) that "fakes" the patch version as a 0, that could work

Am I understanding this all correctly?

Many thanks Matt

MatthewJohn commented 1 year ago

I've been trying to think about how this can be done nicely - unfortunately it won't just be a regex change, as we'd need to manipulate the generated version number to the valid, so it would need to be some templated value, e.g.:

version = "1.0"
git_tag_format = "v{version}"
tag_injection = "{version}.0"
result_tag = "v1.0.0"

One question I would have is whether this should be set at a global level (Terrareg system configuration) or on a per-module basis (as the git tag format currently is).

This implementation also has a problem with beta tags:

version = "1.0-beta"
git_tag_format = "v{version}"
tag_injection = "{version}.0"
result_tag = "v1.0-beta.0"

One thing I can think of doing is using a function (which we have for some other functionality), which will attempt to extract a major/minor/patch, but handles missing values and provides a default. We could have a configuration to allow varying uses of this, with a configuration for provided a default patch version (allowing a missing value), but also becomes more complex if someone wishes to treat v1.2 as v0.1.2 or v1.0.2

MatthewJohn commented 1 year ago

@David-Duong One thought is:

I'm wondering if we can add additional parameters for the "git tag format", e.g. major/minor/patch, so you could set the "git tag format" to: v{major}.{minor}

bmaximuml commented 1 year ago

Hi @MatthewJohn, Thanks for getting back to us on this so quickly.

We hadn't considered that terraform enforces the {major}.{minor}.{patch} versioning system, but it doing so makes me wonder if this is perhaps a bit of an anti-pattern to force it to accept our simplified versioning.

I'm gonna see if there's an easy way for us to just append .0 on the tags on our end, and we can pick this up again if not.

(also just wanna say I think terrareg is really cool and, this issue aside, seems to be exactly what we have been looking for in a private module registry)

MatthewJohn commented 1 year ago

Hey @bmaximuml,

I've started working on a possible fix, I'm mostly there in terms of implementation (wanted to flush through some ideas to see if I could get my head round how it would work), though isn't quite complete, nor tested and nor have I added tests - have been working on a different project and hoping to come back to it this weekend)

I'm gonna see if there's an easy way for us to just append .0 on the tags on our end, and we can pick this up again if not.

Absolutely, if you think it could work for you, it would be a lot simpler in terms of the "responsibility of Terrareg", i.e. it won't be manipulating tags and versions quite as much, but I'm happy either way :)

If I don't hear back in the next day or two, I'll do some more work on it this weekend and will post an update. If you do and you don't require the feature, I'm somewhat tempted to implement it anyway as it may help some other people, but I'll see if it has to get any hackier :laughing:

(also just wanna say I think terrareg is really cool and, this issue aside, seems to be exactly what we have been looking for in a private module registry)

That's lovely to hear, thank you :smile:

Matt

Edit:

We hadn't considered that terraform enforces the {major}.{minor}.{patch} versioning system, but it doing so makes me wonder if this is perhaps a bit of an anti-pattern to force it to accept our simplified versioning.

Whilst yes, terraform internally does expect versions to contain a full semantic version.. you could somewhat hide this with "default version format" (https://github.com/MatthewJohn/terrareg/blob/main/docs/CONFIG.md#terraform_example_version_template) in Terrareg and promoting the use of the ~> 2.0 formatting - not 100% ideal, but could work :)

MatthewJohn commented 1 year ago

Hey @bmaximuml and @David-Duong,

I've raised a PR, which I think should be good to go: https://gitlab.dockstudios.co.uk/pub/terrareg/-/merge_requests/345

I'm not sure if you've made a decision on whether to change your tagging approach, but the branch is there if you wish to test it. I'll do some final checks and look to add some more end-to-end tests for it, but will hopefully merge soon :)

Many thanks Matt

MatthewJohn commented 1 year ago

I've been having a bit of a think, and I've certainly seen some confusion around module indexing before, i.e. you have to provide the resulting semantic version, rather than the source git tag.

I'm wondering if:

Add an API to index by git tag
If a git tag format either doesn't contain a 'version' placeholder or doesn't contain all of the major, minor, patch (and build, perhaps) placeholders, then the original API for indexing by semantic version cannot be used.

I've been adding some information around the issues that could arise if using this format (i.e. if you have a format 'v{minor}' and then index 0.0.1, 0.0.2, 1.0.0, 2.0.5, they'd all be the same git tag, which could result in a lot of confusion. Where-as if we index by git tag, then they're limited to only indexing versions that their git tag format is capable of (so, in this example, it would only be possible to index 0.1.0, 0.2.0 etc.)

This new API would also mean that, if you did have the versioning v{major}, if you have a build/release pipeline, for these modules, you don't have to parse the git tag and determine the semantic number yourself to call the API, the git tag can simply be pushed.

Do you think this would make sense?

MatthewJohn commented 1 year ago

Hey :)

Apologies for the incremental updates...

I've managed to get this working in the PR: https://gitlab.dockstudios.co.uk/pub/terrareg/-/merge_requests/345 I've added a new API (and, at least, suggested deprecating the old one), which can now take a git tag or a version and disabled the indexing of modules by version, if the git tag format doesn't contain {version}.

So in summary, for this use case, once merged, you should be able to:

Create modules with a git tag format of {major}.{minor} (adding any prefixes/suffixes that you may have in your tags)
Use the github/bitbucket hooks (if you wish), or
Use the new /v1/terrareg/modules/<string:namespace>/<string:name>/<string:provider>/import endpoint to import modules with a payload: {"git_tag": "1.2"}

I hope this helps :)

Many thanks Matt

MatthewJohn commented 1 year ago

Hey, I've now merged the change, it it's released in v2.75.0 :)

Many thanks Matt

bmaximuml commented 1 year ago

Thanks @MatthewJohn, super quick turnaround! I'm off this week but I'll give this a try once I get back

MatthewJohn / terrareg

Feature Request: Allow for Simplified Version Tagging #12