[Question/Feedback]: Azure DevOps repository sync with GitHub and vice versa

[vjmanda]

Check for previous/existing GitHub issues

• [X] I have checked for previous/existing GitHub issues

Feature request

Allow syncing/tracking between Azure DevOps repositories and GitHub and vice versa.

Description

After an enterprise adopts AVM, there might be a need to customise a given module’s functionality due to some specific internal use case. In such scenarios it makes perfect sense to create a fork on GitHub and apply the necessary customisation.

However, this could be challenging in the long run for enterprises who have adopted Azure DevOps repositories. After importing the repositories into AzDO, there would be no easy way to track / sync changes from the upstream repository and therefore could risk missing out on important security and feature updates. The workaround to synchronise multiple remote Git repositories offers a solution but is a little cumbersome. Therefore, it would be nice to allow syncing/tracking between AzDO repos and GitHub and vice versa.

[matebarabas]

@Azure/avm-core-team-technical-bicep, @Azure/avm-core-team-technical-terraform, can you please take a look and advise? Thanks!

[AlexanderSehr]

Hey @vjmanda, is this an ask for a new product feature, i.e., a GitHub/AzureDevOps-native built-in feature that allows the functionality? If so, I'm not sure this is the best location to raise this as our team is not developing either product.

If not - I can only speak to the Bicep-side of things: If it is an ask for a CI + added feature (e.g. via a workflow) it's a different story. I'd still argue it's not something we could implement anytime soon as the BRM respository is not indendet to be integrated with othr CI environments (at least today). Added by the fact that if we'd add this feature, we'd also need to continuously validate that it works.

That being said, this could become a part of the CARML-CI story - who's compatbility with AVM we want to invest in in the period going forward.

So much for my thoughts. Ideally the other maintainers also add their thoughts (cc: @Azure/avm-core-team-technical-bicep, @Azure/avm-core-team-technical-terraform)

[jaredfholgate]

Like you say there is no native feature to do this, only a one time import facility. We can discuss this internally, but we could help with a pattern, guidance and perhaps leverage something like the Azure DevOps provider or API as a potential example pattern to refresh repos using the import function. https://registry.terraform.io/providers/microsoft/azuredevops/latest/docs/resources/git_repository#create-import-from-another-git-repository

I don’t think we would be able to provide an officially supported solution at this stage, but we can certainly provide guidance around repo structure. As in make sure your copy of our AVM repo is immutable and only reference it from your own module repo.

[vjmanda]

Thanks @AlexanderSehr and @jaredfholgate, Just to clarify, this is not an AVM (Bicep / Terraform) related feature request. The concerns are not about importing the repo or the structure but more on the lines of what happens after the repo is imported into Azure DevOps. How would the imported repo keep in sync with the latest updates to the upstream repo. Currently there is no way to track these.

I totally agree that this might not be the right place for a feature request relating to GitHub/Azure DevOps. I only wanted to highlight the potential issue organisations currently using Azure DevOps repos will face in the long run after they adopt AVM. The intention was to bring this to your attention so that you can escalate and discuss internally.

Looping in @jtracey93 / @cbezenco

[matt-FFFFFF]

Hi,

Both azdo and gh use git.

This seems like standard forking procedure.

As long as you don't make changes to your fork, you can configure git to have multiple remotes and synchronise them on a schedule. You would pull the upstream changes into a branch (for review) and then rebase those onto main once happy.

This way you maintain the same commit hash as upstream and can be sure you have an identical copy.

A simple pipeline could do this on a schedule for you.

[johnlokerse]

I would like to share my view on this. I know what you are trying to achieve @vjmanda, because I have been in the same situation (using Bicep AVM, not Terraform AVM).

I would not recommend automatically updating a cloned AVM repository because your infrastructure deployment will become unstable. Every module update will be unversioned and you will always get the latest version in your repository. This can (will) cause problems within your Bicep templates. A simple example, a new required parameter of a module is introduced and this is expected to be in your bicepparam file if you don't have this then your deployment will fail. This is just one of many problems that can occur.

Besides that the amount of changes that you get is a lot. The AVM repository is very active and daily changes happen, so this will add administrative load to the repository. You can do tagging to overcome a lot of problems, but there still is a lot to manage (automation, testing, does it break your Bicep templates). Additionally, having automation is tricky too, because the folder structure can change and this will make the automation unreliable as you don't have any control over all of this.

If the above does not apply to you, then I would go for @matt-FFFFFF's suggested solution. 😃

[AlexanderSehr]

I would like to share my view on this. I know what you are trying to achieve @vjmanda, because I have been in the same situation (using Bicep AVM, not Terraform AVM).

I would not recommend automatically updating a cloned AVM repository because your infrastructure deployment will become unstable. Every module update will be unversioned and you will always get the latest version in your repository. This can (will) cause problems within your Bicep templates. A simple example, a new required parameter of a module is introduced and this is expected to be in your bicepparam file if you don't have this then your deployment will fail. This is just one of many problems that can occur.

Besides that the amount of changes that you get is a lot. The AVM repository is very active and daily changes happen, so this will add administrative load to the repository. You can do tagging to overcome a lot of problems, but there still is a lot to manage (automation, testing, does it break your Bicep templates). Additionally, having automation is tricky too, because the folder structure can change and this will make the automation unreliable as you don't have any control over all of this.

If the above does not apply to you, then I would go for @matt-FFFFFF's suggested solution. 😃

Thanks for sharing your thoughts @johnlokerse.

One thing I'd like to mention is a great solution @ahmadabdalla implemented which effectively adds a middle layer. If I recall it correctly, he implemented some automation to pull the latest changes into a repository from where they'd be published into a private registry. And only those versioned entities would be used going further. @ahmadabdalla, please correct me if I'm wrong - it's in any case a vast simplification 😄

[ahmadabdalla]

Thanks @AlexanderSehr

I've referred to this as an overlay pattern where basically you have two repos, a clone of AVM (previously it was the CARML repo) that is constantly synced with Upstream as the team outlined. This clone has minimal customisation to avoid conflicts. The clone basically publishes to an internal bicep registry. The other repo (overlay) consumes the published module but now adds its own customisation like hardcoding parameters and not exposing them to end users. It has its own publishing workflow. This helps create an internal versioning model and would still be backed by AVM. This does bring additional management btw so the team maintaining it needs to weigh in the pros and cons.