[Question/Feedback]: Request for Version Information in Terraform Resource Modules CSV for Automated Pipeline Integration

[tomaszurek]

Check for previous/existing GitHub issues

• [X] I have checked for previous/existing GitHub issues

Description

Hi, I am currently working on automating the process of forking Terraform modules from the provided CSV into our internal Azure DevOps (ADO) repositories. My goal is to implement internal module library with a system that allows us to check for breaking changes and conduct unit testing before merging these modules into our main branch. This would enable us as platform team to offer these modules internally to our app teams with greater confidence in their stability.

However, we have encountered a significant challenge in this process. The CSV file does not include version information for each module. Without versioning, it becomes extremely difficult for our pipeline to determine whether a module has changed and whether it should be re-forked and re-tested.

Request:

Could you please consider adding a version field to the CSV file? This would greatly enhance our ability to automate and streamline the process of module management, allowing us to track changes more effectively and ensure that we are always working with the most up-to-date versions of each module.

We believe this enhancement would not only benefit our team but also others who are integrating these modules into their CI/CD pipelines.

Thank you for considering this request.

Best Tomas

[microsoft-github-policy-service[bot]]

[!IMPORTANT] The "Needs: Triage :mag:" label must be removed once the triage process is complete!

[!TIP] For additional guidance on how to triage this issue/PR, see the AVM Issue Triage documentation.

[matebarabas]

@tomaszurek, thank you for your patience!

@@matt-FFFFFF, can you please take a look at this? Thanks!

[jaredfholgate]

In order to achieve this we would need to automate the generation of the CSV, which is currently manual. I think we can probably expose an API that reads the version directly from GitHub or Terraform registry rather than trying to add it to the CSV.

In the meantime, you can use this API call and page through the results to get what you need: https://registry.terraform.io/v1/modules/search?q=avm-&namespace=Azure&limit=100

[jaredfholgate]

Here is some example PowerShell to create a CSV:

$resultsPerPage = 15
$url = "https://registry.terraform.io/v1/modules/search?q=avm-&namespace=Azure&limit=$resultsPerPage"

$results = @()
$result = Invoke-RestMethod -Uri $url

$results += $result.modules

while ($result.meta.next_url) {
    $result = Invoke-RestMethod -Uri $result.meta.next_url
    $results += $result.modules
}

$results | Export-Csv -Path "C:\temp\microsoft-azure-avm-modules.csv" -NoTypeInformation

[matebarabas]

@tomaszurek, I can also echo what Jared summarized above. The intent behind our CSV's is to drive our module admittance (known as initial module triage process), therefore they're not meant to track version information. In the online version of the AVM module index (shown on the AVM website, here), we're however always dynamically reflecting the latest available module version for each module.

You can also see how this is done at the code level here.