azurerm_automation_module - module import into the automation account fails when it shouldn't. module version not handled

eissko commented 4 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.12.29
azurerm_provider: 2.25.0
Affected Resource(s)

azurerm_automation_module

Terraform Configuration Files

resource "azurerm_automation_module" "ComputerManagementDsc" {
  name                    = "ComputerManagementDsc"
  resource_group_name     = azurerm_resource_group.g.name
  automation_account_name = azurerm_automation_account.aac.name

  module_link {
    uri = "https://psg-prod-eastus.azureedge.net/packages/computermanagementdsc.8.4.0.nupkg"
  }
}

Debug Output

Panic Output

Expected Behavior

Import of the module with different version into automation account is successful even if there is already present module with the same name but different version. The terraform GET which is checking the existence of the resource should take into account VERSION property from HTTP response.

Actual Behavior

Actual Behavior is that when importing module which already exists with the same name but different version within the automation account, the module import fails with error message:

"/subscriptions/<id>/resourceGroups/<rg>/providers/Microsoft.Automation/automationAccounts/<name>/modules/ComputerManagementDsc" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_automation_module" for more information.

Which is by analyzing the DEBUG log caused by the fact that the executed terraform API GET call just check the existence against name of the resource and does not care what version of module is there.

Steps to Reproduce

The steps to reproduce the problem are quite easy. Having automation account and trying to deploy the module ComputerManagementDsc by given terraform code.

terraform apply

Important Factoids

References

Here is official reference of particular API:

https://docs.microsoft.com/en-us/rest/api/automation/module/get

In given HTTP response, there is VERSION property returned as well.


{
  "id": "/subscriptions/subid/resourceGroups/rg/providers/Microsoft.Automation/automationAccounts/myAutomationAccount33/modules/OmsCompositeResources",
  "name": "OmsCompositeResources",
  "type": "Microsoft.Automation/AutomationAccounts/Modules",
  "location": "East US 2",
  "tags": {},
  "etag": null,
  "properties": {
    "isGlobal": false,
    "version": null,
    "sizeInBytes": 0,
    "activityCount": 0,
    "creationTime": "2017-03-29T15:41:47.003+00:00",
    "lastModifiedTime": "2017-03-29T15:42:10.567+00:00",
    "error": {
      "code": null,
      "message": ""
    },
    "provisioningState": "Creating",
    "isComposite": true
  }
}

njuCZ commented 4 years ago

@eissko the rest api does not accept an input parameter "version", it could only query the latest version automation module. So there seems no point to keep an older version automation module. May I ask why do you need to do like this?

eissko commented 4 years ago

@njuCZ Hello hello, I don't know if I explain the problem well. Let me once again. When you deploy automation account, by default it contains some "predefined" modules, and those can be with old or very old version. And you don't want to use old versions, you want use newer version of module.

As my example - the module ComputerManagementDsc. It is by default with version 5.0.0.0 which is very old. I want to use version 8.4.0 but terraform is not capable of importing module which already exists in the automation account (with different old version). It fails with the message I posted above.

Why do I need this like this? - Because currently I have to use workaround with ARM template when I want to import module which already exists with lower version in automation account.

If it is still not clear, pls ping me . P.

njuCZ commented 4 years ago

@eissko in this way, you should import the resource first by using terraform import, and then modify the resource and execute terraform apply. you could refer to this https://www.terraform.io/docs/import/index.html

eissko commented 4 years ago

@njuCZ that's what I don't want. I though it could be done clever way. If we could achieve that resource ComputerManagementDsc/5.0.0.0 is not considered be equally the same as ComputerManagementDsc/8.4.0.

Import is something you don't really want to do. You want to be able deploy from a scratch, from a code.

That's why I described the problem above, if there would be nicer and more clever way to handle the version. Http response on GET contains in the propery the specific version of module.

The question is if that is possible way -> When terraform doing first check of existence, if that could consider VERSION property in response as the 2nd attribute as addition to Name of the module.

I really don't want to use terraform import. I will always prefer ARM template deploy then using azurerm_template_deployment.

Thanks, P.

ruandersMSFT commented 4 years ago

@eissko is referring to the Default Modules that are present in a new automation account. Version is a property of the API and can be found in various API calls like Get (see ContentLink and Module definitions), Create/Update. Without tracking Version as part of the Terraform State, how would Terraform know that it needs to do an update of the module to release the newer version? Version should be adequately tracked to allow version upgrade of modules (including but not limited to default modules), at which point there would never be a need to try to manual import (which defeats the purpose of automating the module deployment).

ruandersMSFT commented 4 years ago

Given that there are already default modules present post creation of a new Automation Account, Terraform should be automatically importing those into Terraform State so that there is an accurate 1:1 match between the Azure Automation account as it exists in Azure and in Terraform State. Additionally, azurerm_automation_module then needs to be able to successfully release a new version of any module (regardless of it being a built in or user created module).

eissko commented 4 years ago

Given that there are already default modules present post creation of a new Automation Account, Terraform should be automatically importing those into Terraform State so that there is an accurate 1:1 match between the Azure Automation account as it exists in Azure and in Terraform State. Additionally, azurerm_automation_module then needs to be able to successfully release a new version of any module (regardless of it being a built in or user created module).

@ruandersMSFT hm, I understand your point. There is about 23 default modules which I even don't need to use. The impact is on size of tfstate. The best would be if automation account can be deployed with 0 default modules and only having those which are explicitly define by customer.

Anyway as you mentioned, handling the version of module by terraform / azurerm_automation_module would be perfect step ahead.

Thank you, P.

ruandersMSFT commented 4 years ago

hm, I understand your point. There is about 23 default modules which I even don't need to use. The impact is on size of tfstate. The best would be if automation account can be deployed with 0 default modules and only having those which are explicitly define by customer.

The "impact on size of tfstate" should be of little to no concern from 23 default modules, as this then implies that there is a fundamental problem in tfstate size which will only get bigger in an Azure Automation Account in which a customer needs to load 100, 250 or 500 Modules? Deployment with 0 default modules is not possible.

I am also running into this issue as a road block with using Terraform as I am unable to automate the deploy the updated version of ComputerManagementDSC v8.4.0 to the automation account.

richshadman commented 4 years ago

For those of you looking for a workaround, whip out your trusty azurerm_template_deployment from the toolbelt. Here is the json deployment template. I put it in a file called AutomationAccounts.module.json under a folder automation.

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters" :
    {
        "automationAccount":
        {
            "type": "string"
        },
        "moduleName":
        {
            "type": "string"
        },
        "contentLink":
        {
            "type": "string"
        },
        "region":
        {
            "type": "string"
        }

    },
    "resources":
    [
        {
            "name": "[concat(parameters('automationAccount'), '/', parameters('moduleName'))]",
            "type": "Microsoft.Automation/automationAccounts/modules",
            "apiVersion": "2015-10-31",
            "properties":
            {
                "contentLink":
                {
                    "uri": "[parameters('contentLink')]"
                }
            },
            "location": "[parameters('region')]"
        }
    ]
}

And here is the resource (for my example I am updating AzureRM.Profile but it works with any global module):

resource "azurerm_template_deployment" "AzureRM_Profile" {
    name = "AzureRM.Profile_deployment_${substr(tostring(uuid()), 0, 8)}"
    resource_group_name = "your resource group"
    deployment_mode = "Incremental"
    template_body = file("${path.root}/automation/AutomationAccounts.module.json")

    parameters = {
        automationAccount = "your automation account"
        moduleName = "AzureRM.Profile"
        contentLink = "https://www.powershellgallery.com/api/v2/package/AzureRM.Profile/5.8.3"
        region = "your region"
    }

    lifecycle {
        ignore_changes = [
            name
        ]
    }
}

NOTE: Azure does not like redeploying the same named deployment so I make the name dynamic and add it to the lifecycle ignore_changes. That way the resource only redeploys if one of the other parameters changes.

ruandersMSFT commented 4 years ago

@richshadman Good temporary workaround for now until a v3 release can correctly handle the default state natively. Thanks.

samuelgautier commented 3 years ago

My automation deployments fails because some modules needs an updated version of a default module (ie : Az.Accounts, Az.Network, Az.storage ....) that i cannot update automatically in my Terraform code :( I would prefer to keep my IaaC pure Terraform ! I love Terraform but I hate Azurerm templates ! :)

jhulick commented 2 years ago

Thanks @richshadman. Your method worked for me. And I found this oss module, which uses the same template method, for those looking for a solution for multiple module imports/updates: https://github.com/scalair/terraform-azure-automation-module

hashicorp / terraform-provider-azurerm