Terraform import within "resource" to import programmatically in the same repo (not as a command line utility but as an attribute within HCL)

[tolgaingenc]

Current Terraform Version

0.12.8

Use-cases

Having terraform import as a command line utility not always enough. We need to be able to import and manipulate resources which are created automatically like cloud9 instances or sometimes EMR instances and similar stuff.

For cloud9 for example you create a cloud9 instance resource (aws_cloud9_environment_ec2) and it creates and ec2 instance. I want to attached an IAM role or do any other manipulation. I can use data to address the instance which is just created but can't import it to manupulate.

Attempted Solutions

there are bad work arounds like manually importing outside of the repo or writing some kind of wrapper which will call terraform import and terraform apply

Proposal

Adding a new attribute to resource called resource_to_import or import which will import the resource instead of creating it if it wasn't imported already. It should start updating after importing like the import cli command.

resource "aws_cloud9_environment_ec2" "main" { instance_type = "${var.cloud9_instance_type}" name = "my_cloud9" .... }

data "aws_instance" "cloud9" { filter { name = "tag:Name" values = ["my_cloud9-*"] } }

resource "aws_instance" "my_instance" { ami = "${var.ami_id}" ... ... resource_to_import = "${data.aws_instance.cloud9.instance_id}" }

References

No references

[rismoney]

What I like about this conceptually is it provides a means to import that is within a conventional workflow of terraform plan then apply.

terraform import doesn't quite fit into a normal pipeline. and requires interactive use. Typically this is handled inside a container inside a CI workflow. Having to interactively handle this or wrapper is heavy handed and defeats the point of all the automation.

[iaacautomation]

Another example : EKS cluster is created and it creates certain resources with it. To implement this https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-cni-walkthrough.html its requried that that the aws-node service account is modified. But terraform cannot modify the aws-node service account so its not implementable.

So an implementation which is a simple import "kubernetes_service_account" "aws-node" { metadata { name = "aws-node" namespace = "kube-system" } or if no import resource "kubernetes_service_account" "aws-node"{ modify = true/false }

Where modify assumes that the resource exists and does the "import" and modify

The documentation says "After adding inline IAM Policies (e.g. aws_iam_role_policy resource) or attaching IAM Policies (e.g. aws_iam_policy resource and aws_iam_role_policy_attachment resource) with the desired permissions to the IAM Role, annotate the Kubernetes service account (e.g. kubernetes_service_account resource) and recreate any pods."

But how to annonate a resource no created by terraform?

[winmillwill]

It would be nice for providers to have a sub-block like identification_policy where you could specify a set of labels that form a unique id, cluing terraform into your organization's conventions. Being able to guarantee that you know the identity of any resource before it's created would also allow things like detecting a renamed resource, like when you introduce for_each to handle resources that were separate before.

You may still want something else to capture the expectation that a resource already exists and avoid accidental creation.

[tolgaingenc]

But how to annonate a resource no created by terraform?

You define the resource anyway to use cli terraform import. So you will define the resource in your code and you will say import this and this time terraform apply will import this particular resource for you instead of creating it if it doesn't exist. But it should update if it exists.

So basically apply works in "Create if it doesn't exist, update if it exists" mode. This time, if I set this flag and name the cloud resource, it will work in "import if it doesn't exist, update if it exists" mode.

[shawnferry]

I have a similar problem with azure resources. Some resources "fail" but also create a resource. If I plan and apply again I get the this resource must be imported errors. Problematic resources for me seem to regularly include azurerm_application_gateway and azurerm_app_service_custom_hostname_binding

I would like to see something like the following which could be added to problematic resources.

lifecycle {
 import_existing = true
}

Also the possibility of a command line to import for add all existing

[queglay]

There is a definite need for this: For example, sometimes IAM roles, may or may not already exist like the aws-ec2-spot-fleet-tagging-role. A turn key solution to run code cannot function without some ability to auto import existing resources, when we just need to ensure they exist. Presently If I identify these types of resources I have to use Ansible instead to just ensure the resources exist which is unfortunate.

[queglay]

This is definitely not the answer, however for some scenarios, and if you are using terragrunt, this bash script example seems to work if you want to run it in a before_script prior to terraform apply:

#!/bin/bash

state_path='aws_iam_role.service_role'
resource_id='aws-ec2-spot-fleet-tagging-role'

echo "Determining if resource already exists in state file..."
output=$((terragrunt state list | grep -m 1 $state_path) 2>&1) && exit_status=0 || exit_status=$?

if [[ ! exit_status -eq 0 ]]; then # if not, then attempt import
    output=$((terragrunt import $state_path $resource_id) 2>&1) && exit_status=0 || exit_status=$?
    if [[ ! exit_status -eq 0 ]]; then # if import failed, assume we will be able to create it with terraform
        echo
        echo "The resource $resource_id will be created by terraform"
        echo
    else
        echo "$output"
        echo
        echo "The resource was imported."
        echo
    fi
else
    echo
    echo "Resource already exists, no auto-import required"
    echo
fi

It will attempt import if the resource if it doesn't exist in terraform state. If that import fails, assuming the id has no match, terraform is left to create the resource.

[nickcaballero]

Pulumi has something similar to this - https://www.pulumi.com/docs/guides/adopting/import/#pulumi-import-resource-operation.

[pmoody-]

I'd like to throw in one more, "this would be nice to have" comment. applying an iam policy to a running instance (which may or may not be managed) is exactly what I'm trying to do. It'd be nice if I could do this in the same way that that I can apply a tag to an instance without actually having the instance's state managed. eg.

data "aws_instances" "instances" {
  filter {
    # some filter that catches managed and unmanaged instances
  }
}

resource "aws_ec2_tag" "some_tag" {
  for_each = toset(flatten(data.aws_instances.isntances[*].ids))
  resource_id = each.key
  # the tag info
}

[jennings]

I've been bringing existing infrastructure under Terraform's control and I would love to have imports happen automatically when branches merge.

Since IDs are not usually sensitive, I thought it would be fine to use the same identifier that you would normally pass to terraform import, and have an import block similar to the moved block:

resource "azurerm_storage_account" "storage" { ... }

import {
  resource = azurerm_storage_account.storage
  id       = "/subscriptions/00000/...."
}

This would have the following effects:

• If azurerm_storage_account.storage needs to be created, Terraform imports the resource first then plans an update.
• If the resource already exists in state, Terraform ignores the import block and an update is planned as normal, regardless of whether the ID matches the resource in state.
• If the resource needs to be destroyed and re-created, Terraform ignores the import block.

To import a resource defined in a module, you could write the import in the root module:

module "foobar" { ... }

import {
  resource = module.foobar.azurerm_storage_account.storage
  id       = "/subscriptions/00000/...."
}

To import resources that use count or for_each:

import {
  resource = azurerm_storage_account.storage[0]
  id       = "/subscriptions/00000/...."
}
import {
  resource = azurerm_storage_account.storage[1]
  id       = "/subscriptions/00000/...."
}

Alternative syntax

I also considered having import_on_create be a lifecycle option:

resource "azurerm_storage_account" "storage" {
  lifecycle {
    import_on_create = "/subscriptions/00000/...."
  }
}

But, this would be awkward or impossible to use with count/for_each. It also might be harder to use with modules. Modules would need to accept import IDs as variables unless there was some kind of syntax like this:

module "foobar" {
  lifecycle {
    import_on_create = {
      # the key is the resource address defined inside the module
      "azurerm_storage_account.storage" = "/subscriptions/00000/..."
    }
  }
}

[kmoe]

The import block is available in Terraform v1.5 (docs).

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.