Terraform Fails to Import Resources that Reference a File when using Remote Execution

[jd4883]

• when attempting to import a resource in terraform, if that resource references a file, import commands will fail when using a remote_exec.

• I built a relatively large scale IAM user/role/keypair management framework for my company, completely in terraform 13.5 (now updated to 14.5), and except for issues running imports, the framework has worked exceptionally well. After this being an issue for quite some time we've been working around instead of having a fix, I did a lot of digging and am pretty sure I've isolated the problem. Below is a code snippet of what we use that reliably shows this issue:

  ## MODULE CALL
  module decoded-iam-entities {
  source = "../yaml/decode"
  path = "${path.root}/configs"
  filename = local.entity-type
  }

  ## MODULE CODE
  data local_file yaml {filename = "${var.path}/${var.filename}.${var.extension}"}
  output yaml {value = yamldecode(file(data.local_file.yaml.filename))}
  variable path {type = string}
  variable filename {type = string}
  variable extension {default = "yaml"}

  There is a lot more code that obscured the problem when trying to diagnose why imports would always fail. In terraform 13.5, there was always an error indicating a race condition was met and therefore resources could not be imported. When testing in 14.5, the error messaging became much more helpful:

  Error: Invalid function argument
  
  on ../../../../../modules/yaml/decode/outputs.tf line 1, in output "yaml":
  1: output yaml {value = yamldecode(file(data.local_file.yaml.filename))}
  |----------------
  | data.local_file.yaml.filename is "/terraform/envs/production/iam/roles/roles-test/configs/roles.yaml"

This lead me to believe and investigate why the file being refenced worked perfectly for plan/apply but always failed to import. My job uses Terraform Enterprise with all workspaces set to remote_exec. I had done similar projects outside of work without ever encountering this issue. When looking at the data lookup error, it becomes clear that the path referenced relates to the remote execution location and not my local path.

I believe the issue is that somewhere in the logic for how terraform import handles resources that there must be some discrepancy in how remote_exec handles references to a file, as these are a non-issue when a resource does not interact with a file reference.

For the purpose of this issue, I don't think it is relevant but th yaml file we are reading has a set of parameters that relate to an IAM user or role, which I use submodules to parse out the correct values with defaults where items are not set and create standardized IAM roles/users. I can include other code if necessary but I don't think it's going to be helpful to reproduce the issue and recreate it.

Per our account representatives suggestion, I tested with local_exec instead of remomte_exec, which initially failed. If I then take a look at my state entries, and remove the data block referencing the file, and plan/apply again, everything works. From this point on I can import. This tells me that the state file reference is always taking the remote path and does not ever seem to check the local path of any files referenced. I believe this is a bug and not a feature and would really appreciate if this can be fixed. I am able to unblock myself from being able to import resources at this time, but the approach listed above is not the most efficient and ultimately should not be needed.

Please let me know if I can provide any further data points or evidence of this issue and let me know about when a resolution is ready for a test run. I'm a huge fan of terraform, and this issue has caused a lot of pain points in being able to write more sophisticated large scale frameworks that can reliably be scaled up.

[jbardin]

Hi @jd4883

Thanks for filing the issue. I'm not certain what the exact cause of the filature is here without a way to reproduce the issue, but my hunch is that the local environment in which you are running import does not quite match the remote environment. Any arguments to functions like file are essentially considered part of the configuration, so if the value of data.local_file.yaml.filename is not valid in in the local environment, the import may fail.

Since we are working on improvements to both the remote execution model and the import workflow in general, I'm going to tag this to be included in those efforts.

[jd4883]

Hi @jbardin,

From what I can tell, it seems that remote_exec vs. local_exec is the key component. since the file is read in functionally the same in both cases, but I can do the import without issue when using local_exec, this aligns with what I am seeing.

My proposed solution, is that import does some sort of compare to what is local and what is remote when using remote_exec so that it can correctly calculate what is in a given file. It seems import is only checking what is local and since the path does in fact not match, this errors out.

Please keep me in the loop on what upcoming changes/improvements may be coming as this can be worked around, but is very cumbersome unless doing large bulk operations (we have to convert the remote_exec to local_exec, prune the data block from state relating to the file, and import will work since the data block is re-read in and updated in state.

Cheers,

Jacob