Running in a Terraform Cloud Agent

[yohanb]

Hi! First off, thanks for creating this tool. I don't understand how Terraform can't have better support for state migration (moved blocks can only do so much). I have a use case where I would have liked to run tfmigrate in the context of Terraform Cloud and it seems like the best way would be to add the functionality to a self-hosted Terraform Cloud Agent.

Has anyone been able to use tfmigrate successfully in a tfc-agent? Getting stuck with this issue for the moment:

2022/09/13 13:10:23 [DEBUG] [main] start: tfmigrate plan tfmigrate.hcl
2022/09/13 13:10:23 [DEBUG] [main] tfmigrate version: 0.3.7
2022/09/13 13:10:23 [DEBUG] [command] load configuration file: .tfmigrate.hcl
2022/09/13 13:10:23 [DEBUG] [command] config: &config.TfmigrateConfig{MigrationDir:".", IsBackendTerraformCloud:false, History:(*history.Config)(nil)}
2022/09/13 13:10:23 [DEBUG] [command] option: &tfmigrate.MigratorOption{ExecPath:"", PlanOut:"", IsBackendTerraformCloud:false, BackendConfig:[]string(nil)}
2022/09/13 13:10:23 [INFO] [runner] load migration file: tfmigrate.hcl
2022/09/13 13:10:23 [INFO] [migrator] start state migrator plan
2022/09/13 13:10:23 [DEBUG] [executor@.]$ terraform version
2022/09/13 13:10:23 [INFO] [migrator@.] terraform version: 1.2.9
2022/09/13 13:10:23 [INFO] [migrator@.] initialize work dir
2022/09/13 13:10:23 [DEBUG] [executor@.]$ terraform init -input=false -no-color
2022/09/13 13:11:08 [DEBUG] [executor@.]$ terraform workspace show
2022/09/13 13:11:08 [DEBUG] [migrator@.] currentWorkspace = ***, workspace = ***
2022/09/13 13:11:08 [INFO] [migrator@.] get the current remote state
2022/09/13 13:11:08 [DEBUG] [executor@.]$ terraform state pull
2022/09/13 13:11:15 [INFO] [migrator@.] override backend to local
2022/09/13 13:11:15 [INFO] [executor@.] create an override file
2022/09/13 13:11:15 [INFO] [migrator@.] creating local workspace folder in: terraform.tfstate.d/***
2022/09/13 13:11:15 [INFO] [executor@.] switch backend to local
2022/09/13 13:11:15 [DEBUG] [executor@.]$ terraform init -input=false -no-color -reconfigure
2022/09/13 13:11:20 [DEBUG] [executor@.] failed to run command: (*exec.ExitError)(0xc0000d8040)(exit status 1)
failed to switch backend to local: failed to run command (exited 1): terraform init -input=false -no-color -reconfigure
stdout:
Initializing modules...

Initializing Terraform Cloud...

stderr:

Error: Invalid command-line option

The -reconfigure option is for in-place reconfiguration of state backends
only, and is not needed when changing Terraform Cloud settings.

When using Terraform Cloud, initialization automatically activates any new
Cloud configuration settings.

[minamijoyo]

Hi @yohanb, Thank you for opening this!

Have you tried the is_backend_terraform_cloud = true flag? https://github.com/minamijoyo/tfmigrate/tree/v0.3.7#is_backend_terraform_cloud

[yohanb]

Hi @minamijoyo! Yes I have and same result.

[minamijoyo]

I’m aware that some folks use the tfmigrate with the Terraform Cloud. That is why the is_backend_terraform_cloud flag exists, which was added by the community. However, honestly, I don’t know how to integrate them because I’m not a TFC user :upside_down_face:

I suggest you provide more context on what you are trying and how to reproduce the error so that someone can identify the difference between you and others.

[mtamiola-priv]

I'm also trying to use tool with TF cloud, but trying to run that locally.I guess that could work?. It passes successfully switch to local, plan phase and it fails at switching back to remote. I'm trying multistate migration, but I guess it is not matter here

2022/11/30 08:26:37 [DEBUG] [main] tfmigrate version: 0.3.8
2022/11/30 08:26:37 [DEBUG] [command] load configuration file: .tfmigrate.hcl
2022/11/30 08:26:37 [DEBUG] [command] config: &config.TfmigrateConfig{MigrationDir:"./tfmigrate", IsBackendTerraformCloud:true, History:(*history.Config)(nil)}
2022/11/30 08:26:37 [DEBUG] [command] option: &tfmigrate.MigratorOption{ExecPath:"", PlanOut:"", IsBackendTerraformCloud:false, BackendConfig:[]string(nil)}
2022/11/30 08:26:37 [INFO] [runner] load migration file: tfmigrate/migration.hcl
2022/11/30 08:26:37 [INFO] [migrator] start multi state migrator plan phase for apply
2022/11/30 08:26:37 [DEBUG] [executor@workspaceA]$ terraform version
2022/11/30 08:26:39 [INFO] [migrator@workspaceA] terraform version: 1.2.2
2022/11/30 08:26:39 [INFO] [migrator@workspaceA] initialize work dir
2022/11/30 08:26:39 [DEBUG] [executor@workspaceA]$ terraform init -input=false -no-color
2022/11/30 08:26:53 [DEBUG] [executor@workspaceA]$ terraform workspace show
2022/11/30 08:26:55 [DEBUG] [migrator@workspaceA] currentWorkspace = dev, workspace = dev
2022/11/30 08:26:55 [INFO] [migrator@workspaceA] get the current remote state
2022/11/30 08:26:55 [DEBUG] [executor@workspaceA]$ terraform state pull
2022/11/30 08:27:04 [INFO] [migrator@workspaceA] override backend to local
2022/11/30 08:27:04 [INFO] [executor@workspaceA] create an override file
2022/11/30 08:27:04 [INFO] [migrator@workspaceA] creating local workspace folder in: workspaceA/terraform.tfstate.d/dev
2022/11/30 08:27:04 [INFO] [executor@workspaceA] switch backend to local
2022/11/30 08:27:04 [DEBUG] [executor@workspaceA]$ terraform init -input=false -no-color -reconfigure
2022/11/30 08:27:12 [DEBUG] [executor@workspaceB]$ terraform version
2022/11/30 08:27:14 [INFO] [migrator@workspaceB] terraform version: 1.2.2
2022/11/30 08:27:14 [INFO] [migrator@workspaceB] initialize work dir
2022/11/30 08:27:14 [DEBUG] [executor@workspaceB]$ terraform init -input=false -no-color
2022/11/30 08:27:23 [DEBUG] [executor@workspaceB]$ terraform workspace show
2022/11/30 08:27:25 [DEBUG] [migrator@workspaceB] currentWorkspace = dev, workspace = dev
2022/11/30 08:27:25 [INFO] [migrator@workspaceB] get the current remote state
2022/11/30 08:27:25 [DEBUG] [executor@workspaceB]$ terraform state pull
2022/11/30 08:27:30 [INFO] [migrator@workspaceB] override backend to local
2022/11/30 08:27:30 [INFO] [executor@workspaceB] create an override file
2022/11/30 08:27:30 [INFO] [migrator@workspaceB] creating local workspace folder in: workspaceB/terraform.tfstate.d/dev
2022/11/30 08:27:30 [INFO] [executor@workspaceB] switch backend to local
2022/11/30 08:27:30 [DEBUG] [executor@workspaceB]$ terraform init -input=false -no-color -reconfigure
2022/11/30 08:27:36 [INFO] [migrator] compute new states (workspaceA => workspaceB)
2022/11/30 08:27:36 [DEBUG] [executor@workspaceA]$ terraform state mv -state=/var/folders/yh/7vgmx6qs6b33f6rs3xb0cy5c0000gn/T/tmp4017198946 -state-out=/var/folders/yh/7vgmx6qs6b33f6rs3xb0cy5c0000gn/T/tmp47193629 -backup=/dev/null module.audit_logs module.audit_logs
2022/11/30 08:27:41 [DEBUG] [executor@workspaceB]$ terraform state mv -state=/var/folders/yh/7vgmx6qs6b33f6rs3xb0cy5c0000gn/T/tmp4096028056 -state-out=/var/folders/yh/7vgmx6qs6b33f6rs3xb0cy5c0000gn/T/tmp2540095428 -backup=/dev/null module.audit_logs module.audit_logs
2022/11/30 08:27:43 [INFO] [migrator@workspaceA] check diffs
2022/11/30 08:27:43 [DEBUG] [executor@workspaceA]$ terraform plan -state=/var/folders/yh/7vgmx6qs6b33f6rs3xb0cy5c0000gn/T/tmp1484509652 -out=/var/folders/yh/7vgmx6qs6b33f6rs3xb0cy5c0000gn/T/tfplan3394747495 -input=false -no-color -detailed-exitcode
2022/11/30 08:30:08 [INFO] [migrator@workspaceB] check diffs
2022/11/30 08:30:08 [DEBUG] [executor@workspaceB]$ terraform plan -state=/var/folders/yh/7vgmx6qs6b33f6rs3xb0cy5c0000gn/T/tmp550612920 -out=/var/folders/yh/7vgmx6qs6b33f6rs3xb0cy5c0000gn/T/tfplan2407686944 -input=false -no-color -detailed-exitcode
2022/11/30 08:30:25 [INFO] [executor@workspaceB] remove the override file
2022/11/30 08:30:25 [INFO] [executor@workspaceB] remove the workspace state folder
2022/11/30 08:30:25 [INFO] [executor@workspaceB] switch back to remote
2022/11/30 08:30:25 [DEBUG] [executor@workspaceB]$ terraform init -input=false -no-color
2022/11/30 08:30:26 [DEBUG] [executor@workspaceB] failed to run command: (*exec.ExitError)(0xc000152040)(exit status 1)
2022/11/30 08:30:26 [ERROR] [executor@workspaceB] failed to switch back to remote: failed to run command (exited 1): terraform init -input=false -no-color

note missing --reconfigure at line.

2022/11/30 08:30:25 [DEBUG] [executor@workspaceB]$ terraform init -input=false -no-color at that point I need to run terraform init -reconfigure manually to fix workspace. So is it just a bug when switch local->remote is done?

Also, what could be helpful to save generated state somewhere (seems tmp files are removed after run). As workaround we can push such state manually, it will make migration much easier anyway than playing with import/rm command for each resource.

tfmigrate.hcl is simple as

tfmigrate {
  migration_dir = "./tfmigrate"
  is_backend_terraform_cloud = true
}

and migration.hcl

migration "multi_state" "mv_dir1_dir2" {
  from_dir = "workspaceA"
  from_workspace = "dev"
  to_dir   = "workspaceB"
  to_workspace = "dev"
  actions = [
    "mv module.audit_logs module.audit_logs"
  ]
}

[minamijoyo]

HI @yohanb, @mtamiola-priv,

Finally, I also had the opportunity to use Terraform Cloud and confirmed that tfmigrate works, at least for non-self-hosted Terraform Cloud. One thing to add other than is_backend_terraform_cloud = true is that I also needed to set the Terraform Cloud workspace name in a migration file.

I'm not sure exactly what environment the original problem occurred in, but I expect there is no additional configuration for self-hosted Terraform Cloud agents. This is because the terraform init and plan commands run in the same local environment as tfmigrate, not in the agent environment.

I will close this issue, but if you continue to be able to reproduce the problem, feel free to reopen it with a reproduction method so that we can investigate further. Thanks!