Tfautomv does not perform the ignore completely

[RonaldoDS27]

First of all, thank you very much for this tool, it makes the process of automating the TF Move much easier.

The "Problem" we currently have is the following. We are running the tool, in order to detect the changes in iaC modules that we have altered, however, our iac plan, gives us a total of 41 Destroys and 42 Creates, which translates to 41 terraform state moves.

Plan: 42 to add, 9 to change, 41 to destroy.

We identified that it is for some parameters that it cannot move, nevertheless, we have created a Script that automatically detects those parameters for which it cannot move the resource, and by means of that script, we constructed the ignore with which the tfautomv would be executed, that is to say, something like this: tfautomv --ignore="everything:aws_instance:credit_specification.#" --ignore="everything: aws_rds_cluster: availability_zones.#"

However, it is not doing the process to ignore them, so we always get a lower number in the state moves. 33 moves written to moves.tf.

Additionally, it was evidenced that when having 1,2,3,...(or higher) Matches in the resources, it omits the creation of state move

│ 3 matches for module.backend.module.apigw_keys.aws_api_gateway_usage_plan_key.this[1] (delete) in current directory │ │ module.backend.module.apigw_keys.aws_api_gateway_usage_plan_key.this["plan-test-1"] (create) in current directory │ │ module.backend.module.apigw_keys.aws_api_gateway_usage_plan_key.this["plan-test-2"] (create) in current directory │ │ module.backend.module.apigw_keys.aws_api_gateway_usage_plan_key.this["plan-test-3"] (create) in current directory.

We would like to know, if it is possible that from the functionality of the same tool, you can add a feature, through which you can perform a complete ignore on all resources that are being detected as changing, because in fairly large infrastructures, can run only 50% in the terraform state move.

I appreciate your support and response.

[busser]

Thanks for reaching out! Your issue touches on a lot of things and I'll try to address them all :)

Originally we added the --ignore flag to solve a very specific problem: providers magically transforming certain attributes in the "apply" step. These transformations caused tfautomv to see differences that weren't actually differences and miss certain resource matches. The use-case you describe is a bit different, and I think that's why the --ignore flag is insufficient. I've been thinking about adding a more generic --rule flag that would allow tweaking tfautomv's engine in different ways. I haven't started work on it yet. I want a better understanding of issues people face before implementing such a big feature.

However, even if a new flag allowed tfautomv to ignore resources planned for change, as you request, I'm afraid that would only be a superficial fix. I think there's a bit of a mismatch between how tfautomv works and how you're using it. tfautomv works on the assumption that if it finds all the necessary moves then the resulting plan will be empty. In other words, it assumes that some resources were moved but that none were changed. This assumption is at the core of tfautomv's logic. It can be bent by the --ignore flag but without that assumption I'm not so sure how tfautomv would work.

As you mentioned in your issue, if tfautomv finds more than 1 match for a resource, it will not generate a move to or from that resource. This issue is pretty common when we use the --ignore flag but it can also happen without it. We've known about this limitation since the very beginning. We have a couple ideas on how to solve it but haven't decided on one yet. Both of these ideas only work with the assumption mentioned above: resources were moved but not changed.

All that being said, I'd like to find a solution to your problem. If in your use-case it's possible to first reorganize resources without changing their attributes, then run tfautomv, and then make code changes that change resource attributes, I would recommend that.

If that isn't possible, then I'd like to better understand the sort of differences in attributes that tfautomv is seeing. Would it be OK for you to share tfautomv's output at maximum verbosity (-vvv or --verbosity 3)?

[RonaldoDS27]

Thanks for the reply. Sure, look, here I leave you a little of the result of what you requested.

It should be noted that this is a practice Infrastructure, and we are dedicating efforts to correctly apply the tool. We hope this issue can be useful to you.

[busser]

Hi @RonaldoDS27, thanks for sharing more details!

In this particular case, it's pretty clear what moves tfautomv should find. I believe it isn't finding them because the resources planned for creation have the following fields set:

throttle_settings {
  burst_limit = 200
  rate_limit  = 100
}

but the resources planned for deletion do not.

In the short-term, I'd say you have two options:

1. Remove this field from the code, run tfautomv so it finds all matches, then put the field back.
2. Use the --ignore flag to ignore these values:

   tfautomv \
   --ignore 'everything:aws_api_gateway_usage_plan:throttle_settings.#' \
   --ignore 'everything:aws_api_gateway_usage_plan:throttle_settings.burst_limit' \
   --ignore 'everything:aws_api_gateway_usage_plan:throttle_settings.rate_limit'

Let me know what option you decide on and what the result is :)

---

Your situation points to multiple ways we could improve tfautomv.

The first is that tfautomv could perhaps suggest the --ignore flags above. We could imagine having a sort of search algorithm that could find them.

The second is that the --ignore flag should support some form of wildcard. Instead of having to use it multiple times like above, it would be convenient to be able to write something like this:

tfautomv --ignore 'everything:aws_api_gateway_usage_plan:throttle_settings.*'

The third is that it would be convenient to be able to tell tfautomv that to match resources of a given type, it should look only at specific fields. For instance, I suppose that the name field is enough to match resources together, so all other fields could be ignored. This currently isn't possible with the --ignore flag. I plan to build a replacement for the --ignore flag that would have this capability, among others. However I can't commit on this replacement being released soon.