Changes not detected when altering provisioner commands

[davidvartan]

Terraform Version

0.9.4

Affected Resource(s)

• provisioner

Terraform Configuration Files

provisioner "local-exec" {
  command = "echo Hello"
}

provisioner "remote-exec" {
  inline = [
    "touch /home/ec2-user/ok",
  ]
}

Debug Output

N/A

Panic Output

N/A

Expected Behavior

Changing provisioner commands (command, inline, script, scripts) should be detected and cause the instance to be re-created.

(Of course, this is not a request for Terraform to check what is in the scripts themselves. Only expecting text changes made in the .tf files to be detected.)

Actual Behavior

Need to manually taint instances after editing provisioners.

Steps to Reproduce

1. Create a remote-exec provisioner with an inline command
2. terraform apply
3. Edit the inline command to something else
4. terraform apply

Important Factoids

N/A

References

N/A

[bobbytables]

I'm seeing this as well:

Terraform v0.11.1
+ provider.aws v1.6.0
+ provider.kubernetes v1.0.1
+ provider.template v1.0.0

[apparentlymart]

Hi @davidvartan and @bobbytables! Sorry for the long silence here... looks like we missed this one the first time around. :confounded:

It's intended behavior that currently provisioner and connection blocks are considered only during the "create" action, but having Terraform check for changes to these settings and treat them like (forces new resource) attributes is an interesting idea.

It's not possible with our current internals because Terraform does not retain the provisioner settings in the state. We'd need to approach this with some care because these blocks often contain secrets and it would be bad to suddenly start writing these to state after they had been in-memory-only for so long.

I expect we won't be able to change anything here in the short term since our attention is focused elsewhere, but we'll keep this idea in mind for the future and see if we can find a reasonable way to implement it. Thanks for asking about this!

[derekyle]

Seems like the easiest method would be to simply store a hash of the inline commands. Then, the hash is checked and compared on apply. Any change to the commands would cause a rebuild. I find myself doing this all the time. I'm not sure of what the use case would be where changing these initial commands would NOT intend to create a rebuild.

The other option would be just a simple version number parameter that when manually changed will force a reset.

[JackDavidson]

This caught me by surprise when my resource was not recreated after I modified the provisioner, considering that terraform destroys and recreates at nearly any change.

Perhaps we could have a lifecycle setting that will cause this to happen? This seems like a pretty normal thing to me, and when I have jenkins spawning and updating infrastructure for me I don't want to have to worry about destroying it first.

When updating my infrastructure using terraform apply, I was surprised to see that nothing happened when I intended for the instance to be destroyed and recreated to run my new docker image.

[kristofferremback]

I too would want something like this. Perhaps the ability to specify what action to take (ignore, modify, recreate?) when something's changed?

For now, I hacked it by having our user_data have a null_resource's id as a variable:

resource "null_resource" "droplet_trigger" {
  triggers = {
    "droplet_ssl_certificate_path" : "${var.droplet_ssl_certificate_path}",
    "droplet_ssl_certificate_key_path" : "${var.droplet_ssl_certificate_key_path}",
    "remote_setup" : "${data.template_file.remote_setup.rendered}",
    "systemd_service" : "${data.template_file.systemd_service.rendered}",
  }
}

data "template_file" "cloud_init" {
  template = "${file("${path.module}/files/cloud-init.sh.tpl")}"
  vars = {
    // Will trigger a replace to the droplet when any of the triggers are updated
    droplet_trigger = "${null_resource.droplet_trigger.id}"

    project_name      = "${var.project_name}"
    service_name      = "${var.droplet_name}"
    service_port      = "${var.droplet_service_port}"
    git_repository    = "${var.git_repository}"
    health_check_path = "${var.health_check_path}"
  }
}

[jpweng]

Seems like the easiest method would be to simply store a hash of the inline commands. Then, the hash is checked and compared on apply. Any change to the commands would cause a rebuild. I find myself doing this all the time. I'm not sure of what the use case would be where changing these initial commands would NOT intend to create a rebuild.

The other option would be just a simple version number parameter that when manually changed will force a reset.

[Newbie question] How do you implement that?

[TomKulakov]

@apparentlymart Hi!

I expect we won't be able to change anything here in the short term since our attention is focused elsewhere, but we'll keep this idea in mind for the future and see if we can find a reasonable way to implement it.

It's 2022 and that still is a problem.

resource "null_resource" "test" {
 provisioner "local-exec" {
    command = "cat /etc/passwd" # :) 
  }
}

Changing command does not make the plan or apply to do anything.

State itself is important, but if it's difficult to implement don't you feel like having command in state and checking if at least command has changed, can be an good addition to TF? Also please remember to advice to remove command outcome manually or something like that, because I bet that at 99% of the time TF automation won't be able to tell what user has intended to do with the local-exec and is unable to check the state of the resource created that way.

[apparentlymart]

Hi all,

In the meantime since I last replied here we've learned that there are a number of essentially-unresolvable challenges with provisioners because they are an imperative concept wedged into what is otherwise a declarative "desired state" system; it seems that there is no possible complete resolution to the conflicts that causes, and so we are preserving the existing provisioner mechanism primarily for backward compatibility but recommending other approaches for new designs.

Along with the specific practical recommendations described there, I would also note that a create-time provisioner is, from Terraform Core's perspective alone, functionally equivalent to a managed resource type that only takes actions during its "create" and does absolutely nothing when asked to create a plan for an updated configuration or a removal. Therefore in situations where you really do want to run an arbitrary command in response to different changes to the configuration it is possible to achieve that by writing a provider that offers a resource type that runs arbitrary external commands in response to those changes. There are a few of these already available in the Terraform Registry for those who are interested, though I've not personally used any in production and so I can't make a specific recommendation. If you search the registry for terms like "exec" and "shell" you should see some possible providers appear, which you can then evaluate based on their documentation. These will all have design tradeoffs due to the aforementioned conflict between declarative and imperative, but the open ecosystem of providers means that many possible designs can coexist that prioritize different goals.

I don't expect that anything will change with regard to this particular issue in the foreseeable future. Provisioners are essentially a legacy system and so preserving compatibility with how they already behave is our priority, to ensure that existing modules can continue to work while new modules use other techniques instead.

[5imun]

I think that this is actually a feature and not a bug, in my view local exec is not a config state that should be tracked it is just a command that will be executed that can cause changes in some config, I would assume that by now a lot of existing code is relying on that fact.

Also if someone wants to recreate resource on command change that should be easy to achieve with already existing things like null_resource.triggers or depends_on, something like example below:

resource "null_resource" "test" {
  triggers = {
    bash_command = "echo 'test'"
  }
  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]
    command = self.triggers.bash_command
  }
}