Host IP of google_compute_instance connection is not populated when destroying with terraform plan

franfran3 commented 5 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment
If an issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to "hashibot", a community member has claimed the issue already.

Terraform Version

Terraform v0.11.13

provider.google v2.3.0

Affected Resource(s)

google_compute_instance

Terraform Configuration Files

provider "google" {
  version = "~> 2.1"
  zone    = "europe-west1-b"
}

locals {
  publickey = "${file("${path.module}/keys/provisioner.pub")}"
}

resource "google_compute_instance" "test_machine" {
  name                      = "test-machine"
  allow_stopping_for_update = true

  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-9"
      size  = 10
      type  = "pd-standard"
    }
  }

  metadata {
    ssh-keys       = "provisioner:${local.publickey}"
    enable-oslogin = "true"
  }

  metadata_startup_script = <<EOS
#!/bin/bash
# setup provisioner user
useradd --create-home --shell /bin/bash provisioner
mkdir --mode 0700 /home/provisioner/.ssh
cat -- > /home/provisioner/.ssh/authorized_keys <<EOKEY
${local.publickey}
EOKEY
chown -R provisioner: /home/provisioner
echo "provisioner ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/provisioner
touch /.init-script-done
echo "Init script finished"
EOS

  machine_type = "n1-standard-1"
  zone         = "europe-west1-b"

  network_interface {
    subnetwork = "default"

    access_config {
      nat_ip = ""
    }
  }

  connection {
    type        = "ssh"
    user        = "provisioner"
    private_key = "${file("${path.module}/keys/provisioner")}"
    agent       = false
    timeout     = "30s"
  }

  provisioner "remote-exec" {
    inline = ["sudo touch /.startup-provisioner"]
  }

  provisioner "remote-exec" {
    when   = "destroy"
    inline = ["sudo touch /.destroy-provisioner", "echo 'destroy provisoner finished'"]
  }
}

output "public_ip" {
  value = "${google_compute_instance.test_machine.network_interface.0.access_config.0.nat_ip}"
}

Debug Output

https://gist.github.com/franfran3/89e736c25200605055ce984bc47a32bb

Panic Output

Expected Behavior

At creation time:

terraform apply should work
terraform plan -out plan-file && terraform apply plan-file should work

After any modification that requires the instance destruction:

terraform apply should work
terraform plan -out plan-file && terraform apply plan-file should work

Actual Behavior

At creation time:

terraform apply works
terraform plan -out plan-file && terraform apply plan-file works

After any modification that requires the instance destruction:

terraform apply works
terraform plan -out plan-file && terraform apply plan-file fails with ssh: handshake failed: ssh: unable to authenticate error

Steps to Reproduce

terraform plan -out plan-file
terraform apply plan-file
make any modification that requires destruction of the instance
terraform plan -out plan-file
terraform apply plan-file

Important Factoids

After making modification, running terraform plan -out plan-file && terraform apply plan-file fails with the output of interest (the Host value is empty):

google_compute_instance.test_machine: Destroying... (ID: test-machine)                                                                                                                                             
google_compute_instance.test_machine: Provisioning with 'remote-exec'...                                                                                                                                           
google_compute_instance.test_machine (remote-exec): Connecting to remote host via SSH...                                                                                                                           
google_compute_instance.test_machine (remote-exec):   Host:                                                                                                                                                        
google_compute_instance.test_machine (remote-exec):   User: provisioner                                                                                                                                            
google_compute_instance.test_machine (remote-exec):   Password: false                                                                                                                                              
google_compute_instance.test_machine (remote-exec):   Private key: true                                                                                                                                            
google_compute_instance.test_machine (remote-exec):   SSH Agent: false                                                                                                                                             
google_compute_instance.test_machine (remote-exec):   Checking Host Key: false                                                                                                                                     
google_compute_instance.test_machine (remote-exec): Connecting to remote host via SSH...                                                                                                                           
...

while terraform apply is successful with the output of interest (the Host value is populated):

google_compute_instance.test_machine: Destroying... (ID: test-machine)                                                                                                                                             
google_compute_instance.test_machine: Provisioning with 'remote-exec'...                                                                                                                                           
google_compute_instance.test_machine (remote-exec): Connecting to remote host via SSH...                                                                                                                           
google_compute_instance.test_machine (remote-exec):   Host: 35.195.205.230                                                                                                                                         
google_compute_instance.test_machine (remote-exec):   User: provisioner                                                                                                                                            
google_compute_instance.test_machine (remote-exec):   Password: false                                                                                                                                              
google_compute_instance.test_machine (remote-exec):   Private key: true                                                                                                                                            
google_compute_instance.test_machine (remote-exec):   SSH Agent: false                                                                                                                                             
google_compute_instance.test_machine (remote-exec):   Checking Host Key: false                                                                                                                                     
google_compute_instance.test_machine (remote-exec): Connected!                                                                                                                                                     
google_compute_instance.test_machine (remote-exec): destroy provisoner finished

A simple actual fix is to explicitly set the Host of the connection:

  connection {
    host        = "${self.network_interface.0.access_config.0.nat_ip}"
    type        = "ssh"
    user        = "provisioner"
    private_key = "${file("${path.module}/keys/provisioner")}"
    agent       = false
    timeout     = "30s"
  }

References

b/308756266

rileykarson commented 5 years ago

Hmm, looks like an issue with the provisioner / connection behaviour? @paddycarver do you mind looking in to this when you have cycles?

paddycarver commented 5 years ago

Hey @franfran3, sorry for the delayed response on this. Are you still seeing this issue on 0.12?

pn-y commented 4 years ago

Can confirm the same behaviour with newest TF version.

Terraform v0.12.18
+ provider.google v2.14.0
+ provider.local v1.3.0
+ provider.template v2.1.2

google_compute_instance.gitlab-ci-runner: Destroying... [id=runner]
google_compute_instance.gitlab-ci-runner: Provisioning with 'remote-exec'...
google_compute_instance.gitlab-ci-runner (remote-exec): Connecting to remote host via SSH...
google_compute_instance.gitlab-ci-runner (remote-exec):   Host:
google_compute_instance.gitlab-ci-runner (remote-exec):   User: root
google_compute_instance.gitlab-ci-runner (remote-exec):   Password: false
google_compute_instance.gitlab-ci-runner (remote-exec):   Private key: false
google_compute_instance.gitlab-ci-runner (remote-exec):   Certificate: false
google_compute_instance.gitlab-ci-runner (remote-exec):   SSH Agent: false
google_compute_instance.gitlab-ci-runner (remote-exec):   Checking Host Key: false
google_compute_instance.gitlab-ci-runner (remote-exec): Connecting to remote host via SSH...
google_compute_instance.gitlab-ci-runner (remote-exec):   Host:
google_compute_instance.gitlab-ci-runner (remote-exec):   User: root
google_compute_instance.gitlab-ci-runner (remote-exec):   Password: false
google_compute_instance.gitlab-ci-runner (remote-exec):   Private key: false
google_compute_instance.gitlab-ci-runner (remote-exec):   Certificate: false
google_compute_instance.gitlab-ci-runner (remote-exec):   SSH Agent: false
google_compute_instance.gitlab-ci-runner (remote-exec):   Checking Host Key: false
google_compute_instance.gitlab-ci-runner (remote-exec): Connecting to remote host via SSH...

Shriyut commented 4 years ago

Has anyone figured out how to use remote-exec provisioner with gce. I'm getting a known hosts key mismatch error when i try to do the same.

karolgorc commented 3 months ago

This got fixed some time ago. The connection block now requires a host specified and won't run without it.

connection {
    host        = self.network_interface.0.access_config.0.nat_ip
    type        = "ssh"
    user        = "provisioner"
    private_key = "${file("${path.module}/keys/provisioner")}"
    agent       = false
    timeout     = "30s"
  }

Terraform destroy:

google_compute_instance.test_machine: Destroying... [id=projects/iac-poc-krakow/zones/europe-west1-b/instances/test-machine]
google_compute_instance.test_machine: Provisioning with 'remote-exec'...
google_compute_instance.test_machine (remote-exec): Connecting to remote host via SSH...
google_compute_instance.test_machine (remote-exec):   Host: 34.78.213.106
google_compute_instance.test_machine (remote-exec):   User: provisioner
google_compute_instance.test_machine (remote-exec):   Password: false
google_compute_instance.test_machine (remote-exec):   Private key: true
google_compute_instance.test_machine (remote-exec):   Certificate: false
google_compute_instance.test_machine (remote-exec):   SSH Agent: false
google_compute_instance.test_machine (remote-exec):   Checking Host Key: false
google_compute_instance.test_machine (remote-exec):   Target Platform: unix
google_compute_instance.test_machine (remote-exec): Connected!
google_compute_instance.test_machine (remote-exec): destroy provisoner finished

@rileykarson I think this issue can be closed.

hashicorp / terraform-provider-google