Updating image inside the example doesn't cause container to restart

[morgante]

Changing the image name should cause a new container to be spun up.

Reproduction:

1. Open the simple-example
2. Run terraform apply
3. Change the image ID
4. Run terraform apply again - the new container should be started instead.

[ivankorn]

@morgante @aaron-lane this issue is no longer reproducible.

Created container-vmf from simple example:

[15:53][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ terraform init
Initializing modules...
- gce-container in ../..

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "random" (terraform-providers/random) 2.1.2...
- Downloading plugin for provider "google" (terraform-providers/google) 2.12.0...
- Downloading plugin for provider "external" (terraform-providers/external) 1.2.0...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.external: version = "~> 1.2"
* provider.google: version = "~> 2.12"
* provider.random: version = "~> 2.1"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
[15:57][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.google_compute_zones.available: Refreshing state...
module.gce-container.data.google_compute_image.coreos: Refreshing state...
module.gce-container.data.external.spec_as_yaml: Refreshing state...

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # google_compute_instance.vm will be created
  + resource "google_compute_instance" "vm" {
      + can_ip_forward       = false
      + cpu_platform         = (known after apply)
      + deletion_protection  = false
      + guest_accelerator    = (known after apply)
      + id                   = (known after apply)
      + instance_id          = (known after apply)
      + label_fingerprint    = (known after apply)
      + labels               = {
          + "container-vm" = "cos-stable-75-12105-97-0"
        }
      + machine_type         = "n1-standard-1"
      + metadata             = {
          + "gce-container-declaration" = "---\nspec:\n  containers:\n  - env:\n    - name: TEST_VAR\n      value: Hello World!\n    image: gcr.io/google-samples/hello-app:1.0\n    volumeMounts:\n    - mountPath: \"/cache\"\n      name: tempfs-0\n      readOnly: false\n  restartPolicy: Always\n  volumes:\n  - emptyDir:\n      medium: Memory\n    name: tempfs-0\n"
        }
      + metadata_fingerprint = (known after apply)
      + name                 = "container-vm-example-test"
      + project              = "gl-ivankorniienko-seed"
      + self_link            = (known after apply)
      + tags                 = [
          + "container-vm-example",
        ]
      + tags_fingerprint     = (known after apply)
      + zone                 = (known after apply)

      + boot_disk {
          + auto_delete                = true
          + device_name                = (known after apply)
          + disk_encryption_key_sha256 = (known after apply)
          + kms_key_self_link          = (known after apply)
          + source                     = (known after apply)

          + initialize_params {
              + image  = "https://www.googleapis.com/compute/v1/projects/cos-cloud/global/images/cos-stable-75-12105-97-0"
              + labels = (known after apply)
              + size   = (known after apply)
              + type   = (known after apply)
            }
        }

      + network_interface {
          + address            = (known after apply)
          + name               = (known after apply)
          + network            = (known after apply)
          + network_ip         = (known after apply)
          + subnetwork         = "default"
          + subnetwork_project = "gl-ivankorniienko-seed"

          + access_config {
              + assigned_nat_ip = (known after apply)
              + nat_ip          = (known after apply)
              + network_tier    = (known after apply)
            }
        }

      + scheduling {
          + automatic_restart   = (known after apply)
          + on_host_maintenance = (known after apply)
          + preemptible         = (known after apply)

          + node_affinities {
              + key      = (known after apply)
              + operator = (known after apply)
              + values   = (known after apply)
            }
        }

      + service_account {
          + email  = (known after apply)
          + scopes = [
              + "https://www.googleapis.com/auth/cloud-platform",
            ]
        }
    }

  # random_shuffle.zone will be created
  + resource "random_shuffle" "zone" {
      + id           = (known after apply)
      + input        = [
          + "us-east4-a",
          + "us-east4-b",
          + "us-east4-c",
        ]
      + result       = (known after apply)
      + result_count = 1
    }

Plan: 2 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

[16:00][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ 
[16:00][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ echo $?
0
[16:04][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ terraform apply
module.gce-container.data.external.spec_as_yaml: Refreshing state...
module.gce-container.data.google_compute_image.coreos: Refreshing state...
data.google_compute_zones.available: Refreshing state...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # google_compute_instance.vm will be created
  + resource "google_compute_instance" "vm" {
      + can_ip_forward       = false
      + cpu_platform         = (known after apply)
      + deletion_protection  = false
      + guest_accelerator    = (known after apply)
      + id                   = (known after apply)
      + instance_id          = (known after apply)
      + label_fingerprint    = (known after apply)
      + labels               = {
          + "container-vm" = "cos-stable-75-12105-97-0"
        }
      + machine_type         = "n1-standard-1"
      + metadata             = {
          + "gce-container-declaration" = "---\nspec:\n  containers:\n  - env:\n    - name: TEST_VAR\n      value: Hello World!\n    image: gcr.io/google-samples/hello-app:1.0\n    volumeMounts:\n    - mountPath: \"/cache\"\n      name: tempfs-0\n      readOnly: false\n  restartPolicy: Always\n  volumes:\n  - emptyDir:\n      medium: Memory\n    name: tempfs-0\n"
        }
      + metadata_fingerprint = (known after apply)
      + name                 = "container-vm-example-test"
      + project              = "gl-ivankorniienko-seed"
      + self_link            = (known after apply)
      + tags                 = [
          + "container-vm-example",
        ]
      + tags_fingerprint     = (known after apply)
      + zone                 = (known after apply)

      + boot_disk {
          + auto_delete                = true
          + device_name                = (known after apply)
          + disk_encryption_key_sha256 = (known after apply)
          + kms_key_self_link          = (known after apply)
          + source                     = (known after apply)

          + initialize_params {
              + image  = "https://www.googleapis.com/compute/v1/projects/cos-cloud/global/images/cos-stable-75-12105-97-0"
              + labels = (known after apply)
              + size   = (known after apply)
              + type   = (known after apply)
            }
        }

      + network_interface {
          + address            = (known after apply)
          + name               = (known after apply)
          + network            = (known after apply)
          + network_ip         = (known after apply)
          + subnetwork         = "default"
          + subnetwork_project = "gl-ivankorniienko-seed"

          + access_config {
              + assigned_nat_ip = (known after apply)
              + nat_ip          = (known after apply)
              + network_tier    = (known after apply)
            }
        }

      + scheduling {
          + automatic_restart   = (known after apply)
          + on_host_maintenance = (known after apply)
          + preemptible         = (known after apply)

          + node_affinities {
              + key      = (known after apply)
              + operator = (known after apply)
              + values   = (known after apply)
            }
        }

      + service_account {
          + email  = (known after apply)
          + scopes = [
              + "https://www.googleapis.com/auth/cloud-platform",
            ]
        }
    }

  # random_shuffle.zone will be created
  + resource "random_shuffle" "zone" {
      + id           = (known after apply)
      + input        = [
          + "us-east4-a",
          + "us-east4-b",
          + "us-east4-c",
        ]
      + result       = (known after apply)
      + result_count = 1
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

random_shuffle.zone: Creating...
random_shuffle.zone: Creation complete after 0s [id=-]
google_compute_instance.vm: Creating...
google_compute_instance.vm: Still creating... [10s elapsed]
google_compute_instance.vm: Creation complete after 11s [id=container-vm-example-test]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

container = {
  "env" = [
    {
      "name" = "TEST_VAR"
      "value" = "Hello World!"
    },
  ]
  "image" = "gcr.io/google-samples/hello-app:1.0"
  "volumeMounts" = [
    {
      "mountPath" = "/cache"
      "name" = "tempfs-0"
      "readOnly" = "false"
    },
  ]
}
instance_name = container-vm-example-test
ipv4 = 35.221.36.163
project_id = gl-ivankorniienko-seed
vm_container_label = cos-stable-75-12105-97-0
volumes = [
  {
    "emptyDir" = {
      "medium" = "Memory"
    }
    "name" = "tempfs-0"
  },
]
zone = us-east4-c
[16:05][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ 

Noted the image gcr.io/google-samples/hello-app:1.0:

[16:05][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ grep hello-app main.tf 
    image = "gcr.io/google-samples/hello-app:1.0"

Updated the image with gcr.io/google-samples/hello-app:2.0:

[16:06][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ sed -i 's/hello-app:1.0/hello-app:2.0/g' main.tf 
[16:07][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ grep hello-app main.tf 
    image = "gcr.io/google-samples/hello-app:2.0"
[16:07][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ 

Re-applied the template: it worked, output reflects changes (note image v2.0)

[16:07][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ terraform apply
data.google_compute_zones.available: Refreshing state...
module.gce-container.data.google_compute_image.coreos: Refreshing state...
module.gce-container.data.external.spec_as_yaml: Refreshing state...
random_shuffle.zone: Refreshing state... [id=-]
google_compute_instance.vm: Refreshing state... [id=container-vm-example-test]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # google_compute_instance.vm will be updated in-place
  ~ resource "google_compute_instance" "vm" {
        can_ip_forward       = false
        cpu_platform         = "Intel Broadwell"
        deletion_protection  = false
        guest_accelerator    = []
        id                   = "container-vm-example-test"
        instance_id          = "8808858470383199004"
        label_fingerprint    = "16h2vNHPdWY="
        labels               = {
            "container-vm" = "cos-stable-75-12105-97-0"
        }
        machine_type         = "n1-standard-1"
      ~ metadata             = {
          ~ "gce-container-declaration" = <<~EOT
                ---
                spec:
                  containers:
                  - env:
                    - name: TEST_VAR
                      value: Hello World!
              -     image: gcr.io/google-samples/hello-app:1.0
              +     image: gcr.io/google-samples/hello-app:2.0
                    volumeMounts:
                    - mountPath: "/cache"
                      name: tempfs-0
                      readOnly: false
                  restartPolicy: Always
                  volumes:
                  - emptyDir:
                      medium: Memory
                    name: tempfs-0
            EOT
        }
        metadata_fingerprint = "mLtKvl0__xU="
        name                 = "container-vm-example-test"
        project              = "gl-ivankorniienko-seed"
        self_link            = "https://www.googleapis.com/compute/v1/projects/gl-ivankorniienko-seed/zones/us-east4-c/instances/container-vm-example-test"
        tags                 = [
            "container-vm-example",
        ]
        tags_fingerprint     = "fHIXWKpF9LM="
        zone                 = "us-east4-c"

        boot_disk {
            auto_delete = true
            device_name = "persistent-disk-0"
            source      = "https://www.googleapis.com/compute/v1/projects/gl-ivankorniienko-seed/zones/us-east4-c/disks/container-vm-example-test"

            initialize_params {
                image  = "https://www.googleapis.com/compute/v1/projects/cos-cloud/global/images/cos-stable-75-12105-97-0"
                labels = {}
                size   = 10
                type   = "pd-standard"
            }
        }

        network_interface {
            name               = "nic0"
            network            = "https://www.googleapis.com/compute/v1/projects/gl-ivankorniienko-seed/global/networks/default"
            network_ip         = "10.150.0.2"
            subnetwork         = "https://www.googleapis.com/compute/v1/projects/gl-ivankorniienko-seed/regions/us-east4/subnetworks/default"
            subnetwork_project = "gl-ivankorniienko-seed"

            access_config {
                nat_ip       = "35.221.36.163"
                network_tier = "PREMIUM"
            }
        }

        scheduling {
            automatic_restart   = true
            on_host_maintenance = "MIGRATE"
            preemptible         = false
        }

        service_account {
            email  = "764811539058-compute@developer.gserviceaccount.com"
            scopes = [
                "https://www.googleapis.com/auth/cloud-platform",
            ]
        }

        shielded_instance_config {
            enable_integrity_monitoring = true
            enable_secure_boot          = false
            enable_vtpm                 = true
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

google_compute_instance.vm: Modifying... [id=container-vm-example-test]
google_compute_instance.vm: Modifications complete after 10s [id=container-vm-example-test]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Outputs:

container = {
  "env" = [
    {
      "name" = "TEST_VAR"
      "value" = "Hello World!"
    },
  ]
  "image" = "gcr.io/google-samples/hello-app:2.0"
  "volumeMounts" = [
    {
      "mountPath" = "/cache"
      "name" = "tempfs-0"
      "readOnly" = "false"
    },
  ]
}
instance_name = container-vm-example-test
ipv4 = 35.221.36.163
project_id = gl-ivankorniienko-seed
vm_container_label = cos-stable-75-12105-97-0
volumes = [
  {
    "emptyDir" = {
      "medium" = "Memory"
    }
    "name" = "tempfs-0"
  },
]
zone = us-east4-c
[16:08][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ 

Double checked with gcloud: we got the image updated, as expected:

[16:24][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ gcloud beta compute instances describe container-vm-example-test --project=gl-ivankorniienko-seed --zone=us-east4-c | grep containers -A 4 
        containers:
        - env:
          - name: TEST_VAR
            value: Hello World!
          image: gcr.io/google-samples/hello-app:2.0
[16:25][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ 

[ivankorn]

Double-checked again - the issue is still valid.

Applied template:

[16:48][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ terraform apply
module.gce-container.data.external.spec_as_yaml: Refreshing state...
data.google_compute_zones.available: Refreshing state...
module.gce-container.data.google_compute_image.coreos: Refreshing state...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # google_compute_instance.vm will be created
  + resource "google_compute_instance" "vm" {
      + can_ip_forward       = false
      + cpu_platform         = (known after apply)
      + deletion_protection  = false
      + guest_accelerator    = (known after apply)
      + id                   = (known after apply)
      + instance_id          = (known after apply)
      + label_fingerprint    = (known after apply)
      + labels               = {
          + "container-vm" = "cos-stable-75-12105-97-0"
        }
      + machine_type         = "n1-standard-1"
      + metadata             = {
          + "gce-container-declaration" = "---\nspec:\n  containers:\n  - env:\n    - name: TEST_VAR\n      value: Hello World!\n    image: gcr.io/google-samples/hello-app:1.0\n    volumeMounts:\n    - mountPath: \"/cache\"\n      name: tempfs-0\n      readOnly: false\n  restartPolicy: Always\n  volumes:\n  - emptyDir:\n      medium: Memory\n    name: tempfs-0\n"
        }
      + metadata_fingerprint = (known after apply)
      + name                 = "container-vm-example-test"
      + project              = "gl-ivankorniienko-seed"
      + self_link            = (known after apply)
      + tags                 = [
          + "container-vm-example",
        ]
      + tags_fingerprint     = (known after apply)
      + zone                 = (known after apply)

      + boot_disk {
          + auto_delete                = true
          + device_name                = (known after apply)
          + disk_encryption_key_sha256 = (known after apply)
          + kms_key_self_link          = (known after apply)
          + source                     = (known after apply)

          + initialize_params {
              + image  = "https://www.googleapis.com/compute/v1/projects/cos-cloud/global/images/cos-stable-75-12105-97-0"
              + labels = (known after apply)
              + size   = (known after apply)
              + type   = (known after apply)
            }
        }

      + network_interface {
          + address            = (known after apply)
          + name               = (known after apply)
          + network            = (known after apply)
          + network_ip         = (known after apply)
          + subnetwork         = "default"
          + subnetwork_project = "gl-ivankorniienko-seed"

          + access_config {
              + assigned_nat_ip = (known after apply)
              + nat_ip          = (known after apply)
              + network_tier    = (known after apply)
            }
        }

      + scheduling {
          + automatic_restart   = (known after apply)
          + on_host_maintenance = (known after apply)
          + preemptible         = (known after apply)

          + node_affinities {
              + key      = (known after apply)
              + operator = (known after apply)
              + values   = (known after apply)
            }
        }

      + service_account {
          + email  = (known after apply)
          + scopes = [
              + "https://www.googleapis.com/auth/cloud-platform",
            ]
        }
    }

  # random_shuffle.zone will be created
  + resource "random_shuffle" "zone" {
      + id           = (known after apply)
      + input        = [
          + "us-east4-a",
          + "us-east4-b",
          + "us-east4-c",
        ]
      + result       = (known after apply)
      + result_count = 1
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

random_shuffle.zone: Creating...
random_shuffle.zone: Creation complete after 0s [id=-]
google_compute_instance.vm: Creating...
google_compute_instance.vm: Still creating... [10s elapsed]
google_compute_instance.vm: Creation complete after 11s [id=container-vm-example-test]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

container = {
  "env" = [
    {
      "name" = "TEST_VAR"
      "value" = "Hello World!"
    },
  ]
  "image" = "gcr.io/google-samples/hello-app:1.0"
  "volumeMounts" = [
    {
      "mountPath" = "/cache"
      "name" = "tempfs-0"
      "readOnly" = "false"
    },
  ]
}
instance_name = container-vm-example-test
ipv4 = 35.221.36.163
project_id = gl-ivankorniienko-seed
vm_container_label = cos-stable-75-12105-97-0
volumes = [
  {
    "emptyDir" = {
      "medium" = "Memory"
    }
    "name" = "tempfs-0"
  },
]
zone = us-east4-c

[16:55][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ gcloud beta compute instances describe container-vm-example-test --project=gl-ivankorniienko-seed --zone=us-east4-c | grep containers -A 4 
        containers:
        - env:
          - name: TEST_VAR
            value: Hello World!
          image: gcr.io/google-samples/hello-app:1.0
[16:55][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ gcloud compute ssh container-vm-example-test --project=gl-ivankorniienko-seed --zone=us-east4-c
Updating project ssh metadata...⠧Updated [https://www.googleapis.com/compute/v1/projects/gl-ivankorniienko-seed].                                                                                                                            
Updating project ssh metadata...done.                                                                                                                                                                                                        
Waiting for SSH key to propagate.
Warning: Permanently added 'compute.6407004692787861186' (ED25519) to the list of known hosts.
  ########################[ Welcome ]########################
  #  You have logged in to the guest OS.                    #
  #  To access your containers use 'docker attach' command  #
  ###########################################################

user@container-vm-example-test ~ $ docker container ls
CONTAINER ID        IMAGE                                 COMMAND             CREATED             STATUS              PORTS               NAMES
16cc18190d70        gcr.io/google-samples/hello-app:1.0   "./hello-app"       7 minutes ago       Up 7 minutes                            klt--abvr
user@container-vm-example-test ~ $ 
user@container-vm-example-test ~ $ exit
logout

Connection to 35.221.36.163 closed.

Modified the image:

[16:57][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ sed -i 's/hello-app:1.0/hello-app:2.0/g' main.tf 
[16:57][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ grep hello-app main.tf 
    image = "gcr.io/google-samples/hello-app:2.0"

Re-applied template (note updated image):

[16:57][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ terraform apply -auto-approve 
module.gce-container.data.external.spec_as_yaml: Refreshing state...
data.google_compute_zones.available: Refreshing state...
module.gce-container.data.google_compute_image.coreos: Refreshing state...
random_shuffle.zone: Refreshing state... [id=-]
google_compute_instance.vm: Refreshing state... [id=container-vm-example-test]
google_compute_instance.vm: Modifying... [id=container-vm-example-test]
google_compute_instance.vm: Still modifying... [id=container-vm-example-test, 10s elapsed]
google_compute_instance.vm: Modifications complete after 13s [id=container-vm-example-test]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Outputs:

container = {
  "env" = [
    {
      "name" = "TEST_VAR"
      "value" = "Hello World!"
    },
  ]
  "image" = "gcr.io/google-samples/hello-app:2.0"
  "volumeMounts" = [
    {
      "mountPath" = "/cache"
      "name" = "tempfs-0"
      "readOnly" = "false"
    },
  ]
}
instance_name = container-vm-example-test
ipv4 = 35.221.36.163
project_id = gl-ivankorniienko-seed
vm_container_label = cos-stable-75-12105-97-0
volumes = [
  {
    "emptyDir" = {
      "medium" = "Memory"
    }
    "name" = "tempfs-0"
  },
]
zone = us-east4-c
[16:58][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ 

Double-checked with gcloud (note updated image)

[16:58][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ gcloud compute instances describe container-vm-example-test --project=gl-ivankorniienko-seed --zone=us-east4-c | grep containers -A 4 
        containers:
        - env:
          - name: TEST_VAR
            value: Hello World!
          image: gcr.io/google-samples/hello-app:2.0
[16:59][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ 

Double checked on the host itself:

[16:59][user@host:~/workspace/google/terraform-google-container-vm/examples/simple_instance]$ gcloud compute ssh container-vm-example-test --project=gl-ivankorniienko-seed --zone=us-east4-c
  ########################[ Welcome ]########################
  #  You have logged in to the guest OS.                    #
  #  To access your containers use 'docker attach' command  #
  ###########################################################

user@container-vm-example-test ~ $ docker container ls
CONTAINER ID        IMAGE                                 COMMAND             CREATED             STATUS              PORTS               NAMES
16cc18190d70        gcr.io/google-samples/hello-app:1.0   "./hello-app"       11 minutes ago      Up 11 minutes                           klt--abvr
user@container-vm-example-test ~ $ exit

The issue is still valid !

[YermakovvD]

Hi guys, the issue still valid. Step for reproducing is the same

1. Open the simple-example (or any other)
2. Run terraform apply
3. Change the image ID
4. Run terraform apply again - the new container should be started instead.

@ivankorn , issue still valid, please fix it ASAP

@aaron-lane , please review changes more deeply

[ivankorn]

@den-gl

@ivankorn , issue still valid, please fix it ASAP

Do you have your tests output?

The issue was checked here as you can see it's not reproducible since then

The image you referenced is not a proof of the fix, it's a proof of the bug, after that it was fixed with tests outputs of successful results in the PR not in the issue.

cc: @aaron-lane @morgante

[YermakovvD]

@ivankorn, you absolutely right, but issue still valid, and you didn't fix their.

And this issue not about your tests, your tests its your problem, but functionality of that module absolutely doesn't work as expected. Try reproduce this issue with the following steps you find above.

If you can't reproduce it, please ping me and I'll provide all my logs.

P.S. double checked and I can't find any word in a title / issue description about broken test you mention I am sure this issue about broken functionality

[ivankorn]

@ivankorn, you absolutely right, but issue still valid, and you didn't fix their. @YermakovvD

I'm 100% sure it was passing(I posted link to output above), but what I didn't do is running the check a number of times to confirm subsequent passes..

P.S. double checked and I can't find any word in a title / issue description about broken test you mention I am sure this issue about broken functionality

I spoke to @kopachevsky who discussed that with you guys. Looks like what you're seeing is intermittent failure. Is it the case? Like it's randomly reproducible, one pass, one fail etc ? Your're using the latest version right and you tried pulling the code or pointing to a git repo instead of terraform registry which may have older release?

[YermakovvD]

yes, we work with the latest version of this module

[YermakovvD]

@ivankorn, any updates with fixing this bug ?

[ivankorn]

@aaron-lane @morgante

I looked at the code @YermakovvD is reporting an issue with.

He didn't include workaround for this container image in it and still uses

locals {
  instance_name = var.instance_name
}

instead of

locals {
  instance_name = format("%s-%s", var.instance_name, substr(md5(module.gce-container.container.image), 0, 8))
}

@YermakovvD is still concerned about workaround @morgante proposed earlier for the fix and he doesn't want whole VM to be re-created in order to update the container image.

However the mentioned workaround merged with #31 still works.

@YermakovvD, can you please confirm if I understood your concern correctly?

Thanks

[Ahmedn1]

I also have the same issue. Currently I'm solving it by running gcloud compute instances update-container $instance_name --container-image $TF_VAR_image_id command everytime I run terraform apply

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days

[Redmega]

Facing this issue not only when image updates, but also when any image config (e.x. env) updates.

[derhally]

Ran into this issue as well where anytime we update environment variables defined in the metadata, we have to restart the instance

[theCrius]

Can confirm that this issue is still there. Simply changing the container image reference doesn't actually make the change happen inside the VM.

The Web Dashboard shows the new image being correctly assigned but when accessing the VM itself and checking, it still load the previous image.

Only by using the workaround reported by @Ahmedn1 it seems to be fine but that's triggering a restart of the whole VM it seems while it would suffice if there was a docker refresh of sort.

Is there anyone working on this? Can someone point to where to go looking to understand how to help?

[JBuschAtMonoLets]

Yes this is still an issue.

[dbousamra]

Any updates?

[dbousamra]

I think we want to run sudo systemctl start konlet-startup anytime the metadata changes. Is this possible using remote-exec perhaps? I am not very good at Terraform.

[m0ar]

This is still a problem, and a very annoying one. Having dynamic resource names means we can't just communicate between VMs using their name, because that will unexpectedly change with infra updates :facepalm:

[ArnaudD]

This dirty workaround seems to work for me :

locals {
  envs = [{
    name  = "FOO"
    value = "BAR"
  }]

  config_sha = sha1("${join("", local.envs.*.value)}-${local.image}")
  image    = "gcr.io/..."
  zone     = "..."
}

module "my_gce_container" {
  source  = "terraform-google-modules/container-vm/google"
  # ...

  container = {
    image = local.image
    env   = local.envs
    # ...
  }
}

resource "google_compute_instance" "gce_instance" {
  project = data.google_project.project.project_id
  zone    = local.zone
  # ...

}

resource "null_resource" "gce_null_instance" {
  triggers = {
    config_sha = local.config_sha
  }

  provisioner "local-exec" {
    command = "gcloud compute ssh --project=${data.google_project.project.project_id} --zone=${local.zone} ${google_compute_instance.gce_instance.name} --command 'sudo systemctl start konlet-startup'"
  }

  depends_on = [
    google_compute_instance.gce_instance
  ]
}

It should also work with remote-exec @dbousamra .

[slatkovic]

+1

EDIT: Although the instance group gets updated with a new template via tf, compute instances still point to the previous template.

One workaround I found is to manually run gcloud compute instance-groups managed set-instance-template with the new template version.

EDIT 2: After digging through this a bit more, the issue seems to be that rolling update is not triggered for some reason after updating the template on the instance group.

A better workaround is to run gcloud compute instance-groups managed rolling-action start-update via local exec in google_compute_instance_group_manager as it doesn't destroy instances as the previous command.

 provisioner "local-exec" {
    command = <<EOL
      gcloud compute instance-groups managed rolling-action start-update \
        ${google_compute_instance_group_manager.my_group_manager.name} \
        --version template=${google_compute_instance_template.my_instance_template.id} \
        --max-unavailable=1 \
        --project=${var.my_project_id} \
        --zone=${var.my_zone}"
    EOL
  }

EDIT 3: It seems like the same effect can be achieved by adding an update_policy to google_compute_instance_group_manager. Before update_policy was introduced, a rolling update would run automatically, which is not the case anymore.

 update_policy {
    type = "PROACTIVE"
    minimal_action = "REPLACE"
    max_unavailable_fixed = 1
 }

[nilebox]

Another workaround described in https://cloud.google.com/blog/topics/developers-practitioners/force-terraform-resource-recreation is hashing the metadata content and putting the hash into a GCE field which forces its replacement (e.g. destroy and create), the description field for example.

This of course means a downtime, so may not work for everyone.

[hadim]

I ended up using the update_policy trick from @slatkovic .

It works well but create a new VM instance each time. It's not ideal but acceptable for my usecase.