Mac address taints terraform state

[Xel-Naha]

Hello,

I have found that immediately after deploying two ct their are state seems tainted:

# module.envs.module.dev.proxmox_virtual_environment_container.ubuntu_container[0] is tainted, so must be replaced
-/+ resource "proxmox_virtual_environment_container" "ubuntu_container" {
      ~ id            = "101" -> (known after apply)
        tags          = [
            "media",
        ]
        # (6 unchanged attributes hidden)

      ~ network_interface {
          - mac_address = "C6:3A:FF:2C:4A:7D" -> null
            name        = "veth0"
            # (6 unchanged attributes hidden)
        }

        # (2 unchanged blocks hidden)
    }

  # module.envs.module.dev.proxmox_virtual_environment_container.ubuntu_container[1] is tainted, so must be replaced
-/+ resource "proxmox_virtual_environment_container" "ubuntu_container" {
      ~ id            = "105" -> (known after apply)
        tags          = [
            "media",
        ]
        # (6 unchanged attributes hidden)

      ~ network_interface {
          - mac_address = "16:92:73:45:D7:F6" -> null
            name        = "veth0"
            # (6 unchanged attributes hidden)
        }

        # (2 unchanged blocks hidden)
    }

both mac_address and id / tags seem to be tainted. Might this be a bug?

[M0NsTeRRR]

Hello,

Can confirm, with qemu guest agent enabled it's not the only attribute :

• ipv4_addresses
• ipv6_addresses
• mac_addresses
• network_interface_names

Terraform will perform the following actions:

  # proxmox_virtual_environment_vm.vm is tainted, so must be replaced
-/+ resource "proxmox_virtual_environment_vm" "vm" {
      ~ id                      = "105" -> (known after apply)
      ~ ipv4_addresses          = [
          - [
              - "127.0.0.1",
            ],
          - [
              - "192.168.10.91",
            ],
        ] -> (known after apply)
      ~ ipv6_addresses          = [
          - [
              - "::1",
            ],
          - [
              - "2a0c:b641:2c0:110::91",
              - "fe80::be24:11ff:fe17:8eaf",
            ],
        ] -> (known after apply)
      ~ mac_addresses           = [
          - "00:00:00:00:00:00",
          - "BC:24:11:17:8E:AF",
        ] -> (known after apply)
        name                    = "worker1.unicornafk.fr"
      ~ network_interface_names = [
          - "lo",
          - "enp6s18",
        ] -> (known after apply)
        tags                    = [
            "ansible_managed",
            "kubernetes",
            "kubernetes_worker",
        ]
      ~ vm_id                   = 105 -> (known after apply)
        # (20 unchanged attributes hidden)

      ~ disk {
          ~ file_format       = "raw" -> (known after apply)
          ~ path_in_datastore = "vm-105-disk-1" -> (known after apply)
            # (7 unchanged attributes hidden)
        }

      ~ network_device {
          ~ mac_address = "BC:24:11:17:8E:AF" -> (known after apply)
            # (8 unchanged attributes hidden)
        }

        # (5 unchanged blocks hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

here is my terraform configuration if needed

[lazywebm]

Same here. It happens when I want to add a 2nd disk via terraform (either dynamic or as a 2nd static disk block).

If I add the disk manually in Proxmox first, and let terraform run then, it adopts the new disk as it's supposed to be.

[lazywebm]

Adding to my comment above:

I've managed to refactor my terraform code switching from the Telmate Proxmox provider to bpg, which mostly works well.

But importing existing VMs yields a similar "forces replacement" error for existing disks:

Terraform will perform the following actions:

  # module.proxmox_monero-j[0].proxmox_virtual_environment_vm.vm must be replaced
-/+ resource "proxmox_virtual_environment_vm" "vm" {
      ~ id                      = "101" -> (known after apply)
      ~ ipv4_addresses          = [
          - [
              - "127.0.0.1",
            ],
          - [
              - "10.10.4.10",
            ],
        ] -> (known after apply)
      ~ ipv6_addresses          = [
          - [
              - "::1",
            ],
          - [
              - "fe80::38f0:f8ff:fe56:b90b",
            ],
        ] -> (known after apply)
      ~ mac_addresses           = [
          - "00:00:00:00:00:00",
          - "3A:F0:F8:56:B9:0B",
        ] -> (known after apply)
        name                    = "monero-j01"
      ~ network_interface_names = [
          - "lo",
          - "eth0",
        ] -> (known after apply)
      - tags                    = [] -> null
      + vm_id                   = (known after apply)
        # (21 unchanged attributes hidden)

      + clone {
          + datastore_id = "vmpool_zfs_encrypted"
          + full         = true
          + node_name    = "atlas"
          + retries      = 1
          + vm_id        = 126
        }

      ~ cpu {
          - flags        = [] -> null
            # (7 unchanged attributes hidden)
        }

      + disk { # forces replacement
          + cache             = "writeback"
          + datastore_id      = "vmpool_zfs_encrypted"
          + discard           = "on"
          + file_format       = "raw" # forces replacement
          + interface         = "scsi0"
          + iothread          = false
          + path_in_datastore = (known after apply)
          + size              = 40
          + ssd               = true
        }
      + disk { # forces replacement
          + cache             = "writeback"
          + datastore_id      = "vmpool_zfs_encrypted"
          + discard           = "on"
          + file_format       = "raw" # forces replacement
          + interface         = "scsi1"
          + iothread          = false
          + path_in_datastore = (known after apply)
          + size              = 250
          + ssd               = true
        }

      ~ initialization {
          - interface    = "ide0" -> null
            # (1 unchanged attribute hidden)

            # (2 unchanged blocks hidden)
        }

      ~ network_device {
          ~ mac_address = "3A:F0:F8:56:B9:0B" -> (known after apply)
            # (8 unchanged attributes hidden)
        }

        # (4 unchanged blocks hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

The file_format variable seems to have a problem. It does not matter if the value is explicitly set to raw or if file_format is omitted entirely. The only way to mitigate at the moment is to ignore the disk resource globally, which leaves a clean state after terraform apply.

  lifecycle {
    ignore_changes = [
      clone,
      disk,
    ]
  }

Proxmox 8.0.4 Terraform 1.6.3 bpg/proxmox 0.38.1

[otopetrik]

Hello,

it looks like there are multiple things mixed together in this issue.

There a two different reasons why terraform might want to replace a VM (or any other resource).

1. Resource can be 'tainted'.

That means that its creation was not successfuly completed. In plan it can be seen as "is tainted, so must be replaced", e.g:

  # proxmox_virtual_environment_vm.vm_test is tainted, so must be replaced
-/+ resource "proxmox_virtual_environment_vm" "vm_test" {
      ~ id                      = "102" -> (known after apply)
      ~ ipv4_addresses          = [] -> (known after apply)
      ~ ipv6_addresses          = [] -> (known after apply)

Because terraform HTTP API does not provide all operations needed, this provider also uses SSH. When VM configured with disks using file_id argument, the VM is first created using HTTP API, then using SSH the provider copies the disk image and attaches it.

If the SSH part fails (e.g. because SSH is not configured, does not work, etc...) the VM is not in a good known state, next time terraform is run, it sees it as tainted and correctly wants to recreate it.

Stopping terraform using Ctrl-C or failing provisioner can leave behind tainted resources

2. Resource description changed in a way which requires complete replacement.

E.g. adding clone.vm_id argument to VM clearly requires VM to be replaced. There is no reasonable way to modify existing VM to match the new resource description.

In plan it can be seen as "must be replaced" with comments "# forces replacement" after arguments which force that replacement, e.g:

  # proxmox_virtual_environment_vm.vm_test must be replaced
-/+ resource "proxmox_virtual_environment_vm" "vm_test" {
      ~ id                      = "102" -> (known after apply)
      ~ ipv4_addresses          = [] -> (known after apply)
      ~ ipv6_addresses          = [] -> (known after apply)

...

     + clone {
          + full    = true # forces replacement
          + retries = 1 # forces replacement
          + vm_id   = 1234 # forces replacement
        }

The problem is, that at the moment, many operations modifying VM's disk force VM replacement (see #103), some of those operations should not require VM replacement.

It is a bit tricky issue, because Proxmox API does not have a stable ID for a disk. Moving disk local-zfs:vm-102-disk-3 (datastore local-zfs, path_in_datastore vm-102-disk-3) to datastore another-zfs causes automatic change of path_in_datastore to vm-102-disk-0 (assuming that there are no other disks of VM 102 in another-zfs). The disk number at the end of path_in_datastore is not stable! Using interface argument as stable id is also problematic, as users might want to switch between e.g. scsi and virtio.

It probably should be possible to not-require VM replacement when adding disks.

Supporting removing disks from VM without VM replacement would likely require stable disk identifier, maybe auto-genrating 'serial' argument of the disk (at the moment, it looks like the provider does not use this part of proxmox API).

@Xel-Naha The plan shows resources which are tainted, which suggest that their creation did not finish. Is it possible that provisioner failed to run ? (I use mostly VMs, so have no experience with proxmox_virtual_environment_container failure modes)

@M0NsTeRRR The linked terraform configuration uses bios = "ovmf" and

initialization {
    interface            = "ide2"

This will result in a configuration when cloud-init does not reliably find the configuration data. Use interface = "scsi1" or similar, details here.

That alone should not cause VM to be tainted, but combined with

agent {
    enabled = true
}

The provider waits for qemu-guest-agent to run, and for VM get at least one reasonable IP address. Without the cloud init, the static address is not set. Assuming that terraform either times out or is stopped using Ctrl-C, it would explain the tainted resources.

Also note that qemu-guest-agent is not installed by default in most cloud images, it can be installed and started during cloud-init. Looking at the linked configuration it looks like you are using custom packer-build image with qemu-guest-agent baked in, so that should not be the problem.

@lazywebm Importing existing resources is relatively recent part of the provider. I think it badly interacts with the default value file_format = "qcow2" somewhere. Depending on your risk tolerance, manually changing file_format value in terraform.tfstate file to raw might be a workaround. Then it should be possible to remove disks from ignore changes. Probably the only reason to do that would be to gain the ability to increase VM disks sizes from terraform configuration.

[M0NsTeRRR]

@M0NsTeRRR The linked terraform configuration uses bios = "ovmf" and

initialization {
    interface            = "ide2"

This will result in a configuration when cloud-init does not reliably find the configuration data. Use interface = "scsi1" or similar, details here.

That alone should not cause VM to be tainted, but combined with

agent {
    enabled = true
}

The provider waits for qemu-guest-agent to run, and for VM get at least one reasonable IP address. Without the cloud init, the static address is not set. Assuming that terraform either times out or is stopped using Ctrl-C, it would explain the tainted resources.

Also note that qemu-guest-agent is not installed by default in most cloud images, it can be installed and started during cloud-init. Looking at the linked configuration it looks like you are using custom packer-build image with qemu-guest-agent baked in, so that should not be the problem.

No my setup with IDE and cloud init is applied without issue and cloud init don't require this requirement as you said. If you are willing to test again i'm using a packer template that build a latest ubuntu 22.04 server and I clean at the end the cloud init instance with cloud-init clean --seed --machine-id. Maybe some people faced a bug (as I see on the other issue) but it works without issue on my side. I will add a comment in the linked issue.

After deploying my instance I've looked into the tfstate and all IPs are set in it. So I don't understand why the next plan wants to remove them if they are available from qemu guest agent.

Yes my packer template has qemu-guest-agent and IPs are shown on proxmox interface.

[M0NsTeRRR]

Here is the full output of a fresh machine from this morning

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # proxmox_virtual_environment_vm.vm will be created
  + resource "proxmox_virtual_environment_vm" "vm" {
      + acpi                    = true
      + bios                    = "ovmf"
      + boot_order              = [
          + "scsi0",
        ]
      + id                      = (known after apply)
      + ipv4_addresses          = (known after apply)
      + ipv6_addresses          = (known after apply)
      + keyboard_layout         = "en-us"
      + mac_addresses           = (known after apply)
      + migrate                 = false
      + name                    = "test.unicornafk.fr"
      + network_interface_names = (known after apply)
      + node_name               = "server3"
      + on_boot                 = true
      + reboot                  = false
      + scsi_hardware           = "virtio-scsi-single"
      + started                 = true
      + tablet_device           = true
      + tags                    = [
          + "ansible_managed",
          + "dhcp",
        ]
      + template                = false
      + timeout_clone           = 1800
      + timeout_create          = 1800
      + timeout_migrate         = 1800
      + timeout_move_disk       = 1800
      + timeout_reboot          = 1800
      + timeout_shutdown_vm     = 1800
      + timeout_start_vm        = 1800
      + timeout_stop_vm         = 300
      + vm_id                   = (known after apply)

      + agent {
          + enabled = true
          + timeout = "15m"
          + trim    = false
          + type    = "virtio"
        }

      + clone {
          + full    = true
          + retries = 1
          + vm_id   = 114
        }

      + cpu {
          + architecture = "x86_64"
          + cores        = 4
          + flags        = []
          + hotplugged   = 0
          + numa         = false
          + sockets      = 1
          + type         = "x86-64-v2-AES"
          + units        = 1024
        }

      + disk {
          + cache             = "none"
          + datastore_id      = "SERVER3-SSD1"
          + discard           = "ignore"
          + file_format       = (known after apply)
          + interface         = "scsi0"
          + iothread          = true
          + path_in_datastore = (known after apply)
          + size              = 20
          + ssd               = false
        }

      + initialization {
          + datastore_id         = "local-lvm"
          + interface            = "ide2"
          + meta_data_file_id    = "local:snippets/test.unicornafk.fr-ci-meta_data.yml"
          + network_data_file_id = "local:snippets/test.unicornafk.fr-ci-network.yml"
          + user_data_file_id    = "local:snippets/test.unicornafk.fr-ci-user.yml"
        }

      + memory {
          + dedicated = 4096
          + floating  = 0
          + shared    = 0
        }

      + network_device {
          + bridge      = "vmbr1"
          + enabled     = true
          + firewall    = false
          + mac_address = (known after apply)
          + model       = "virtio"
          + mtu         = 0
          + queues      = 0
          + rate_limit  = 0
          + vlan_id     = 10
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + vm_id       = (known after apply)
proxmox_virtual_environment_vm.vm: Creating...
proxmox_virtual_environment_vm.vm: Still creating... [10s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [20s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [30s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [40s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [50s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [1m0s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [1m10s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [1m20s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [1m30s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [1m40s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [1m50s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [2m0s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [2m10s elapsed]
proxmox_virtual_environment_vm.vm: Provisioning with 'local-exec'...
proxmox_virtual_environment_vm.vm (local-exec): Executing: ["/bin/sh" "-c" "ansible all -i 192.168.10.234, -m shell -a '/usr/bin/cloud-init status --wait' --ssh-common-args='-o StrictHostKeyChecking=no -o userknownhostsfile=/dev/null'"]
proxmox_virtual_environment_vm.vm: Still creating... [2m20s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [2m30s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [2m40s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [2m50s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [3m0s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [3m10s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [3m20s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [3m30s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [3m40s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [3m50s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [4m0s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [4m10s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [4m20s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [4m30s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [4m40s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [4m50s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [5m0s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [5m10s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [5m20s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [5m30s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [5m40s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [5m50s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [6m0s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [6m10s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [6m20s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [6m30s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [6m40s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [6m50s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [7m0s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [7m10s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [7m18s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [7m28s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [7m38s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [7m48s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [7m58s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [8m8s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [8m18s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [8m28s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [8m38s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [8m48s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [8m58s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [9m8s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [9m18s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [9m28s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [9m38s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [9m48s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [9m58s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [10m8s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [10m18s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [10m28s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [10m38s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [10m48s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [10m58s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [11m8s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [11m18s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [11m28s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [11m38s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [11m48s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [11m58s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [12m8s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [12m18s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [12m28s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [12m38s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [12m48s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [12m58s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [13m8s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [13m18s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [13m28s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [13m38s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [13m48s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [13m58s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [14m8s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [14m18s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [14m28s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [14m38s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [14m48s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [14m58s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [15m8s elapsed]
proxmox_virtual_environment_vm.vm: Still creating... [15m18s elapsed]
proxmox_virtual_environment_vm.vm (local-exec): 192.168.10.234 | CHANGED | rc=0 >>
proxmox_virtual_environment_vm.vm (local-exec): ..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
proxmox_virtual_environment_vm.vm (local-exec): status: done
proxmox_virtual_environment_vm.vm: Creation complete after 15m19s [id=104]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

domain = "unicornafk.fr"
server_used = "server3"
vm_hostname = "test"
vm_id = "104"
vm_ip = "192.168.10.234"
vm_ip6 = "2a0c:b641:2c0:110::234"
vm_tags = tolist([
  "ansible_managed",
  "dhcp",
])

The new plan gives

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # proxmox_virtual_environment_vm.vm must be replaced
-/+ resource "proxmox_virtual_environment_vm" "vm" {
      ~ id                      = "104" -> (known after apply)
      ~ ipv4_addresses          = [
          - [
              - "127.0.0.1",
            ],
          - [
              - "192.168.10.234",
            ],
        ] -> (known after apply)
      ~ ipv6_addresses          = [
          - [
              - "::1",
            ],
          - [
              - "2a0c:b641:2c0:110::234",
              - "fe80::be24:11ff:fe87:2d39",
            ],
        ] -> (known after apply)
      ~ mac_addresses           = [
          - "00:00:00:00:00:00",
          - "BC:24:11:87:2D:39",
        ] -> (known after apply)
        name                    = "test.unicornafk.fr"
      ~ network_interface_names = [
          - "lo",
          - "enp6s18",
        ] -> (known after apply)
        tags                    = [
            "ansible_managed",
            "dhcp",
        ]
      ~ vm_id                   = 104 -> (known after apply)
        # (20 unchanged attributes hidden)

      ~ clone {
          ~ vm_id   = 114 -> 111 # forces replacement
            # (2 unchanged attributes hidden)
        }

      ~ disk {
          ~ file_format       = "raw" -> (known after apply)
          ~ path_in_datastore = "vm-104-disk-1" -> (known after apply)
            # (7 unchanged attributes hidden)
        }

      ~ network_device {
          ~ mac_address = "BC:24:11:87:2D:39" -> (known after apply)
            # (8 unchanged attributes hidden)
        }

        # (4 unchanged blocks hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Changes to Outputs:
  ~ vm_id       = "104" -> (known after apply)

[otopetrik]

Note that there is a difference: Old plan:

  # proxmox_virtual_environment_vm.vm is tainted, so must be replaced

The resource was not completely created.

New plan:

  # proxmox_virtual_environment_vm.vm must be replaced
...
~ clone {
          ~ vm_id   = 114 -> 111 # forces replacement
            # (2 unchanged attributes hidden)
        }
...

Today's run created resource completely. Note that there is no "is tainted," string.

But after it was created, its configuration was changed to require to be cloned from completely different VM (it was cloned from 114, but now it should be cloned from 111).

This does not look stable. It looks like between creating the VM and running the plan again, another VM with lower VM id and "correct tags to be a template" was created. Maybe a packer run at the same time ?

[M0NsTeRRR]

Note that there is a difference: Old plan:

  # proxmox_virtual_environment_vm.vm is tainted, so must be replaced

The resource was not completely created.

New plan:

  # proxmox_virtual_environment_vm.vm must be replaced
...
~ clone {
          ~ vm_id   = 114 -> 111 # forces replacement
            # (2 unchanged attributes hidden)
        }
...

Today's run created resource completely. Note that there is no "is tainted," string.

But after it was created, its configuration was changed to require to be cloned from completely different VM (it was cloned from 114, but now it should be cloned from 111).

This does not look stable. It looks like between creating the VM and running the plan again, another VM with lower VM id and "correct tags to be a template" was created. Maybe a packer run at the same time ?

It's my fault i've not explained but it's not related to terraform, i've a local exec script (ansible wait cloud init) as you can see in the log and I got an issue with pam.d that doesn't allow user except root to login to when the machine is booting up, it's why the ressource got tainted on first run as terraform stopped to execute on this error. Sorry for the confusion. To fix it i've changed pam.d configuration in /etc/pam.d/sshd

# comment the line below
#account required pam_nologin.so
# add the line below
auth required pam_wheel.so

Will look into your tip.

[M0NsTeRRR]

I've tested with this change again and it works (not a daily terraform user :sweat_smile:). It was a wrong understanding on my side (changing packer template of course need recreation but I though the "-" in log mean recreation too). Sorry for the noise in this issue, I will help in the other issue regarding IDE2, sorry for wasting your time @otopetrik.

data.proxmox_virtual_environment_vms.template: Reading...
proxmox_virtual_environment_file.cloud_network_config: Refreshing state... [id=local:snippets/test1.unicornafk.fr-ci-network.yml]
proxmox_virtual_environment_file.cloud_user_config: Refreshing state... [id=local:snippets/test1.unicornafk.fr-ci-user.yml]
proxmox_virtual_environment_file.cloud_meta_config: Refreshing state... [id=local:snippets/test1.unicornafk.fr-ci-meta_data.yml]
data.proxmox_virtual_environment_vms.template: Read complete after 0s [id=068e1c27-478f-41d2-9d69-8f79cb214f3d]
proxmox_virtual_environment_vm.vm: Refreshing state... [id=104]

No changes. Your infrastructure matches the configuration.

PS : As an user the provider works really well and is well documented compared to Telmate provider good job bpg and other contributors :)

[lazywebm]

@lazywebm Importing existing resources is relatively recent part of the provider. I think it badly interacts with the default value file_format = "qcow2" somewhere. Depending on your risk tolerance, manually changing file_format value in terraform.tfstate file to raw might be a workaround. Then it should be possible to remove disks from ignore changes. Probably the only reason to do that would be to gain the ability to increase VM disks sizes from terraform configuration.

You are quite right, thank you for your detailed answer and getting me on the right track. I edited the raw Terraform state - a little tedious since it's a Postgres DB - and was able to include disks and manage them via Terraform.

For reference - TF state in Postgres is still JSON, but each resource is one (very long) line.

So within the resource for the existing VM, somewhere within that line, this:

"disk":[]\n

turns into:

"disk":[\n {\n "cache": "writeback",\n "datastore_id": "vmpool_zfs_encrypted",\n "discard": "on",\n "file_format": "raw",\n "file_id": "",\n "interface": "scsi0",\n "iothread": false,\n "path_in_datastore": "101/vm-101-disk-0.raw",\n "size": 40,\n "speed": [],\n "ssd": true\n },\n {\n "cache": "writeback",\n "datastore_id": "vmpool_zfs_encrypted",\n "discard": "on",\n "file_format": "raw",\n "file_id": "",\n "interface": "scsi1",\n "iothread": false,\n "path_in_datastore": "101/vm-101-disk-1.raw",\n "size": 250,\n "speed": [],\n "ssd": true\n }\n ]\n

[Xel-Naha]

@Xel-Naha The plan shows resources which are tainted, which suggest that their creation did not finish. Is it possible that provisioner failed to run ? (I use mostly VMs, so have no experience with proxmox_virtual_environment_container failure modes)

I've retested this just now. The container completes fine and immediately afterwards I get no problems, however if I come back later on I do see the same error that the container state was tainted.

I will keep this ct running and retry later on.

[bpg-autobot[bot]]

Marking this issue as stale due to inactivity in the past 180 days. This helps us focus on the active issues. If this issue is reproducible with the latest version of the provider, please comment. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!