Failed to create qemu VMs: 599 Too Many Redirections

[alex4108]

tldr; When trying to create 3 qemu VMs on the same host (slowly), the provider is unable to handle the transient 599 error raised by the proxmox API, causing an error during apply.

proposed solution to handle & retry on 599 errors

Terraform code:

variable "template_id" {
  description = "The VM ID of the template node to deploy from"
  default     = "kube-ubuntu-2004-20230529-220351"
  type        = string
}

variable "controlplane_memory" {
  description = "MB of RAM to allocate for control planes"
  type        = number
  default     = 2048
}

variable "controlplane_cpu" {
  description = "CPU cores to allocate for control planes"
  type        = number
  default     = 2
}

locals {
  timestamp   = replace(replace(timestamp(), "-", ""), ":", "")
  target_node = "pve-1"
  controlplanes = {
    "1" = {
      target_node = "pve-1"
      hastate     = "" # no HA
      hagroup     = ""
      id          = 9010
    },
    "2" = {
      target_node = "pve-1"
      hastate     = "" # no HA
      hagroup     = ""
      id          = 9011
    },
    "3" = {
      target_node = "pve-1"
      hastate     = ""
      hagroup     = ""
      id          = 9012
    }
  }
}

resource "proxmox_vm_qemu" "controlplane" {
  for_each    = local.controlplanes
  vmid        = each.value.id
  name        = "k8s-master-${local.timestamp}-${each.key}"
  target_node = each.value.target_node
  clone       = var.template_id
  onboot      = true # Start nodes on boot
  agent       = 1    # Qemu guest agent
  hastate     = each.value.hastate
  hagroup     = each.value.hagroup
  memory      = var.controlplane_memory
  cores       = var.controlplane_cpu
  scsihw      = "virtio-scsi-pci"
  cicustom    = "network=local:snippets/kube-cloudinit-network.yml,user=local:snippets/kube-cloudinit-user.yml"
  vga {
    type = "virtio"
  }
}

Plan:

~ cd ./infra/controlplane/terraform && bash run.sh

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of telmate/proxmox from the dependency lock file
- Using previously-installed telmate/proxmox v2.9.14

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
proxmox_vm_qemu.controlplane["3"]: Refreshing state... [id=pve-1/qemu/9012]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the
following symbols:
  + create

Terraform will perform the following actions:

  # proxmox_vm_qemu.controlplane["1"] will be created
  + resource "proxmox_vm_qemu" "controlplane" {
      + additional_wait           = 5
      + agent                     = 1
      + automatic_reboot          = true
      + balloon                   = 0
      + bios                      = "seabios"
      + boot                      = (known after apply)
      + bootdisk                  = (known after apply)
      + cicustom                  = "network=local:snippets/kube-cloudinit-network.yml,user=local:snippets/kube-cloudinit-user.yml"
      + clone                     = "kube-ubuntu-2004-20230530-013945"
      + clone_wait                = 10
      + cores                     = 2
      + cpu                       = "host"
      + default_ipv4_address      = (known after apply)
      + define_connection_info    = true
      + force_create              = false
      + full_clone                = true
      + guest_agent_ready_timeout = 100
      + hotplug                   = "network,disk,usb"
      + id                        = (known after apply)
      + kvm                       = true
      + memory                    = 2048
      + name                      = (known after apply)
      + nameserver                = (known after apply)
      + onboot                    = true
      + oncreate                  = true
      + preprovision              = true
      + reboot_required           = (known after apply)
      + scsihw                    = "virtio-scsi-pci"
      + searchdomain              = (known after apply)
      + sockets                   = 1
      + ssh_host                  = (known after apply)
      + ssh_port                  = (known after apply)
      + tablet                    = true
      + target_node               = "pve-1"
      + unused_disk               = (known after apply)
      + vcpus                     = 0
      + vlan                      = -1
      + vmid                      = 9010

      + vga {
          + type = "virtio"
        }
    }

  # proxmox_vm_qemu.controlplane["2"] will be created
  + resource "proxmox_vm_qemu" "controlplane" {
      + additional_wait           = 5
      + agent                     = 1
      + automatic_reboot          = true
      + balloon                   = 0
      + bios                      = "seabios"
      + boot                      = (known after apply)
      + bootdisk                  = (known after apply)
      + cicustom                  = "network=local:snippets/kube-cloudinit-network.yml,user=local:snippets/kube-cloudinit-user.yml"
      + clone                     = "kube-ubuntu-2004-20230530-013945"
      + clone_wait                = 10
      + cores                     = 2
      + cpu                       = "host"
      + default_ipv4_address      = (known after apply)
      + define_connection_info    = true
      + force_create              = false
      + full_clone                = true
      + guest_agent_ready_timeout = 100
      + hotplug                   = "network,disk,usb"
      + id                        = (known after apply)
      + kvm                       = true
      + memory                    = 2048
      + name                      = (known after apply)
      + nameserver                = (known after apply)
      + onboot                    = true
      + oncreate                  = true
      + preprovision              = true
      + reboot_required           = (known after apply)
      + scsihw                    = "virtio-scsi-pci"
      + searchdomain              = (known after apply)
      + sockets                   = 1
      + ssh_host                  = (known after apply)
      + ssh_port                  = (known after apply)
      + tablet                    = true
      + target_node               = "pve-1"
      + unused_disk               = (known after apply)
      + vcpus                     = 0
      + vlan                      = -1
      + vmid                      = 9011

      + vga {
          + type = "virtio"
        }
    }

  # proxmox_vm_qemu.controlplane["3"] will be created
  + resource "proxmox_vm_qemu" "controlplane" {
      + additional_wait           = 5
      + agent                     = 1
      + automatic_reboot          = true
      + balloon                   = 0
      + bios                      = "seabios"
      + boot                      = (known after apply)
      + bootdisk                  = (known after apply)
      + cicustom                  = "network=local:snippets/kube-cloudinit-network.yml,user=local:snippets/kube-cloudinit-user.yml"
      + clone                     = "kube-ubuntu-2004-20230530-013945"
      + clone_wait                = 10
      + cores                     = 2
      + cpu                       = "host"
      + default_ipv4_address      = (known after apply)
      + define_connection_info    = true
      + force_create              = false
      + full_clone                = true
      + guest_agent_ready_timeout = 100
      + hotplug                   = "network,disk,usb"
      + id                        = (known after apply)
      + kvm                       = true
      + memory                    = 2048
      + name                      = (known after apply)
      + nameserver                = (known after apply)
      + onboot                    = true
      + oncreate                  = true
      + preprovision              = true
      + reboot_required           = (known after apply)
      + scsihw                    = "virtio-scsi-pci"
      + searchdomain              = (known after apply)
      + sockets                   = 1
      + ssh_host                  = (known after apply)
      + ssh_port                  = (known after apply)
      + tablet                    = true
      + target_node               = "pve-1"
      + unused_disk               = (known after apply)
      + vcpus                     = 0
      + vlan                      = -1
      + vmid                      = 9012

      + vga {
          + type = "virtio"
        }
    }

Plan: 3 to add, 0 to change, 0 to destroy.
──────────────────────────────────────
Saved the plan to: plan.out

Apply the plan, note parallelism 1 to try and slow things down.

~ terraform apply -parallelism=1 plan.out 
proxmox_vm_qemu.controlplane["3"]: Creating...
proxmox_vm_qemu.controlplane["3"]: Still creating... [10s elapsed]
proxmox_vm_qemu.controlplane["3"]: Still creating... [20s elapsed]
proxmox_vm_qemu.controlplane["3"]: Still creating... [30s elapsed]
proxmox_vm_qemu.controlplane["3"]: Still creating... [40s elapsed]
proxmox_vm_qemu.controlplane["3"]: Still creating... [50s elapsed]
proxmox_vm_qemu.controlplane["3"]: Still creating... [1m0s elapsed]
proxmox_vm_qemu.controlplane["3"]: Still creating... [1m10s elapsed]
proxmox_vm_qemu.controlplane["3"]: Still creating... [1m20s elapsed]
proxmox_vm_qemu.controlplane["3"]: Still creating... [1m30s elapsed]
proxmox_vm_qemu.controlplane["3"]: Still creating... [1m40s elapsed]
proxmox_vm_qemu.controlplane["3"]: Creation complete after 1m44s [id=pve-1/qemu/9012]
proxmox_vm_qemu.controlplane["1"]: Creating...
proxmox_vm_qemu.controlplane["1"]: Still creating... [10s elapsed]
proxmox_vm_qemu.controlplane["1"]: Still creating... [20s elapsed]
proxmox_vm_qemu.controlplane["1"]: Still creating... [30s elapsed]
proxmox_vm_qemu.controlplane["1"]: Still creating... [40s elapsed]
virt-customize --truncate /etc/machine-info --truncate /etc/machine-idproxmox_vm_qemu.controlplane["1"]: Still creating... [50s elapsed]
proxmox_vm_qemu.controlplane["1"]: Still creating... [1m0s elapsed]
proxmox_vm_qemu.controlplane["1"]: Still creating... [1m10s elapsed]
proxmox_vm_qemu.controlplane["1"]: Still creating... [1m20s elapsed]
proxmox_vm_qemu.controlplane["1"]: Still creating... [1m30s elapsed]
proxmox_vm_qemu.controlplane["1"]: Still creating... [1m40s elapsed]
proxmox_vm_qemu.controlplane["1"]: Still creating... [1m50s elapsed]
proxmox_vm_qemu.controlplane["1"]: Creation complete after 1m58s [id=pve-1/qemu/9010]
proxmox_vm_qemu.controlplane["2"]: Creating...
proxmox_vm_qemu.controlplane["2"]: Still creating... [10s elapsed]
proxmox_vm_qemu.controlplane["2"]: Still creating... [20s elapsed]
╷
│ Error: 599 Too many redirections
│
│   with proxmox_vm_qemu.controlplane["2"],
│   on main.tf line 1, in resource "proxmox_vm_qemu" "controlplane":
│    1: resource "proxmox_vm_qemu" "controlplane" {
│
╵

Relevant tflog:

2023-05-30T02:10:54.337-0600 [INFO]  Starting apply for proxmox_vm_qemu.controlplane["1"]
2023-05-30T02:10:54.337-0600 [DEBUG] proxmox_vm_qemu.controlplane["1"]: applying the planned Create change
2023-05-30T02:10:54.338-0600 [INFO]  provider.terraform-provider-proxmox_v2.9.14: 2023/05/30 02:10:54 [DEBUG] setting computed for "unused_disk" from ComputedKeys: timestamp=2023-05-30T02:10:54.338-0600
2023-05-30T02:10:54.339-0600 [INFO]  provider.terraform-provider-proxmox_v2.9.14: 2023/05/30 02:10:54 [DEBUG][QemuVmCreate] checking for duplicate name: k8s-master-20230530T080704Z-1: timestamp=2023-05-30T02:10:54.339-0600
2023-05-30T02:10:54.346-0600 [INFO]  provider.terraform-provider-proxmox_v2.9.14: 2023/05/30 02:10:54 [DEBUG][QemuVmCreate] cloning VM: timestamp=2023-05-30T02:10:54.346-0600
proxmox_vm_qemu.controlplane["1"]: Still creating... [10s elapsed]
2023-05-30T02:11:14.243-0600 [ERROR] provider.terraform-provider-proxmox_v2.9.14: Response contains error diagnostic: diagnostic_summary="599 Too many redirections" diagnostic_severity=ERROR tf_proto_version=5.3 tf_provider_addr=registry.terraform.io/telmate/proxmox tf_req_id=22ff0f17-0ebb-8454-5a12-3f6ef9dbcdad tf_resource_type=proxmox_vm_qemu @caller=github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/diag/diagnostics.go:55 @module=sdk.proto diagnostic_detail= tf_rpc=ApplyResourceChange timestamp=2023-05-30T02:11:14.243-0600
2023-05-30T02:11:14.250-0600 [ERROR] vertex "proxmox_vm_qemu.controlplane[\"1\"]" error: 599 Too many redirections
╷

[sorquan]

I also have this error on creating six VMs

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity. Please update the provider to the latest version and, in the issue persist, provide full configuration and debug logs

[github-actions[bot]]

This issue was closed because it has been inactive for 5 days since being marked as stale.

[MEschenbacher]

We have the same issue on latest master.

[RobertSkawinski]

same issue on v3.0.1-rc1

[AlanJumeaucourt]

same issue on v3.0.1-rc1

Same issue here on v3.0.1-rc1 when creating 5 vm at once then launching small "remote-exec" bash command. The four first vm are OK then the last one get this issue.

Hopefully, my terraform is idempotent so I can just relaunch the terraform apply.

Did you find any fix or workaround ?

[qume]

We are seeing this too, however when creating just a single LXC container

[holooloo]

i have the same issue the creating more than 1 VM