different boot disks with cloud-init

[braham2019]

I want to create 3 identical swarm nodes with a 8 GB boot disk and 16 GB data disk. This is a lab setup to practice my terraform and ansible skills.

The VM's are clones from a cloud init template.

The problem I'm facing is that every time a VM is created, I get a different boot disk. Sometimes it's sda, sometimes it's sdb. I've destroyed and redone this several times. The results are always different.

Example for first server

swarmie@swarm-1:~$ lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda       8:0    0    8G  0 disk
├─sda1    8:1    0  7.9G  0 part /
├─sda14   8:14   0    3M  0 part
└─sda15   8:15   0  124M  0 part /boot/efi
sdb       8:16   0   16G  0 disk
sr0      11:0    1    4M  0 rom

And the second server:

swarmie@swarm-2:~$ lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda       8:0    0   16G  0 disk
sdb       8:16   0    8G  0 disk
├─sdb1    8:17   0  7.9G  0 part /
├─sdb14   8:30   0    3M  0 part
└─sdb15   8:31   0  124M  0 part /boot/efi
sr0      11:0    1    4M  0 rom

I have tried to change "disks" section to a "disk" section , but then cloud-init fails altogether. Commenting out the boot section or enabling it is the same result.

boot = "order=scsi0;scsi1"

This is my swarm.tf file.

resource "proxmox_vm_qemu" "srv-swarm" {
  count = var.ci_swarm_count
  name = "swarm-${count.index + 1}"
  desc = "Swarm Node ${count.index + 1}"
  vmid = var.ci_start_vmid + (count.index + var.ci_swarm_count)
  target_node = "morpheus"

  clone = "debian-cloud-init"

  agent = 1
  cores = 2
  sockets = 1
  cpu = "host"
  memory = 4096
  bootdisk = "scsi0"
  boot = "order=scsi0;scsi1"
  scsihw = "virtio-scsi-single"
  onboot = false
  os_type = "cloud-init"
  #ipconfig0 = "ip=dhcp"
  ipconfig0 = "ip=${var.ci_swarm_base_ip}${count.index}/${var.ci_network_cidr},gw=${var.ci_ip_gateway}"
  nameserver = "172.16.0.1"
  searchdomain = "lan"
  ciuser = var.ci_user
  cipassword = var.ci_password
  sshkeys = <<EOF
  ${file(var.ci_ssh_public_key)}
  EOF

  network {
    bridge = "vmbr0"
    model = "virtio"
    tag = "404"
  }

  disks {
    scsi {
      scsi0 {
        disk {
          size = 8
          storage = "local-zfs"
          iothread = true
          discard = true
        }
      }
      scsi1 {
        disk {
          size = 16
          storage = "local-zfs"
          iothread = true
          discard = true
        }
      }
    }
    ide {
      ide3 {
        cloudinit {
          storage = "local-zfs"
        }
      }
    }
  }
}

[Tinyblargon]

@braham2019 the Terraform looks good. You could change boot to boot = "order=scsi0", not sure what bootdisk does in this context.

The disks swapping probably has to do with the os in the image as /dev/sda and /dev/sdb are based on the order is which GRUB discovered the "hardware".

As workaround, you could add a serial: to the disks. In /dev/disk/by-id, a simlink will be created to /dev/sda and /dev/sdb.