some-random-username-123
commented
1 year ago PS:I don't really know if this is not implemented, of if its just a issue from my side.
Open some-random-username-123 opened 1 year ago
some-random-username-123
commented
1 year ago PS:I don't really know if this is not implemented, of if its just a issue from my side.
bpg
commented
1 year ago Hi @some-random-username-123! 👋🏼 (nice GH username btw :)
- See error
Could you pls. post here the error that that you saw?
Also I'd like to clarify whther the original VM that you're cloning has a cloudinit config?
some-random-username-123
commented
1 year ago Hi @bpg, Sure I can give you more insights:
Also I'd like to clarify whther the original VM that you're cloning has a cloudinit config?
I Tried both, so i have a emplate with a Cloudinit drive, and i also have a template with no cloudinit drive.
I have the issue with both tamplates.
The exact issue is, that when I clone my vm with the TF provider, after the vm starts the cloudinit config (user and hostname, etc) is not applied in the template.
But when I clone it manually and press the button, all the cloudinit configuration is applied.
If you want to have a look at my code feel free to look at my repo: https://github.com/some-random-username-123/Proxmox-setup/tree/main
I hope this gives you some more insight.
Regards
0xinterface
commented
1 year ago @bpg I took a brief look at the example @some-random-username-123 provided and apparently the packer template is built by specifying bios to ovmf therefore I think this is related to #575
webtroter
commented
1 year ago Hello,
I'm also having this problem
I found this small thread in the Proxmox forums about the Regeneration process. -> https://forum.proxmox.com/threads/regenerate-cloud-init-image-using-ansible.89964/
It is basically a remove/re-add the cloudinit drive.
I hope it could help solve this problem.
Thank you
bpg
commented
1 year ago Hey @some-random-username-123 👋🏼
I'm trying to reproduce this issue with a minimal TF configuration, but everything seems to be working for me (tm)
Here is my template:
resource "proxmox_virtual_environment_vm" "ubuntu_vm" {
name = "586-template"
node_name = var.virtual_environment_node_name
vm_id = 586
machine = "q35"
bios = "ovmf"
agent {
enabled = true
}
efi_disk {
datastore_id = "local-lvm"
file_format = "raw"
type = "4m"
}
disk {
datastore_id = "local-lvm"
file_id = proxmox_virtual_environment_file.ubuntu_cloud_image.id
interface = "virtio0"
iothread = true
discard = "on"
size = 20
}
network_device {
bridge = "vmbr0"
}
operating_system {
type = "l26"
}
started = false
}
resource "proxmox_virtual_environment_file" "ubuntu_cloud_image" {
content_type = "iso"
datastore_id = "local"
node_name = var.virtual_environment_node_name
source_file {
path = "https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img"
}
}
data "local_file" "ssh_public_key" {
filename = "./id_rsa.pub"
}
resource "proxmox_virtual_environment_file" "cloud_config" {
content_type = "snippets"
datastore_id = "local"
node_name = var.virtual_environment_node_name
source_raw {
data = <<EOF
#cloud-config
runcmd:
- apt update
- apt install -y qemu-guest-agent net-tools
- timedatectl set-timezone America/Toronto
- systemctl enable qemu-guest-agent
- systemctl start qemu-guest-agent
- echo "done" > /tmp/vendor-cloud-init-done
EOF
file_name = "cloud-config.yaml"
}
}
resource "proxmox_virtual_environment_vm" "ubuntu_vm_cloned" {
depends_on = [proxmox_virtual_environment_vm.ubuntu_vm]
name = "587-cloned"
node_name = var.virtual_environment_node_name
vm_id = 587
clone {
vm_id = 586
}
initialization {
# interface = "scsi0"
ip_config {
ipv4 {
address = "dhcp"
}
}
user_account {
username = "ubuntu"
keys = [trimspace(data.local_file.ssh_public_key.content)]
}
vendor_data_file_id = proxmox_virtual_environment_file.cloud_config.id
}
started = true
}Both VMs got deployed, the "cloned" one has cloudinit:
Without pressing "regenerate" there, I was able to login into the VM using configured public key:
So far, no ideas what could be wrong 🤔
bpg
commented
1 year ago @some-random-username-123 this could be the reason. If you're not overriding vm_ci_interface variable at apply, the default ide0 may not work in all configurations. I would recommend switching to ide2 as per documentation.
some-random-username-123
commented
1 year ago @bpg Thanks for the response. I tried it with the other interface and all, but it still wont work. This might be a issue with the distribution, as you areusing ubuntu and it works for you, idk. I will try your ubuntu template and clone next and see it has something to do with the linux distribution, alternatively I will also try and reinstall proxmox and see if this might fixes the issue.
bpg
commented
1 year ago Hi @webtroter, @tobiasehlert! 👋🏼
I see you upvoted this issue, do you have a similar problem in your deployments? If yes, would you mind sharing your configuration?
Thanks!
webtroter
commented
1 year ago On my part, after your other reply I seemed to made it work (see my librenms IAC repo)
But I need to retry the provider to be certain or at least report what I did wrong.
On Wed, Oct 11, 2023, 22:28 Pavel Boldyrev @.***> wrote:
Hi @webtroter https://github.com/webtroter, @tobiasehlert https://github.com/tobiasehlert! 👋🏼
I see you upvoted this issue, do you have a similar problem in your deployments? If yes, would you mind sharing your configuration?
Thanks!
— Reply to this email directly, view it on GitHub https://github.com/bpg/terraform-provider-proxmox/issues/586#issuecomment-1758815983, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB77R6BIXTOD2RODIAG7G53X65ITZANCNFSM6AAAAAA5DIECAY . You are receiving this because you were mentioned.Message ID: @.***>
tobiasehlert
commented
1 year ago @bpg, well slightly the same.
Example terraform file:
resource "proxmox_virtual_environment_file" "vm_cloud_init" {
content_type = "snippets"
datastore_id = "local"
node_name = var.proxmox_settings.node
source_raw {
file_name = "cloud-init.user-data.yml"
data = file("cloud-init/user-data.yml")
}
}Example cloud-init/user-data.yml file:
#cloud-config
package_upgrade: true
packages:
- qemu-guest-agent
timezone: Europe/Stockholm
users:
- name: myusername
groups: [adm, cdrom, dip, plugdev, lxd, sudo]
lock-passwd: false
sudo: ALL=(ALL) NOPASSWD:ALL
shell: /bin/bash
ssh_authorized_keys:
- ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINyDI3c8l5Se4w0B2ejL36WJh/PdD5MKKsL9zX0a9sT
power_state:
delay: now
mode: reboot
message: Rebooting after cloud-init completion
condition: trueProxmox GUI:
My expectation was to see my user that was provisioned and the ssh-key, but that's not shown.
I run version Proxmox version 8.0.3 and Terraform provider bpg/proxmox version 0.33.0.
tobiasehlert
commented
1 year ago Also (maybe related to cloud-init issue), the _proxmox_virtual_environmentfile resource for the cloud-init file shown in comment returns following error to me when applying:
Planning failed. Terraform encountered an error while generating this plan.
╷
│ Error: no such file: "local:snippets/cloud-init.user-data.yml"
│
│ with proxmox_virtual_environment_file.vm_cloud_init,
│ on cloud_init.tf line 1, in resource "proxmox_virtual_environment_file" "vm_cloud_init":
│ 1: resource "proxmox_virtual_environment_file" "vm_cloud_init" {
│
╵@tobiasehlert could you pls check if snippets content type is enabled on the local storage? Could be another case of https://github.com/bpg/terraform-provider-proxmox/issues/370#issuecomment-1605266793
ratiborusx
commented
1 year ago I've spent a whole day yesterday and quite a bit of overtime experimenting with provider and testing cloud-init functionality specifically and here's my observations.
PART I In short, CI (cloud-init) cloud-config works as expected at least in userdata part. I didn't test vendor/network/meta because i've got no use for it but i have no reason to believe it won't work. As i already mentioned somewhere in other issues Proxmox executes cloud-init in somewhat "unconventional" way (at least i'm not used to it compared to how cloud providers deal with it) generating an ISO image for it with the data and mounting it in CD-ROM drive. It also allows to regenerate said image and reapply configuration it contains (for example add a new ssh-key for a user), which is not usually the case with the cloud stuff where cloud-config applied only once at the first boot (or more precisely some modules of CI).
I'm not sure if people are aware but things that Proxmox exposes in it's GUI for Cloud-Init is just a small subset of what cloud-config is capable of. These things are what provider can control via "initialization.dns", "initialization.ip_config" and "initialization.user_account" (btw @bpg there seem to be no way for provider to change "Upgrade packages" option as in GUI?):
But that is in no way a full extent of what could be done via cloud-config and by using provider's arguments like "user_data_file_id" (or network/vendor/meta ones) we could inject our own custom cloud-init cloud-config. My own mistake (and probably most people with this issue are doing the same) was that i expected to see changes in exposed parameters of cloud-init in Proxmox's GUI - it won't happen although after booting up all the config's parameters would be applied. I also do not fully understand that weird magic Proxmox does with cloud-init so it confuses me alot. For example (and i think THIS is what provider does or emulates when we inject custom cloud-config via "proxmox_virtual_environment_file") i used a console utility "qm" to inject my custom cloud-config into an image and bake it in template afterwards. Documentation also mentions that you could dump cloud-init info contained inside an image but this somehow never worked for me (for CUSTOM cloud-config anyway, but it shows info you put in the GUI fields of cloud-init). An example:
# freshly baked template with an injected cloud-config shows nothing for it
root@prox-srv1:~# qm cloudinit dump 1011 user
#cloud-config
hostname: template-ubuntu-22
manage_etc_hosts: true
fqdn: template-ubuntu-22.04
chpasswd:
expire: False
users:
- default
package_upgrade: true
# i try to inject cloud-config via console tool but it still shows nothing afterwads
root@prox-srv1:~# qm set 1011 --cicustom "user=local:snippets/userdata-proxmox.yml"
update VM 1011: -cicustom user=local:snippets/userdata-proxmox.yml
root@prox-srv1:~# qm cloudinit dump 1011 user
#cloud-config
hostname: template-ubuntu-22
manage_etc_hosts: true
fqdn: template-ubuntu-22.04
chpasswd:
expire: False
users:
- default
package_upgrade: true
# but if i change "User" field of Cloud-Init in Proxmox's GUI it detects changes
root@prox-srv1:~# qm cloudinit dump 1011 user
#cloud-config
hostname: template-ubuntu-22
manage_etc_hosts: true
fqdn: template-ubuntu-22.04
user: WOLOLO
chpasswd:
expire: False
users:
- default
package_upgrade: trueIn the end it's very weird for me how Proxmox handles cloud-init and i'm not sure what exactly it shows via "qm cloudinit dump". Why it's only reflection of GUI's changes i've no clue... Maybe some Proxmox guru could explain it or @bpg himself (at least on provider's part, how it handles all that stuff). The only thing i want to reiterate again - if you inject custom cloud-config into your vm/template do not expect to see any changes in the Proxmox's GUI in Cloud-Init. It won't be shown in "qm cloudinit dump" either.
PART II That's content of my custom cloud-config:
ratiborus@HOMEWORLD:~/WORKSPACE/terraform_yandex/proxmox$ cat ../keys/userdata-proxmox.yml
#cloud-config
hostname: px-cloud-config-applied
users:
- name: tf
groups: adm,sudo
shell: /bin/bash
sudo: ['ALL=(ALL) NOPASSWD:ALL']
ssh-authorized-keys:
- ssh-rsa AAAABsomelongstring== tf
packages:
- qemu-guest-agent
package_upgrade: true
runcmd:
- systemctl enable --now qemu-guest-agent
- echo "done" > /tmp/user-cloud-init-done
final_message: |
cloud-init has finished
version: $version
timestamp: $timestamp
datasource: $datasource
uptime: $uptime
Ratiborus was here... And here's 3 different implementations for injecting custom cloud-init cloud-config for userdata, and all of them work:
#==================#
# CLOUD-CONFIG #
#==================#
resource "proxmox_virtual_environment_file" "cloud_config_userdata" {
content_type = "snippets"
datastore_id = "local"
node_name = "prox-srv1"
source_file {
path = "../keys/userdata-proxmox.yml"
}
}
resource "proxmox_virtual_environment_file" "cloud_config_userdata_raw_file" {
content_type = "snippets"
datastore_id = "local"
node_name = "prox-srv1"
source_raw {
data = file("../keys/userdata-proxmox.yml")
file_name = "userdata-proxmox-raw-file.yml"
}
}
resource "proxmox_virtual_environment_file" "cloud_config_userdata_raw" {
content_type = "snippets"
datastore_id = "local"
node_name = "prox-srv1"
source_raw {
data = <<EOF
#cloud-config
hostname: px-cloud-config-applied
packages:
- qemu-guest-agent
users:
- name: tf
groups: sudo
shell: /bin/bash
ssh-authorized-keys:
- ssh-rsa some_long_key
sudo: ALL=(ALL) NOPASSWD:ALL
runcmd:
- systemctl enable --now qemu-guest-agent
- echo "done" > /tmp/user-cloud-init-done
EOF
file_name = "userdata-proxmox-raw.yml"
}
}And now about the issue with the first implementation i mentioned earlier. If you change your external file that is getting imported into Proxmox's snippets datastore these changes are not visible for provider and resource will stay the same. Most likely because provider checks only existence of a file by its name. But if we "import" raw source even as rendered external file then provider sees changes and acts accordingly. The same thing probably happens with cloud images being imported into ISO datastore. Maybe if provider checked and compared checksum of source and target files every plan and tried to recreate resource (re-download file) if changes are detected it would work better. We could also re-download any distribution's cloud images when they get updated. But i'm not sure if it's something feasible or easily implemented or even wanted by most people. What do you think, @bpg?
Here's an example:
module.proxmox.proxmox_virtual_environment_file.cloud_config_userdata_raw_file: Creating...
module.proxmox.proxmox_virtual_environment_file.cloud_config_userdata: Creating...
module.proxmox.proxmox_virtual_environment_file.cloud_config_userdata: Creation complete after 1s [id=local:snippets/userdata-proxmox.yml]
module.proxmox.proxmox_virtual_environment_file.cloud_config_userdata_raw_file: Creation complete after 1s [id=local:snippets/userdata-proxmox-raw-file.yml]
###
do some changes to source file "../keys/userdata-proxmox.yml", i added installation of vim and htop packages
###
ratiborus@HOMEWORLD:~/WORKSPACE/terraform_yandex/proxmox$ terraform apply
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
Terraform will perform the following actions:
# module.proxmox.proxmox_virtual_environment_file.cloud_config_userdata_raw_file must be replaced
-/+ resource "proxmox_virtual_environment_file" "cloud_config_userdata_raw_file" {
+ file_modification_date = (known after apply)
~ file_name = "userdata-proxmox-raw-file.yml" -> (known after apply)
+ file_size = (known after apply)
+ file_tag = (known after apply)
~ id = "local:snippets/userdata-proxmox-raw-file.yml" -> (known after apply)
# (5 unchanged attributes hidden)
~ source_raw {
~ data = <<-EOT # forces replacement
#cloud-config
hostname: px-cloud-config-applied
users:
- name: tf
groups: adm,sudo
shell: /bin/bash
sudo: ['ALL=(ALL) NOPASSWD:ALL']
ssh-authorized-keys:
- ssh-rsa AAAABsomelongstring== tf
packages:
- qemu-guest-agent
+ - vim
+ - htop
package_upgrade: true
runcmd:
- systemctl enable --now qemu-guest-agent
- echo "done" > /tmp/user-cloud-init-done
final_message: |
cloud-init has finished
version: $version
timestamp: $timestamp
datasource: $datasource
uptime: $uptime
Ratiborus was here...
EOT
# (2 unchanged attributes hidden)
}
}
Plan: 1 to add, 0 to change, 1 to destroy.And another longread. I'm sorry...
@bpg I'm not exactly sure what is the reason but it probably needs to be highlighted in documentation. When you create a VM template do not forget to set "started = false". Default is "true" which is sane but not for template considering it's not supposed to be ran at all. Else you'll be getting updates in-place on any plan generation even though VM itself wouldn't be started ever:
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# module.proxmox.proxmox_virtual_environment_vm.template_ubuntu_2204 will be updated in-place
~ resource "proxmox_virtual_environment_vm" "template_ubuntu_2204" {
id = "1011"
~ ipv4_addresses = [] -> (known after apply)
~ ipv6_addresses = [] -> (known after apply)
name = "template-ubuntu-22.04"
~ network_interface_names = [] -> (known after apply)
tags = [
"template-ubuntu-22.04",
"templates",
"terraform",
]
# (21 unchanged attributes hidden)
# (9 unchanged blocks hidden)
}
Plan: 0 to add, 1 to change, 0 to destroy.Also, even with "started = false" i'm still getting this "architecture" update for template VM after it was already created (once on the next plan generation):
...
module.proxmox.proxmox_virtual_environment_vm.template_ubuntu_2204: Creating...
module.proxmox.proxmox_virtual_environment_vm.template_ubuntu_2204: Creation complete after 7s [id=1011]
ratiborus@HOMEWORLD:~/WORKSPACE/terraform_yandex/proxmox$ terraform apply
...
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# module.proxmox.proxmox_virtual_environment_vm.template_ubuntu_2204 will be updated in-place
~ resource "proxmox_virtual_environment_vm" "template_ubuntu_2204" {
id = "1011"
name = "template-ubuntu-22.04"
tags = [
"template-ubuntu-22.04",
"templates",
"terraform",
]
# (24 unchanged attributes hidden)
~ cpu {
+ architecture = "x86_64"
# (7 unchanged attributes hidden)
}
# (8 unchanged blocks hidden)
}
Plan: 0 to add, 1 to change, 0 to destroy.It happens only once after template VM resource creation and also does not happen if i recreate resource (i.e. with "terraform taint"). No difference whether "template = true" or "false". I also tried it on other Linux distribution (Astra) and the result is the same. Here's test resources:
resource "proxmox_virtual_environment_file" "cloud_image_ubuntu_2204" {
content_type = "iso"
datastore_id = "local"
node_name = "prox-srv1"
source_file {
path = "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img"
}
}
resource "proxmox_virtual_environment_vm" "template_ubuntu_2204" {
name = "template-ubuntu-22.04"
description = "Ubuntu 22.04 template"
tags = sort(["terraform", "templates", "template-ubuntu-22.04"])
pool_id = proxmox_virtual_environment_pool.templates.id
node_name = "prox-srv1"
vm_id = 1011
migrate = true
on_boot = false
started = false
template = true
scsi_hardware = "virtio-scsi-single"
agent {
enabled = true
trim = true
}
cpu {
architecture = "x86_64"
cores = 1
numa = true
sockets = 2
type = "host"
}
memory {
dedicated = 2048
}
disk {
datastore_id = "local"
file_id = proxmox_virtual_environment_file.cloud_image_ubuntu_2204.id
file_format = "qcow2"
interface = "scsi0"
cache = "none"
size = 20
discard = "on"
iothread = true
ssd = true
}
initialization {
datastore_id = "local"
interface = "ide2"
ip_config {
ipv4 {
address = "dhcp"
}
}
user_data_file_id = proxmox_virtual_environment_file.cloud_config_userdata.id
}
network_device {
bridge = "vmbr0"
}
operating_system {
type = "l26"
}
serial_device {
device = "socket"
}
vga {
enabled = true
memory = 32
type = "serial0"
}
# lifecycle {
# ignore_changes = [
# initialization[0].user_data_file_id
# ]
# }
}Not sure but maybe all that info should be covered in a dedicated issue? Just feel like i'm a bit off-topic here and deluting main question :)
bpg
commented
1 year ago Thanks for the wealth of information, @ratiborusx. That's a really impressive investigation! I'll need to set aside some time to go over all the details here. I'll get back to you at some point over the weekend :)
ratiborusx
commented
1 year ago Just a little bit of new information that was revealed at today's past-midnight meeting and demo. If i inject my custom cloud-config like this there's no way to set any cloud-init configuration through Proxmox's GUI - it never gets applied even if i regenerate image:
initialization {
datastore_id = "local"
interface = "ide2"
ip_config {
ipv4 {
address = "dhcp"
}
}
user_data_file_id = proxmox_virtual_environment_file.cloud_config_userdata["prox-srv1"].id
}An example:
ratiborus@HOMEWORLD:~$ ssh -i ~/.ssh/tf.key tf@10.177.144.35
tf@px-cloud-config-applied:~$ getent passwd | grep bash
root:x:0:0:root:/root:/bin/bash
tf:x:1000:1000::/home/tf:/bin/bashratiborus@HOMEWORLD:~$ ssh -i ~/.ssh/tf.key abc123@10.177.144.35
abc123@10.177.144.35: Permission denied (publickey).I saw those in resource description but thought they just overwrite each other.
So if after applying my custom cloud-config i reapply setting from GUI (with image regeneration) they will overwrite old ones from custom config because both are supposed to put that data in that small 4Mb cloud-init image that resides alongside with disk's one?
UPD.
Now, after reading some docs of PVE again i see that they call stuff available through GUI as "automatically generated config" and that custom and automatic stuff can be mixed but only on "silo" level - i.e. user, network, vendor and meta data should be unique (either custom or automatic).
After all that i'm still not sure whether i could use GUI's Cloud-Init settings to overwrite ones injected through custom cloud-config. It's not that i'd need that but others who access Proxmox via GUI would need it sometimes to drop in their own creds.
But ofc it has nothing to do with provider :)
@tobiasehlert Your 1st issue is that you expect info from your custom cloud-config to be shown in GUI - it won't be visible there but it should be applied by cloud-init, your should check it. Your 2nd issue with an error about missing file may be because you removed that file manually via GUI but resource still present in terraform's state file. The easiest way to fix would be to remove said resource from state file via "terraform state rm 'resource_name'" and recreate it. If I'm not wrong this issue (it also affected disk images) should be fixed in 0.36.0, i mentioned this in #643.
tobiasehlert
commented
1 year ago @tobiasehlert could you pls check if
snippetscontent type is enabled on thelocalstorage? Could be another case of #370 (comment)
Yes, that fixed my problem with snippet, thanks @bpg! New to Proxmox, so didn't know about that parameter :)
Fabiosilvero
commented
1 year ago Hello !
I also have this issue (I didn't wanted to open a duplicate, feel free to ask me to open my own issue).
Host :
Template :
Terraform code (redacted) :
terraform {
required_providers {
proxmox = {
source = "bpg/proxmox"
version = "0.37.0"
}
}
}
provider "proxmox" {
endpoint = "https://my.awesome.server:8006/"
api_token = "user@pve!token1=my-awesome-token"
insecure = true
tmp_dir = "/var/tmp"
}
resource "proxmox_virtual_environment_vm" "ubuntu_vm" {
name = "terraform-provider-proxmox-ubuntu-vm"
description = "Managed by Terraform"
tags = ["terraform", "ubuntu"]
node_name = "node1"
vm_id = 4321
agent {
# read 'Qemu guest agent' section, change to true only when ready
enabled = true
}
clone {
vm_id = "101"
}
bios = "ovmf"
reboot = true
boot_order = ["ide2","virtio0","net0"]
disk {
datastore_id = "datastore1"
file_format = "raw"
interface = "virtio0"
size = 25
}
disk {
datastore_id = "datastore1"
file_format = "raw"
interface = "virtio1"
size = 4
}
network_device {
bridge = "vmbr0"
vlan_id = "1"
}
operating_system {
type = "l26"
}
scsi_hardware = "virtio-scsi-single"
initialization {
datastore_id = "datastore1"
interface = "ide2"
ip_config {
ipv4 {
address = "1.2.3.4/24"
gateway = "1.2.3.1"
}
}
}
}
The VM is created from template with correct boot order, but after disk resize, the boot order is changed to virtio0;net0 although I specified it in my Terraform code.
I tried to remove the Cloud Init Drive to let Terraform create it (Packer create this drive at ide0 at the last step) and set it to ide2, but I have the same issue.
Since the Cloud init drive doesn't appear in boot order, the VM doesn't see it, so Cloud Init never configure the IP address. And the Terraform execution hangs up undefinitively (well, I killed it at 2min and at 10min before).
When I create a VM with qm clone 101 1015 --full --name toto && qm set 1015 --ipconfig0 ip=1.2.3.4/24,gw=1.2..3.1 && qm start 1015, that works fine and respect the previous boot order set up in Packer boot = "order=virtio0;ide2;net0;ide0" (but I don't resize any disk).
ratiborusx
commented
1 year ago @Fabiosilvero Hey! I don't remember ever booting from Cloud-Init drive. It is my understanding that it only keeps cloud-config on it. You probably don't need it in the boot order at all, but i'm not sure if UEFI has any influence on the whole thing (never used "OVMF").
I don't think Terraform's provider will be able to save whatever you backed in your image about boot order, it will always rewrite it. Even if you don't specify it in the resource i think it'll default to "boot_order = []" which most likely means to boot up from the first drive it detects. I'd recommend you to remove "ide2" from load order or try to comment out "boot order" line as a whole (so provider will switch to defaults) and check if it works.
Just to be sure i'd recommend you to try to create a template with an official cloud image of Debian 12 and see if the issue is image-related. I used this one and it works without any problems with cloud-config: https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-generic-amd64.qcow2
UPD.
Also, this:
Fabiosilvero
commented
1 year ago Hello and thanks for your reply !
I can't get the cloud image to boot with OVMF even with the efi_disk fully configured on terraform side (I used the tar.xz image as qcow2 wasn't valid for the provider - maybe setting the type to snipplet would have been sufficient tho). I'll try again later but I don't think the template is the issue as it works perfectly with Telmate provider.
I get what you mean by asking to remove the boot_order directive, that was a workaround since with OVMF, it seems that if disks aren't present in boot_order, they aren't properly recognized (see https://forum.proxmox.com/threads/cloud-init-drive-not-auto-regenerating.117318/ and https://forum.proxmox.com/threads/ide-cloud-init-volumes-are-not-usable-when-cold-booting-a-vm.117345/).
I tried first without setting it, and the result was the same. My configuration quoted was misleading, that was a fluke but I tried with virtio0 first (and ide2 at the end) and I tried without ide2 : still same thing.
I don't think that this provider enforce the order in the Terraform config at the very end of the execution, but instead before any disks resize - and the boot order is altered at that moment again, causing config revert to default (that's just a guess).
Maybe it's a "Regenerate Image" trigger issue in this provider : if I stop the VM then click on "Regenerate Image" on the Web GUI, the Cloud Init drive appears in Boot Order and the Cloud Init service kicks in at startup.
I'll try to generate a Seabios template and see if the issue persists.
EDIT : It works fine with BIOS Template, so I think it's a problem with UEFI/OVMF-based templates :/
otopetrik
commented
1 year ago Combination cloud-init + ovmf + cloud-init drive @ ide does not work,
combination cloud-init + ovmf + cloud-init drive @ sata also does not work,
but combination cloud-init + ovmf + cloud-init drive @ scsi works (debian 12 genericcloud image).
Not sure why.
Cloud-init docs do not mention anything about different requirements when booted using UEFI.
Maybe OVMF does not support CD drive on ide or sata interfaces ?
Added 2 lines to a debian example:
...
bios = "ovmf" # added
...
initialization {
interface = "scsi0" # added
..
}and created the VM. Cloud-init correctly applied the configuration.
It is likely unsafe to modify the provider to change initialization.interface default to depend on bios value.
Any scsiN value could potentially colide with actual VM disks.
But it might be a good idea to detect when the user is using bios = "ovmf" with non-scsi interface, and display a warning/suggestion to use one ofscsi0, ..., scsi30 values. (and also add it to the documentation)
some-random-username-123
commented
1 year ago Hey again,
After a long time of testing around with different configs, I still got no results.
But now I tried the Config from @otopetrik (thank you btw) and it works.
When i use scsi1 here, all the CI config is getting applied to the VM on first boot.
So thanks to all of you that helped.
Fabiosilvero
commented
1 year ago Well, thank you ! I just changed the interface from ide0/2 to scsi1, and it worked flawlessly in OVMF setting.
Maybe a VM with OVMF requires that IDE drives should be present at boot order to be available in VMs at all, and SCSI drives don't need that (I don't see the Cloud Init drive in Boot order).
windowsrefund
commented
1 year ago I've learned some things about Proxmox+cloud-init which may add some value to some areas of this thread.
qm cloudinit dump to view one or more types, only that small subset which is "used" by the Proxmox GUI is shown. In other words, the dump approach is useless when working with a custom cloud-init.--cicustom.In general, the clobbering nature of proxmox_virtual_environment_vm.initialization.ip_config and proxmox_virtual_environment_vm.initialization.user_account over their *_data_file_id counterparts seems really finicky and just ripe for pointless confusion. I'm personally going to avoid using them and just stick with the data files since that's where all the cloud-init power lies anyway. I understand the value in the perception end users are quickly bootstrapping all the things but wonder if that's enough reason to continue supporting these conflicting elements. Food for thought.
ratiborusx
commented
1 year ago @windowsrefund Yep, that's basically what i wrote somewhere at the beginning of the thread. Not as concise though, i must admit :)
windowsrefund
commented
1 year ago @ratiborusx Yea, it was definitely your earlier comments that perked my interest and got me heading down the rabbit hole. I'm glad I did as I have a better understanding of things. I really had to spend a few hours doing things by hand (via qm) as well as mounting the read-only cloudinit itself in order to cat the files inside. Thanks for helping to shed the light.
bpg
commented
1 year ago Awesome work testing all of these! ❤️ If anyone is up for the challenge to summarize cloud init testing in docs, I'll greatly appreciate a PR! If not, I'll try to find some time in the next few weeks to test and document these use cases, with examples.
bpg
commented
1 year ago @all-contributors please add @Fabiosilvero and @windowsrefund for testing
allcontributors[bot]
commented
1 year ago @bpg
I've put up a pull request to add @Fabiosilvero! :tada:
bpg
commented
1 year ago @all-contributors please add @windowsrefund for testing
allcontributors[bot]
commented
1 year ago @bpg
I've put up a pull request to add @windowsrefund! :tada:
M0NsTeRRR
commented
1 year ago Hello guys,
Cloud init with IDE2 (default) and EFI is working on my side, you can find the attached apply i've posted in an other issue https://github.com/bpg/terraform-provider-proxmox/issues/734#issuecomment-1826280294 You can find my terraform config here (I use terragrunt to boostrap everything)
Tested with Proxmox Virtual Environment 8.0.9, Terraform v1.6.4 and Ubuntu 22.04.3.
I'm creating a template with packer, with some ansible scripts (in the same repo), at the end I reset cloud init instance with cloud-init clean --seed --machine-id and I deploy this template with Terraform.
Note : I use cloud init file
initialization {
interface = "ide2"
network_data_file_id = proxmox_virtual_environment_file.cloud_network_config.id
user_data_file_id = proxmox_virtual_environment_file.cloud_user_config.id
meta_data_file_id = proxmox_virtual_environment_file.cloud_meta_config.id
}
cursedforever
commented
10 months ago Hello guys, I don't really understand why the cloud-init configuration files are split in this way? Can't we just put one cloud-init file and then --cicustom in the backend? The way it's handled here seems cumbersome to me
bpg
commented
10 months ago Hi @cursedforever! That's what the Proxmox API provides us:
10.8.3. Custom Cloud-Init Configuration
The Cloud-Init integration also allows custom config files to be used instead of the automatically generated configs. This is done via the cicustom option on the command line:
qm set 9000 --cicustom "user=<volume>,network=<volume>,meta=<volume>"The custom config files have to be on a storage that supports snippets and have to be available on all nodes the VM is going to be migrated to. Otherwise the VM won’t be able to start. For example:
qm set 9000 --cicustom "user=local:snippets/userconfig.yaml"There are three kinds of configs for Cloud-Init. The first one is the
userconfig as seen in the example above. The second is thenetworkconfig and the third themetaconfig. They can all be specified together or mixed and matched however needed. The automatically generated config will be used for any that don’t have a custom config file specified.
Well, they missed vendor config in this section, but mentioned it a bit further in the doc.
I'm not a cloud-init expert by any means, so can't comment on "why" it's done this way. Tough I did try passing user config in vendor attribute, and it seemed to be working, by YMMV, and I suspect it may depend on the distro you're using.
cursedforever
commented
10 months ago Ok thank you very much, I've tried to make a global cloud-init file and pass it to the user_config_file variable and it works.
michaelfranzl
commented
8 months ago Using v0.49.0 with Proxmox 8.1.4, I am experiencing the same issue as described in the very first comment. Both bios settings ovmf and seabios are affected.
Adding initialization.interface = "scsi1" as suggested by @otopetrik above fixes the issue. Unfortunately, doing so also exposes issue https://github.com/bpg/terraform-provider-proxmox/issues/1083 where the resource is rebooted every apply even when nothing changed.
michaelfranzl
commented
8 months ago Just for reference, more Proxmox users seem to have related issues with the cloud-init IDE interface, where switching to SCSI fixes the issue:
https://forum.proxmox.com/threads/cloud-init-image-only-applies-configuration-on-second-boot.93414/post-452830 https://forum.proxmox.com/threads/scsi0-cloud-init-drive-is-already-attached-at-ide0.131159/post-590571
Maybe the configuration default should be changed from IDE to SCSI?
ratiborusx
commented
8 months ago I personally never had problems with IDE interface for cloud-init but i must admit i never used UEFI ('bios = ovmf') either because i didn't need PCI devices pass-through so far. As a workaround for #1083 you may try to use something like this (i think it's unlikely someone would want to change these settings on already deployed instance):
lifecycle {
ignore_changes = [
initialization[0].datastore_id,
initialization[0].interface
]
}But i'm absolutely not against someone smart fixing that thing for good. I'm just mentally handicapped in that relation.
bpg-autobot[bot]
commented
2 months ago Marking this issue as stale due to inactivity in the past 180 days. This helps us focus on the active issues. If this issue is reproducible with the latest version of the provider, please comment. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!
ElForastero
commented
2 months ago I'm using 0.64.0 and the issue is still there. Just writing this to prevent it from being closed as stale.
Default cloud-image configuration from snippets isn't applied.
bpg
commented
2 months ago Hey @ElForastero 👋🏼
As you can see from the 40+ comments, this issue is quite complex, and many factors could be affecting your specific setup.
The bottom line is that, in most cases discussed above, the problem wasn't with the provider but with VM configuration, cloud-init settings, the base OS image, etc. I'm keeping this issue open mainly to summarize those aspects in the documentation.
I suggest opening a new issue and explaining your use case so we can investigate and offer guidance.
Thanks!
Describe the bug My issue is: When applying a terraform manifest that clones a vm and adds cloud init configuration, the cloudinit image doesnt get regenerated. I the UI this is done via the Button Regererate image:
There is also a API call that can do this: API Refference
To Reproduce Steps to reproduce the behavior:
terraform applyMy TF Module:
Screenshots If applicable, add screenshots to help explain your problem.
Additional context Add any other context about the problem here.