New resource proxmox_virtual_environment_server_settings

[TheNotary]

Is your feature request related to a problem? Please describe.

In order to use PCI passthrough, some configurations need to be made to the Proxmox hypervisor followed by a system reboot. I think it would add a lot of value if we had a resource that represented that configuration item so 100% of the pci passthrough configuration can be done through Terraform alone.

Describe the solution you'd like

(main.tf)

.
.
.
resource "proxmox_virtual_environment_server_settings" "cmdline" {
    grub_cmdline_linux_default = "quiet iommu=pt intel_iommu=on"

    modules = "vfio vfio_iommu_type1 vfio_pci vfio_virqfd"
    force_reboot = false
}
.
.
.

Applying that resource should update the needed files, running the associated commands, and result in the user being prompted that they need to make preparations to reboot the server to complete the configuration. A force_reboot option might be helpful to someone, but I actually wouldn't use that on either my "production" or testing machine.

Describe alternatives you've considered

This is just one api. I wondered about two resources, one for grub, and another for modules. I don't have strong opinions around the api design just that the configurations can be defined and applied through terraform.

Additional context

Here's the complete list of minimal configurations required to allow a complete PCI passthrough. (Note: The official documentation indicated block-listing the device from proxmox was needed, but I'm not seeing that requirement currently on my end so I've omitted the configuration suggestion to /etc/modprobe.d/blacklist.conf.)

(/etc/default/grub)

.
.
.
# GRUB_CMDLINE_LINUX_DEFAULT="quiet"  # default settings look like this
GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt intel_iommu=on"  # for intel passthrough set like this
.
.
.

Follow up grub command:

update-grub

Edit

For zfs systems, the command line entails the below alternate configurations:

(/etc/kernel/cmdline)

root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt

Subsequently this command should be run for zfs:

/etc/kernel/postinst.d/zz-proxmox-boot

(/etc/modules)

.
.
.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Follow up modules command:

update-initramfs -u -k all

[otopetrik]

Interesting idea, it could be useful for setting "vfio-pci.ids=..." on kernel cmdline. Unfortunately there are two issues:

1) Booting Proxmox is a bit more complex, sometimes Grub is not used.

For EFI booting system, Proxmox uses systemd-boot. When ZFS mirror is used for booting, there are multiple EFI Service Partitions, one ESP on each disk. Proxmox uses separate tool proxmox-boot-tool to ensure that both are in sync.

On systems booting using systemd-boot, kernel options are in /etc/kernel/cmdline, file /etc/default/grub is then ignored.

2) Changes are applied on reboot, therefore VMs defined in the same terraform configuration would fail.

The best case scenario, they would just fail to start; worse would be partially configured. If force_reboot is set, and reboot is performed while VMs are still being created there could be various "intersting" failures modes. E.g. Proxmox creates VM (with dynamic vm id allocation), and reboots before terraform provider completes configuring the VM, resource creation fails and no VM ID is saved to terraform state file. VM then exists in partially configured state, but not managed by terraform.

It looks like a case for layered configuration (e.g. terragrunt). First layer to provision the host(s), next layer to provision VMs. In such case, it makes sense to use separate providers.

It might be easier to do the same thing using a provider, to just copy and run a shell script (full of sed -i calls) on the host. This provider looks like it might work (have not tried that provider myself).

Chances are, the user might want to perform additional configuration on the host anyway (e.g. configure UPS monitoring or setup GlusterFS for shared storage for ISO files).

Instead of additional resources, the documentation could then contain a "GPU passthrough / VFIO" documentation page with recommended script template for users to use.

It might be a good idea to modify the organization of documentation pages, closer to to e.g. azure provider. Instead of a single group "Virtual Environment" with "Resources" and "Data Sources" subgroups, it might be a good idea to add "Guides" (and possibly "Examples") sections at the same level as "Virtual Environment" (which could be renamed to "Reference").

[TheNotary]

These are good points. Thanks for that tip around the ssh provider.

Point 2 feels like much more of a show stopper so I'll focus my thoughts there (regarding point 1, I'm presuming detection around zfs and automation is possible).

I'd have to test this, but I have a hunch that VMs with PCI passthrough would succeed, though the PCI device would not actually show up for the VM. That might make the situation not so bad and resolved following a reboot.

If the VMs do need the server configuration change to be applied before they pass, I wonder if we could take a snapshot of the servers boot up time, always force reboots, and then force all resources to depend on proxmox_virtual_environment_server_settings. Maybe we could set the provider to initiate the reboot, and wait for the system to come back online with a new bootup timestamp?

IDK, it would be messy, you're right, heh. I'll need to think about that setup for a while to decide if I would even use the feature or not. It might be making more sense for me to define an Ansible provisioner since I'm already finding myself cloning down projects every time I setup a new build.

I like the idea of having a guide section. In an ideal world any user new to Proxmox could just do a default install, and then paste in and tweak some examples and just be online and operational without needing to go through any tutorials. But I appreciate the stickiness of this issue and a concise, well written guide can be a solid stand-in.

[bpg-autobot[bot]]

Marking this issue as stale due to inactivity in the past 180 days. This helps us focus on the active issues. If this issue is reproducible with the latest version of the provider, please comment. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!