feat(provider): reliable sequential and random `vm_id` generation

Rework the ID generation logic to be more resilient to generating duplicates when creating resources with high parallelism.

Now the providers uses a file-based mutex in conjunction with ID tracking (also in a file) to ensure sequential ID allocation. In addition to that, an opt-in random id generation mode was added to further improve robustness in concurrent scenarios. See updated documentation for more details.

Contributor's Note

[x] I have added / updated documentation in /docs for any user-facing features or additions.
[x] I have added / updated acceptance tests in /fwprovider/tests for any new or updated resources / data sources.
[x] I have ran make example to verify that the change works as expected.

Proof of Work

resource "proxmox_virtual_environment_vm" "test_vm" {
  count     = 20
  node_name = "pve"
  started   = false
}

apply with default provider settings:

❯ tofu apply -auto-approve

OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

OpenTofu will perform the following actions:

<SKIPPED>

Plan: 20 to add, 0 to change, 0 to destroy.
proxmox_virtual_environment_vm.test_vm[13]: Creating...
proxmox_virtual_environment_vm.test_vm[16]: Creating...
proxmox_virtual_environment_vm.test_vm[10]: Creating...
proxmox_virtual_environment_vm.test_vm[0]: Creating...
proxmox_virtual_environment_vm.test_vm[4]: Creating...
proxmox_virtual_environment_vm.test_vm[8]: Creating...
proxmox_virtual_environment_vm.test_vm[5]: Creating...
proxmox_virtual_environment_vm.test_vm[17]: Creating...
proxmox_virtual_environment_vm.test_vm[3]: Creating...
proxmox_virtual_environment_vm.test_vm[12]: Creating...
proxmox_virtual_environment_vm.test_vm[13]: Creation complete after 0s [id=100]
proxmox_virtual_environment_vm.test_vm[14]: Creating...
proxmox_virtual_environment_vm.test_vm[16]: Creation complete after 0s [id=101]
proxmox_virtual_environment_vm.test_vm[10]: Creation complete after 0s [id=102]
proxmox_virtual_environment_vm.test_vm[6]: Creating...
proxmox_virtual_environment_vm.test_vm[18]: Creating...
proxmox_virtual_environment_vm.test_vm[0]: Creation complete after 0s [id=103]
proxmox_virtual_environment_vm.test_vm[9]: Creating...
proxmox_virtual_environment_vm.test_vm[8]: Creation complete after 0s [id=104]
proxmox_virtual_environment_vm.test_vm[5]: Creation complete after 0s [id=105]
proxmox_virtual_environment_vm.test_vm[11]: Creating...
proxmox_virtual_environment_vm.test_vm[1]: Creating...
proxmox_virtual_environment_vm.test_vm[4]: Creation complete after 0s [id=106]
proxmox_virtual_environment_vm.test_vm[17]: Creation complete after 0s [id=107]
proxmox_virtual_environment_vm.test_vm[2]: Creating...
proxmox_virtual_environment_vm.test_vm[7]: Creating...
proxmox_virtual_environment_vm.test_vm[6]: Creation complete after 0s [id=111]
proxmox_virtual_environment_vm.test_vm[3]: Creation complete after 0s [id=108]
proxmox_virtual_environment_vm.test_vm[19]: Creating...
proxmox_virtual_environment_vm.test_vm[15]: Creating...
proxmox_virtual_environment_vm.test_vm[14]: Creation complete after 0s [id=110]
proxmox_virtual_environment_vm.test_vm[18]: Creation complete after 0s [id=112]
proxmox_virtual_environment_vm.test_vm[9]: Creation complete after 0s [id=113]
proxmox_virtual_environment_vm.test_vm[11]: Creation complete after 0s [id=114]
proxmox_virtual_environment_vm.test_vm[1]: Creation complete after 0s [id=115]
proxmox_virtual_environment_vm.test_vm[2]: Creation complete after 0s [id=116]
proxmox_virtual_environment_vm.test_vm[7]: Creation complete after 0s [id=117]
proxmox_virtual_environment_vm.test_vm[19]: Creation complete after 0s [id=118]
proxmox_virtual_environment_vm.test_vm[12]: Creation complete after 1s [id=109]
proxmox_virtual_environment_vm.test_vm[15]: Creation complete after 1s [id=119]

Apply complete! Resources: 20 added, 0 changed, 0 destroyed.

With adding random_vm_ids = true to the provider's config:

❯ tofu apply -auto-approve

OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

OpenTofu will perform the following actions:

<SKIPPED>

Plan: 20 to add, 0 to change, 0 to destroy.
proxmox_virtual_environment_vm.test_vm[19]: Creating...
proxmox_virtual_environment_vm.test_vm[18]: Creating...
proxmox_virtual_environment_vm.test_vm[5]: Creating...
proxmox_virtual_environment_vm.test_vm[0]: Creating...
proxmox_virtual_environment_vm.test_vm[13]: Creating...
proxmox_virtual_environment_vm.test_vm[11]: Creating...
proxmox_virtual_environment_vm.test_vm[14]: Creating...
proxmox_virtual_environment_vm.test_vm[1]: Creating...
proxmox_virtual_environment_vm.test_vm[6]: Creating...
proxmox_virtual_environment_vm.test_vm[9]: Creating...
proxmox_virtual_environment_vm.test_vm[19]: Creation complete after 0s [id=29686]
proxmox_virtual_environment_vm.test_vm[0]: Creation complete after 0s [id=51434]
proxmox_virtual_environment_vm.test_vm[12]: Creating...
proxmox_virtual_environment_vm.test_vm[16]: Creating...
proxmox_virtual_environment_vm.test_vm[18]: Creation complete after 0s [id=53084]
proxmox_virtual_environment_vm.test_vm[17]: Creating...
proxmox_virtual_environment_vm.test_vm[11]: Creation complete after 0s [id=43713]
proxmox_virtual_environment_vm.test_vm[6]: Creation complete after 0s [id=67428]
proxmox_virtual_environment_vm.test_vm[8]: Creating...
proxmox_virtual_environment_vm.test_vm[7]: Creating...
proxmox_virtual_environment_vm.test_vm[12]: Creation complete after 0s [id=61633]
proxmox_virtual_environment_vm.test_vm[14]: Creation complete after 0s [id=41484]
proxmox_virtual_environment_vm.test_vm[5]: Creation complete after 0s [id=42957]
proxmox_virtual_environment_vm.test_vm[16]: Creation complete after 0s [id=57649]
proxmox_virtual_environment_vm.test_vm[15]: Creating...
proxmox_virtual_environment_vm.test_vm[4]: Creating...
proxmox_virtual_environment_vm.test_vm[2]: Creating...
proxmox_virtual_environment_vm.test_vm[8]: Creation complete after 0s [id=54035]
proxmox_virtual_environment_vm.test_vm[3]: Creating...
proxmox_virtual_environment_vm.test_vm[7]: Creation complete after 0s [id=33116]
proxmox_virtual_environment_vm.test_vm[10]: Creating...
proxmox_virtual_environment_vm.test_vm[15]: Creation complete after 0s [id=85399]
proxmox_virtual_environment_vm.test_vm[2]: Creation complete after 0s [id=43957]
proxmox_virtual_environment_vm.test_vm[4]: Creation complete after 0s [id=37329]
proxmox_virtual_environment_vm.test_vm[3]: Creation complete after 0s [id=41337]
proxmox_virtual_environment_vm.test_vm[10]: Creation complete after 0s [id=83950]
proxmox_virtual_environment_vm.test_vm[13]: Creation complete after 1s [id=44217]
proxmox_virtual_environment_vm.test_vm[1]: Creation complete after 1s [id=82976]
proxmox_virtual_environment_vm.test_vm[9]: Creation complete after 1s [id=13515]
proxmox_virtual_environment_vm.test_vm[17]: Creation complete after 1s [id=83993]

Apply complete! Resources: 20 added, 0 changed, 0 destroyed.

Community Note

Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Relates #1521

bpg / terraform-provider-proxmox