search

ansible-collections / cloud.terraform

The collection automates the management and provisioning of infrastructure as code (IaC) using the Terraform CLI tool within Ansible playbooks and Execution Environment runtimes.
GNU General Public License v3.0
100 stars 35 forks source link

Issues creating inventory from state file with terraform_state plugin #142

Open seanw7 opened 5 months ago

seanw7 commented 5 months ago
SUMMARY

I'm running into issues when attempting to retrieve a GitLab remote state file with the cloud.terraform.terraform_state inventory plugin.

I'm able to retrieve it with terraform show cmd without issue, however, I receive an error when attempting to use the terraform_state inventory plugin stderr: Failed to marshal state to json: unsupported attribute "disks", please see example terraform show output below

Any help would be appreciated, thank you!

ISSUE TYPE
COMPONENT NAME

cloud.terraform.terraform_state

ANSIBLE VERSION
ansible [core 2.16.7]
  config file = None
  configured module search path = ['/home/username/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/username/Documents/lab_automation/venv/lib/python3.11/site-packages/ansible
  ansible collection location = /home/username/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/username/Documents/lab_automation/venv/bin/ansible
  python version = 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] (/home/username/Documents/lab_automation/venv/bin/python3)
  jinja version = 3.1.4
  libyaml = True
COLLECTION VERSION
ansible-galaxy collection list cloud.terraform

# /home/username/.ansible/collections/ansible_collections
Collection      Version
--------------- -------
cloud.terraform 3.0.0
CONFIGURATION
CONFIG_FILE() = None
OS / ENVIRONMENT

Raspbian running Terraform & Ansible against Proxmox v8

STEPS TO REPRODUCE
# Example terraform_state.yaml
plugin: cloud.terraform.terraform_state
backend_type: http
backend_config:
  address: https://gitlab.com/api/v4/projects/foo/terraform/state/default
  username: REDACTED
  password: REDACTED
# Exerpt from proxmox main.tf where disks block exists
...
  disks {
    scsi {
      scsi0 {
        disk {
          size    = "40"
          storage = "local-lvm"
        }
      }
    }
  }
...
# Example terraform show output 
terraform show -json
{"format_version":"1.0","terraform_version":"1.8.4","values":{"outputs":{"vm_ips":{"sensitive":false,"value":{"k3s-01-agent-01":"192.168.1.144","k3s-01-server-01":"192.168.1.143"},"type":["object",{"k3s-01-agent-01":"string","k3s-01-server-01":"string"}]}},"root_module":{"resources":[{"address":"data.terraform_remote_state.dev-tfstate","mode":"data","type":"terraform_remote_state","name":"dev-tfstate","provider_name":"terraform.io/builtin/terraform","schema_version":0,"values":{"backend":"http","config":{"address":"https://gitlab.com/api/v4/projects/foo/terraform/state/default","password":"REDACTED","username":"username"},"defaults":null,"outputs":{"vm_ips":{"k3s-01-agent-01":"192.168.1.144","k3s-01-server-01":"192.168.1.143"}},"workspace":null},"sensitive_values":{"config":{},"outputs":{"vm_ips":{}}}}],"child_modules":[{"resources":[{"address":"module.terraform-proxmox-k3s.proxmox_vm_qemu.cloudinit-vm[\"k3s-01-agent-01\"]","mode":"managed","type":"proxmox_vm_qemu","name":"cloudinit-vm","index":"k3s-01-agent-01","provider_name":"registry.terraform.io/telmate/proxmox","schema_version":0,"values":{"additional_wait":5,"agent":1,"args":"","automatic_reboot":true,"balloon":4096,"bios":"seabios","boot":"order=scsi0;net0;ide2","bootdisk":"","bridge":null,"ci_wait":null,"cicustom":"","cipassword":"password","ciuser":"username","clone":"vm101","clone_wait":10,"cloudinit_cdrom_storage":"local-lvm","cores":4,"cpu":"x86-64-v2-AES","default_ipv4_address":"192.168.1.144","define_connection_info":true,"desc":"k3s-01-agent-01","disk":[],"disk_gb":null,"disks":[{"ide":[],"sata":[],"scsi":[{"scsi0":[{"cdrom":[],"disk":[{"asyncio":"","backup":true,"cache":"","discard":false,"emulatessd":false,"format":"raw","id":0,"iops_r_burst":0,"iops_r_burst_length":0,"iops_r_concurrent":0,"iops_wr_burst":0,"iops_wr_burst_length":0,"iops_wr_concurrent":0,"iothread":false,"linked_disk_id":-1,"mbps_r_burst":0,"mbps_r_concurrent":0,"mbps_wr_burst":0,"mbps_wr_concurrent":0,"readonly":false,"replicate":false,"serial":"","size":40,"storage":"local-lvm","wwn":""}],"passthrough":[]}],"scsi1":[],"scsi10":[],"scsi11":[],"scsi12":[],"scsi13":[],"scsi14":[],"scsi15":[],"scsi16":[],"scsi17":[],"scsi18":[],"scsi19":[],"scsi2":[],"scsi20":[],"scsi21":[],"scsi22":[],"scsi23":[],"scsi24":[],"scsi25":[],"scsi26":[],"scsi27":[],"scsi28":[],"scsi29":[],"scsi3":[],"scsi30":[],"scsi4":[],"scsi5":[],"scsi6":[],"scsi7":[],"scsi8":[],"scsi9":[]}],"virtio":[]}],"efidisk":[],"force_create":false,"force_recreate_on_change_of":null,"full_clone":true,"guest_agent_ready_timeout":100,"hagroup":"","hastate":"","hostpci":[],"hotplug":"network,disk,usb","id":"proxmox-01/qemu/104","ipconfig0":"ip=192.168.1.144/24,gw=192.168.1.254","ipconfig1":"","ipconfig10":"","ipconfig11":"","ipconfig12":"","ipconfig13":"","ipconfig14":"","ipconfig15":"","ipconfig2":"","ipconfig3":"","ipconfig4":"","ipconfig5":"","ipconfig6":"","ipconfig7":"","ipconfig8":"","ipconfig9":"","iso":"","kvm":true,"linked_vmid":0,"mac":null,"machine":"","memory":4096,"name":"k3s-01-agent-01","nameserver":"192.168.1.253","network":[{"bridge":"vmbr0","firewall":false,"link_down":false,"macaddr":"FOO","model":"virtio","mtu":0,"queues":0,"rate":0,"tag":-1}],"nic":null,"numa":false,"onboot":false,"oncreate":false,"os_network_config":null,"os_type":"cloud-init","pool":"","preprovision":true,"pxe":null,"qemu_os":"l26","reboot_required":false,"scsihw":"virtio-scsi-single","searchdomain":"home.arpa","serial":[],"smbios":[{"family":"","manufacturer":"","product":"","serial":"","sku":"","uuid":"foo","version":""}],"sockets":2,"ssh_forward_ip":null,"ssh_host":"192.168.1.144","ssh_port":"22","ssh_private_key":null,"ssh_user":null,"sshkeys":"ssh-rsa ","startup":"","storage":null,"storage_type":null,"tablet":true,"tags":"k3s-01;k3s-agent","target_node":"proxmox-01","target_nodes":null,"timeouts":null,"unused_disk":[],"usb":[],"vcpus":0,"vga":[],"vlan":-1,"vm_state":"running","vmid":null},"sensitive_values":{"cipassword":true,"disk":[],"disks":[{"ide":[],"sata":[],"scsi":[{"scsi0":[{"cdrom":[],"disk":[{}],"passthrough":[]}],"scsi1":[],"scsi10":[],"scsi11":[],"scsi12":[],"scsi13":[],"scsi14":[],"scsi15":[],"scsi16":[],"scsi17":[],"scsi18":[],"scsi19":[],"scsi2":[],"scsi20":[],"scsi21":[],"scsi22":[],"scsi23":[],"scsi24":[],"scsi25":[],"scsi26":[],"scsi27":[],"scsi28":[],"scsi29":[],"scsi3":[],"scsi30":[],"scsi4":[],"scsi5":[],"scsi6":[],"scsi7":[],"scsi8":[],"scsi9":[]}],"virtio":[]}],"efidisk":[],"hostpci":[],"network":[{}],"serial":[],"smbios":[{}],"ssh_private_key":true,"unused_disk":[],"usb":[],"vga":[]},"tainted":true},
...
}
EXPECTED RESULTS

Expected to see an inventory of the two VMs that are in my remote state file

ACTUAL RESULTS
ansible-inventory [core 2.16.7]
  config file = None
  configured module search path = ['/home/username/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/username/Documents/venv/lib/python3.11/site-packages/ansible
  ansible collection location = /home/username/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/username/Documents/venv/bin/ansible-inventory
  python version = 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] (/home/username/Documents/venv/bin/python3)
  jinja version = 3.1.4
  libyaml = True
No config file found; using defaults
setting up inventory plugins
Loading collection ansible.builtin from 
host_list declined parsing /home/username/Documents/tf/applications/k3s-module/terraform_state.yaml as it did not pass its verify_file() method
script declined parsing /home/username/Documents/tf/applications/k3s-module/terraform_state.yaml as it did not pass its verify_file() method
Loading collection cloud.terraform from /home/username/.ansible/collections/ansible_collections/cloud/terraform
Using inventory plugin 'ansible_collections.cloud.terraform.plugins.inventory.terraform_state' to process inventory source '/home/username/Documents/tf/applications/k3s-module/terraform_state.yaml'
toml declined parsing /home/username/Documents/tf/applications/k3s-module/terraform_state.yaml as it did not pass its verify_file() method
[WARNING]:  * Failed to parse /home/username/Documents/tf/applications/k3s-module/terraform_state.yaml with auto plugin: Could not get
Terraform show from path: /tmp/tmphg4ifq1m.  stdout:  stderr: Failed to marshal state to json: unsupported attribute "disks"
  File "/home/username/Documents/venv/lib/python3.11/site-packages/ansible/inventory/manager.py", line 293, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/home/username/Documents/venv/lib/python3.11/site-packages/ansible/plugins/inventory/auto.py", line 59, in parse
    plugin.parse(inventory, loader, path, cache=cache)
  File "/home/username/.ansible/collections/ansible_collections/cloud/terraform/plugins/inventory/terraform_state.py", line 573, in parse
    instances = self._query(
                ^^^^^^^^^^^^
  File "/home/username/.ansible/collections/ansible_collections/cloud/terraform/plugins/inventory/terraform_state.py", line 507, in _query
    raise TerraformError(e.message)
[WARNING]:  * Failed to parse /home/username/Documents/tf/applications/k3s-module/terraform_state.yaml with yaml plugin: Plugin configuration
YAML file, not YAML inventory
  File "/home/username/Documents/venv/lib/python3.11/site-packages/ansible/inventory/manager.py", line 293, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/home/username/Documents/venv/lib/python3.11/site-packages/ansible/plugins/inventory/yaml.py", line 114, in parse
    raise AnsibleParserError('Plugin configuration YAML file, not YAML inventory')
[WARNING]:  * Failed to parse /home/username/Documents/tf/applications/k3s-module/terraform_state.yaml with ini plugin: Invalid host pattern
'plugin:' supplied, ending in ':' is not allowed, this character is reserved to provide a port.
  File "/home/username/Documents/venv/lib/python3.11/site-packages/ansible/inventory/manager.py", line 293, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/home/username/Documents/venv/lib/python3.11/site-packages/ansible/plugins/inventory/ini.py", line 138, in parse
    raise AnsibleParserError(e)
[WARNING]: Unable to parse /home/username/Documents/tf/applications/k3s-module/terraform_state.yaml as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
@all:
  |--@ungrouped:
gravesm commented 3 months ago

Would you mind retrying with main branch? The way the terraform_state plugin handles getting and reading state was changed a couple weeks ago. It's possible this may also fix your problem.