search

kubealex / libvirt-k8s-provisioner

Automate your k8s installation
MIT License
425 stars 76 forks source link

fatal: [localhost]: FAILED! => {"changed": false, "command": "/usr/local/bin/terraform apply #20

Closed organicnz closed 3 years ago

organicnz commented 3 years ago

Caught one more.

TASK [terraform] *******************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "command": "/usr/local/bin/terraform apply -no-color -input=false -auto-approve=true -lock=true /tmp/tmpekvbdoti.tfplan", "msg": "Failure when executing Terraform command. Exited 1.\nstdout: libvirt_domain.k8s-master[1]: Creating...\nlibvirt_domain.k8s-master[0]: Creating...\nlibvirt_domain.k8s-master[2]: Creating...\n\nstderr: \nError: Error defining libvirt domain: virError(Code=9, Domain=20, Message='operation failed: domain 'k8s-test-master-1' already exists with uuid d4a59107-7719-4beb-84ef-349ff7fabf47')\n\n  on k8s-master.tf line 68, in resource \"libvirt_domain\" \"k8s-master\":\n  68: resource \"libvirt_domain\" \"k8s-master\" {\n\n\n\nError: Error defining libvirt domain: virError(Code=9, Domain=20, Message='operation failed: domain 'k8s-test-master-0' already exists with uuid 70a8ecb8-5c0a-4ea0-b151-63e650054a7a')\n\n  on k8s-master.tf line 68, in resource \"libvirt_domain\" \"k8s-master\":\n  68: resource \"libvirt_domain\" \"k8s-master\" {\n\n\n\nError: Error defining libvirt domain: virError(Code=9, Domain=20, Message='operation failed: domain 'k8s-test-master-2' already exists with uuid fcbcf5dc-dc02-4714-aed1-e9b796a7c538')\n\n  on k8s-master.tf line 68, in resource \"libvirt_domain\" \"k8s-master\":\n  68: resource \"libvirt_domain\" \"k8s-master\" {\n\n\n"}
kubealex commented 3 years ago

could you please run:

ansible-playbook 99_cleanup.yml -v

And try again launching the main playbook?

organicnz commented 3 years ago

Destroying is failed.

UPD: Terraform v0.14.3

Welcome to Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-58-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

0 updates can be installed immediately.
0 of these updates are security updates.

Your Hardware Enablement Stack (HWE) is supported until April 2025.
ansible-playbook 99_cleanup.yml -v

Using /home/organic/dev/libvirt-k8s-provisioner/ansible.cfg as config file

PLAY [Cleanup playbook] ************************************************************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Destroy cluster VM] **********************************************************************************************************************************************************************************
failed: [localhost] (item=workers) => {"ansible_loop_var": "item", "changed": false, "item": "workers", "msg": "Failed to initialize Terraform modules:\r\n\u001b[31m\n\u001b[1m\u001b[31mError: \u001b[0m\u001b[0m\u001b[1mFailed to query available provider packages\u001b[0m\n\n\u001b[0mCould not retrieve the list of available versions for provider\ndmacvicar/libvirt: provider registry registry.terraform.io does not have a\nprovider named registry.terraform.io/dmacvicar/libvirt\n\nIf you have just upgraded directly from Terraform v0.12 to Terraform v0.14\nthen please upgrade to Terraform v0.13 first and follow the upgrade guide for\nthat release, which might help you address this problem.\n\u001b[0m\u001b[0m\n"}
failed: [localhost] (item=workers-rook) => {"ansible_loop_var": "item", "changed": false, "item": "workers-rook", "msg": "Failed to initialize Terraform modules:\r\n\u001b[31m\n\u001b[1m\u001b[31mError: \u001b[0m\u001b[0m\u001b[1mFailed to query available provider packages\u001b[0m\n\n\u001b[0mCould not retrieve the list of available versions for provider\ndmacvicar/libvirt: provider registry registry.terraform.io does not have a\nprovider named registry.terraform.io/dmacvicar/libvirt\n\nIf you have just upgraded directly from Terraform v0.12 to Terraform v0.14\nthen please upgrade to Terraform v0.13 first and follow the upgrade guide for\nthat release, which might help you address this problem.\n\u001b[0m\u001b[0m\n"}
failed: [localhost] (item=masters) => {"ansible_loop_var": "item", "changed": false, "item": "masters", "msg": "Failed to initialize Terraform modules:\r\n\u001b[31m\n\u001b[1m\u001b[31mError: \u001b[0m\u001b[0m\u001b[1mFailed to query available provider packages\u001b[0m\n\n\u001b[0mCould not retrieve the list of available versions for provider\ndmacvicar/libvirt: provider registry registry.terraform.io does not have a\nprovider named registry.terraform.io/dmacvicar/libvirt\n\nIf you have just upgraded directly from Terraform v0.12 to Terraform v0.14\nthen please upgrade to Terraform v0.13 first and follow the upgrade guide for\nthat release, which might help you address this problem.\n\u001b[0m\u001b[0m\n\u001b[31m\n\u001b[1m\u001b[31mError: \u001b[0m\u001b[0m\u001b[1mFailed to install provider\u001b[0m\n\n\u001b[0mError while installing hashicorp/template v2.2.0: open\n.terraform/providers/registry.terraform.io/hashicorp/template/2.2.0/linux_amd64/terraform-provider-template_v2.2.0_x4:\npermission denied\n\u001b[0m\u001b[0m\n"}
failed: [localhost] (item=loadbalancer) => {"ansible_loop_var": "item", "changed": false, "item": "loadbalancer", "msg": "Failed to initialize Terraform modules:\r\n\u001b[31m\n\u001b[1m\u001b[31mError: \u001b[0m\u001b[0m\u001b[1mFailed to query available provider packages\u001b[0m\n\n\u001b[0mCould not retrieve the list of available versions for provider\ndmacvicar/libvirt: provider registry registry.terraform.io does not have a\nprovider named registry.terraform.io/dmacvicar/libvirt\n\nIf you have just upgraded directly from Terraform v0.12 to Terraform v0.14\nthen please upgrade to Terraform v0.13 first and follow the upgrade guide for\nthat release, which might help you address this problem.\n\u001b[0m\u001b[0m\n"}

PLAY RECAP *************************************************************************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
kubealex commented 3 years ago

Supported terraform version at the moment is the one bundled with the playbooks. 0.13.x.

organicnz commented 3 years ago

Should the Terraform version be even lower? Drops the same error.

terraform --version
Terraform v0.13.5

Your version of Terraform is out of date! The latest version
is 0.14.3. You can update by downloading from https://www.terraform.io/downloads.html
kubealex commented 3 years ago

Bumped version to 0.14.3 on all branches. with #21 #22 #23 BTW didn't you try to apply the plan without running the playbook? I just ran and destroyed 3 clean installations and didn't get the error

organicnz commented 3 years ago

Pulled changes, updated to 0.14.3, but it doesn't help :( Nope, I ran straightforward ansible-playbook main.yml from quickstart.

UPD: trying to delete VMs manually.

kubealex commented 3 years ago

Pulled changes, updated to 0.14.3, but it doesn't help :( Nope, I ran straightforward ansible-playbook main.yml from quickstart.

UPD: trying to delete VMs manually.

Yep, let me know if it works!

organicnz commented 3 years ago

Undefining of shut off VMs doesn't solve it as well. So VMs appear in shut off mode after failure. I can reinstall Ubuntu 20.04 and try it out on a fresh OS.

virsh net-list --all
 Name                    State      Autostart   Persistent
------------------------------------------------------------
 default                 active     yes         yes
 k8s-test                active     yes         yes
 Ubuntu0                 inactive   no          yes
 vagrant-libvirt         inactive   no          yes
 vagrant-provisioning0   inactive   no          yes
sudo virsh list --all
 Id   Name                State
------------------------------------
 -    k8s-test-master-0   shut off
 -    k8s-test-master-1   shut off
 -    k8s-test-master-2   shut off
organicnz commented 3 years ago

Failed on another SSD with Ubuntu 20.04. Have you tested on CentOS yesterday? Maybe I should too :)

terraform -v
Terraform v0.14.3
TASK [terraform] *******************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to initialize Terraform modules:\r\n\u001b[31m\n\u001b[1m\u001b[31mError: \u001b[0m\u001b[0m\u001b[1mFailed to query available provider packages\u001b[0m\n\n\u001b[0mCould not retrieve the list of available versions for provider\ndmacvicar/libvirt: no available releases match the given constraints 0.6.2\n\u001b[0m\u001b[0m\n"}

PLAY RECAP *************************************************************************************************************************************************************************************************
localhost                  : ok=28   changed=22   unreachable=0    failed=1    skipped=13   rescued=0    ignored=0
kubealex commented 3 years ago

No I tested both Ubuntu and centos actually.. would you mind pulling again the master branch?

organicnz commented 3 years ago

But it's Already up to date. mate :) Do you mean to delete it and git clone the repo again?

UPD: swapped back to my previous SSD.

kubealex commented 3 years ago

But it's Already up to date. mate :) Do you mean to delete it and git clone the repo again?

UPD: swapped back to my previous SSD.

Thank you for pointing that out, I forgot to push and merge the branch with the fix to the version, I was sure I did 🗡️

organicnz commented 3 years ago

On Stackoverflow someone's pointing out that it can be related with parsing/passing of an Ansible string list itself. That's Terrable :)

Now with sudo it cleanup passes though almost successfully except item=masters
What's your tested Ubuntu version, maybe I could potentially try to avoid this error on another version like 18.04?

$ sudo ansible-playbook 99_cleanup.yml -v

Using /home/organic/dev/libvirt-k8s-provisioner/ansible.cfg as config file

PLAY [Cleanup playbook] ************************************************************************************************************************************************************************************

TASK [Gathering Facts] *************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Destroy cluster VM] **********************************************************************************************************************************************************************************
ok: [localhost] => (item=workers) => {"ansible_loop_var": "item", "changed": false, "command": "/usr/local/bin/terraform destroy -no-color -force -lock=true", "item": "workers", "outputs": {}, "state": "absent", "stderr": "", "stderr_lines": [], "stdout": "\nDestroy complete! Resources: 0 destroyed.\n", "stdout_lines": ["", "Destroy complete! Resources: 0 destroyed."], "workspace": "default"}
ok: [localhost] => (item=workers-rook) => {"ansible_loop_var": "item", "changed": false, "command": "/usr/local/bin/terraform destroy -no-color -force -lock=true", "item": "workers-rook", "outputs": {}, "state": "absent", "stderr": "", "stderr_lines": [], "stdout": "\nDestroy complete! Resources: 0 destroyed.\n", "stdout_lines": ["", "Destroy complete! Resources: 0 destroyed."], "workspace": "default"}
failed: [localhost] (item=masters) => {"ansible_loop_var": "item", "changed": false, "item": "masters", "msg": "Failed to initialize Terraform modules:\r\n\u001b[31m\n\u001b[1m\u001b[31mError: \u001b[0m\u001b[0m\u001b[1mFailed to query available provider packages\u001b[0m\n\n\u001b[0mCould not retrieve the list of available versions for provider\ndmacvicar/libvirt: locked provider registry.terraform.io/dmacvicar/libvirt\n0.6.2 does not match configured version constraint 0.6.3; must use terraform\ninit -upgrade to allow selection of new versions\n\u001b[0m\u001b[0m\n"}
ok: [localhost] => (item=loadbalancer) => {"ansible_loop_var": "item", "changed": false, "command": "/usr/local/bin/terraform destroy -no-color -force -lock=true", "item": "loadbalancer", "outputs": {}, "state": "absent", "stderr": "", "stderr_lines": [], "stdout": "\nDestroy complete! Resources: 0 destroyed.\n", "stdout_lines": ["", "Destroy complete! Resources: 0 destroyed."], "workspace": "default"}

PLAY RECAP *************************************************************************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

$ sudo virsh list --all

 Id   Name   State
--------------------

$ sudo  ansible-playbook main.yml

...

TASK [terraform] *******************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to initialize Terraform modules:\r\n\u001b[31m\n\u001b[1m\u001b[31mError: \u001b[0m\u001b[0m\u001b[1mFailed to query available provider packages\u001b[0m\n\n\u001b[0mCould not retrieve the list of available versions for provider\ndmacvicar/libvirt: locked provider registry.terraform.io/dmacvicar/libvirt\n0.6.2 does not match configured version constraint 0.6.3; must use terraform\ninit -upgrade to allow selection of new versions\n\u001b[0m\u001b[0m\n"}

PLAY RECAP *************************************************************************************************************************************************************************************************
localhost                  : ok=28   changed=6    unreachable=0    failed=1    skipped=13   rescued=0    ignored=0
kubealex commented 3 years ago

If you created the machines before my push, you should try deleting it manually. I tested it with Ubuntu 20.04 and 20.10

organicnz commented 3 years ago

Freshly installed Ubuntu 20.04 similar error :) Anyway, the bug was already fixed and tested hence I'm doing something wrong. Thanks for your massive help, Alex :) 

TASK [terraform] *******************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "command": "/usr/bin/terraform apply -no-color -input=false -auto-approve=true -lock=true /tmp/tmp38sihrhd.tfplan", "msg": "Failure when executing Terraform command. Exited 1.\nstdout: libvirt_volume.storage_image[2]: Creating...\nlibvirt_volume.os_image[1]: Creating...\nlibvirt_volume.os_image[0]: Creating...\nlibvirt_volume.os_image[2]: Creating...\nlibvirt_volume.storage_image[0]: Creating...\nlibvirt_volume.storage_image[1]: Creating...\nlibvirt_cloudinit_disk.commoninit[0]: Creating...\nlibvirt_cloudinit_disk.commoninit[2]: Creating...\nlibvirt_cloudinit_disk.commoninit[1]: Creating...\nlibvirt_volume.os_image[0]: Creation complete after 0s [id=/var/lib/libvirt/images/k8s-test/k8s-test-master-os_image-0]\nlibvirt_volume.os_image[2]: Creation complete after 1s [id=/var/lib/libvirt/images/k8s-test/k8s-test-master-os_image-2]\nlibvirt_volume.storage_image[1]: Creation complete after 1s [id=/var/lib/libvirt/images/k8s-test/k8s-test-master-1-storage_image]\nlibvirt_volume.storage_image[2]: Creation complete after 1s [id=/var/lib/libvirt/images/k8s-test/k8s-test-master-2-storage_image]\nlibvirt_volume.storage_image[0]: Creation complete after 1s [id=/var/lib/libvirt/images/k8s-test/k8s-test-master-0-storage_image]\nlibvirt_volume.os_image[1]: Creation complete after 2s [id=/var/lib/libvirt/images/k8s-test/k8s-test-master-os_image-1]\nlibvirt_cloudinit_disk.commoninit[0]: Creation complete after 2s [id=/var/lib/libvirt/images/k8s-test/k8s-test-master-0-commoninit.iso;b6f53d72-9cdb-4fef-81e8-c3c27f540657]\nlibvirt_cloudinit_disk.commoninit[2]: Creation complete after 3s [id=/var/lib/libvirt/images/k8s-test/k8s-test-master-2-commoninit.iso;cb2fea35-1266-4529-bac2-e56d6e21dc66]\nlibvirt_cloudinit_disk.commoninit[1]: Creation complete after 4s [id=/var/lib/libvirt/images/k8s-test/k8s-test-master-1-commoninit.iso;603438cd-b0e4-4b0f-8328-25f9d3d5a9a0]\nlibvirt_domain.k8s-master[0]: Creating...\nlibvirt_domain.k8s-master[2]: Creating...\nlibvirt_domain.k8s-master[1]: Creating...\n\nstderr: \nError: Error creating libvirt domain: virError(Code=1, Domain=10, Message='internal error: process exited while connecting to monitor: 2021-01-03T13:40:23.084644Z qemu-system-x86_64: -blockdev {\"driver\":\"file\",\"filename\":\"/var/lib/libvirt/images/k8s-test/k8s-test-master-os_image-2\",\"node-name\":\"libvirt-3-storage\",\"auto-read-only\":true,\"discard\":\"unmap\"}: Could not open '/var/lib/libvirt/images/k8s-test/k8s-test-master-os_image-2': Permission denied')\n\n  on k8s-master.tf line 68, in resource \"libvirt_domain\" \"k8s-master\":\n  68: resource \"libvirt_domain\" \"k8s-master\" {\n\n\n\nError: Error creating libvirt domain: virError(Code=1, Domain=10, Message='internal error: process exited while connecting to monitor: 2021-01-03T13:40:23.114183Z qemu-system-x86_64: -blockdev {\"driver\":\"file\",\"filename\":\"/var/lib/libvirt/images/k8s-test/k8s-test-master-os_image-0\",\"node-name\":\"libvirt-3-storage\",\"auto-read-only\":true,\"discard\":\"unmap\"}: Could not open '/var/lib/libvirt/images/k8s-test/k8s-test-master-os_image-0': Permission denied')\n\n  on k8s-master.tf line 68, in resource \"libvirt_domain\" \"k8s-master\":\n  68: resource \"libvirt_domain\" \"k8s-master\" {\n\n\n\nError: Error creating libvirt domain: virError(Code=1, Domain=10, Message='internal error: process exited while connecting to monitor: 2021-01-03T13:40:23.118456Z qemu-system-x86_64: -blockdev {\"driver\":\"file\",\"filename\":\"/var/lib/libvirt/images/k8s-test/k8s-test-master-os_image-1\",\"node-name\":\"libvirt-3-storage\",\"auto-read-only\":true,\"discard\":\"unmap\"}: Could not open '/var/lib/libvirt/images/k8s-test/k8s-test-master-os_image-1': Permission denied')\n\n  on k8s-master.tf line 68, in resource \"libvirt_domain\" \"k8s-master\":\n  68: resource \"libvirt_domain\" \"k8s-master\" {\n\n\n"}

PLAY RECAP *************************************************************************************************************************************************************************************************
localhost                  : ok=28   changed=19   unreachable=0    failed=1    skipped=13   rescued=0    ignored=0