Proxmox plugin breaks UUID creation on foreman for libvirt

[santachago]

Describe the bug After upgrading foreman to 2.4.0 we can no longer create hosts using libvirt compute resource. When creating the host the VM is created on libvirt side (with the correct UUID) but for some reason the UUID stored for the vm on foreman side is wrong it adds “8_” in front of the UUID. This generates an error since foreman is not able to start the VM cause it cannot find it with the incorrect UUID on libvirt. I confirmed this is triggered by the proxmox plugin as i removed it and VM creation for libvirt started working. Added it and failed again.

part of the log:

2021-04-23T09:53:01 [I|aud|644f05ab] Host::Base (606) create event on use_image 
2021-04-23T09:53:01 [I|aud|644f05ab] Host::Base (606) create event on image_file 
2021-04-23T09:53:01 [I|aud|644f05ab] Host::Base (606) create event on uuid 8_f193b5c0-a69c-41f6-a07e-724db2efecca
2021-04-23T09:53:01 [I|aud|644f05ab] Host::Base (606) create event on compute_resource_id 8
2021-04-23T09:53:01 [I|aud|644f05ab] Host::Base (606) create event on puppet_proxy_id 1
....
2021-04-23T09:53:03 [I|app|66957c2e] Backtrace for 'Failed retrieving libvirt vm by uuid 8_f193b5c0-a69c-41f6-a07e-724db2efecca' error (Libvirt::RetrieveError): Call to virDomainLookupByUUID failed: Invalid UUID
 66957c2e | /usr/share/foreman/vendor/ruby/2.5.0/gems/fog-libvirt-0.8.0/lib/fog/libvirt/requests/compute/list_domains.rb:9:in `lookup_domain_by_uuid'
 66957c2e | /usr/share/foreman/vendor/ruby/2.5.0/gems/fog-libvirt-0.8.0/lib/fog/libvirt/requests/compute/list_domains.rb:9:in `list_domains'
 66957c2e | /usr/share/foreman/vendor/ruby/2.5.0/gems/fog-libvirt-0.8.0/lib/fog/libvirt/models/compute/servers.rb:15:in `get'
 66957c2e | /usr/share/foreman/app/models/compute_resource.rb:183:in `find_vm_by_uuid'
 66957c2e | /usr/share/foreman/app/models/compute_resources/foreman/model/libvirt.rb:43:in `find_vm_by_uuid'
 66957c2e | /usr/share/foreman/app/controllers/hosts_controller.rb:280:in `vm'
 66957c2e | /usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-6.0.3.5/lib/action_controller/metal/basic_implicit_render.rb:

Expected behavior Create the host and associate the vm with the correct UUID

Foreman version: 2.4.0 Proxy versions: 2.4.0

Distribution and version: Distributor ID: Ubuntu Description: Ubuntu 18.04.5 LTS Release: 18.04 Codename: bionic

[tbrisker]

Looks like a regression introduced by https://github.com/theforeman/foreman_fog_proxmox/commit/53dfda68e6d3ad94df38594a0ebdb52ffed46952

[eLvErDe]

Hello,

Good catch guys, it's actually even worse, I guess it breaks ALL other compute resources (VMWare below):

2021-04-30T08:53:45 [I|app|9ac2a604] Backtrace for 'Error has occurred while communicating to ESX (VMware)' error (ActiveRecord::RecordNotFound): ActiveRecord::RecordNotFound
 9ac2a604 | /usr/share/foreman/app/models/compute_resource.rb:183:in `find_vm_by_uuid'
 9ac2a604 | /usr/share/foreman/app/services/power_manager/virt.rb:9:in `block in initialize'

I tried to revert the patch and rebuild the RPM but it's failing. Looks way to complicated for my RPM packaging skills sadly. I'll just rollback the foreman VM to 2.3.3.

Regards, Adam.

[santachago]

I dont think this is related to foreman upgrade, this is related to the latest version of this module, but if you downgrade the module without downgrading foreman it should be ok.

[eLvErDe]

Yes but the only other available version in foreman's yum repo is 0.12.x which does not have cloud-init support, so, useless to me. Morever I don't think using an old version will be enough as the broken patch does have a rails migration so you'll probably end up with a broken system due to mismatch between code and DB scheme

[eLvErDe]

@tristanrobert any "feeling" about this bug ? Do you think it can be fixed on ruby side or the issue is more complex ?

[eLvErDe]

Btw, would be happy to make a donation if you're willing to fix bug. I might also be able to do something more official (and more €) if you have an european VAT number.

[tristanrobert]

I could fix it next week. It's planned!

[eLvErDe]

Very cool :)

[eLvErDe]

If the patch is ruby only font hesitate to contact me, I have a sandbox to test it

[eLvErDe]

Any news ? ;)

[tristanrobert]

I am working on it. I need to know why other compute resources than proxmox have hosts uuid like cluster_id+_+vmid.

[tristanrobert]

I have a clue: https://github.com/theforeman/foreman_fog_proxmox/blob/53dfda68e6d3ad94df38594a0ebdb52ffed46952/app/models/concerns/orchestration/proxmox/compute.rb#L57

[tristanrobert]

I have found. I am working on the patch.

[eLvErDe]

Awesome :)

[eLvErDe]

Hello,

I think sadly this issue is not fully fixed ;-)

2021-05-28T11:36:10 [I|app|62187d60] Started POST "/compute_resources/2-Proxmox01/images" for 10.1.1.1 at 2021-05-28 11:36:10 +0200
2021-05-28T11:36:10 [I|app|62187d60] Processing by ImagesController#create as HTML
2021-05-28T11:36:10 [I|app|62187d60]   Parameters: {"utf8"=>"✓", "authenticity_token"=>"token", "image"=>{"name"=>"test", "compute_resource_id"=>"2", "operatingsystem_id"=>"9", "architecture_id"=>"1", "username"=>"root", "password"=>"[FILTERED]", "user_data"=>"0", "uuid"=>"9001"}, "commit"=>"Submit", "compute_resource_id"=>"2-Proxmox01"}
2021-05-28T11:36:10 [E|app|62187d60] Failed to save: UUID could not be found in Proxmox01

I cannot create and image anymore, the drop-down show all images from all nodes but it fails to validate when I attempt to save !

Regards, Adam.

[eLvErDe]

Which is funny because 9000 has been selected from the drop down :D

[taniahagan]

I've noticed this also breaks my setup and had to remove the plugin for now. My notes were first asked at https://community.theforeman.org/t/unable-to-complete-vm-build-in-aws/24379 where I've posted the error I was receiving.