search

apache / cloudstack

Apache CloudStack is an opensource Infrastructure as a Service (IaaS) cloud computing platform
https://cloudstack.apache.org/
Apache License 2.0
1.84k stars 1.07k forks source link

Unable to list VMs on the remote KVM host, for import #8704

Closed sureshanaparti closed 3 weeks ago

sureshanaparti commented 4 months ago

Failed to list the VMs in remote KVM host, for importing them to CloudStack.

ISSUE TYPE
COMPONENT NAME
VM Import
CLOUDSTACK VERSION
4.19.0
CONFIGURATION

KVM hypervisor / Advanced Zone

OS / ENVIRONMENT

Tried on MS, KVM Hosts on Ubuntu 22.04 & MS / KVM Hosts on Rocky Linux 8

SUMMARY

Failed to list the VMs in remote KVM host, for importing them to CloudStack.

API cmd has the following parameters - zoneid, host, username, password, hypervisor=kvm.

API Response:

{
    "listvmsforimportresponse": {
        "uuidList": [],
        "errorcode": 530,
        "cserrorcode": 4250,
        "errortext": "Error while listing remote Vms"
    }
}

Exception in Logs:

2024-02-26 06:12:25,881 ERROR [c.c.a.ApiServer] (qtp1753127384-473:ctx-f57f9de7 ctx-67aa987a) (logid:2887196b) unhandled exception executing api command: [Ljava.lang.String;@106067a6
com.cloud.utils.exception.CloudRuntimeException: Error while listing remote Vms
    at org.apache.cloudstack.vm.UnmanagedVMsManagerImpl.getRemoteVms(UnmanagedVMsManagerImpl.java:2489)
    at org.apache.cloudstack.vm.UnmanagedVMsManagerImpl.listVmsForImport(UnmanagedVMsManagerImpl.java:2464)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
    at com.sun.proxy.$Proxy459.listVmsForImport(Unknown Source)
    at org.apache.cloudstack.api.command.admin.vm.ListVmsForImportCmd.execute(ListVmsForImportCmd.java:121)
STEPS TO REPRODUCE
EXPECTED RESULTS
The stopped VMs on the remote KVM host should be listed, for import
ACTUAL RESULTS
Unable to list VMs on the remote KVM, fails with Request Failed (530) error.
kiranchavala commented 4 months ago

Tested the issue and was able to reproduce the issue

weizhouapache commented 4 months ago

@sureshanaparti @kiranchavala I run into this issue last week. It seems to be because that the libvirt service on remote host listens on TLS port (16514), but the API call tries to get VMs on TCP port (16509). To access the TLS port, the cert/key are required. However they are not supported for now.

@sureshanaparti can you retry after the changes below ?

sureshanaparti commented 4 months ago

@weizhouapache Tried with listen_tcp = 1, but still unable to see list the stopped VMs.

with listen_tcp = 0, it's throwing exception and fails with Request Failed (530) error. I think, it's better to show the appropriate message if failed to connect to the remote host's qemu.

kiranchavala commented 4 months ago

@weizhouapache

the issue is still present

Steps followed

  1. Create a KVM external environment

Launch a centos 7 vm and install the following packages 

yum install qemu-kvm libvirt libvirt-python libguestfs-tools virt-install

Systemctl enable libvirtd

systemctl start libvirtd
  1. Edit the /etc/libvirtd/libvirtd.conf

listen_tls = 0 listen_tcp = 1 tcp_port = "16509" auth_tcp = "none" mdns_adv = 0

key_file="/etc/pki/libvirt/private/serverkey.pem"

cert_file="/etc/pki/libvirt/servercert.pem"

ca_file="/etc/pki/CA/cacert.pem"

3. Edit the  /etc/sysconfig/libvirtd

LIBVIRTD_ARGS="--listen"'


4. Launch a vm with the following command 

`virt-install --virt-type=kvm --name centos8 --vcpus 2 --ram 2048 --os-variant=rhel8.0 --cdrom=/var/lib/libvirt/images/centos8.iso --network=default  --disk size=16`

5. Shutdown the vm

`virsh shutdown <name >
`
Agent Logs

2024-02-26 10:08:04,738 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) (logid:da44af16) Processing command: com.cloud.agent.api.GetRemoteVmsCommand 2024-02-26 10:08:04,739 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) (logid:da44af16) Looking for libvirtd connection at: qemu+tcp://10.0.32.71/system 2024-02-26 10:08:04,745 DEBUG [resource.wrapper.LibvirtGetRemoteVmsCommandWrapper] (agentRequest-Handler-5:null) (logid:da44af16) VM testkiran: powerstate = VIR_DOMAIN_SHUTOFF; vm state=PowerOff 2024-02-26 10:08:04,749 DEBUG [resource.wrapper.LibvirtGetRemoteVmsCommandWrapper] (agentRequest-Handler-5:null) (logid:da44af16) Unable to retrieve unmanaged instance info. java.lang.NullPointerException at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.getTagValue(LibvirtDomainXMLParser.java:431) at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.extractCpuModeDef(LibvirtDomainXMLParser.java:518) at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.parseDomainXML(LibvirtDomainXMLParser.java:390) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.getUnmanagedInstance(LibvirtGetRemoteVmsCommandWrapper.java:90) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.execute(LibvirtGetRemoteVmsCommandWrapper.java:71) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.execute(LibvirtGetRemoteVmsCommandWrapper.java:46) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1903) at com.cloud.agent.Agent.processRequest(Agent.java:683) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106) at com.cloud.utils.nio.Task.call(Task.java:83) at com.cloud.utils.nio.Task.call(Task.java:29) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-02-26 10:08:04,749 ERROR [resource.wrapper.LibvirtGetRemoteVmsCommandWrapper] (agentRequest-Handler-5:null) (logid:da44af16) Error while fetching instance details com.cloud.utils.exception.CloudRuntimeException: Unable to retrieve unmanaged instance info. null at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.getUnmanagedInstance(LibvirtGetRemoteVmsCommandWrapper.java:110) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.execute(LibvirtGetRemoteVmsCommandWrapper.java:71) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.execute(LibvirtGetRemoteVmsCommandWrapper.java:46) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1903) at com.cloud.agent.Agent.processRequest(Agent.java:683) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106) at com.cloud.utils.nio.Task.call(Task.java:83) at com.cloud.utils.nio.Task.call(Task.java:29) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

weizhouapache commented 4 months ago

2024-02-26 10:08:04,749 DEBUG [resource.wrapper.LibvirtGetRemoteVmsCommandWrapper] (agentRequest-Handler-5:null) (logid:da44af16) Unable to retrieve unmanaged instance info. java.lang.NullPointerException at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.getTagValue(LibvirtDomainXMLParser.java:431)

@kiranchavala it looks the VMs have been listed successfully.

but the vm is not compatible with cloudstack. can you share the vm xml dump ?

virsh dumpxml --security-info <vm name>
kiranchavala commented 4 months ago

@weizhouapache

virsh dumpxml --security-info testkiran

dumpxml


<domain type='kvm'>
  <name>testkiran</name>
  <uuid>aafaaabc-8657-4efc-9c52-3422d4e04088</uuid>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/ubuntu-22.04.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/testkiran.qcow2'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:09:73:b8'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'>
      <listen type='address'/>
    </graphics>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
      <image compression='off'/>
    </graphics>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </memballoon>
  </devices>
</domain>
weizhouapache commented 4 months ago 
<model fallback='allow'/>

@kiranchavala the vm does not have a model specified. maybe that caused the issue. looks like a bug

weizhouapache commented 4 months ago 
2024-02-26 10:08:04,738 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) (logid:da44af16) Processing command: com.cloud.agent.api.GetRemoteVmsCommand
2024-02-26 10:08:04,739 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) (logid:da44af16) Looking for libvirtd connection at: qemu+tcp://10.0.32.71/system
2024-02-26 10:08:04,745 DEBUG [resource.wrapper.LibvirtGetRemoteVmsCommandWrapper] (agentRequest-Handler-5:null) (logid:da44af16) VM testkiran: powerstate = VIR_DOMAIN_SHUTOFF; vm state=PowerOff
2024-02-26 10:08:04,749 DEBUG [resource.wrapper.LibvirtGetRemoteVmsCommandWrapper] (agentRequest-Handler-5:null) (logid:da44af16) Unable to retrieve unmanaged instance info.
java.lang.NullPointerException
    at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.getTagValue(LibvirtDomainXMLParser.java:431)
    at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.extractCpuModeDef(LibvirtDomainXMLParser.java:518)
    at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.parseDomainXML(LibvirtDomainXMLParser.java:390)
    at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.getUnmanagedInstance(LibvirtGetRemoteVmsCommandWrapper.java:90)
    at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.execute(LibvirtGetRemoteVmsCommandWrapper.java:71)
    at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.execute(LibvirtGetRemoteVmsCommandWrapper.java:46)
    at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
    at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1903)

@kiranchavala I've created a PR for this issue #8720

rohityadavcloud commented 3 months ago

@sureshanaparti are you going to work on this issue, or should we have @vishesh92 work on this? Thanks.

weizhouapache commented 3 months ago

@weizhouapache

the issue is still present

Steps followed

  1. Create a KVM external environment

Launch a centos 7 vm and install the following packages

yum install qemu-kvm libvirt libvirt-python libguestfs-tools virt-install

Systemctl enable libvirtd

systemctl start libvirtd
  1. Edit the /etc/libvirtd/libvirtd.conf

listen_tls = 0
listen_tcp = 1
tcp_port = "16509"
auth_tcp = "none"
mdns_adv = 0

key_file="/etc/pki/libvirt/private/serverkey.pem"

cert_file="/etc/pki/libvirt/servercert.pem"

ca_file="/etc/pki/CA/cacert.pem"
  1. Edit the /etc/sysconfig/libvirtd
`LIBVIRTD_ARGS="--listen"' `
  1. Launch a vm with the following command

virt-install --virt-type=kvm --name centos8 --vcpus 2 --ram 2048 --os-variant=rhel8.0 --cdrom=/var/lib/libvirt/images/centos8.iso --network=default --disk size=16

  1. Shutdown the vm

virsh shutdown <name > Agent Logs


2024-02-26 10:08:04,738 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) (logid:da44af16) Processing command: com.cloud.agent.api.GetRemoteVmsCommand
2024-02-26 10:08:04,739 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) (logid:da44af16) Looking for libvirtd connection at: qemu+tcp://10.0.32.71/system
2024-02-26 10:08:04,745 DEBUG [resource.wrapper.LibvirtGetRemoteVmsCommandWrapper] (agentRequest-Handler-5:null) (logid:da44af16) VM testkiran: powerstate = VIR_DOMAIN_SHUTOFF; vm state=PowerOff
2024-02-26 10:08:04,749 DEBUG [resource.wrapper.LibvirtGetRemoteVmsCommandWrapper] (agentRequest-Handler-5:null) (logid:da44af16) Unable to retrieve unmanaged instance info.
java.lang.NullPointerException
  at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.getTagValue(LibvirtDomainXMLParser.java:431)
  at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.extractCpuModeDef(LibvirtDomainXMLParser.java:518)
  at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.parseDomainXML(LibvirtDomainXMLParser.java:390)
  at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.getUnmanagedInstance(LibvirtGetRemoteVmsCommandWrapper.java:90)
  at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.execute(LibvirtGetRemoteVmsCommandWrapper.java:71)
  at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.execute(LibvirtGetRemoteVmsCommandWrapper.java:46)
  at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
  at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1903)
  at com.cloud.agent.Agent.processRequest(Agent.java:683)
  at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106)
  at com.cloud.utils.nio.Task.call(Task.java:83)
  at com.cloud.utils.nio.Task.call(Task.java:29)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  at java.base/java.lang.Thread.run(Thread.java:829)
2024-02-26 10:08:04,749 ERROR [resource.wrapper.LibvirtGetRemoteVmsCommandWrapper] (agentRequest-Handler-5:null) (logid:da44af16) Error while fetching instance details
com.cloud.utils.exception.CloudRuntimeException: Unable to retrieve unmanaged instance info. null
  at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.getUnmanagedInstance(LibvirtGetRemoteVmsCommandWrapper.java:110)
  at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.execute(LibvirtGetRemoteVmsCommandWrapper.java:71)
  at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.execute(LibvirtGetRemoteVmsCommandWrapper.java:46)
  at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
  at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1903)
  at com.cloud.agent.Agent.processRequest(Agent.java:683)
  at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106)
  at com.cloud.utils.nio.Task.call(Task.java:83)
  at com.cloud.utils.nio.Task.call(Task.java:29)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  at java.base/java.lang.Thread.run(Thread.java:829)

@kiranchavala can you test with latest 4.19 or main ? the vm import issue should be fixed by #8720

sureshanaparti commented 3 weeks ago

Tested with latest 4.19, able to list the VMs (with listen_tcp = 1). Noticed inappropriate error messages when remote host is not reachable, etc - improved messages / logs in this PR #9204.

ListVMsRemoteKVMHost
sureshanaparti commented 3 weeks ago

@weizhouapache Tried with listen_tcp = 1, but still unable to see list the stopped VMs.

with listen_tcp = 0, it's throwing exception and fails with Request Failed (530) error. I think, it's better to show the appropriate message if failed to connect to the remote host's qemu.

Appropriate messages, are addressed in PR https://github.com/apache/cloudstack/pull/9204.

DaanHoogland commented 3 weeks ago

fixed in #9204