I have got an exception during the launch of an instance. I get the following stacktrace in nova-compute.log:
2011-04-21 22:47:12,887 DEBUG nova [-] injecting network info to xs for vm: |OpaqueRef:31c1cdee-b215-470a-8a93-07599fb4ab28| from (pid=2349) _inject_network_info /usr/lib/python2.6/site-packages/nova/virt/xenapi/vmops.py:859
2011-04-21 22:47:13,606 WARNING nova.virt.xenapi [-] Task [Async.host.call_plugin] OpaqueRef:6897e357-a275-828b-fe22-74ffe95df056 status: failure ['XENAPI_PLUGIN_FAILURE', 'non-zero exit', '', '']
2011-04-21 22:47:13,612 ERROR nova.compute.manager [58X1QE5AKBX4UWIBNJRX dashboard dashboard] Instance '1' failed to spawn. Is virtualization enabled in the BIOS?
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 234, in run_instance
(nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
(nova.compute.manager): TRACE: File "/usr/lib/python2.6/site-packages/nova/virt/xenapi_conn.py", line 188, in spawn
(nova.compute.manager): TRACE: self._vmops.spawn(instance)
(nova.compute.manager): TRACE: File "/usr/lib/python2.6/site-packages/nova/virt/xenapi/vmops.py", line 118, in spawn
(nova.compute.manager): TRACE: vm_ref = self._create_vm(instance, vdi_uuid, network_info)
(nova.compute.manager): TRACE: File "/usr/lib/python2.6/site-packages/nova/virt/xenapi/vmops.py", line 179, in _create_vm
(nova.compute.manager): TRACE: self.inject_network_info(instance, network_info, vm_ref)
(nova.compute.manager): TRACE: File "/usr/lib/python2.6/site-packages/nova/virt/xenapi/vmops.py", line 850, in inject_network_info
(nova.compute.manager): TRACE: self._inject_network_info(instance, network_info, vm_ref)
(nova.compute.manager): TRACE: File "/usr/lib/python2.6/site-packages/nova/virt/xenapi/vmops.py", line 875, in _inject_network_info
(nova.compute.manager): TRACE: vm_ref)
(nova.compute.manager): TRACE: File "/usr/lib/python2.6/site-packages/nova/virt/xenapi/vmops.py", line 977, in _make_plugin_call
(nova.compute.manager): TRACE: err_msg = err_trace.splitlines()-1: TRACE: IndexError: list index out of range
(nova.compute.manager): TRACE:
Two points here:
1) "Instance '1' failed to spawn. Is virtualization enabled in the BIOS?" as error message is misleading and does not really a clue of what's going on
2) There's an unhandled exception within the catch block in vmops.py:977
Revision 989 (Cactus).
I have got an exception during the launch of an instance. I get the following stacktrace in nova-compute.log:
2011-04-21 22:47:12,887 DEBUG nova [-] injecting network info to xs for vm: |OpaqueRef:31c1cdee-b215-470a-8a93-07599fb4ab28| from (pid=2349) _inject_network_info /usr/lib/python2.6/site-packages/nova/virt/xenapi/vmops.py:859 2011-04-21 22:47:13,606 WARNING nova.virt.xenapi [-] Task [Async.host.call_plugin] OpaqueRef:6897e357-a275-828b-fe22-74ffe95df056 status: failure ['XENAPI_PLUGIN_FAILURE', 'non-zero exit', '', ''] 2011-04-21 22:47:13,612 ERROR nova.compute.manager [58X1QE5AKBX4UWIBNJRX dashboard dashboard] Instance '1' failed to spawn. Is virtualization enabled in the BIOS? (nova.compute.manager): TRACE: Traceback (most recent call last): (nova.compute.manager): TRACE: File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 234, in run_instance (nova.compute.manager): TRACE: self.driver.spawn(instance_ref) (nova.compute.manager): TRACE: File "/usr/lib/python2.6/site-packages/nova/virt/xenapi_conn.py", line 188, in spawn (nova.compute.manager): TRACE: self._vmops.spawn(instance) (nova.compute.manager): TRACE: File "/usr/lib/python2.6/site-packages/nova/virt/xenapi/vmops.py", line 118, in spawn (nova.compute.manager): TRACE: vm_ref = self._create_vm(instance, vdi_uuid, network_info) (nova.compute.manager): TRACE: File "/usr/lib/python2.6/site-packages/nova/virt/xenapi/vmops.py", line 179, in _create_vm (nova.compute.manager): TRACE: self.inject_network_info(instance, network_info, vm_ref) (nova.compute.manager): TRACE: File "/usr/lib/python2.6/site-packages/nova/virt/xenapi/vmops.py", line 850, in inject_network_info (nova.compute.manager): TRACE: self._inject_network_info(instance, network_info, vm_ref) (nova.compute.manager): TRACE: File "/usr/lib/python2.6/site-packages/nova/virt/xenapi/vmops.py", line 875, in _inject_network_info (nova.compute.manager): TRACE: vm_ref) (nova.compute.manager): TRACE: File "/usr/lib/python2.6/site-packages/nova/virt/xenapi/vmops.py", line 977, in _make_plugin_call (nova.compute.manager): TRACE: err_msg = err_trace.splitlines()-1: TRACE: IndexError: list index out of range (nova.compute.manager): TRACE:
Two points here:
1) "Instance '1' failed to spawn. Is virtualization enabled in the BIOS?" as error message is misleading and does not really a clue of what's going on 2) There's an unhandled exception within the catch block in vmops.py:977
As for /var/log/messages on the host, I get:
Apr 22 03:00:39 localhost fe: 20110422T03:00:39.664Z||5861|Args after replacement = [/etc/xapi.d/plugins/xenstore.py;write_record OpaqueRef:8985f82c-2c1a-bf2f-d452-8d19b2589b3e path vm-data/networking/02163e0e00b9 dom_id -1 value {"label": "public", "broadcast": "10.0.0.31", "ips": [{"ip": "10.0.0.2", "netmask": "255.255.255.224", "enabled": "1"}], "mac": "02:16:3e:0e:00:b9", "dns": ["192.168.1.1"], "gateway": "10.0.0.1"} ]
Apr 22 03:00:39 localhost fe: 20110422T03:00:39.664Z||5861|I've received the following fds: [2;1]
Apr 22 03:00:39 localhost fe: 20110422T03:00:39.666Z||5862|Caught unexpected exception: Unix.Unix_error(1, "execve", "/etc/xapi.d/plugins/xenstore.py")
Apr 22 03:00:39 localhost fe: 5862 (/etc/xapi.d/plugins/xenstore.py write_record</methodN...) exitted with code 1
No exception handling, that let us get a clue of what's going on.
Any idea of what might have gone wrong? Thanks, Armando