notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: image: [ami-00000003] notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: KEYPAIR mykey notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: euca-run-instances ami-00000003 -k mykey -t m1.tiny notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: instance: [i-00000001] notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Running instance failed because the instance status wasn't "launching" after 120 seconds. err: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: change from notrun to 0 failed: /var/lib/nova/test.sh image_kvm.tgz 2>&1 returned 1 instead of one of [0] at /etc/puppet/modules/nova_e/manifests/nova_api/test.pp:18 notice: Finished catalog run in 202.38 seconds

It seems that some service wasn't started.

You can confirm the status of nova service with the following command on nova-api server.

nova-manage service list

In order to find the cause of the error, could you look for error messsages in /var/log/nova-compute.log, /var/log/nova-api.log, /var/log/nova-scheduler.log? If there are some error messages, please paste them.

Ok just one second and I will put them.

****root@server1:/var/log/nova# nova-manage service list 2012-07-05 22:09:54 DEBUG nova.utils [req-13b90320-78b5-4361-88b3-f5e99f962c94 None None] backend <module 'nova.db.sqlalchemy.api' from '/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.pyc'> from (pid=9768) __get_backend /usr/lib/python2.7/dist-packages/nova/utils.py:658 Binary Host Zone Status State Updated_At nova-consoleauth server1.faveones.com nova enabled :-) 2012-07-06 02:09:51 nova-cert server1.faveones.com nova enabled :-) 2012-07-06 02:09:46 nova-volume server1.faveones.com nova enabled :-) 2012-07-06 02:09:47 nova-scheduler server1.faveones.com nova enabled :-) 2012-07-06 02:09:47 nova-compute server3.faveones.com nova enabled :-) 2012-07-06 02:09:45 nova-compute server5.faveones.com nova enabled :-) 2012-07-06 02:09:49

/var/log/nova-compute.log: On Compute Server #1 2012-07-05 22:00:30 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 343, in call 2012-07-05 22:00:30 TRACE nova.rpc.amqp rv = list(rv) 2012-07-05 22:00:30 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 304, in iter 2012-07-05 22:00:30 TRACE nova.rpc.amqp self.done() 2012-07-05 22:00:30 TRACE nova.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in exit 2012-07-05 22:00:30 TRACE nova.rpc.amqp self.gen.next() 2012-07-05 22:00:30 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 301, in iter 2012-07-05 22:00:30 TRACE nova.rpc.amqp self._iterator.next() 2012-07-05 22:00:30 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 572, in iterconsume 2012-07-05 22:00:30 TRACE nova.rpc.amqp yield self.ensure(_error_callback, _consume) 2012-07-05 22:00:30 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 503, in ensure 2012-07-05 22:00:30 TRACE nova.rpc.amqp error_callback(e) 2012-07-05 22:00:30 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 553, in _error_callback 2012-07-05 22:00:30 TRACE nova.rpc.amqp raise rpc_common.Timeout() 2012-07-05 22:00:30 TRACE nova.rpc.amqp Timeout: Timeout while waiting on RPC response. 2012-07-05 22:00:30 TRACE nova.rpc.amqp 2012-07-05 22:00:51 ERROR nova.rpc.common [-] Timed out waiting for RPC response: timed out 2012-07-05 22:00:51 TRACE nova.rpc.common Traceback (most recent call last): 2012-07-05 22:00:51 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 490, in ensure 2012-07-05 22:00:51 TRACE nova.rpc.common return method(_args, _kwargs) 2012-07-05 22:00:51 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 567, in _consume 2012-07-05 22:00:51 TRACE nova.rpc.common return self.connection.drain_events(timeout=timeout) 2012-07-05 22:00:51 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in drain_events 2012-07-05 22:00:51 TRACE nova.rpc.common return self.transport.drain_events(self.connection, kwargs) 2012-07-05 22:00:51 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in drain_events 2012-07-05 22:00:51 TRACE nova.rpc.common return connection.drain_events(kwargs) 2012-07-05 22:00:51 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in drain_events 2012-07-05 22:00:51 TRACE nova.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout) 2012-07-05 22:00:51 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in wait_multi 2012-07-05 22:00:51 TRACE nova.rpc.common chanmap.keys(), allowed_methods, timeout=timeout) 2012-07-05 22:00:51 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in _wait_multiple 2012-07-05 22:00:51 TRACE nova.rpc.common channel, method_sig, args, content = read_timeout(timeout) 2012-07-05 22:00:51 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 94, in read_timeout 2012-07-05 22:00:51 TRACE nova.rpc.common return self.method_reader.read_method() 2012-07-05 22:00:51 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method 2012-07-05 22:00:51 TRACE nova.rpc.common raise m 2012-07-05 22:00:51 TRACE nova.rpc.common timeout: timed out 2012-07-05 22:00:51 TRACE nova.rpc.common 2012-07-05 22:00:51 DEBUG nova.manager [-] Skipping ComputeManager._run_image_cache_manager_pass, 39 ticks left until next run from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:147 2012-07-05 22:00:51 DEBUG nova.manager [-] Running periodic task ComputeManager._reclaim_queued_deletes from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:00:51 DEBUG nova.compute.manager [-] FLAGS.reclaim_instance_interval <= 0, skipping... from (pid=2575) _reclaim_queued_deletes /usr/lib/python2.7/dist-packages/nova/compute/manager.py:2386 2012-07-05 22:00:51 DEBUG nova.manager [-] Running periodic task ComputeManager._report_driver_status from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:00:51 INFO nova.compute.manager [-] Updating host status 2012-07-05 22:00:51 DEBUG nova.virt.libvirt.connection [-] Updating host stats from (pid=2575) update_status /usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py:2504 2012-07-05 22:00:51 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_unconfirmed_resizes from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:01:51 DEBUG nova.manager [-] Running periodic task ComputeManager._publish_service_capabilities from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:01:51 DEBUG nova.manager [-] Notifying Schedulers of capabilities ... from (pid=2575) _publish_service_capabilities /usr/lib/python2.7/dist-packages/nova/manager.py:203 2012-07-05 22:01:51 DEBUG nova.rpc.amqp [-] Making asynchronous fanout cast... from (pid=2575) fanout_cast /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:359 2012-07-05 22:01:51 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rescued_instances from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:01:51 DEBUG nova.manager [-] Skipping ComputeManager._sync_power_states, 8 ticks left until next run from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:147 2012-07-05 22:01:51 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_bandwidth_usage from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:01:51 DEBUG nova.manager [-] Running periodic task ComputeManager.update_available_resource from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:01:52 INFO nova.virt.libvirt.connection [-] Compute_service record updated for server3.faveones.com 2012-07-05 22:01:52 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rebooting_instances from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:01:52 DEBUG nova.manager [-] Skipping ComputeManager._cleanup_running_deleted_instances, 28 ticks left until next run from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:147 2012-07-05 22:01:52 DEBUG nova.manager [-] Running periodic task ComputeManager._heal_instance_info_cache from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:01:52 DEBUG nova.rpc.amqp [-] Making asynchronous call on network ... from (pid=2575) multicall /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:326 2012-07-05 22:01:52 DEBUG nova.rpc.amqp [-] MSG_ID is 2aa712c2f9a0443289a8bafc081bed17 from (pid=2575) multicall /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:329 2012-07-05 22:02:52 ERROR nova.rpc.common [-] Timed out waiting for RPC response: timed out 2012-07-05 22:02:52 TRACE nova.rpc.common Traceback (most recent call last): 2012-07-05 22:02:52 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 490, in ensure 2012-07-05 22:02:52 TRACE nova.rpc.common return method(_args, _kwargs) 2012-07-05 22:02:52 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 567, in _consume 2012-07-05 22:02:52 TRACE nova.rpc.common return self.connection.drain_events(timeout=timeout) 2012-07-05 22:02:52 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in drain_events 2012-07-05 22:02:52 TRACE nova.rpc.common return self.transport.drain_events(self.connection, kwargs) 2012-07-05 22:02:52 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in drain_events 2012-07-05 22:02:52 TRACE nova.rpc.common return connection.drain_events(kwargs) 2012-07-05 22:02:52 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in drain_events 2012-07-05 22:02:52 TRACE nova.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout) 2012-07-05 22:02:52 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in wait_multi 2012-07-05 22:02:52 TRACE nova.rpc.common chanmap.keys(), allowed_methods, timeout=timeout) 2012-07-05 22:02:52 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in _wait_multiple 2012-07-05 22:02:52 TRACE nova.rpc.common channel, method_sig, args, content = read_timeout(timeout) 2012-07-05 22:02:52 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 94, in read_timeout 2012-07-05 22:02:52 TRACE nova.rpc.common return self.method_reader.read_method() 2012-07-05 22:02:52 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method 2012-07-05 22:02:52 TRACE nova.rpc.common raise m 2012-07-05 22:02:52 TRACE nova.rpc.common timeout: timed out 2012-07-05 22:02:52 TRACE nova.rpc.common 2012-07-05 22:02:52 DEBUG nova.manager [-] Skipping ComputeManager._run_image_cache_manager_pass, 38 ticks left until next run from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:147 2012-07-05 22:02:52 DEBUG nova.manager [-] Running periodic task ComputeManager._reclaim_queued_deletes from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:02:52 DEBUG nova.compute.manager [-] FLAGS.reclaim_instance_interval <= 0, skipping... from (pid=2575) _reclaim_queued_deletes /usr/lib/python2.7/dist-packages/nova/compute/manager.py:2386 2012-07-05 22:02:52 DEBUG nova.manager [-] Running periodic task ComputeManager._report_driver_status from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:02:52 INFO nova.compute.manager [-] Updating host status 2012-07-05 22:02:52 DEBUG nova.virt.libvirt.connection [-] Updating host stats from (pid=2575) update_status /usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py:2504 2012-07-05 22:02:52 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_unconfirmed_resizes from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:03:52 DEBUG nova.manager [-] Running periodic task ComputeManager._publish_service_capabilities from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:03:52 DEBUG nova.manager [-] Notifying Schedulers of capabilities ... from (pid=2575) _publish_service_capabilities /usr/lib/python2.7/dist-packages/nova/manager.py:203 2012-07-05 22:03:52 DEBUG nova.rpc.amqp [-] Making asynchronous fanout cast... from (pid=2575) fanout_cast /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:359 2012-07-05 22:03:52 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rescued_instances from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:03:52 DEBUG nova.manager [-] Skipping ComputeManager._sync_power_states, 7 ticks left until next run from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:147 2012-07-05 22:03:52 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_bandwidth_usage from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:03:52 DEBUG nova.manager [-] Running periodic task ComputeManager.update_available_resource from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:03:53 INFO nova.virt.libvirt.connection [-] Compute_service record updated for server3.faveones.com 2012-07-05 22:03:53 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rebooting_instances from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:03:53 DEBUG nova.manager [-] Skipping ComputeManager._cleanup_running_deleted_instances, 27 ticks left until next run from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:147 2012-07-05 22:03:53 DEBUG nova.manager [-] Running periodic task ComputeManager._heal_instance_info_cache from (pid=2575) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:03:53 DEBUG nova.rpc.amqp [-] Making asynchronous call on network ... from (pid=2575) multicall /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:326 2012-07-05 22:03:53 DEBUG nova.rpc.amqp [-] MSG_ID is a32b5c4d65a04375ae8b02a20ec192cf from (pid=2575) multicall /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:329

*****/var/log/nova-compute.log: On Compute Server #2

2012-07-05 22:02:54 DEBUG nova.rpc.amqp [-] Making asynchronous fanout cast... from (pid=2694) fanout_cast /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:359 2012-07-05 22:02:54 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rescued_instances from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:02:54 DEBUG nova.manager [-] Skipping ComputeManager._sync_power_states, 6 ticks left until next run from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:147 2012-07-05 22:02:54 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_bandwidth_usage from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:02:54 DEBUG nova.manager [-] Running periodic task ComputeManager.update_available_resource from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:02:54 INFO nova.virt.libvirt.connection [-] Compute_service record updated for server5.faveones.com 2012-07-05 22:02:54 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rebooting_instances from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:02:54 DEBUG nova.manager [-] Skipping ComputeManager._cleanup_running_deleted_instances, 26 ticks left until next run from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/no va/manager.py:147 2012-07-05 22:02:54 DEBUG nova.manager [-] Running periodic task ComputeManager._heal_instance_info_cache from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:02:54 DEBUG nova.manager [-] Skipping ComputeManager._run_image_cache_manager_pass, 36 ticks left until next run from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/ma nager.py:147 2012-07-05 22:02:54 DEBUG nova.manager [-] Running periodic task ComputeManager._reclaim_queued_deletes from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:02:54 DEBUG nova.compute.manager [-] FLAGS.reclaim_instance_interval <= 0, skipping... from (pid=2694) _reclaim_queued_deletes /usr/lib/python2.7/dist-packages/nova/compute/manager.py: 2386 2012-07-05 22:02:54 DEBUG nova.manager [-] Running periodic task ComputeManager._report_driver_status from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:02:54 INFO nova.compute.manager [-] Updating host status 2012-07-05 22:02:54 DEBUG nova.virt.libvirt.connection [-] Updating host stats from (pid=2694) update_status /usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py:2504 2012-07-05 22:02:54 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_unconfirmed_resizes from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:03:54 DEBUG nova.manager [-] Running periodic task ComputeManager._publish_service_capabilities from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:03:54 DEBUG nova.manager [-] Notifying Schedulers of capabilities ... from (pid=2694) _publish_service_capabilities /usr/lib/python2.7/dist-packages/nova/manager.py:203 2012-07-05 22:03:54 DEBUG nova.rpc.amqp [-] Making asynchronous fanout cast... from (pid=2694) fanout_cast /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:359 2012-07-05 22:03:55 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rescued_instances from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:03:55 DEBUG nova.manager [-] Skipping ComputeManager._sync_power_states, 5 ticks left until next run from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:147 2012-07-05 22:03:55 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_bandwidth_usage from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:03:55 DEBUG nova.manager [-] Running periodic task ComputeManager.update_available_resource from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:03:55 INFO nova.virt.libvirt.connection [-] Compute_service record updated for server5.faveones.com 2012-07-05 22:03:55 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rebooting_instances from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:03:55 DEBUG nova.manager [-] Skipping ComputeManager._cleanup_running_deleted_instances, 25 ticks left until next run from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/no va/manager.py:147 2012-07-05 22:03:55 DEBUG nova.manager [-] Running periodic task ComputeManager._heal_instance_info_cache from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:03:55 DEBUG nova.manager [-] Skipping ComputeManager._run_image_cache_manager_pass, 35 ticks left until next run from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/ma nager.py:147 2012-07-05 22:03:55 DEBUG nova.manager [-] Running periodic task ComputeManager._reclaim_queued_deletes from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:03:55 DEBUG nova.compute.manager [-] FLAGS.reclaim_instance_interval <= 0, skipping... from (pid=2694) _reclaim_queued_deletes /usr/lib/python2.7/dist-packages/nova/compute/manager.py: 2386 2012-07-05 22:03:55 DEBUG nova.manager [-] Running periodic task ComputeManager._report_driver_status from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:03:55 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_unconfirmed_resizes from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:04:55 DEBUG nova.manager [-] Running periodic task ComputeManager._publish_service_capabilities from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:04:55 DEBUG nova.manager [-] Notifying Schedulers of capabilities ... from (pid=2694) _publish_service_capabilities /usr/lib/python2.7/dist-packages/nova/manager.py:203 2012-07-05 22:04:55 DEBUG nova.rpc.amqp [-] Making asynchronous fanout cast... from (pid=2694) fanout_cast /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:359 2012-07-05 22:04:55 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rescued_instances from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:04:55 DEBUG nova.manager [-] Skipping ComputeManager._sync_power_states, 4 ticks left until next run from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:147 2012-07-05 22:04:55 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_bandwidth_usage from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:04:55 DEBUG nova.manager [-] Running periodic task ComputeManager.update_available_resource from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:04:56 INFO nova.virt.libvirt.connection [-] Compute_service record updated for server5.faveones.com 2012-07-05 22:04:56 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rebooting_instances from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:04:56 DEBUG nova.manager [-] Skipping ComputeManager._cleanup_running_deleted_instances, 24 ticks left until next run from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/no va/manager.py:147 2012-07-05 22:04:56 DEBUG nova.manager [-] Running periodic task ComputeManager._heal_instance_info_cache from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:04:56 DEBUG nova.manager [-] Skipping ComputeManager._run_image_cache_manager_pass, 34 ticks left until next run from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/ma nager.py:147 2012-07-05 22:04:56 DEBUG nova.manager [-] Running periodic task ComputeManager._reclaim_queued_deletes from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:04:56 DEBUG nova.compute.manager [-] FLAGS.reclaim_instance_interval <= 0, skipping... from (pid=2694) _reclaim_queued_deletes /usr/lib/python2.7/dist-packages/nova/compute/manager.py: 2386 2012-07-05 22:04:56 DEBUG nova.manager [-] Running periodic task ComputeManager._report_driver_status from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:04:56 INFO nova.compute.manager [-] Updating host status 2012-07-05 22:04:56 DEBUG nova.virt.libvirt.connection [-] Updating host stats from (pid=2694) update_status /usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py:2504 2012-07-05 22:04:56 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_unconfirmed_resizes from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:05:56 DEBUG nova.manager [-] Running periodic task ComputeManager._publish_service_capabilities from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:05:56 DEBUG nova.manager [-] Notifying Schedulers of capabilities ... from (pid=2694) _publish_service_capabilities /usr/lib/python2.7/dist-packages/nova/manager.py:203 2012-07-05 22:05:56 DEBUG nova.rpc.amqp [-] Making asynchronous fanout cast... from (pid=2694) fanout_cast /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:359 2012-07-05 22:05:56 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rescued_instances from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:05:56 DEBUG nova.manager [-] Skipping ComputeManager._sync_power_states, 3 ticks left until next run from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:147 2012-07-05 22:05:56 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_bandwidth_usage from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:05:56 DEBUG nova.manager [-] Running periodic task ComputeManager.update_available_resource from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:05:56 INFO nova.virt.libvirt.connection [-] Compute_service record updated for server5.faveones.com 2012-07-05 22:05:56 DEBUG nova.manager [-] Running periodic task ComputeManager._poll_rebooting_instances from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152 2012-07-05 22:05:56 DEBUG nova.manager [-] Skipping ComputeManager._cleanup_running_deleted_instances, 23 ticks left until next run from (pid=2694) periodic_tasks /usr/lib/python2.7/dist-packages/no va/manager.py:147

****/var/log/nova-api.log: Dodai-Deploy server (NOVA)

2012-07-05 22:00:00 DEBUG nova.api.ec2.apirequest [req-818cde77-cd5c-49ee-8cba-cca56fc3066d 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] <?xml version="1.0" ?>req-818cde77-cd5c-49ee-8cba-cca56fc3066d68200876aadf451fa5afd3241a8a9629defaultr-1o37yuo8novainstance-storemykey (68200876aadf451fa5afd3241a8a9629, server3.faveones.com)i-000000010pendingami-00000003server-1server-12012-07-06T01:59:29.000Z0/dev/sda1aki-00000001ari-00000002m1.tiny from (pid=6966) _render_response /usr/lib/python2.7/dist-packages/nova/api/ec2/apirequest.py:105 2012-07-05 22:00:00 INFO nova.api.ec2 [req-818cde77-cd5c-49ee-8cba-cca56fc3066d 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] 0.42883s 192.168.1.151 POST /services/Cloud/ CloudController:DescribeInstances 200 [Boto/2.2.2 (linux2)] application/x-www-form-urlencoded text/xml 2012-07-05 22:00:10 DEBUG nova.api.ec2 [req-c3c9a807-2310-45b7-b784-c549edd20e3a 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] action: DescribeInstances from (pid=6966) call /usr/lib/python2.7/dist-packages/nova/api/ec2/init.py:435 2012-07-05 22:00:10 DEBUG nova.compute.api [req-c3c9a807-2310-45b7-b784-c549edd20e3a 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] Searching by: {'deleted': False} from (pid=6966) get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:1064 2012-07-05 22:00:10 DEBUG nova.api.ec2.apirequest [req-c3c9a807-2310-45b7-b784-c549edd20e3a 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] <?xml version="1.0" ?>req-c3c9a807-2310-45b7-b784-c549edd20e3a68200876aadf451fa5afd3241a8a9629defaultr-1o37yuo8novainstance-storemykey (68200876aadf451fa5afd3241a8a9629, server3.faveones.com)i-000000010pendingami-00000003server-1server-12012-07-06T01:59:29.000Z0/dev/sda1aki-00000001ari-00000002m1.tiny from (pid=6966) _render_response /usr/lib/python2.7/dist-packages/nova/api/ec2/apirequest.py:105 2012-07-05 22:00:10 INFO nova.api.ec2 [req-c3c9a807-2310-45b7-b784-c549edd20e3a 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] 0.43345s 192.168.1.151 POST /services/Cloud/ CloudController:DescribeInstances 200 [Boto/2.2.2 (linux2)] application/x-www-form-urlencoded text/xml 2012-07-05 22:00:20 DEBUG nova.api.ec2 [req-6b5315a1-a038-4fae-817a-978c762c44d7 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] action: DescribeInstances from (pid=6966) call /usr/lib/python2.7/dist-packages/nova/api/ec2/init.py:435 2012-07-05 22:00:20 DEBUG nova.compute.api [req-6b5315a1-a038-4fae-817a-978c762c44d7 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] Searching by: {'deleted': False} from (pid=6966) get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:1064 2012-07-05 22:00:20 DEBUG nova.api.ec2.apirequest [req-6b5315a1-a038-4fae-817a-978c762c44d7 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] <?xml version="1.0" ?>req-6b5315a1-a038-4fae-817a-978c762c44d768200876aadf451fa5afd3241a8a9629defaultr-1o37yuo8novainstance-storemykey (68200876aadf451fa5afd3241a8a9629, server3.faveones.com)i-000000010pendingami-00000003server-1server-12012-07-06T01:59:29.000Z0/dev/sda1aki-00000001ari-00000002m1.tiny from (pid=6966) _render_response /usr/lib/python2.7/dist-packages/nova/api/ec2/apirequest.py:105 2012-07-05 22:00:20 INFO nova.api.ec2 [req-6b5315a1-a038-4fae-817a-978c762c44d7 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] 0.39085s 192.168.1.151 POST /services/Cloud/ CloudController:DescribeInstances 200 [Boto/2.2.2 (linux2)] application/x-www-form-urlencoded text/xml 2012-07-05 22:00:30 DEBUG nova.api.ec2 [req-e76d5f93-4de9-4ffd-a441-f4ab35bff654 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] action: DescribeInstances from (pid=6966) call /usr/lib/python2.7/dist-packages/nova/api/ec2/init.py:435 2012-07-05 22:00:30 DEBUG nova.compute.api [req-e76d5f93-4de9-4ffd-a441-f4ab35bff654 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] Searching by: {'deleted': False} from (pid=6966) get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:1064 2012-07-05 22:00:30 DEBUG nova.api.ec2.apirequest [req-e76d5f93-4de9-4ffd-a441-f4ab35bff654 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] <?xml version="1.0" ?>req-e76d5f93-4de9-4ffd-a441-f4ab35bff65468200876aadf451fa5afd3241a8a9629defaultr-1o37yuo8novainstance-storemykey (68200876aadf451fa5afd3241a8a9629, server3.faveones.com)i-000000010errorami-00000003server-1server-12012-07-06T01:59:29.000Z0/dev/sda1aki-00000001ari-00000002m1.tiny from (pid=6966) _render_response /usr/lib/python2.7/dist-packages/nova/api/ec2/apirequest.py:105 2012-07-05 22:00:30 INFO nova.api.ec2 [req-e76d5f93-4de9-4ffd-a441-f4ab35bff654 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] 0.41331s 192.168.1.151 POST /services/Cloud/ CloudController:DescribeInstances 200 [Boto/2.2.2 (linux2)] application/x-www-form-urlencoded text/xml 2012-07-05 22:00:40 DEBUG nova.api.ec2 [req-0e2697f3-9410-4e94-ae21-5342bec08a3b 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] action: DescribeInstances from (pid=6966) call /usr/lib/python2.7/dist-packages/nova/api/ec2/init.py:435 2012-07-05 22:00:40 DEBUG nova.compute.api [req-0e2697f3-9410-4e94-ae21-5342bec08a3b 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] Searching by: {'deleted': False} from (pid=6966) get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:1064 2012-07-05 22:00:40 DEBUG nova.api.ec2.apirequest [req-0e2697f3-9410-4e94-ae21-5342bec08a3b 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] <?xml version="1.0" ?>req-0e2697f3-9410-4e94-ae21-5342bec08a3b68200876aadf451fa5afd3241a8a9629defaultr-1o37yuo8novainstance-storemykey (68200876aadf451fa5afd3241a8a9629, server3.faveones.com)i-000000010errorami-00000003server-1server-12012-07-06T01:59:29.000Z0/dev/sda1aki-00000001ari-00000002m1.tiny from (pid=6966) _render_response /usr/lib/python2.7/dist-packages/nova/api/ec2/apirequest.py:105 2012-07-05 22:00:40 INFO nova.api.ec2 [req-0e2697f3-9410-4e94-ae21-5342bec08a3b 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] 0.43770s 192.168.1.151 POST /services/Cloud/ CloudController:DescribeInstances 200 [Boto/2.2.2 (linux2)] application/x-www-form-urlencoded text/xml 2012-07-05 22:00:50 DEBUG nova.api.ec2 [req-e1a4dbb8-e8e8-41e8-bac0-154f5008b43f 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] action: DescribeInstances from (pid=6966) call /usr/lib/python2.7/dist-packages/nova/api/ec2/init.py:435 2012-07-05 22:00:50 DEBUG nova.compute.api [req-e1a4dbb8-e8e8-41e8-bac0-154f5008b43f 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] Searching by: {'deleted': False} from (pid=6966) get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:1064 2012-07-05 22:00:50 DEBUG nova.api.ec2.apirequest [req-e1a4dbb8-e8e8-41e8-bac0-154f5008b43f 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] <?xml version="1.0" ?>req-e1a4dbb8-e8e8-41e8-bac0-154f5008b43f68200876aadf451fa5afd3241a8a9629defaultr-1o37yuo8novainstance-storemykey (68200876aadf451fa5afd3241a8a9629, server3.faveones.com)i-000000010errorami-00000003server-1server-12012-07-06T01:59:29.000Z0/dev/sda1aki-00000001ari-00000002m1.tiny from (pid=6966) _render_response /usr/lib/python2.7/dist-packages/nova/api/ec2/apirequest.py:105 2012-07-05 22:00:50 INFO nova.api.ec2 [req-e1a4dbb8-e8e8-41e8-bac0-154f5008b43f 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] 0.38719s 192.168.1.151 POST /services/Cloud/ CloudController:DescribeInstances 200 [Boto/2.2.2 (linux2)] application/x-www-form-urlencoded text/xml 2012-07-05 22:01:00 DEBUG nova.api.ec2 [req-80cab79e-ceb7-4b27-8ede-18ca6264f4ed 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] action: DescribeInstances from (pid=6966) call /usr/lib/python2.7/dist-packages/nova/api/ec2/init.py:435 2012-07-05 22:01:00 DEBUG nova.compute.api [req-80cab79e-ceb7-4b27-8ede-18ca6264f4ed 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] Searching by: {'deleted': False} from (pid=6966) get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:1064 2012-07-05 22:01:00 DEBUG nova.api.ec2.apirequest [req-80cab79e-ceb7-4b27-8ede-18ca6264f4ed 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] <?xml version="1.0" ?>req-80cab79e-ceb7-4b27-8ede-18ca6264f4ed68200876aadf451fa5afd3241a8a9629defaultr-1o37yuo8novainstance-storemykey (68200876aadf451fa5afd3241a8a9629, server3.faveones.com)i-000000010errorami-00000003server-1server-12012-07-06T01:59:29.000Z0/dev/sda1aki-00000001ari-00000002m1.tiny from (pid=6966) _render_response /usr/lib/python2.7/dist-packages/nova/api/ec2/apirequest.py:105 2012-07-05 22:01:00 INFO nova.api.ec2 [req-80cab79e-ceb7-4b27-8ede-18ca6264f4ed 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] 0.46095s 192.168.1.151 POST /services/Cloud/ CloudController:DescribeInstances 200 [Boto/2.2.2 (linux2)] application/x-www-form-urlencoded text/xml 2012-07-05 22:01:11 DEBUG nova.api.ec2 [req-73e3f943-8262-49b0-9df7-927398c6f51b 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] action: DescribeInstances from (pid=6966) call /usr/lib/python2.7/dist-packages/nova/api/ec2/init.py:435 2012-07-05 22:01:11 DEBUG nova.compute.api [req-73e3f943-8262-49b0-9df7-927398c6f51b 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] Searching by: {'deleted': False} from (pid=6966) get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:1064 2012-07-05 22:01:11 DEBUG nova.api.ec2.apirequest [req-73e3f943-8262-49b0-9df7-927398c6f51b 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] <?xml version="1.0" ?>req-73e3f943-8262-49b0-9df7-927398c6f51b68200876aadf451fa5afd3241a8a9629defaultr-1o37yuo8novainstance-storemykey (68200876aadf451fa5afd3241a8a9629, server3.faveones.com)i-000000010errorami-00000003server-1server-12012-07-06T01:59:29.000Z0/dev/sda1aki-00000001ari-00000002m1.tiny from (pid=6966) _render_response /usr/lib/python2.7/dist-packages/nova/api/ec2/apirequest.py:105 2012-07-05 22:01:11 INFO nova.api.ec2 [req-73e3f943-8262-49b0-9df7-927398c6f51b 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] 0.39393s 192.168.1.151 POST /services/Cloud/ CloudController:DescribeInstances 200 [Boto/2.2.2 (linux2)] application/x-www-form-urlencoded text/xml 2012-07-05 22:01:21 DEBUG nova.api.ec2 [req-add03ac3-5e13-4ada-b037-b64728d2e53c 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] action: DescribeInstances from (pid=6966) call /usr/lib/python2.7/dist-packages/nova/api/ec2/init.py:435 2012-07-05 22:01:21 DEBUG nova.compute.api [req-add03ac3-5e13-4ada-b037-b64728d2e53c 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] Searching by: {'deleted': False} from (pid=6966) get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:1064 2012-07-05 22:01:21 DEBUG nova.api.ec2.apirequest [req-add03ac3-5e13-4ada-b037-b64728d2e53c 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] <?xml version="1.0" ?>req-add03ac3-5e13-4ada-b037-b64728d2e53c68200876aadf451fa5afd3241a8a9629defaultr-1o37yuo8novainstance-storemykey (68200876aadf451fa5afd3241a8a9629, server3.faveones.com)i-000000010errorami-00000003server-1server-12012-07-06T01:59:29.000Z0/dev/sda1aki-00000001ari-00000002m1.tiny from (pid=6966) _render_response /usr/lib/python2.7/dist-packages/nova/api/ec2/apirequest.py:105 2012-07-05 22:01:21 INFO nova.api.ec2 [req-add03ac3-5e13-4ada-b037-b64728d2e53c 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] 0.42760s 192.168.1.151 POST /services/Cloud/ CloudController:DescribeInstances 200 [Boto/2.2.2 (linux2)] application/x-www-form-urlencoded text/xml 2012-07-05 22:01:31 DEBUG nova.api.ec2 [req-7817569e-4b53-40f6-8b62-ca6f7dbdfb4f 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] action: DescribeInstances from (pid=6966) call /usr/lib/python2.7/dist-packages/nova/api/ec2/init.py:435 2012-07-05 22:01:31 DEBUG nova.compute.api [req-7817569e-4b53-40f6-8b62-ca6f7dbdfb4f 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] Searching by: {'deleted': False} from (pid=6966) get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:1064 2012-07-05 22:01:31 DEBUG nova.api.ec2.apirequest [req-7817569e-4b53-40f6-8b62-ca6f7dbdfb4f 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] <?xml version="1.0" ?>req-7817569e-4b53-40f6-8b62-ca6f7dbdfb4f68200876aadf451fa5afd3241a8a9629defaultr-1o37yuo8novainstance-storemykey (68200876aadf451fa5afd3241a8a9629, server3.faveones.com)i-000000010errorami-00000003server-1server-12012-07-06T01:59:29.000Z0/dev/sda1aki-00000001ari-00000002m1.tiny from (pid=6966) _render_response /usr/lib/python2.7/dist-packages/nova/api/ec2/apirequest.py:105 2012-07-05 22:01:31 INFO nova.api.ec2 [req-7817569e-4b53-40f6-8b62-ca6f7dbdfb4f 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] 0.42384s 192.168.1.151 POST /services/Cloud/ CloudController:DescribeInstances 200 [Boto/2.2.2 (linux2)] application/x-www-form-urlencoded text/xml 2012-07-05 22:01:41 DEBUG nova.api.ec2 [req-1cdf6f75-6904-4433-b81b-7625ed90f78f 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] action: DescribeInstances from (pid=6966) call /usr/lib/python2.7/dist-packages/nova/api/ec2/init.py:435 2012-07-05 22:01:41 DEBUG nova.compute.api [req-1cdf6f75-6904-4433-b81b-7625ed90f78f 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] Searching by: {'deleted': False} from (pid=6966) get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:1064 2012-07-05 22:01:41 DEBUG nova.api.ec2.apirequest [req-1cdf6f75-6904-4433-b81b-7625ed90f78f 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] <?xml version="1.0" ?>req-1cdf6f75-6904-4433-b81b-7625ed90f78f68200876aadf451fa5afd3241a8a9629defaultr-1o37yuo8novainstance-storemykey (68200876aadf451fa5afd3241a8a9629, server3.faveones.com)i-000000010errorami-00000003server-1server-12012-07-06T01:59:29.000Z0/dev/sda1aki-00000001ari-00000002m1.tiny from (pid=6966) _render_response /usr/lib/python2.7/dist-packages/nova/api/ec2/apirequest.py:105 2012-07-05 22:01:41 INFO nova.api.ec2 [req-1cdf6f75-6904-4433-b81b-7625ed90f78f 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] 0.41163s 192.168.1.151 POST /services/Cloud/ CloudController:DescribeInstances 200 [Boto/2.2.2 (linux2)] application/x-www-form-urlencoded text/xml

*****/var/log/nova-scheduler.log: Dodai-Deploy server (NOVA) 2012-07-05 21:57:45 DEBUG nova.service [req-ae2a22eb-3c47-4dd5-87c1-8dd94f126344 None None] Creating Consumer connection for Service scheduler from (pid=9180) start /usr/lib/python2.7/dist-packages/nova/service.py:178 2012-07-05 21:59:29 DEBUG nova.rpc.amqp [-] received {u'_context_roles': [u'admin', u'KeystoneServiceAdmin', u'KeystoneAdmin'], u'_context_request_id': u'req-7a111aa7-a5a3-48ae-803d-da1780bcc0a4', u'_context_read_deleted': u'no', u'args': {u'request_spec': {u'num_instances': 1, u'block_device_mapping': [], u'image': {u'status': u'active', u'name': u'mybucket/image.manifest.xml', u'deleted': False, u'container_format': u'ami', u'created_at': u'2012-07-06 01:58:28.765683', u'disk_format': u'ami', u'updated_at': u'2012-07-06 01:58:30.341644', u'properties': {u'kernel_id': u'46dd4290-5b6f-49bf-bada-728150c36549', u'image_location': u'mybucket/image.manifest.xml', u'image_state': u'available', u'ramdisk_id': u'810f9108-0ab9-47d5-967f-ac73d92b2580', u'architecture': u'x86_64'}, u'min_ram': u'0', u'checksum': u'10047a119149e08fb206eea89832eee0', u'min_disk': u'0', u'is_public': False, u'deleted_at': None, u'id': u'a3944cf0-a690-4092-8102-23f41350c8d7', u'size': 25165824}, u'instance_type': {u'root_gb': 0, u'name': u'm1.tiny', u'flavorid': u'1', u'deleted': False, u'created_at': None, u'ephemeral_gb': 0, u'updated_at': None, u'memory_mb': 512, u'vcpus': 1, u'extra_specs': {}, u'swap': 0, u'rxtx_factor': 1.0, u'deleted_at': None, u'vcpu_weight': None, u'id': 2}, u'instance_properties': {u'vm_state': u'building', u'availability_zone': None, u'launch_time': u'2012-07-06T01:59:29Z', u'ephemeral_gb': 0, u'instance_type_id': 2, u'user_data': u'', u'vm_mode': None, u'reservation_id': u'r-1o37yuo8', u'root_device_name': None, u'display_name': None, u'uuid': u'38c08e1e-eb01-4406-aec1-4febb3bd1e3f', u'display_description': u'', u'key_data': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQC/Xnq8ps5/neljJICaRQ1uLTzmRTRO2XH2MhX7LbdYdX8dfk26dDglai6T9j2CLZuzGewFfOyPvDlCH76Pp+TpFKgwk408V4flZ65sGqMOVxxILfdIW46fhxCJl1x4xsS9e7jUYIqfJWgJCvRZ4JhzDd8TACwGMoT62BbsnOQEdw== nova@server1.faveones.com\n', u'power_state': 0, u'progress': 0, u'project_id': u'68200876aadf451fa5afd3241a8a9629', u'metadata': {}, u'ramdisk_id': u'810f9108-0ab9-47d5-967f-ac73d92b2580', u'access_ip_v6': None, u'access_ip_v4': None, u'kernel_id': u'46dd4290-5b6f-49bf-bada-728150c36549', u'key_name': u'mykey', u'user_id': u'007870be38a346ad95087222e7f195e2', u'config_drive_id': u'', u'root_gb': 0, u'locked': False, u'launch_index': 0, u'memory_mb': 512, u'vcpus': 1, u'image_ref': u'a3944cf0-a690-4092-8102-23f41350c8d7', u'architecture': None, u'auto_disk_config': None, u'os_type': None, u'config_drive': u''}, u'security_group': u'default'}, u'is_first_time': True, u'filter_properties': {u'scheduler_hints': None}, u'topic': u'compute', u'admin_password': '', u'injected_files': None, u'requested_networks': None}, u'_context_auth_token': '', u'_context_is_admin': True, u'_context_project_id': u'68200876aadf451fa5afd3241a8a9629', u'_context_timestamp': u'2012-07-06T01:59:29.208940', u'_context_user_id': u'007870be38a346ad95087222e7f195e2', u'method': u'run_instance', u'_context_remote_address': u'192.168.1.151'} from (pid=9180) _safe_log /usr/lib/python2.7/dist-packages/nova/rpc/common.py:160 2012-07-05 21:59:29 DEBUG nova.rpc.amqp [req-7a111aa7-a5a3-48ae-803d-da1780bcc0a4 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] unpacked context: {'user_id': u'007870be38a346ad95087222e7f195e2', 'roles': [u'admin', u'KeystoneServiceAdmin', u'KeystoneAdmin'], 'timestamp': '2012-07-06T01:59:29.208940', 'auth_token': '', 'remote_address': u'192.168.1.151', 'is_admin': True, 'request_id': u'req-7a111aa7-a5a3-48ae-803d-da1780bcc0a4', 'project_id': u'68200876aadf451fa5afd3241a8a9629', 'read_deleted': u'no'} from (pid=9180) _safe_log /usr/lib/python2.7/dist-packages/nova/rpc/common.py:160 2012-07-05 21:59:29 DEBUG nova.rpc.amqp [req-7a111aa7-a5a3-48ae-803d-da1780bcc0a4 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] Making asynchronous cast on compute.server3.faveones.com... from (pid=9180) cast /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:351 2012-07-05 21:59:29 DEBUG nova.rpc.amqp [req-7a111aa7-a5a3-48ae-803d-da1780bcc0a4 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] Pool creating new connection from (pid=9180) create /usr/lib/python2.7/dist-packages/nova/rpc/amqp.py:60 2012-07-05 21:59:29 INFO nova.rpc.common [req-7a111aa7-a5a3-48ae-803d-da1780bcc0a4 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] Connected to AMQP server on 192.168.1.151:5672 2012-07-05 21:59:29 DEBUG nova.scheduler.driver [req-7a111aa7-a5a3-48ae-803d-da1780bcc0a4 007870be38a346ad95087222e7f195e2 68200876aadf451fa5afd3241a8a9629] Casted 'run_instance' to compute 'server3.faveones.com' from (pid=9180) cast_to_compute_host /usr/lib/python2.7/dist-packages/nova/scheduler/driver.py:80 2012-07-05 21:59:50 DEBUG nova.rpc.amqp [-] received {u'_context_roles': [u'admin'], u'_context_request_id': u'req-19112e17-290b-480a-82b8-a781d5e2cd65', u'_context_read_deleted': u'no', u'args': {u'service_name': u'compute', u'host': u'server3.faveones.com', u'capabilities': {u'disk_available': 28, u'vcpus_used': 0, u'hypervisor_type': u'QEMU', u'disk_total': 29, u'host_memory_free': 3760, u'vcpus': 8, u'disk_used': 1, u'host_memory_total': 3954, u'hypervisor_version': 1000000, u'cpu_info': {u'arch': u'x86_64', u'model': u'qemu32', u'vendor': u'Intel', u'features': [u'lahf_lm', u'lm', u'nx', u'syscall', u'hypervisor', u'popcnt', u'cx16', u'ht', u'clflush', u'pse36', u'mca', u'mtrr'], u'topology': {u'cores': u'8', u'threads': u'1', u'sockets': u'1'}}}}, u'_context_auth_token': '', u'_context_is_admin': True, u'_context_project_id': None, u'_context_timestamp': u'2012-07-06T01:59:50.769543', u'_context_user_id': None, u'method': u'update_service_capabilities', u'_context_remote_address': None} from (pid=9180) _safe_log /usr/lib/python2.7/dist-packages/nova/rpc/common.py:160 2012-07-05 21:59:50 DEBUG nova.rpc.amqp [req-19112e17-290b-480a-82b8-a781d5e2cd65 None None] unpacked context: {'user_id': None, 'roles': [u'admin'], 'timestamp': '2012-07-06T01:59:50.769543', 'auth_token': '', 'remote_address': None, 'is_admin': True, 'request_id': u'req-19112e17-290b-480a-82b8-a781d5e2cd65', 'project_id': None, 'read_deleted': u'no'} from (pid=9180) _safe_log /usr/lib/python2.7/dist-packages/nova/rpc/common.py:160 2012-07-05 21:59:50 DEBUG nova.scheduler.host_manager [req-19112e17-290b-480a-82b8-a781d5e2cd65 None None] Received compute service update from server3.faveones.com. from (pid=9180) update_service_capabilities /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:273 2012-07-05 21:59:52 DEBUG nova.rpc.amqp [-] received {u'_context_roles': [u'admin'], u'_context_request_id': u'req-d0dca018-afd8-47a5-b693-221b867e86fb', u'_context_read_deleted': u'no', u'args': {u'service_name': u'compute', u'host': u'server5.faveones.com', u'capabilities': {u'disk_available': 28, u'vcpus_used': 0, u'hypervisor_type': u'QEMU', u'disk_total': 29, u'host_memory_free': 3758, u'vcpus': 8, u'disk_used': 1, u'host_memory_total': 3954, u'hypervisor_version': 1000000, u'cpu_info': {u'arch': u'x86_64', u'model': u'qemu32', u'vendor': u'Intel', u'features': [u'lahf_lm', u'lm', u'nx', u'syscall', u'hypervisor', u'popcnt', u'cx16', u'ht', u'clflush', u'pse36', u'mca', u'mtrr'], u'topology': {u'cores': u'8', u'threads': u'1', u'sockets': u'1'}}}}, u'_context_auth_token': '', u'_context_is_admin': True, u'_context_project_id': None, u'_context_timestamp': u'2012-07-06T01:59:52.124574', u'_context_user_id': None, u'method': u'update_service_capabilities', u'_context_remote_address': None} from (pid=9180) _safe_log /usr/lib/python2.7/dist-packages/nova/rpc/common.py:160 2012-07-05 21:59:52 DEBUG nova.rpc.amqp [req-d0dca018-afd8-47a5-b693-221b867e86fb None None] unpacked context: {'user_id': None, 'roles': [u'admin'], 'timestamp': '2012-07-06T01:59:52.124574', 'auth_token': '', 'remote_address': None, 'is_admin': True, 'request_id': u'req-d0dca018-afd8-47a5-b693-221b867e86fb', 'project_id': None, 'read_deleted': u'no'} from (pid=9180) _safe_log /usr/lib/python2.7/dist-packages/nova/rpc/common.py:160 2012-07-05 21:59:52 DEBUG nova.scheduler.host_manager [req-d0dca018-afd8-47a5-b693-221b867e86fb None None] Received compute service update from server5.faveones.com. from (pid=9180) update_service_capabilities /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:273 2012-07-05 22:00:52 DEBUG nova.rpc.amqp [-] received {u'_context_roles': [u'admin'], u'_context_request_id': u'req-adf7b984-d210-465e-8635-df3a153a6842', u'_context_read_deleted': u'no', u'args': {u'service_name': u'compute', u'host': u'server5.faveones.com', u'capabilities': {u'disk_available': 28, u'vcpus_used': 0, u'hypervisor_type': u'QEMU', u'disk_total': 29, u'host_memory_free': 3758, u'vcpus': 8, u'disk_used': 1, u'host_memory_total': 3954, u'hypervisor_version': 1000000, u'cpu_info': {u'arch': u'x86_64', u'model': u'qemu32', u'vendor': u'Intel', u'features': [u'lahf_lm', u'lm', u'nx', u'syscall', u'hypervisor', u'popcnt', u'cx16', u'ht', u'clflush', u'pse36', u'mca', u'mtrr'], u'topology': {u'cores': u'8', u'threads': u'1', u'sockets': u'1'}}}}, u'_context_auth_token': '', u'_context_is_admin': True, u'_context_project_id': None, u'_context_timestamp': u'2012-07-06T02:00:52.648874', u'_context_user_id': None, u'method': u'update_service_capabilities', u'_context_remote_address': None} from (pid=9180) _safe_log /usr/lib/python2.7/dist-packages/nova/rpc/common.py:160 2012-07-05 22:00:52 DEBUG nova.rpc.amqp [req-adf7b984-d210-465e-8635-df3a153a6842 None None] unpacked context: {'user_id': None, 'roles': [u'admin'], 'timestamp': '2012-07-06T02:00:52.648874', 'auth_token': '', 'remote_address': None, 'is_admin': True, 'request_id': u'req-adf7b984-d210-465e-8635-df3a153a6842', 'project_id': None, 'read_deleted': u'no'} from (pid=9180) _safe_log /usr/lib/python2.7/dist-packages/nova/rpc/common.py:160 2012-07-05 22:00:52 DEBUG nova.scheduler.host_manager [req-adf7b984-d210-465e-8635-df3a153a6842 None None] Received compute service update from server5.faveones.com. from (pid=9180) update_service_capabilities /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:273 2012-07-05 22:01:51 DEBUG nova.rpc.amqp [-] received {u'_context_roles': [u'admin'], u'_context_request_id': u'req-24d008be-ca87-4d9d-98fd-473a5e8be209', u'_context_read_deleted': u'no', u'args': {u'service_name': u'compute', u'host': u'server3.faveones.com', u'capabilities': {u'disk_available': 28, u'vcpus_used': 0, u'hypervisor_type': u'QEMU', u'disk_total': 29, u'host_memory_free': 3756, u'vcpus': 8, u'disk_used': 1, u'host_memory_total': 3954, u'hypervisor_version': 1000000, u'cpu_info': {u'arch': u'x86_64', u'model': u'qemu32', u'vendor': u'Intel', u'features': [u'lahf_lm', u'lm', u'nx', u'syscall', u'hypervisor', u'popcnt', u'cx16', u'ht', u'clflush', u'pse36', u'mca', u'mtrr'], u'topology': {u'cores': u'8', u'threads': u'1', u'sockets': u'1'}}}}, u'_context_auth_token': '', u'_context_is_admin': True, u'_context_project_id': None, u'_context_timestamp': u'2012-07-06T02:01:51.714623', u'_context_user_id': None, u'method': u'update_service_capabilities', u'_context_remote_address': None} from (pid=9180) _safe_log /usr/lib/python2.7/dist-packages/nova/rpc/common.py:160 2012-07-05 22:01:51 DEBUG nova.rpc.amqp [req-24d008be-ca87-4d9d-98fd-473a5e8be209 None None] unpacked context: {'user_id': None, 'roles': [u'admin'], 'timestamp': '2012-07-06T02:01:51.714623', 'auth_token': '', 'remote_address': None, 'is_admin': True, 'request_id': u'req-24d008be-ca87-4d9d-98fd-473a5e8be209', 'project_id': None, 'read_deleted': u'no'} from (pid=9180) _safe_log /usr/lib/python2.7/dist-packages/nova/rpc/common.py:160 2012-07-05 22:01:51 DEBUG nova.scheduler.host_manager [req-24d008be-ca87-4d9d-98fd-473a5e8be209 None None] Received compute service update from server3.faveones.com. from (pid=9180) update_service_capabilities /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:273 2012-07-05 22:01:53 DEBUG nova.rpc.amqp [-] received {u'_context_roles': [u'admin'], u'_context_request_id': u'req-1183faff-db38-4a2d-9e9d-2372b0fa7209', u'_context_read_deleted': u'no', u'args': {u'service_name': u'compute', u'host': u'server5.faveones.com', u'capabilities': {u'disk_available': 28, u'vcpus_used': 0, u'hypervisor_type': u'QEMU', u'disk_total': 29, u'host_memory_free': 3756, u'vcpus': 8, u'disk_used': 1, u'host_memory_total': 3954, u'hypervisor_version': 1000000, u'cpu_info': {u'arch': u'x86_64', u'model': u'qemu32', u'vendor': u'Intel', u'features': [u'lahf_lm', u'lm', u'nx', u'syscall', u'hypervisor', u'popcnt', u'cx16', u'ht', u'clflush', u'pse36', u'mca', u'mtrr'], u'topology': {u'cores': u'8', u'threads': u'1', u'sockets': u'1'}}}}, u'_context_auth_token': '', u'_context_is_admin': True, u'_context_project_id': None, u'_context_timestamp': u'2012-07-06T02:01:53.551596', u'_context_user_id': None, u'method': u'update_service_capabilities', u'_context_remote_address': None} from (pid=9180) _safe_log /usr/lib/python2.7/dist-packages/nova/rpc/common.py:160 2012-07-05 22:01:53 DEBUG nova.rpc.amqp [req-1183faff-db38-4a2d-9e9d-2372b0fa7209 None None] unpacked context: {'user_id': None, 'roles': [u'admin'], 'timestamp': '2012-07-06T02:01:53.551596', 'auth_token': '', 'remote_address': None, 'is_admin': True, 'request_id': u'req-1183faff-db38-4a2d-9e9d-2372b0fa7209', 'project_id': None, 'read_deleted': u'no'} from (pid=9180) _safe_log /usr/lib/python2.7/dist-packages/nova/rpc/common.py:160 2012-07-05 22:01:53 DEBUG nova.scheduler.host_manager [req-1183faff-db38-4a2d-9e9d-2372b0fa7209 None None] Received compute service update from server5.faveones.com. from (pid=9180) update_service_capabilities /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:273 2012-07-05 22:02:54 DEBUG nova.rpc.amqp [-] received {u'_context_roles': [u'admin'], u'_context_request_id': u'req-e25b7789-fe18-4a44-944f-7a466bee9463', u'_context_read_deleted': u'no', u'args': {u'service_name': u'compute', u'host': u'server5.faveones.com', u'capabilities': {u'disk_available': 28, u'vcpus_used': 0, u'hypervisor_type': u'QEMU', u'disk_total': 29, u'host_memory_free': 3756, u'vcpus': 8, u'disk_used': 1, u'host_memory_total': 3954, u'hypervisor_version': 1000000, u'cpu_info': {u'arch': u'x86_64', u'model': u'qemu32', u'vendor': u'Intel', u'features': [u'lahf_lm', u'lm', u'nx', u'syscall', u'hypervisor', u'popcnt', u'cx16', u'ht', u'clflush', u'pse36', u'mca', u'mtrr'], u'topology': {u'cores': u'8', u'threads': u'1', u'sockets': u'1'}}}}, u'_context_auth_token': '', u'_context_is_admin': True, u'_context_project_id': None, u'_context_timestamp': u'2012-07-06T02:02:54.079374', u'_context_user_id': None, u'method': u'update_service_capabilities', u'_context_remote_address': None} from (pid=9180) _safe_log /usr/lib/python2.7/dist-packages/nova/rpc/common.py:160 2012-07-05 22:02:54 DEBUG nova.rpc.amqp [req-e25b7789-fe18-4a44-944f-7a466bee9463 None None] unpacked context: {'user_id': None, 'roles': [u'admin'], 'timestamp': '2012-07-06T02:02:54.079374', 'auth_token': '', 'remote_address': None, 'is_admin': True, 'request_id': u'req-e25b7789-fe18-4a44-944f-7a466bee9463', 'project_id': None, 'read_deleted': u'no'} from (pid=9180) _safe_log /usr/lib/python2.7/dist-packages/nova/rpc/common.py:160 2012-07-05 22:02:54 DEBUG nova.scheduler.host_manager [req-e25b7789-fe18-4a44-944f-7a466bee9463 None None] Received compute service update from server5.faveones.com. from (pid=9180) update_service_capabilities /usr/lib/python2.7/dist-packages/nova/scheduler/host_manager.py:273 2012-07-05 22:03:52 DEBUG nova.rpc.amqp [-] received {u'_context_roles': [u'admin'], u'_context_request_id': u'req-40fe940d-b62b-4ba5-a04f-048598c08c57', u'_context_read_deleted': u'no', u'args': {u'service_name': u'compute', u'host': u'server3.faveones.com', u'capabilities': {u'disk_available': 28, u'vcpus_used': 0, u'hyperviso

Also here is the /var/log/syslog from server1 (NOVA and dobai-deploy)

Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Part: kernel.part.00 Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Generating manifest /tmp/kernel.manifest.xml Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Checking bucket: mybucket Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Uploading manifest file Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Uploading part: kernel.part.00 Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Uploaded image as mybucket/kernel.manifest.xml Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Checking image Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Encrypting image Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Splitting image... Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Part: ramdisk.part.00 Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Generating manifest /tmp/ramdisk.manifest.xml Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Checking bucket: mybucket Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Uploading manifest file Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Uploading part: ramdisk.part.00 Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Uploaded image as mybucket/ramdisk.manifest.xml Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Checking image Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Encrypting image Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Splitting image... Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Part: image.part.00 Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Part: image.part.01 Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Generating manifest /tmp/image.manifest.xml Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Checking bucket: mybucket Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Uploading manifest file Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Uploading part: image.part.00 Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Uploading part: image.part.01 Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Uploaded image as mybucket/image.manifest.xml Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) image: [ami-00000003] Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) KEYPAIR#011mykey Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) euca-run-instances ami-00000003 -k mykey -t m1.tiny Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) instance: [i-00000001] Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Instance status: pending Jul 5 22:01:41 puppet-agent[9238]: last message repeated 11 times Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Instance status: error Jul 5 22:01:41 puppet-agent[9238]: last message repeated 15 times Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) Running instance failed because the instance status wasn't "launching" after 120 seconds. Jul 5 22:01:41 server1 puppet-agent[9238]: (/Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns) change from notrun to 0 failed: /var/lib/nova/test.sh image_kvm.tgz 2>&1 returned 1 instead of one of [0] at /etc/puppet/modules/nova_e/manifests/nova_api/test.pp:18 Jul 5 22:01:41 server1 puppet-agent[9238]: Finished catalog run in 199.87 seconds

And last this is the logs from the nova test on the http://server1:3000.

notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Uploading part: image.part.01 notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Uploaded image as mybucket/image.manifest.xml notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: image: [ami-00000003] notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: KEYPAIR mykey notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: euca-run-instances ami-00000003 -k mykey -t m1.tiny notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: instance: [i-00000001] notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: pending notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Instance status: error notice: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: Running instance failed because the instance status wasn't "launching" after 120 seconds. err: /Stage[main]/Nova_e::Nova_api::Test/Exec[/var/lib/nova/test.sh image_kvm.tgz 2>&1]/returns: change from notrun to 0 failed: /var/lib/nova/test.sh image_kvm.tgz 2>&1 returned 1 instead of one of [0] at /etc/puppet/modules/nova_e/manifests/nova_api/test.pp:18 notice: Finished catalog run in 199.87 seconds

The service nova-network wasn't started. Could you confirm it with the command below.

sudo status nova-network

If it was down, pls start it with the command below.

sudo start nova-network

Well it is actually running on the nova server.

root@server1:~# sudo status nova-network nova-network start/running, process 9190

Note: It is not running on the nova-compute nodes.

I have installed this 5 times today and every time I end up with the same problem. Just do not what to do.

Have you installed glance, and changed the value of "Config item: glance" from localhost to the actual fqdn of the glance server?

Yes Sir, completely installed. I can even upload image to glance and they show in the dashboard.

Of course I cannot start any instance with this image because it always fail.

Could you paste contents of file /etc/nova/nova.conf and /etc/nova/nova-compute.conf in server3.faveones.com?

/etc/nova/nova.conf /ON NOVA SERVER/

--dhcpbridge_flagfile=/etc/nova/nova.conf --dhcpbridge=/usr/bin/nova-dhcpbridge --logdir=/var/log/nova --state_path=/var/lib/nova --lock_path=/var/lock/nova --allow_admin_api=true --use_deprecated_auth=false --auth_strategy=keystone --scheduler_driver=nova.scheduler.simple.SimpleScheduler --s3_host=192.168.1.151 --ec2_host=192.168.1.151 --rabbit_host=192.168.1.151 --cc_host=192.168.1.151 --nova_url=http://192.168.1.151:8774/v1.1/ --routing_source_ip=192.168.1.151 --glance_api_servers=server2:9292 --image_service=nova.image.glance.GlanceImageService --iscsi_ip_prefix=192.168.22 --sql_connection=mysql://root:nova@192.168.1.151/nova --ec2_url=http://192.168.1.151:8773/services/Cloud --keystone_ec2_url=http://server1:5000/v2.0/ec2tokens --api_paste_config=/etc/nova/api-paste.ini --libvirt_type=kvm --libvirt_use_virtio_for_bridges=true --start_guests_on_host_boot=true --resume_guests_state_on_host_boot=true --vnc_enabled=true --novncproxy_base_url=http://192.168.1.151:6080/vnc_auto.html

network specific settings

--network_manager=nova.network.manager.VlanManager --public_interface=eth0 --flat_interface=eth0 --vlan_interface=eth0 --flat_network_bridge=br100 --fixed_range=192.168.22.32/27 --floating_range=10.42.0.32/27 --network_size=32

--flat_network_dhcp_start=192.168.22.33

--flat_injected=False --force_dhcp_release --iscsi_helper=tgtadm --connection_type=libvirt --root_helper=sudo nova-rootwrap --verbose /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

/etc/nova/nova-compute.conf /ON ONE COMPUTE NODE/

--libvirt_type=kvm --vncserver_proxyclient_address=192.168.1.155 --vncserver_listen=192.168.1.155

*****/etc/nova/nova.conf

network specific settings

--flat_network_dhcp_start=192.168.22.33

--flat_injected=False --force_dhcp_release --iscsi_helper=tgtadm --connection_type=libvirt --root_helper=sudo nova-rootwrap --verbose

///////////////////////////////////////////////////////////////////////////////////////////////////

/etc/nova/nova-compute.conf /ON ONE COMPUTE NODE/

--libvirt_type=kvm --vncserver_proxyclient_address=192.168.1.153 --vncserver_listen=192.168.1.153root@server3:/etc/nova#

*****/etc/nova/nova.conf

network specific settings

--flat_network_dhcp_start=192.168.22.33

--flat_injected=False --force_dhcp_release --iscsi_helper=tgtadm --connection_type=libvirt --root_helper=sudo nova-rootwrap --verbose

The conf files seems correct.

According to the following error messages,

2012-07-05 22:00:30 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 343, in call 2012-07-05 22:00:30 TRACE nova.rpc.amqp rv = list(rv) 2012-07-05 22:00:30 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 304, in iter 2012-07-05 22:00:30 TRACE nova.rpc.amqp self.done() 2012-07-05 22:00:30 TRACE nova.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in exit 2012-07-05 22:00:30 TRACE nova.rpc.amqp self.gen.next() 2012-07-05 22:00:30 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 301, in iter 2012-07-05 22:00:30 TRACE nova.rpc.amqp self._iterator.next() 2012-07-05 22:00:30 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 572, in iterconsume 2012-07-05 22:00:30 TRACE nova.rpc.amqp yield self.ensure(_error_callback, _consume) 2012-07-05 22:00:30 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 503, in ensure 2012-07-05 22:00:30 TRACE nova.rpc.amqp error_callback(e) 2012-07-05 22:00:30 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 553, in _error_callback 2012-07-05 22:00:30 TRACE nova.rpc.amqp raise rpc_common.Timeout() 2012-07-05 22:00:30 TRACE nova.rpc.amqp Timeout: Timeout while waiting on RPC response. 2012-07-05 22:00:30 TRACE nova.rpc.amqp

The direct cause of the test failure may be that nova-compute couldn't connect to RabbitMQ server or connect to other server via RabbitMQ. Therefore, the following things should be confirm.

The status RabbitMQ service.
The status of glance services.

status glance-api status glance-registry
Whether access ports of glance from nova-compute(server3.faveones.com) with the following command.

telnet server2 9292 # glance-api telnet server2 9191 # glance-registry

1- root@server1:~# /etc/init.d/rabbitmq-server status Status of node rabbit@server1 ... [{pid,5114}, {running_applications,[{rabbit,"RabbitMQ","2.7.1"}, {mnesia,"MNESIA CXC 138 12","4.5"}, {os_mon,"CPO CXC 138 46","2.2.7"}, {sasl,"SASL CXC 138 11","2.1.10"}, {stdlib,"ERTS CXC 138 10","1.17.5"}, {kernel,"ERTS CXC 138 10","2.14.5"}]}, {os,{unix,linux}}, {erlang_version,"Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:8:8] [rq:8] [async-threads:30] [kernel-poll:true]\n"}, {memory,[{total,26183840}, {processes,10760888}, {processes_used,10734800}, {system,15422952}, {atom,1124441}, {atom_used,1120220}, {binary,97368}, {code,11134417}, {ets,812856}]}, {vm_memory_high_watermark,0.4}, {vm_memory_limit,840056832}] ...done.

2- root@server2:~# status glance-api glance-api start/running, process 2683 root@server2:~# status glance-registry glance-registry start/running, process 2698

3- root@server3:/etc/nova# telnet server2 9292 Trying 192.168.1.152... Connected to server2.faveones.com. Escape character is '^]'. ^C^Zquit Connection closed by foreign host. root@server3:/etc/nova# telnet server2 9291 Trying 192.168.1.152... telnet: Unable to connect to remote host: Connection refused

Are you sure the port for glance-registry is 9291. Well let me try with 9191.

root@server3:/etc/nova# telnet server2 9191 Trying 192.168.1.152... Connected to server2.faveones.com. Escape character is '^]'. quit

The port is 9191. It seems all things are right. Could you start a instance from openstack dashboard?

I wish I could send you a picture for you to see. Any email I can email you a picture?.

Of course you can. Please.

Let me know if these pictures help.

Where are the pictures?

On Fri, Jul 6, 2012 at 1:09 PM, kirelgt reply@reply.github.com wrote:

Let me know if these pictures help.

Reply to this email directly or view it on GitHub: https://github.com/nii-cloud/dodai-deploy/issues/18#issuecomment-6796942

I hit Reply and it looks like the pictures did not get to you. Is there any other email I can email them to you?.

I couldn't see any thing. Could you send pictures to the guanxiaohua2k6@gmail.com .

On Fri, Jul 6, 2012 at 1:24 PM, kirelgt reply@reply.github.com wrote:

Still around???.

Reply to this email directly or view it on GitHub: https://github.com/nii-cloud/dodai-deploy/issues/18#issuecomment-6797040

The task of instance was stopped at networking. So there was something wrong with nova-network. Could you paste /var/log/nova/nova-network.log?

root@server1:/var/log/nova# tail -n 100 nova-network.log 2012-07-05 21:57:44 DEBUG nova.service [-] volume_api_class : nova.volume.api.API from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] report_interval : 10 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] scheduler_manager : nova.scheduler.manager.SchedulerManager from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] ec2_port : 8773 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] monkey_patch_modules : ['nova.api.ec2.cloud:nova.notifier.api.notify_decorator', 'nova.compute.api:nova.notifier.api.notify_decorator'] from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] rabbit_retry_backoff : 2 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] auth_token_ttl : 3600 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] quota_volumes : 10 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] ec2_listen_port : 8773 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] ec2_scheme : http from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] vpn_image_id : 0 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] disable_process_locking : False from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] auto_assign_floating_ip : False from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] cert_topic : cert from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] quota_floating_ips : 10 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] fake_call : False from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] instance_dns_manager : nova.network.dns_driver.DNSDriver from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] sql_idle_timeout : 3600 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] glance_host : 192.168.1.151 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] default_image : ami-11111 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] aws_secret_access_key : FLAG SET from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:409 2012-07-05 21:57:44 DEBUG nova.service [-] use_ipv6 : False from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] log_format : %(asctime)s %(levelname)8s [%(name)s] %(message)s from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] fake_network : False from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] metadata_port : 8775 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] force_dhcp_release : True from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] dhcpbridge_flagfile : /etc/nova/nova.conf from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] quota_gigabytes : 1000 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] region_list : [] from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] dhcpbridge : /usr/bin/nova-dhcpbridge from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] num_networks : 1 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] network_manager : nova.network.manager.VlanManager from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] root_helper : sudo nova-rootwrap from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] memcached_servers : None from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] logging_default_format_string : %(asctime)s %(levelname)s %(name)s [-] %(instance)s%(message)s from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] osapi_volume_ext_list : [] from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] find_host_timeout : 30 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] quota_max_injected_file_path_bytes : 255 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] fake_rabbit : False from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] log_date_format : %Y-%m-%d %H:%M:%S from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] rabbit_host : 192.168.1.151 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] policy_default_rule : default from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] use_syslog : False from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] rpc_thread_pool_size : 1024 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] osapi_path : /v1.1/ from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] syslog_log_facility : LOG_USER from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] ec2_path : /services/Cloud from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] linuxnet_interface_driver : nova.network.linux_net.LinuxBridgeInterfaceDriver from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] bindir : /usr/lib/python2.7/dist-packages/bin from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] rabbit_use_ssl : False from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] rabbit_retry_interval : 1 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] config_file : ['/tmp/nova-conf-cc1OLw/tmpbDEvZj.conf'] from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] network_api_class : nova.network.api.API from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] node_availability_zone : nova from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] db_driver : nova.db from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] create_unique_mac_address_attempts : 5 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] volume_manager : nova.volume.manager.VolumeManager from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] start_guests_on_host_boot : True from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] pybasedir : /usr/lib/python2.7/dist-packages from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] vlan_start : 100 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] metadata_manager : nova.api.manager.MetadataManager from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] ipv6_backend : rfc2462 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] isolated_images : [] from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] api_paste_config : /etc/nova/api-paste.ini from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] quota_security_group_rules : 20 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] rabbit_virtual_host : / from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] network_driver : nova.network.linux_net from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] quota_instances : 10 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] image_service : nova.image.glance.GlanceImageService from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] use_single_default_gateway : False from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] control_exchange : nova from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] cnt_vpn_clients : 0 from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] log_file : None from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] compute_manager : nova.compute.manager.ComputeManager from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 DEBUG nova.service [-] network_topic : network from (pid=9193) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 21:57:44 AUDIT nova.service [-] Starting network node (version 2012.1-LOCALBRANCH:LOCALREVISION) 2012-07-05 21:57:44 DEBUG nova.utils [-] Found lockfile nova-iptables.lock with link count 2 from (pid=9193) cleanup_file_locks /usr/lib/python2.7/dist-packages/nova/utils.py:1023 2012-07-05 21:57:44 DEBUG nova.network.l3 [-] Initializing linux_net L3 driver from (pid=9193) initialize /usr/lib/python2.7/dist-packages/nova/network/l3.py:83 2012-07-05 21:57:44 DEBUG nova.utils [-] Attempting to grab semaphore "iptables" for method "apply"... from (pid=9193) inner /usr/lib/python2.7/dist-packages/nova/utils.py:927 2012-07-05 21:57:44 DEBUG nova.utils [-] Got semaphore "iptables" for method "apply"... from (pid=9193) inner /usr/lib/python2.7/dist-packages/nova/utils.py:931 2012-07-05 21:57:44 DEBUG nova.utils [-] Attempting to grab file lock "iptables" for method "apply"... from (pid=9193) inner /usr/lib/python2.7/dist-packages/nova/utils.py:935

2012-07-05 23:11:53 DEBUG nova.service [-] sql_max_retries : 10 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] default_instance_type : m1.small from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] metadata_listen : 0.0.0.0 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] firewall_driver : nova.virt.firewall.IptablesFirewallDriver from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] periodic_interval : 60 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] vpn_key_suffix : FLAG SET from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:409 2012-07-05 23:11:53 DEBUG nova.service [-] use_cow_images : True from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] osapi_volume_listen_port : 8776 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] gateway : None from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] null_kernel : nokernel from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] instance_format : [instance: %(uuid)s] from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] osapi_compute_link_prefix : None from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] rpc_response_timeout : 60 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] osapi_compute_ext_list : [] from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] publish_errors : False from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] consoleauth_topic : consoleauth from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] service_down_time : 60 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] logging_exception_prefix : %(asctime)s TRACE %(name)s %(instance)s from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] enabled_apis : ['ec2', 'osapi_compute', 'osapi_volume', 'metadata'] from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] volume_api_class : nova.volume.api.API from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] report_interval : 10 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] scheduler_manager : nova.scheduler.manager.SchedulerManager from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] ec2_port : 8773 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] monkey_patch_modules : ['nova.api.ec2.cloud:nova.notifier.api.notify_decorator', 'nova.compute.api:nova.notifier.api.notify_decorator'] from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] rabbit_retry_backoff : 2 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] auth_token_ttl : 3600 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] quota_volumes : 10 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] ec2_listen_port : 8773 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] ec2_scheme : http from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] vpn_image_id : 0 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] disable_process_locking : False from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] auto_assign_floating_ip : False from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] cert_topic : cert from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] quota_floating_ips : 10 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] fake_call : False from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] instance_dns_manager : nova.network.dns_driver.DNSDriver from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] sql_idle_timeout : 3600 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] glance_host : 192.168.1.151 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] default_image : ami-11111 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] aws_secret_access_key : FLAG SET from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:409 2012-07-05 23:11:53 DEBUG nova.service [-] use_ipv6 : False from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] log_format : %(asctime)s %(levelname)8s [%(name)s] %(message)s from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] fake_network : False from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] metadata_port : 8775 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] force_dhcp_release : True from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] dhcpbridge_flagfile : /etc/nova/nova.conf from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] quota_gigabytes : 1000 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] region_list : [] from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] dhcpbridge : /usr/bin/nova-dhcpbridge from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] num_networks : 1 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] network_manager : nova.network.manager.VlanManager from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] root_helper : sudo nova-rootwrap from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] memcached_servers : None from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] logging_default_format_string : %(asctime)s %(levelname)s %(name)s [-] %(instance)s%(message)s from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] osapi_volume_ext_list : [] from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] find_host_timeout : 30 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] quota_max_injected_file_path_bytes : 255 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] fake_rabbit : False from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] log_date_format : %Y-%m-%d %H:%M:%S from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] rabbit_host : 192.168.1.151 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] policy_default_rule : default from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] use_syslog : False from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] rpc_thread_pool_size : 1024 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] osapi_path : /v1.1/ from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] syslog_log_facility : LOG_USER from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] ec2_path : /services/Cloud from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] linuxnet_interface_driver : nova.network.linux_net.LinuxBridgeInterfaceDriver from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] bindir : /usr/lib/python2.7/dist-packages/bin from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] rabbit_use_ssl : False from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] rabbit_retry_interval : 1 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] config_file : ['/tmp/nova-conf-fG8JsM/tmpdKPSvs.conf'] from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] network_api_class : nova.network.api.API from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] node_availability_zone : nova from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] db_driver : nova.db from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] create_unique_mac_address_attempts : 5 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] volume_manager : nova.volume.manager.VolumeManager from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] start_guests_on_host_boot : True from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] pybasedir : /usr/lib/python2.7/dist-packages from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] vlan_start : 100 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] metadata_manager : nova.api.manager.MetadataManager from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] ipv6_backend : rfc2462 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] isolated_images : [] from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] api_paste_config : /etc/nova/api-paste.ini from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] quota_security_group_rules : 20 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] rabbit_virtual_host : / from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] network_driver : nova.network.linux_net from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] quota_instances : 10 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] image_service : nova.image.glance.GlanceImageService from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] use_single_default_gateway : False from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] control_exchange : nova from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] cnt_vpn_clients : 0 from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] log_file : None from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] compute_manager : nova.compute.manager.ComputeManager from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 DEBUG nova.service [-] network_topic : network from (pid=10240) wait /usr/lib/python2.7/dist-packages/nova/service.py:411 2012-07-05 23:11:53 AUDIT nova.service [-] Starting network node (version 2012.1-LOCALBRANCH:LOCALREVISION) 2012-07-05 23:11:53 DEBUG nova.utils [-] Found lockfile nova-iptables.lock with link count 2 from (pid=10240) cleanup_file_locks /usr/lib/python2.7/dist-packages/nova/utils.py:1023 2012-07-05 23:11:53 DEBUG nova.network.l3 [-] Initializing linux_net L3 driver from (pid=10240) initialize /usr/lib/python2.7/dist-packages/nova/network/l3.py:83 2012-07-05 23:11:53 DEBUG nova.utils [-] Attempting to grab semaphore "iptables" for method "apply"... from (pid=10240) inner /usr/lib/python2.7/dist-packages/nova/utils.py:927 2012-07-05 23:11:53 DEBUG nova.utils [-] Got semaphore "iptables" for method "apply"... from (pid=10240) inner /usr/lib/python2.7/dist-packages/nova/utils.py:931 2012-07-05 23:11:53 DEBUG nova.utils [-] Attempting to grab file lock "iptables" for method "apply"... from (pid=10240) inner /usr/lib/python2.7/dist-packages/nova/utils.py:935

According to the following messages,

2012-07-05 23:11:53 DEBUG nova.utils [-] Attempting to grab file lock "iptables" for method "apply"... from (pid=10240) inner /usr/lib/python2.7/dist-packages/nova/utils.py:935

nova-network was trying to get file lock, but it seems that the lock was used by other process, so the service network was waiting.

If the service was started correctly, nova-netwok should be displayed in the output of command "nova-manage service list", and the log should be something like the below.

2012-07-03 17:49:25 AUDIT nova.service [-] Starting network node (version 2012.1-LOCALBRANCH:LOCALREVISION) 2012-07-03 17:49:25 DEBUG nova.network.l3 [-] Initializing linux_net L3 driver from (pid=811) initialize /usr/lib/python2.7/dist-packages/nova/network/l3.py:83 2012-07-03 17:49:25 DEBUG nova.utils [-] Attempting to grab semaphore "iptables" for method "apply"... from (pid=811) inner /usr/lib/python2.7/dist-packages/nova/utils.py:927 2012-07-03 17:49:25 DEBUG nova.utils [-] Got semaphore "iptables" for method "apply"... from (pid=811) inner /usr/lib/python2.7/dist-packages/nova/utils.py:931 2012-07-03 17:49:25 DEBUG nova.utils [-] Attempting to grab file lock "iptables" for method "apply"... from (pid=811) inner /usr/lib/python2.7/dist-packages/nova/utils.py:935 2012-07-03 17:49:25 DEBUG nova.utils [-] Got file lock "iptables" for method "apply"... from (pid=811) inner /usr/lib/python2.7/dist-packages/nova/utils.py:942 2012-07-03 17:49:25 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap iptables-save -t filter from (pid=811) execute /usr/lib/python2.7/dist-packages/nova/utils.py:219 2012-07-03 17:49:33 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap iptables-restore from (pid=811) execute /usr/lib/python2.7/dist-packages/nova/utils.py:219 2012-07-03 17:49:33 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap iptables-save -t nat from (pid=811) execute /usr/lib/python2.7/dist-packages/nova/utils.py:219 2012-07-03 17:49:35 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap iptables-restore from (pid=811) execute /usr/lib/python2.7/dist-packages/nova/utils.py:219

And the problem may be related to the bug https://bugs.launchpad.net/nova/+bug/999698.

As to the solution of the problem, you can add the following line to the nova.conf of nova-network server, then restart it.

--disable_process_locking

--disable_process_locking should be put into the nova.conf of server1 (NOVA, dobai-deploy)???

Got it. Thanks for your time and one last little question, so sorry for asking another question.

When an instance is created, it has an IP address 10.0.0.#, my question is: How can I give it a 192.168.1.# that will allow me to work with this instance from outside.

Sorry, now it allows me to create the instances but they do not start. The status mark as failed and the task never finishes.

The cause is that service nova-network stopped at the place where to get a lock. It seems to be the bug of nova https://bugs.launchpad.net/nova/+bug/999698. After adding the flag "--disable_process_locking" to nova.conf when creating proposal, it will be solved.

nii-cloud / dodai-deploy

Problem when testing nova. #18

network specific settings

--flat_network_dhcp_start=192.168.22.33

network specific settings

--flat_network_dhcp_start=192.168.22.33

network specific settings

--flat_network_dhcp_start=192.168.22.33