search

stackinabox / stackinabox.io

stackinabox.io is a quick way to get up and running with IBM UrbanCode
http://stackinabox.io/
14 stars 6 forks source link

Cannot start any VM in OpenStack #6

Open niklaushirt opened 7 years ago

niklaushirt commented 7 years ago

As of the Jan, 6th commit I am unable to start a VM in OpenStack (be it via UCDP or directly in the interface) (same in the latest Commit, Jan. 19th). The Error is: Error: Failed to perform requested operation on instance "demo_fail", the instance has an error status: Please try again later [Error: Build of instance 979ab17a-3411-4ef4-bded-c1f5bd9bda1c aborted: Image is unacceptable: Bad Image format: Image could not be found.].

I tried various combinations of Vagrant (1.9.1 and 1.8.5)/VirtualBox (5.0.30 and 5.1.12) and Macs (RAM/CPU) config but to no avail and the images themselves in Openstack exist as far as I can tell.

Errors found in n-cpu.log:

2017-01-19 06:14:58.536 DEBUG nova.compute.manager [req-11ea2e21-d952-4ab4-ae43-fcf2403123e4 demo demo] [instance: 15734ad5-a957-41e7-aab7-6278e3338d40] Start spawning the instance on the hypervisor. from (pid=4496) _build_and_run_instance /opt/stack/nova/nova/compute/manager.py:1919
2017-01-19 06:14:58.546 DEBUG nova.virt.lxd.driver [req-11ea2e21-d952-4ab4-ae43-fcf2403123e4 demo demo] [instance: 15734ad5-a957-41e7-aab7-6278e3338d40] setup_image called for instance from (pid=4496) setup_image /opt/stack/nova-lxd/nova/virt/lxd/driver.py:1173
2017-01-19 06:14:58.548 DEBUG oslo_concurrency.lockutils [req-11ea2e21-d952-4ab4-ae43-fcf2403123e4 demo demo] Acquired semaphore "/opt/stack/data/nova/instances/locks" from (pid=4496) lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:212
2017-01-19 06:14:58.548 DEBUG nova.virt.lxd.session [req-11ea2e21-d952-4ab4-ae43-fcf2403123e4 demo demo] [instance: 15734ad5-a957-41e7-aab7-6278e3338d40] image_defined called for instance from (pid=4496) image_defined /opt/stack/nova-lxd/nova/virt/lxd/session.py:462
2017-01-19 06:14:58.669 INFO os_vif [req-11ea2e21-d952-4ab4-ae43-fcf2403123e4 demo demo] Loaded VIF plugin class '<class 'vif_plug_ovs.ovs.OvsPlugin'>' with name 'ovs'
2017-01-19 06:14:58.670 INFO os_vif [req-11ea2e21-d952-4ab4-ae43-fcf2403123e4 demo demo] Loaded VIF plugin class '<class 'vif_plug_linux_bridge.linux_bridge.LinuxBridgePlugin'>' with name 'linux_bridge'
2017-01-19 06:14:58.729 ERROR nova.virt.lxd.driver [req-11ea2e21-d952-4ab4-ae43-fcf2403123e4 demo demo] [instance: 15734ad5-a957-41e7-aab7-6278e3338d40] Failed to upload  to LXD: Image  is unacceptable: Bad Image format: Image  could not be found.
2017-01-19 06:14:58.730 DEBUG oslo_concurrency.lockutils [req-11ea2e21-d952-4ab4-ae43-fcf2403123e4 demo demo] Releasing semaphore "/opt/stack/data/nova/instances/locks" from (pid=4496) lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:225
2017-01-19 06:14:58.730 ERROR nova.compute.manager [req-11ea2e21-d952-4ab4-ae43-fcf2403123e4 demo demo] [instance: 15734ad5-a957-41e7-aab7-6278e3338d40] Instance failed to spawn
2017-01-19 06:14:58.730 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40] Traceback (most recent call last):
2017-01-19 06:14:58.730 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40]   File "/opt/stack/nova/nova/compute/manager.py", line 2082, in _build_resources
2017-01-19 06:14:58.730 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40]     yield resources
build_and_run_instance0 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40]   File "/opt/stack/nova/nova/compute/manager.py", line 1924, in _--More--(61%)
2017-01-19 06:14:58.730 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40]     block_device_info=block_device_info)
2017-01-19 06:14:58.730 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40]   File "/opt/stack/nova-lxd/nova/virt/lxd/driver.py", line 274, in spawn
2017-01-19 06:14:58.730 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40]     self.setup_image(context, instance, image_meta)
2017-01-19 06:14:58.730 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40]   File "/opt/stack/nova-lxd/nova/virt/lxd/driver.py", line 1333, in setup_image
2017-01-19 06:14:58.730 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40]     os.unlink(container_manifest)
2017-01-19 06:14:58.730 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-01-19 06:14:58.730 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40]     self.force_reraise()
2017-01-19 06:14:58.730 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-01-19 06:14:58.730 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40]     six.reraise(self.type_, self.value, self.tb)
2017-01-19 06:14:58.730 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40]   File "/opt/stack/nova-lxd/nova/virt/lxd/driver.py", line 1216, in setup_image
2017-01-19 06:14:58.730 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40]     image_id=instance.image_ref, reason=reason)
2017-01-19 06:14:58.730 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40] ImageUnacceptable: Image  is unacceptable: Bad Image format: Image  could not be found.
2017-01-19 06:14:58.730 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40]

....

2017-01-19 06:15:02.984 DEBUG oslo_messaging._drivers.amqpdriver [req-11ea2e21-d952-4ab4-ae43-fcf2403123e4 demo demo] CALL msg_id: f552ba031e9d4569b78aeeb789006280 exchange 'nova' topic 'conductor' from (pid=4496) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:448
2017-01-19 06:15:03.003 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: f552ba031e9d4569b78aeeb789006280 from (pid=4496) __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:296
2017-01-19 06:15:03.005 WARNING nova.scheduler.client.report [req-11ea2e21-d952-4ab4-ae43-fcf2403123e4 demo demo] No authentication information found for placement API. Placement is optional in Newton, but required in Ocata. Please enable the placement service before upgrading.
2017-01-19 06:15:03.005 WARNING nova.scheduler.client.report [req-11ea2e21-d952-4ab4-ae43-fcf2403123e4 demo demo] Unable to refresh my resource provider record
2017-01-19 06:15:03.005 DEBUG oslo_concurrency.lockutils [req-11ea2e21-d952-4ab4-ae43-fcf2403123e4 demo demo] Lock "compute_resources" released by "nova.compute.resource_tracker.abort_instance_claim" :: held 0.178s from (pid=4496) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:2822017-01-19 06:15:03.006 DEBUG nova.compute.utils [req-11ea2e21-d952-4ab4-ae43-fcf2403123e4 demo demo] [instance: 15734ad5-a957-41e7-aab7-6278e3338d40] Image  is unacceptable: Bad Image format: Image  could not be found. from (pid=4496) notify_about_instance_usage /opt/stack/nova/nova/compute/utils.py:313
2017-01-19 06:15:03.007 DEBUG oslo_messaging._drivers.amqpdriver [req-11ea2e21-d952-4ab4-ae43-fcf2403123e4 demo demo] CAST unique_id: 7a67d3247e9e44078ff8d4d0d815db96 NOTIFY exchange 'nova' topic 'notifications.error' from (pid=4496) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:432
2017-01-19 06:15:03.013 ERROR nova.compute.manager [req-11ea2e21-d952-4ab4-ae43-fcf2403123e4 demo demo] [instance: 15734ad5-a957-41e7-aab7-6278e3338d40] Build of instance 15734ad5-a957-41e7-aab7-6278e3338d40 aborted: Image  is unacceptable: Bad Image format: Image  could not be found.
2017-01-19 06:15:03.013 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40] Traceback (most recent call last):
2017-01-19 06:15:03.013 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40]   File "/opt/stack/nova/nova/compute/manager.py", line 1783, in _do_build_and_run_instance
2017-01-19 06:15:03.013 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40]     filter_properties)
2017-01-19 06:15:03.013 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40]   File "/opt/stack/nova/nova/compute/manager.py", line 1975, in _build_and_run_instance
2017-01-19 06:15:03.013 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40]     reason=e.format_message())
2017-01-19 06:15:03.013 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40] BuildAbortException: Build of instance 15734ad5-a957-41e7-aab7-6278e3338d40 aborted: Image  is unacceptable: Bad Image format: Image  could not be found.
2017-01-19 06:15:03.013 TRACE nova.compute.manager [instance: 15734ad5-a957-41e7-aab7-6278e3338d40]
tpouyer commented 7 years ago

Do you get the same error if you log into the horizon console (http://192.168.27.100 username: demo password: labstack) and try to launch a node from there? Which image are you trying to deploy?

niklaushirt commented 7 years ago

Yep, unfortunately that's exactly what I'm doing. I launch one of the two Ubuntu Images 1404 or 1604 and get the same result. Same thing when using admin.

tpouyer commented 7 years ago

couple of questions:

  1. what OS are you running this on? Windows, Mac, Linux, etc...
  2. what version of the "stackinabox/openstack" vagrant box are you running? e.g. vagrant box list you should see something like this: stackinabox/openstack (virtualbox, 0.9.7)
niklaushirt commented 7 years ago

Mac OS Sierra 11.12.2 on all three machines I tried this on.

Boxes are as follows: stackina-base-box (virtualbox, 0) stackinabox/opdk (virtualbox, 0.9.3) stackinabox/opdk (virtualbox, 0.9.6) stackinabox/openstack (virtualbox, 0.9.7) ubuntu/trusty64 (virtualbox, 20160304.0.1) ubuntu/trusty64 (virtualbox, 20160314.0.1) ubuntu/xenial64 (virtualbox, 20161209.0.0)

tpouyer commented 7 years ago

Can you run the 14.04 image successfully?

niklaushirt commented 7 years ago

Still not: vagrant@openstack:~$ lxc launch ubuntu:16.04 my-ubuntu gives me 

Creating my-ubuntu
Retrieving image: 100%error: Failed to create ZFS filesystem: cannot mount '/var/lib/lxd/images': directory is not empty

But only the first time. The second time it works:

vagrant@openstack:~$ lxc launch ubuntu:16.04 my-ubuntu
Creating my-ubuntu
Starting my-ubuntu

I can download the image from glance without a problem:

vagrant@openstack:~$ glance image-download --file ubuntu1404.tar.xz b169e10a-9c66-449f-84d5-939a3d2666d6
vagrant@openstack:~$ ls
ubuntu1404.tar.xz

But the Instance creation in OpenStack still continuous to fail...

tpouyer commented 7 years ago

can you try running the ubuntu-14.04-amd64 image from the horizon console and let me know if that work for you or not.

sudhakarau1 commented 7 years ago

Just FYI running 0.9.8 on Vagrant 1.9.1, VirtualBox-5.1-5.1.14_112924 on RHEL 7.2 (3.10.x kernel) this is fixed for me, I can launch both 14.04 and 16.04 fine.

tpouyer commented 7 years ago

@sudhakarau1 Thanks for figuring that out! I will update the Vagrantfile to require Vagrant 1.9.1 or higher.

@niklaushirt please upgrade your Vagrant and Virtualbox to 5.1.14 and let me know if it works on your mac.

tpouyer commented 7 years ago

@niklaushirt Can you verify that upgrading your Vagrant to 1.9.1 and your VirtualBox to 5.1.14 on Mac OS X resolves this issue for you? I'd like to close this one out if I can.

niklaushirt commented 7 years ago

Nope, still got the same issue with Vagrant 1.9.1, VB 5.1.14 and stackinabox 0.99. Mystery, I'll have to play some more with my Macs configuration...

tpouyer commented 7 years ago

@niklaushirt damn, let's debug this together in Madrid

tpouyer commented 7 years ago

After discussing with @niklaushirt we have found a work-around for this issue. You must ensure that your environment is setup to use US English localization. I am working on determining why this is happening in @niklaushirt case because he was having the issue when running stackinabox under his native Swedish localization.