rcbops / chef-cookbooks

RCB OPS - Chef Cookbooks
Other
118 stars 102 forks source link

[nova] file injection broken #813

Closed cjoelrun closed 10 years ago

cjoelrun commented 10 years ago

Nova can't boot an instance with file injection . Seen on precise

2014-02-27 09:08:13.110 31311 AUDIT nova.compute.manager [req-ae79ca56-cef9-4719-9195-1d03397ec865 e642301df2c940ee83dda8a933e02676 eb59b574afc048b5a159ab7ae2cc016e] [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] Starting instance...
2014-02-27 09:08:13.954 31311 AUDIT nova.compute.claims [req-ae79ca56-cef9-4719-9195-1d03397ec865 e642301df2c940ee83dda8a933e02676 eb59b574afc048b5a159ab7ae2cc016e] [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] Attempting claim: memory 512 MB, disk 1 GB, VCPUs 1
2014-02-27 09:08:13.954 31311 AUDIT nova.compute.claims [req-ae79ca56-cef9-4719-9195-1d03397ec865 e642301df2c940ee83dda8a933e02676 eb59b574afc048b5a159ab7ae2cc016e] [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] Total Memory: 24097 MB, used: 512.00 MB
2014-02-27 09:08:13.955 31311 AUDIT nova.compute.claims [req-ae79ca56-cef9-4719-9195-1d03397ec865 e642301df2c940ee83dda8a933e02676 eb59b574afc048b5a159ab7ae2cc016e] [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] Memory limit: 36145.50 MB, free: 35633.50 MB
2014-02-27 09:08:13.955 31311 AUDIT nova.compute.claims [req-ae79ca56-cef9-4719-9195-1d03397ec865 e642301df2c940ee83dda8a933e02676 eb59b574afc048b5a159ab7ae2cc016e] [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] Total Disk: 250 GB, used: 0.00 GB
2014-02-27 09:08:13.956 31311 AUDIT nova.compute.claims [req-ae79ca56-cef9-4719-9195-1d03397ec865 e642301df2c940ee83dda8a933e02676 eb59b574afc048b5a159ab7ae2cc016e] [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] Disk limit not specified, defaulting to unlimited
2014-02-27 09:08:13.956 31311 AUDIT nova.compute.claims [req-ae79ca56-cef9-4719-9195-1d03397ec865 e642301df2c940ee83dda8a933e02676 eb59b574afc048b5a159ab7ae2cc016e] [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] Total CPU: 12 VCPUs, used: 0.00 VCPUs
2014-02-27 09:08:13.957 31311 AUDIT nova.compute.claims [req-ae79ca56-cef9-4719-9195-1d03397ec865 e642301df2c940ee83dda8a933e02676 eb59b574afc048b5a159ab7ae2cc016e] [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] CPU limit: 192.00 VCPUs, free: 192.00 VCPUs
2014-02-27 09:08:13.957 31311 AUDIT nova.compute.claims [req-ae79ca56-cef9-4719-9195-1d03397ec865 e642301df2c940ee83dda8a933e02676 eb59b574afc048b5a159ab7ae2cc016e] [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] Claim successful
2014-02-27 09:08:14.342 31311 WARNING nova.network.neutronv2.api [-] [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] No network configured!
2014-02-27 09:08:14.483 31311 INFO nova.virt.libvirt.driver [req-ae79ca56-cef9-4719-9195-1d03397ec865 e642301df2c940ee83dda8a933e02676 eb59b574afc048b5a159ab7ae2cc016e] [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] Creating image
2014-02-27 09:08:17.312 31311 INFO nova.virt.libvirt.driver [req-ae79ca56-cef9-4719-9195-1d03397ec865 e642301df2c940ee83dda8a933e02676 eb59b574afc048b5a159ab7ae2cc016e] [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] Injecting metadata into image 9402ff72-47b8-4bc7-8cd0-66290068c49f
2014-02-27 09:08:17.312 31311 INFO nova.virt.libvirt.driver [req-ae79ca56-cef9-4719-9195-1d03397ec865 e642301df2c940ee83dda8a933e02676 eb59b574afc048b5a159ab7ae2cc016e] [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] Injecting files into image 9402ff72-47b8-4bc7-8cd0-66290068c49f
2014-02-27 09:08:17.583 31311 ERROR nova.virt.libvirt.driver [req-ae79ca56-cef9-4719-9195-1d03397ec865 e642301df2c940ee83dda8a933e02676 eb59b574afc048b5a159ab7ae2cc016e] [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] Error injecting data into image 9402ff72-47b8-4bc7-8cd0-66290068c49f (Error mounting /var/lib/nova/instances/d0c20c9f-467d-4e20-b987-08c832aa83af/disk with libguestfs (external command failed, see earlier error messages))
2014-02-27 09:08:17.583 31311 ERROR nova.compute.manager [req-ae79ca56-cef9-4719-9195-1d03397ec865 e642301df2c940ee83dda8a933e02676 eb59b574afc048b5a159ab7ae2cc016e] [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] Instance failed to spawn
2014-02-27 09:08:17.583 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] Traceback (most recent call last):
2014-02-27 09:08:17.583 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1417, in _spawn
2014-02-27 09:08:17.583 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]     block_device_info)
2014-02-27 09:08:17.583 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2063, in spawn
2014-02-27 09:08:17.583 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]     admin_pass=admin_password)
2014-02-27 09:08:17.583 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2494, in _create_image
2014-02-27 09:08:17.583 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]     instance=instance)
2014-02-27 09:08:17.583 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2488, in _create_image
2014-02-27 09:08:17.583 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]     mandatory=('files',))
2014-02-27 09:08:17.583 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]   File "/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py", line 325, in inject_data
2014-02-27 09:08:17.583 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]     fs.setup()
2014-02-27 09:08:17.583 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]   File "/usr/lib/python2.7/dist-packages/nova/virt/disk/vfs/guestfs.py", line 129, in setup
2014-02-27 09:08:17.583 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]     {'imgfile': self.imgfile, 'e': e})
2014-02-27 09:08:17.583 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] NovaException: Error mounting /var/lib/nova/instances/d0c20c9f-467d-4e20-b987-08c832aa83af/disk with libguestfs (external command failed, see earlier error messages)
2014-02-27 09:08:17.583 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] 
2014-02-27 09:08:17.652 31311 AUDIT nova.compute.manager [req-ae79ca56-cef9-4719-9195-1d03397ec865 e642301df2c940ee83dda8a933e02676 eb59b574afc048b5a159ab7ae2cc016e] [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] Terminating instance
2014-02-27 09:08:18.641 31311 ERROR nova.virt.libvirt.driver [-] [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] During wait destroy, instance disappeared.
2014-02-27 09:08:18.757 31311 INFO nova.virt.libvirt.driver [req-ae79ca56-cef9-4719-9195-1d03397ec865 e642301df2c940ee83dda8a933e02676 eb59b574afc048b5a159ab7ae2cc016e] [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] Deleting instance files /var/lib/nova/instances/d0c20c9f-467d-4e20-b987-08c832aa83af
2014-02-27 09:08:18.758 31311 INFO nova.virt.libvirt.driver [req-ae79ca56-cef9-4719-9195-1d03397ec865 e642301df2c940ee83dda8a933e02676 eb59b574afc048b5a159ab7ae2cc016e] [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] Deletion of /var/lib/nova/instances/d0c20c9f-467d-4e20-b987-08c832aa83af complete
2014-02-27 09:08:18.929 31311 ERROR nova.compute.manager [req-ae79ca56-cef9-4719-9195-1d03397ec865 e642301df2c940ee83dda8a933e02676 eb59b574afc048b5a159ab7ae2cc016e] [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] Error: Error mounting /var/lib/nova/instances/d0c20c9f-467d-4e20-b987-08c832aa83af/disk with libguestfs (external command failed, see earlier error messages)
2014-02-27 09:08:18.929 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] Traceback (most recent call last):
2014-02-27 09:08:18.929 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1037, in _build_instance
2014-02-27 09:08:18.929 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]     set_access_ip=set_access_ip)
2014-02-27 09:08:18.929 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1420, in _spawn
2014-02-27 09:08:18.929 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]     LOG.exception(_('Instance failed to spawn'), instance=instance)
2014-02-27 09:08:18.929 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1417, in _spawn
2014-02-27 09:08:18.929 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]     block_device_info)
2014-02-27 09:08:18.929 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2063, in spawn
2014-02-27 09:08:18.929 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]     admin_pass=admin_password)
2014-02-27 09:08:18.929 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2494, in _create_image
2014-02-27 09:08:18.929 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]     instance=instance)
2014-02-27 09:08:18.929 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2488, in _create_image
2014-02-27 09:08:18.929 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]     mandatory=('files',))
2014-02-27 09:08:18.929 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]   File "/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py", line 325, in inject_data
2014-02-27 09:08:18.929 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]     fs.setup()
2014-02-27 09:08:18.929 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]   File "/usr/lib/python2.7/dist-packages/nova/virt/disk/vfs/guestfs.py", line 129, in setup
2014-02-27 09:08:18.929 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af]     {'imgfile': self.imgfile, 'e': e})
2014-02-27 09:08:18.929 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] NovaException: Error mounting /var/lib/nova/instances/d0c20c9f-467d-4e20-b987-08c832aa83af/disk with libguestfs (external command failed, see earlier error messages)
2014-02-27 09:08:18.929 31311 TRACE nova.compute.manager [instance: d0c20c9f-467d-4e20-b987-08c832aa83af] 
claco commented 10 years ago

Closing. Can't reproduce now. Reopen if necessary for 4.2.3.

breu commented 10 years ago

I am going to reopen this for 4.2.2

breu commented 10 years ago

we need to default this to -2 to disable file injection. This can be done with cloud-init instead of cracking open the image. https://github.com/rcbops-cookbooks/nova/blob/master/attributes/default.rb#L127

@klevenstein doc impact on this one for 4.2.2

claco commented 10 years ago

Fixed in master: https://github.com/rcbops-cookbooks/nova/pull/395 Going into 4.2.2: https://github.com/rcbops-cookbooks/nova/pull/396

cjoelrun commented 10 years ago

This is not fixed with cloud-init. The EC2 provider does not support file injection of the OpenStack kind (--file).

The most sane way to do file injection is using a config drive instead of metadata.

nova boot --config_drive=true --file /root/openrc=/root/openrc --flavor 1 --image cirros-image
kaushikb commented 10 years ago

Reopening this so that the workaround/best practice is documented.

breu commented 10 years ago

--file in the nova cli and API is a OpenStack specific feature.

File injections work in one of three ways:

1) guestfs launches a VM with the instance and uses a communication path to copy the file to the instance. The instance is then stopped, and OpenStack starts a new one kvm process with the modified image file. 2) use cloud-init and config-drive to copy the file into the image. 3) user-data can be specified at instance invocation to supply a bash script that writes the file to the disk.

We feel that launching, copying, tearing down, and relaunching the instance increases the overhead on the hypervisor and can be prone to failure.

cloud-init can copy this file over correctly when using config-drive (or user-data).

We recommend using config-drive and cloud-init to copy files to the instance. This is supported on both RedHat, Ubuntu, cirros, and Windows (third party cloud-init). You can enforce the use of config-drive by setting the following override attribute:

"nova": {
  "config":
    "force_config_drive": true
  }
}
klevenstein commented 10 years ago

if this is a thing we're never going to fix, would it make more sense to put it in the actual install guide, rather than in the release notes?

breu commented 10 years ago

@klevenstein yes - but I think a short note in the release notes to define the change in behavior is still a good idea.

klevenstein commented 10 years ago

How does this sound:

File injection best practice When file injection is needed, Rackspace recommends using config-drive and cloud-init to copy files to an instance. When performed manually, set the --config-drive attribute to true in the nova boot command, as in the following example.

$ nova boot --config_drive=true --file /root/openrc=/root/openrc --flavor 1 --image cirros-image

You can also set an override attribute in your environment that enforces the use of config-drive at all times.

"nova": {
  "config":
    "force_config_drive": true
  }
}
Apsu commented 10 years ago

@klevenstein Close, but the paragraph shouldn't say "using config-drive and cloud-init to copy files", because this suggestion is to config-drive to do the file copying. cloud-init won't be in use when config-drive is enabled, so this bit of documentation is contrasting to the usual case of cloud-init.

klevenstein commented 10 years ago

Ah, okay. So instead: "When file injection is needed, Rackspace recommends using config-drive to copy files to an instance."

Apsu commented 10 years ago

Yep, +1 from me. @rackerjoe?

breu commented 10 years ago

@klevenstein go with your original statement. cloud-init uses config-drive to present and copy the files to the instance.

klevenstein commented 10 years ago

@rackerjoe how about "Rackspace recommends enabling config-drive so that cloud-init can copy files to an instance"? If that's the accurate description of what's going on here.

breu commented 10 years ago

@klevenstein That sounds good.

klevenstein commented 10 years ago

Added to release notes and instructions for instance creation in the Install/Getting Started guide.

https://one.rackspace.com/download/attachments/79960660/rackspace-private-cloud-releasenotes-v4.2-20140314.pdf?version=1&modificationDate=1394826996592&api=v2

https://one.rackspace.com/download/attachments/79960660/rackspace-private-cloud-installation-20140314.pdf?version=1&modificationDate=1394827018501&api=v2

cjoelrun commented 10 years ago

QE Verified. Looks good