Cloud-Init fails to deal with SWAP and Ephemeral if virtio-scsi is enabled

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1705346

Launchpad details

affected_projects = []
assignee = None
assignee_name = None
date_closed = None
date_created = 2017-07-19T20:47:19.153070+00:00
date_fix_committed = None
date_fix_released = None
id = 1705346
importance = medium
is_complete = False
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1705346
milestone = None
owner = martinx
owner_name = Thiago Martins
private = False
status = confirmed
submitter = martinx
submitter_name = Thiago Martins
tags = []
duplicates = []

Launchpad user Thiago Martins(martinx) wrote on 2017-07-19T20:47:19.153070+00:00

Hello!

In my OpenStack environment, everything works great! Including Cloud-Init.

However, I can not use virtio-blk anymore (i.e., /dev/vdX devices), because it is too slow.

I changed to virtio-scsi (i.e., /dev/sdX devices) and the IOPS is more than double!

But, after this change in Glance images, Cloud-Init is failing to make the SWAP and to mount the Ephemeral volumes.

• Here is a normal log (with virtio-blk):

  http://paste.ubuntu.com/25127354/

  So, after changing my Ubuntu image properties on Glance, like this:

  glance image-update --property hw_scsi_model=virtio-scsi --property hw_disk_bus=scsi $UBUNTU_IMAGE_ID

  Cloud Init is now failing to make the SWAP, also Ephemeral is not mounted!

• Here is the error log (with virtio-scsi):

  http://paste.ubuntu.com/25127989/

  Image used in Glance:

  http://uec-images.ubuntu.com/releases/16.04/release/ubuntu-16.04-server-cloudimg-amd64-disk1.img

  If I bring back virtio-blk, cloud-init works again (same Image, same Flavor).

  But, I can't use virtio-blk because it is very slow in my system! The virtio-scsi rocks!

Cheers! Thiago

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2017-07-19T21:57:29.013301+00:00

Can you please collect the config drive that was used ?

at very least we want to get the metadata off it. but just as easy to grab the whole thing.

[ubuntu-server-builder]

Launchpad user Thiago Martins(martinx) wrote on 2017-07-19T22:37:27.531819+00:00

This is the config driver used on this tests. Launchpad attachments: The config.disk (Config Drive)

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2017-07-20T13:56:14.013760+00:00

Thiago, thanks for attaching the config drive.

So here is the net of the problem... (Description as much for me and other cloud-init developers as for you).

On openstack, to determine block device mappings (mapping a string like "ephemeral0" or "swap" to a real block device like /dev/sdb or /dev/sda) cloud-init utilizes the ec2 metadata information present alongside the openstack metadata. That is because openstack doesn't provide any such data.

The ec2 metadata in /ec2/meta-data.json says: "block-device-mapping": {"ami": "sda", "root": "/dev/sda", "swap": "/dev/sdc", "ephemeral0": "/dev/sdb"}

which is clearly wrong.

If that information were not present at all, then cloud-init would attempt to look the info up by a LABEL on the filesystem (which is less than ideal but would probably work in this case).

That code is at https://git.launchpad.net/~cloud-init-dev/cloud-init/tree/cloudinit/sources/helpers/openstack.py?h=master#n129

[ubuntu-server-builder]

Launchpad user Thiago Martins(martinx) wrote on 2017-07-20T18:29:25.889872+00:00

Hmm... So, do you think that this is an OpenStack Nova bug instead?

Do you know if there is a good workaround?

Also, why are you saying that this:

- "block-device-mapping": {"ami": "sda", "root": "/dev/sda", "swap": "/dev/sdc", "ephemeral0": "/dev/sdb"}

Is clearly wrong?

It looks okay to me...

How you think that it must looks like?

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2018-09-13T19:41:00.519062+00:00

I said 'clearly wrong' because the deivce is probably not called '/dev/sdb'. I guess it might be but it could just as well be 'vdb'.

[ubuntu-server-builder]

Launchpad user Thiago Martins(martinx) wrote on 2019-06-19T19:51:33.571946+00:00

Hey Scott,

I completely forgot to update this bug report.

This is still a problem with Ubuntu 18.04!

So, the devices are RIGHT, they're supposed to be called "sda/sdb/sdc" and NOT "vda/vdb/vdc", since I'm using VirtIO-SCSI instead.

Then, this is a Cloud-Init bug, right?

Cheers! Thiago

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2019-06-21T16:11:44.452493+00:00

Could you attach the output of $ sudo blkid --cache=/dev/null and also attach the output of running: $ cloud-init collect-logs