search

canonical / cloud-init

Official upstream for the cloud-init: cloud instance initialization
https://cloud-init.io/
Other
2.9k stars 863 forks source link

cloud-init resizefs fails when booting with root=PARTUUID= #3036

Closed ubuntu-server-builder closed 1 year ago

ubuntu-server-builder commented 1 year ago

This bug was originally filed in Launchpad as LP: #1725067

Launchpad details 
affected_projects = ['cloud-init (Ubuntu)', 'cloud-init (Ubuntu Xenial)', 'cloud-init (Ubuntu Zesty)', 'cloud-init (Ubuntu Artful)']
assignee = chad.smith
assignee_name = Chad Smith
date_closed = 2017-10-24T03:25:01.312856+00:00
date_created = 2017-10-20T00:44:33.794030+00:00
date_fix_committed = 2017-10-24T03:25:01.312856+00:00
date_fix_released = 2017-10-24T03:25:01.312856+00:00
id = 1725067
importance = high
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1725067
milestone = None
owner = vorlon
owner_name = Steve Langasek
private = False
status = fix_released
submitter = vorlon
submitter_name = Steve Langasek
tags = ['regression-proposed', 'verification-done', 'verification-done-artful', 'verification-done-xenial', 'verification-done-zesty']
duplicates = []

Launchpad user Steve Langasek(vorlon) wrote on 2017-10-20T00:44:33.794030+00:00

http://pad.lv/1725067 https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1725067

=== Begin SRU Template === [Impact] Growing the root partition would not resize and leave a Traceback in /var/log/cloud-init.log if both:  a.) The device /dev/root did not exist      -- this case only hit xenial only because zesty already had a /dev/root created in the image so resize succeeded.  b.) the kernel command line included PARTUUID=      -- This potential SRU regression occurred because of code restructuring that still discovered (and logged) but didn't use the discovered kernel_cmdline root device path when resizing.

[Test Case] get-proposed-image is   https://github.com/cloud-init/qa-scripts/blob/master/scripts/get-proposed-cloudimg It downloads a cloud image of a given release, and then creates a -proposed image with cloud-init upgraded.

A script 'recreate.sh' will run each of the steps below automated.   https://github.com/cloud-init/ubuntu-sru/blob/master/bugs/lp-1684869/recreate.sh

NOTE: By default the images downloaded start off as 2Gig partitions, by requesting a 10G sparse qcow2 image resizefs should find the partition and resize it to the full 10Gig. If resizefs fails, we'd still be stuck with a 2GB response for "df -h /". This is what failed on our previous SRU proposal. The disk stayed at 2G instead of being resized to 10G.

1.) get a (proposed) disk image image.   and convert it to raw so you can read the partuuid with sfdisk   (get-proposed-image does this, if not,   'qemu-img convert -O raw orig.img orig.raw')   ./get-proposed-image

2.) get the partition uuid of the first partition    # for xenial images that are dos partition table rather than gpt    # we need to convert that with:    # sgdisk --mbrtogpt $raw    $ raw=yakkety-server-cloudimg-amd64-proposed.raw    $ ptuuid=$(sfdisk --part-uuid $raw 1)

3.) create a nocloud seed    $ printf "%s\n%s\n%s\n%s\n" "#cloud-config" "password: passw0rd" \         "chpasswd: {expire: False}" "ssh_pwauth: True" > my-user-data    $ echo "instance-id: $(uuidgen || echo i-abcdefg)" > my-meta-data    $ cloud-localds my-seed.img my-user-data my-meta-data

4.) extract kernel from inside the image    $ sudo mount-image-callback $raw -- mchroot sh -xc 'cat /boot/vmlinu?-*' > kernel

5.) boot instance with disk backed by the raw disk above.

   $ qemu-img create -f qcow2 -b $raw disk.img 10G    $ qemu-system-x86_64 -enable-kvm \        -drive file=disk.img,if=ide,index=0 -drive file=my-seed.img,if=ide \        -net nic -net user,hostfwd=tcp::2222-:22 \        -snapshot -m 768 -nographic -echr 0x05 \        -kernel kernel \        -append "root=PARTUUID=${ptuuid} ro console=tty1 console=ttyS0"

6.) log in, verify / has been resized.    log in with 'ubuntu' and password 'passw0rd'     $ df -h /     Filesystem Size Used Avail Use% Mounted on     /dev/root 9.6G 1009M 8.6G 11% /

[Regression Potential] Regressions would surface as the root filesystem not being correctly resized. The user would find themselves with not as much disk as expected.

[Other Info] The qemu-system-x86 command above uses ide devices. This is because the ide device emulated by qemu is built into the -generic kernel, while the more common virtio-block or virtio-scsi are not. If you attach those device types, it will fail with 'cant find root'.

Note that this was a regression of changes added in   bug 1684869: growing root partition does not always work with root=PARTUUID=   bug 1677376: growing partitions does not work when booted without initramfs

The issue probably is only seen if using the version of cloud-init in xenial-proposed. Zesty and artful kernels or userspace made the change actually not regress. However we will verify functionality for the uploaded version in each of x, z, a.

[Other Info] Upstream commit at   https://git.launchpad.net/cloud-init/commit/?id=17a15f9e0a

=== End SRU Template ===

A freshly built Ubuntu 17.10 image that's configured to boot without initramfs by passing root=PARTUUID= on the kernel commandline comes up degraded because cloud-init fails to resize the root partition.

Oct 20 00:07:30 ubuntu cloud-init[493]: 2017-10-20 00:07:30,656 - util.py[WARNING]: Failed to resize filesystem (cmd=('resize2fs', '/dev/root')) Oct 20 00:07:30 ubuntu cloud-init[493]: 2017-10-20 00:07:30,662 - util.py[WARNING]: Running module resizefs (<module 'cloudinit.config.cc_resizefs' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_resizefs.py'>) failed

Looking at the code, I see that there are two separate implementations of rootdev_from_cmdline, one in cloudinit/util.py and one in cloudinit/config/cc_resizefs.py; and the second does not handle partuuid.

from cloudinit.util import rootdev_from_cmdline rootdev_from_cmdline('BOOT_IMAGE=/boot/vmlinuz-4.4.0-1007-kvm root=PARTUUID=f122607f-3631-4411-bd34-de4bed76e0f7 ro console=tty1 console=ttyS0 systemd.log_level=debug') '/dev/disk/by-partuuid/f122607f-3631-4411-bd34-de4bed76e0f7' from cloudinit.config.cc_resizefs import rootdev_from_cmdline rootdev_from_cmdline('OT_IMAGE=/boot/vmlinuz-4.4.0-1007-kvm root=PARTUUID=f122607f-3631-4411-bd34-de4bed76e0f7 ro console=tty1 console=ttyS0 systemd.log_level=debug') '/dev/PARTUUID=f122607f-3631-4411-bd34-de4bed76e0f7'

This is related to bug #1684869; I'm not sure if it's the same bug reintroduced or if was never fixed properly on trunk (17.1).

Related bugs:   bug 1684869: growing root partition does not always work with root=PARTUUID=   bug 1685291: RFC: virtio and virtio-scsi should be built in  * bug 1677376: growing partitions does not work when booted without initramfs

ubuntu-server-builder commented 1 year ago

Launchpad user Steve Langasek(vorlon) wrote on 2017-10-20T00:47:01.530903+00:00

Actually, cc_resizefs.rootdev_from_cmdline exists but appears to be unused. I'm not sure why this is failing, then.

ubuntu-server-builder commented 1 year ago

Launchpad user Steve Langasek(vorlon) wrote on 2017-10-20T00:51:20.397859+00:00

This code from cc_resizefs.py in the 0.7.9 SRU is missing from 17.1 in artful.

container = util.is_container()

Ensure the path is a block device.

if (devpth == "/dev/root" and not os.path.exists(devpth) and
        not container):
    devpth = util.rootdev_from_cmdline(util.get_cmdline())
    if devpth is None:
        log.warn("Unable to find device '/dev/root'")
        return
    log.debug("Converted /dev/root to '%s' per kernel cmdline", devpth)
ubuntu-server-builder commented 1 year ago

Launchpad user Steve Langasek(vorlon) wrote on 2017-10-20T16:31:44.529495+00:00

This regression is present in the cloud-init 17.1-18-gd4f70470-0ubuntu1~16.04.2 SRU currently in xenial-proposed. Marking up the bugs appropriately to stop the line on that SRU, since this breaks our GAed 16.04 minimal images.

ubuntu-server-builder commented 1 year ago

Launchpad user Chris Halse Rogers(raof) wrote on 2017-10-25T00:14:38.710906+00:00

An upload of cloud-init to xenial-proposed has been rejected from the upload queue for the following reason: "Please re-upload including the changelog entries from the unreleased version in -proposed in the .changes file. Thanks!".

ubuntu-server-builder commented 1 year ago

Launchpad user Chris Halse Rogers(raof) wrote on 2017-10-25T00:30:18.947803+00:00

An upload of cloud-init to zesty-proposed has been rejected from the upload queue for the following reason: "Please re-upload including the changelog entries from the unreleased version in -proposed in the .changes file. Thanks!".

ubuntu-server-builder commented 1 year ago

Launchpad user Chris Halse Rogers(raof) wrote on 2017-10-25T00:34:19.405252+00:00

Hello Steve, or anyone else affected,

Accepted cloud-init into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/17.1-25-g17a15f9e-0ubuntu1~17.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

ubuntu-server-builder commented 1 year ago

Launchpad user Chris Halse Rogers(raof) wrote on 2017-10-25T03:21:26.677912+00:00

Hello Steve, or anyone else affected,

Accepted cloud-init into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/17.1-25-g17a15f9e-0ubuntu1~17.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

ubuntu-server-builder commented 1 year ago

Launchpad user Chris Halse Rogers(raof) wrote on 2017-10-25T03:30:12.715826+00:00

Hello Steve, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/17.1-25-g17a15f9e-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

ubuntu-server-builder commented 1 year ago

Launchpad user Chad Smith(chad.smith) wrote on 2017-10-25T19:10:27.961221+00:00

=== Abridged SRU verification output ==== ./recreate.sh xenial ... Preparing to unpack .../cloud-init_17.1-25-g17a15f9e-0ubuntu1~16.04.1_all.deb ... ...

login ubuntu/pwd

$ cat /run/cloud-init/result.json { "v1": { "datasource": "DataSourceNoCloud [seed=/dev/sdb][dsmode=net]", "errors": [] } } $ df -h / Filesystem Size Used Avail Use% Mounted on /dev/root 9.6G 979M 8.6G 11% /

$ grep Trace /var/log/cloud-init.log $

$ ./recreate.sh zesty ... Preparing to unpack .../cloud-init_17.1-25-g17a15f9e-0ubuntu1~17.04.1_all.deb ...

login ubuntu/pwd

$ cat /run/cloud-init/result.json { "v1": { "datasource": "DataSourceNoCloud [seed=/dev/sdb][dsmode=net]", "errors": [] } } $ df -h / Filesystem Size Used Avail Use% Mounted on /dev/root 9.6G 1023M 8.6G 11% / $ grep Trace /var/log/cloud-init.log $

$ ./recreate.sh artful ... Preparing to unpack .../cloud-init_17.1-25-g17a15f9e-0ubuntu1~17.10.1_all.deb ...

$ cat /run/cloud-init/result.json { "v1": { "datasource": "DataSourceNoCloud [seed=/dev/sdb][dsmode=net]", "errors": [] } } $ df -h / Filesystem Size Used Avail Use% Mounted on /dev/root 9.6G 1.1G 8.6G 11% / $ grep Trace /var/log/cloud-init.log $

=== End Abridged SRU verification output ====

ubuntu-server-builder commented 1 year ago

Launchpad user Launchpad Janitor(janitor) wrote on 2017-10-26T08:49:10.714791+00:00

This bug was fixed in the package cloud-init - 17.1-25-g17a15f9e-0ubuntu1~17.10.1

cloud-init (17.1-25-g17a15f9e-0ubuntu1~17.10.1) artful-proposed; urgency=medium

ubuntu-server-builder commented 1 year ago

Launchpad user Launchpad Janitor(janitor) wrote on 2017-11-02T17:55:22.957773+00:00

This bug was fixed in the package cloud-init - 17.1-25-g17a15f9e-0ubuntu1~17.10.1

cloud-init (17.1-25-g17a15f9e-0ubuntu1~17.10.1) artful-proposed; urgency=medium

ubuntu-server-builder commented 1 year ago

Launchpad user Brian Murray(brian-murray) wrote on 2017-11-02T17:55:43.410944+00:00

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

ubuntu-server-builder commented 1 year ago

Launchpad user Łukasz Zemczak(sil2100) wrote on 2017-11-07T17:08:16.644091+00:00

Hello Steve, or anyone else affected,

Accepted cloud-init into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/17.1-27-geb292c18-0ubuntu1~17.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

ubuntu-server-builder commented 1 year ago

Launchpad user Łukasz Zemczak(sil2100) wrote on 2017-11-07T17:18:42.525153+00:00

Hello Steve, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/17.1-27-geb292c18-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

ubuntu-server-builder commented 1 year ago

Launchpad user Chad Smith(chad.smith) wrote on 2017-11-07T21:09:09.438351+00:00

=== Abridged SRU verification output ==== ./recreate.sh xenial ... Preparing to unpack .../cloud-init_17.1-27-geb292c18-0ubuntu1~16.04.1_all.deb ... ...

login ubuntu/pwd

ubuntu@ubuntu:~$ cat /run/cloud-init/result.json { "v1": { "datasource": "DataSourceNoCloud [seed=/dev/sdb][dsmode=net]", "errors": [] } } ubuntu@ubuntu:~$ df -h / Filesystem Size Used Avail Use% Mounted on /dev/root 9.6G 996M 8.6G 11% / ubuntu@ubuntu:~$ grep Trace /var/log/cloud-init.log ubuntu@ubuntu:~$

$ ./recreate.sh zesty ... Preparing to unpack .../cloud-init_17.1-27-geb292c18-0ubuntu1~17.04.1_all.deb ...

login ubuntu/pwd

ubuntu@ubuntu:~$ cat /run/cloud-init/result.json { "v1": { "datasource": "DataSourceNoCloud [seed=/dev/sdb][dsmode=net]", "errors": [] } } ubuntu@ubuntu:~$ df -h / Filesystem Size Used Avail Use% Mounted on /dev/root 9.6G 1021M 8.6G 11% / ubuntu@ubuntu:~$ grep Trace /var/log/cloud-init.log ubuntu@ubuntu:~$

$ ./recreate.sh artful ... Preparing to unpack .../cloud-init_17.1-27-geb292c18-0ubuntu1~17.10.1_all.deb ...

ubuntu@ubuntu:~$ cat /run/cloud-init/result.json { "v1": { "datasource": "DataSourceNoCloud [seed=/dev/sdb][dsmode=net]", "errors": [] } } ubuntu@ubuntu:~$ df -h / Filesystem Size Used Avail Use% Mounted on /dev/root 9.6G 1.1G 8.5G 11% / ubuntu@ubuntu:~$ grep Trace /var/log/cloud-init.log ubuntu@ubuntu:~$

=== End Abridged SRU verification output ====

ubuntu-server-builder commented 1 year ago

Launchpad user Launchpad Janitor(janitor) wrote on 2017-11-14T22:23:59.287002+00:00

This bug was fixed in the package cloud-init - 17.1-27-geb292c18-0ubuntu1~17.04.1

cloud-init (17.1-27-geb292c18-0ubuntu1~17.04.1) zesty-proposed; urgency=medium

cloud-init (17.1-25-g17a15f9e-0ubuntu1~17.04.1) zesty-proposed; urgency=medium

cloud-init (17.1-18-gd4f70470-0ubuntu1~17.04.2) zesty-proposed; urgency=medium

cloud-init (17.1-18-gd4f70470-0ubuntu1~17.04.1) zesty-proposed; urgency=medium

ubuntu-server-builder commented 1 year ago

Launchpad user Launchpad Janitor(janitor) wrote on 2017-11-15T01:07:42.077466+00:00

This bug was fixed in the package cloud-init - 17.1-27-geb292c18-0ubuntu1~16.04.1

cloud-init (17.1-27-geb292c18-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

cloud-init (17.1-25-g17a15f9e-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

cloud-init (17.1-18-gd4f70470-0ubuntu1~16.04.2) xenial-proposed; urgency=medium

cloud-init (17.1-18-gd4f70470-0ubuntu1~16.04.1) xenial-proposed; urgency=medium