Oracle: cloud-init openstack local detection too strict for oracle cloud

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1784685

Launchpad details

affected_projects = ['cloud-init (Ubuntu)', 'cloud-init (Ubuntu Xenial)', 'cloud-init (Ubuntu Bionic)']
assignee = chad.smith
assignee_name = Chad Smith
date_closed = 2018-10-02T21:13:35.377862+00:00
date_created = 2018-07-31T17:47:51.962921+00:00
date_fix_committed = 2018-07-31T18:44:18.234309+00:00
date_fix_released = 2018-10-02T21:13:35.377862+00:00
id = 1784685
importance = high
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1784685
milestone = None
owner = chad.smith
owner_name = Chad Smith
private = False
status = fix_released
submitter = chad.smith
submitter_name = Chad Smith
tags = ['regression-proposed', 'verification-done', 'verification-done-bionic', 'verification-done-xenial']
duplicates = []

Launchpad user Chad Smith(chad.smith) wrote on 2018-07-31T17:47:51.962921+00:00

=== Begin SRU Template === [Impact] Cloud-init should detect OpenStack datasource on any OracleCloud instance. Per the bug, it was falling through the DataSourceNone after upgrade to 18.3.9 and a system reboot.

[Test Case]

Deploy an Oracle cloud bionic instance and validate upgrade/reboot path

cat > setup_proposed.sh <<EOF

/bin/bash

mirror=http://archive.ubuntu.com/ubuntu echo deb \$mirror \$(lsb_release -sc)-proposed main | tee /etc/apt/sources.list.d/proposed.list apt-get update -q; apt-get install -qy cloud-init; EOF

for vm in '129.146.86.46'; do    echo '=== BEGIN ' $vm ' ==='    ssh ubuntu@$vm grep CODENAME /etc/os-release;    ssh ubuntu@$vm -- dpkg-query --show cloud-init;    ssh ubuntu@$vm -- cloud-init status --long;    ssh ubuntu@$vm -- cloud-init --version;    ssh ubuntu@$vm -- cloud-init analyze show;    scp setup_proposed.sh ubuntu@$vm:.;    ssh ubuntu@$vm sudo bash ./setup_proposed.sh 2>&1 | egrep 'cloud-init';    ssh ubuntu@$vm -- sudo cloud-init clean --logs --reboot;    echo "After clean reboot, upgrade 18.3.9 should detect OpenStackLocal datasource"    ssh-keygen -f ".ssh/known_hosts" -R $vm;    ssh ubuntu@$vm -- cloud-init status --long;    ssh ubuntu@$vm -- cloud-init --version; done

[Regression Potential] This fixes a critical issue in datasource detection on Oracle platforms only. This should not regress any other platforms and is only a minor code path change to include Oracle's DMI chassis asset tag as a valid OpenStack datasource type.

[Other Info] Upstream commit at   https://git.launchpad.net/cloud-init/commit/?id=0df2b42

=== End SRU Template ===

=== Original Description ===

cloud-init 18.3 cannot detect OpenStack datasource on Oracle cloud across reboots.

18.2 properly detects DataSourceOpenstack, but 18.3 added a detect_openstack helper function which fails to detect openstack datasource from the environment or DMI data matching a limited set of conditions:

  - DMI system-product_name in ['OpenStack Nova', 'OpenStack Compute']       OracleCloud product-name is 'Standard PC (i440FX + PIIX, 1996)'   - DMI chassis_asset_tag in ['OpenTelekomCloud']          Oracle's chassis asset tag is 'OracleCloud.com'

  - proc/1/environ:product_name == 'OpenStack Nova'        Oracle's /proc/1/environ has no product_name key: $ sudo cat /proc/1/environ HOME=/init=/sbin/initrecovery=TERM=linuxdrop_caps=BOOT_IMAGE=/boot/vmlinuz-4.15.0-29-genericPATH=/sbin:/usr/sbin:/bin:/usr/binPWD=/rootmnt=/root

We need a more permissive detect_openstack function to detect Oracle during cloud-init's get_data() method.

[ubuntu-server-builder]

Launchpad user Server Team CI bot(server-team-bot) wrote on 2018-07-31T18:44:17.040376+00:00

This bug is fixed with commit 3cee0bf8 to cloud-init on branch master. To view that commit see the following URL: https://git.launchpad.net/cloud-init/commit/?id=3cee0bf8

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2018-07-31T20:24:46.264044+00:00

fixed in cosmic at 18.3-18-g3cee0bf8-0ubuntu1

[ubuntu-server-builder]

Launchpad user Chris Halse Rogers(raof) wrote on 2018-08-01T01:34:05.122681+00:00

Hello Chad, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/18.3-9-g2e62cb8a-0ubuntu1~16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[ubuntu-server-builder]

Launchpad user Chris Halse Rogers(raof) wrote on 2018-08-01T01:55:58.942027+00:00

Hello Chad, or anyone else affected,

Accepted cloud-init into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/18.3-9-g2e62cb8a-0ubuntu1~18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[ubuntu-server-builder]

Launchpad user Chad Smith(chad.smith) wrote on 2018-08-01T19:55:54.075366+00:00

SRU regresion validation logs for Oracle bionic and xenial Launchpad attachments: lp-1784685.txt

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2018-08-08T01:56:21.499600+00:00

This bug was fixed in the package cloud-init - 18.3-9-g2e62cb8a-0ubuntu1~18.04.2

---

cloud-init (18.3-9-g2e62cb8a-0ubuntu1~18.04.2) bionic-proposed; urgency=medium

• cherry-pick 3cee0bf8: oracle: fix detect_openstack to report True on (LP: #1784685)

cloud-init (18.3-9-g2e62cb8a-0ubuntu1~18.04.1) bionic-proposed; urgency=medium

• New upstream snapshot. (LP: #1777912)
  • docs: note in rtd about avoiding /tmp when writing files
  • ubuntu,centos,debian: get_linux_distro to align with platform.dist
  • Fix boothook docs on environment variable name (INSTANCE_I -> INSTANCE_ID) [Marc Tamsky]
  • update_metadata: a datasource can support network re-config every boot
  • tests: drop salt-minion integration test
  • Retry on failed import of gpg receive keys.
  • tools: Fix run-container when neither source or binary package requested.
  • docs: Fix a small spelling error. [Oz N Tiram]
  • tox: use simplestreams from git repository rather than bzr.

cloud-init (18.3-0ubuntu1~18.04.1) bionic-proposed; urgency=medium

• debian/rules: update version.version_string to contain packaged version. (LP: #1770712)
• debian/patches/openstack-no-network-config.patch add patch to ignore Openstack network_config from network_data.json by default

• New upstream release. (LP: #1777912)

  • release 18.3
  • docs: represent sudo:false in docs for user_groups config module
  • Explicitly prevent sudo access for user module [Jacob Bednarz]
  • lxd: Delete default network and detach device if lxd-init created them.
  • openstack: avoid unneeded metadata probe on non-openstack platforms
  • stages: fix tracebacks if a module stage is undefined or empty [Robert Schweikert]
  • Be more safe on string/bytes when writing multipart user-data to disk.
  • Fix get_proc_env for pids that have non-utf8 content in environment.
  • tests: fix salt_minion integration test on bionic and later
  • tests: provide human-readable integration test summary when --verbose
  • tests: skip chrony integration tests on lxd running artful or older
  • test: add optional --preserve-instance arg to integraiton tests
  • netplan: fix mtu if provided by network config for all rendered types
  • tests: remove pip install workarounds for pylxd, take upstream fix.
  • subp: support combine_capture argument.
  • tests: ordered tox dependencies for pylxd install
  • util: add get_linux_distro function to replace platform.dist [Robert Schweikert]
  • pyflakes: fix unused variable references identified by pyflakes 2.0.0.
  • • Do not use the systemd_prefix macro, not available in this environment [Robert Schweikert]
  • doc: Add config info to ec2, openstack and cloudstack datasource docs
  • Enable SmartOS network metadata to work with netplan via per-subnet routes [Dan McDonald]
  • openstack: Allow discovery in init-local using dhclient in a sandbox.
  • tests: Avoid using https in httpretty, improve HttPretty test case.
  • yaml_load/schema: Add invalid line and column nums to error message
  • Azure: Ignore NTFS mount errors when checking ephemeral drive [Paul Meyer]
  • packages/brpm: Get proper dependencies for cmdline distro.
  • packages: Make rpm spec files patch in package version like in debs.
  • tools/run-container: replace tools/run-centos with more generic.
  • Update version.version_string to contain packaged version.
  • cc_mounts: Do not add devices to fstab that are already present. [Lars Kellogg-Stedman]
  • ds-identify: ensure that we have certain tokens in PATH.
  • tests: enable Ubuntu Cosmic in integration tests [Joshua Powers]
  • read_file_or_url: move to url_helper, fix bug in its FileResponse.
  • cloud_tests: help pylint
  • flake8: fix flake8 errors in previous commit.
  • typos: Fix spelling mistakes in cc_mounts.py log messages [Stephen Ford]
  • tests: restructure SSH and initial connections [Joshua Powers]
  • ds-identify: recognize container-other as a container, test SmartOS.
  • cloud-config.service: run After snap.seeded.service.
  • tests: do not rely on host /proc/cmdline in test_net.py [Lars Kellogg-Stedman]
  • ds-identify: Remove dupe call to is_ds_enabled, improve debug message.
  • SmartOS: fix get_interfaces for nics that do not have addr_assign_type.
  • tests: fix package and ca_cert cloud_tests on bionic
  • ds-identify: make shellcheck 0.4.6 happy with ds-identify.
  • pycodestyle: Fix deprecated string literals, move away from flake8.
  • azure: Add reported ready marker file. [Joshua Chan]
  • tools: Support adding a release suffix through packages/bddeb.
  • FreeBSD: Invoke growfs on ufs filesystems such that it does not prompt. [Harm Weites]
  • tools: Re-use the orig tarball in packages/bddeb if it is around.
  • netinfo: fix netdev_pformat when a nic does not have an address assigned.
  • collect-logs: add -v flag, write to stderr, limit journal to single boot.
  • IBMCloud: Disable config-drive and nocloud only if IBMCloud is enabled.
  • Add reporting events and log_time around early source of blocking time

  -- Chad Smith chad.smith@canonical.com Tue, 31 Jul 2018 12:58:16 -0600

[ubuntu-server-builder]

Launchpad user Chris Halse Rogers(raof) wrote on 2018-08-08T01:56:39.687829+00:00

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2018-08-08T01:57:22.720631+00:00

This bug was fixed in the package cloud-init - 18.3-9-g2e62cb8a-0ubuntu1~16.04.2

---

cloud-init (18.3-9-g2e62cb8a-0ubuntu1~16.04.2) xenial-proposed; urgency=medium

• cherry-pick 3cee0bf8: oracle: fix detect_openstack to report True on (LP: #1784685)

cloud-init (18.3-9-g2e62cb8a-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

• New upstream snapshot. (LP: #1777912)
  • docs: note in rtd about avoiding /tmp when writing files
  • ubuntu,centos,debian: get_linux_distro to align with platform.dist
  • Fix boothook docs on environment variable name (INSTANCE_I -> INSTANCE_ID) [Marc Tamsky]
  • update_metadata: a datasource can support network re-config every boot
  • tests: drop salt-minion integration test
  • Retry on failed import of gpg receive keys.
  • tools: Fix run-container when neither source or binary package requested.
  • docs: Fix a small spelling error. [Oz N Tiram]
  • tox: use simplestreams from git repository rather than bzr.

cloud-init (18.3-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

• debian/rules: update version.version_string to contain packaged version. (LP: #1770712)
• debian/patches/openstack-no-network-config.patch add patch to ignore Openstack network_config from network_data.json by default
• Refresh patches against upstream:
  • azure-use-walinux-agent.patch
  • ds-identify-behavior-xenial.patch

• New upstream release. (LP: #1777912)

  • release 18.3
  • docs: represent sudo:false in docs for user_groups config module
  • Explicitly prevent sudo access for user module [Jacob Bednarz]
  • lxd: Delete default network and detach device if lxd-init created them.
  • openstack: avoid unneeded metadata probe on non-openstack platforms
  • stages: fix tracebacks if a module stage is undefined or empty [Robert Schweikert]
  • Be more safe on string/bytes when writing multipart user-data to disk.
  • Fix get_proc_env for pids that have non-utf8 content in environment.
  • tests: fix salt_minion integration test on bionic and later
  • tests: provide human-readable integration test summary when --verbose
  • tests: skip chrony integration tests on lxd running artful or older
  • test: add optional --preserve-instance arg to integraiton tests
  • netplan: fix mtu if provided by network config for all rendered types
  • tests: remove pip install workarounds for pylxd, take upstream fix.
  • subp: support combine_capture argument.
  • tests: ordered tox dependencies for pylxd install
  • util: add get_linux_distro function to replace platform.dist [Robert Schweikert]
  • pyflakes: fix unused variable references identified by pyflakes 2.0.0.
  • • Do not use the systemd_prefix macro, not available in this environment [Robert Schweikert]
  • doc: Add config info to ec2, openstack and cloudstack datasource docs
  • Enable SmartOS network metadata to work with netplan via per-subnet routes [Dan McDonald]
  • openstack: Allow discovery in init-local using dhclient in a sandbox.
  • tests: Avoid using https in httpretty, improve HttPretty test case.
  • yaml_load/schema: Add invalid line and column nums to error message
  • Azure: Ignore NTFS mount errors when checking ephemeral drive [Paul Meyer]
  • packages/brpm: Get proper dependencies for cmdline distro.
  • packages: Make rpm spec files patch in package version like in debs.
  • tools/run-container: replace tools/run-centos with more generic.
  • Update version.version_string to contain packaged version.
  • cc_mounts: Do not add devices to fstab that are already present. [Lars Kellogg-Stedman]
  • ds-identify: ensure that we have certain tokens in PATH.
  • tests: enable Ubuntu Cosmic in integration tests [Joshua Powers]
  • read_file_or_url: move to url_helper, fix bug in its FileResponse.
  • cloud_tests: help pylint
  • flake8: fix flake8 errors in previous commit.
  • typos: Fix spelling mistakes in cc_mounts.py log messages [Stephen Ford]
  • tests: restructure SSH and initial connections [Joshua Powers]
  • ds-identify: recognize container-other as a container, test SmartOS.
  • cloud-config.service: run After snap.seeded.service.
  • tests: do not rely on host /proc/cmdline in test_net.py [Lars Kellogg-Stedman]
  • ds-identify: Remove dupe call to is_ds_enabled, improve debug message.
  • SmartOS: fix get_interfaces for nics that do not have addr_assign_type.
  • tests: fix package and ca_cert cloud_tests on bionic
  • ds-identify: make shellcheck 0.4.6 happy with ds-identify.
  • pycodestyle: Fix deprecated string literals, move away from flake8.
  • azure: Add reported ready marker file. [Joshua Chan]
  • tools: Support adding a release suffix through packages/bddeb.
  • FreeBSD: Invoke growfs on ufs filesystems such that it does not prompt. [Harm Weites]
  • tools: Re-use the orig tarball in packages/bddeb if it is around.
  • netinfo: fix netdev_pformat when a nic does not have an address assigned.
  • collect-logs: add -v flag, write to stderr, limit journal to single boot.
  • IBMCloud: Disable config-drive and nocloud only if IBMCloud is enabled.
  • Add reporting events and log_time around early source of blocking time
  • IBMCloud: recognize provisioning environment during debug boots.
  • net: detect unstable network names and trigger a settle if needed
  • IBMCloud: improve documentation in datasource.
  • sysconfig: dhcp6 subnet type should not imply dhcpv4 [Vitaly Kuznetsov]
  • packages/debian/control.in: add missing dependency on iproute2.
  • DataSourceSmartOS: add locking of serial device. [Mike Gerdts]
  • DataSourceSmartOS: sdc:hostname is ignored [Mike Gerdts]
  • DataSourceSmartOS: list() should always return a list [Mike Gerdts]
  • schema: in validation, raise ImportError if strict but no jsonschema.
  • set_passwords: Add newline to end of sshd config, only restart if updated.
  • pylint: pay attention to unused variable warnings.
  • doc: Add documentation for AliYun datasource. [Junjie Wang]
  • Schema: do not warn on duplicate items in commands.
  • net: Depend on iproute2's ip instead of net-tools ifconfig or route
  • DataSourceSmartOS: fix hang when metadata service is down [Mike Gerdts]
  • DataSourceSmartOS: change default fs on ephemeral disk from ext3 to ext4. [Mike Gerdts]
  • pycodestyle: Fix invalid escape sequences in string literals.
  • Implement bash completion script for cloud-init command line
  • tools: Fix make-tarball cli tool usage for development
  • renderer: support unicode in render_from_file.
  • Implement ntp client spec with auto support for distro selection
  • Apport: add Brightbox, IBM, LXD, and OpenTelekomCloud to list of clouds.
  • tests: fix ec2 integration network metadata validation

  -- Chad Smith chad.smith@canonical.com Tue, 31 Jul 2018 13:57:21 -0600

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2018-10-02T21:13:37.418281+00:00

This bug is believed to be fixed in cloud-init in version 18.4. If this is still a problem for you, please make a comment and set the state back to New

Thank you.