search

canonical / cloud-init

Official upstream for the cloud-init: cloud instance initialization
https://cloud-init.io/
Other
2.92k stars 871 forks source link

cloudconfig not writing maas data source #3873

Closed ubuntu-server-builder closed 1 year ago

ubuntu-server-builder commented 1 year ago

This bug was originally filed in Launchpad as LP: #1927020

Launchpad details 
affected_projects = ['curtin']
assignee = None
assignee_name = None
date_closed = 2021-05-13T15:26:52.382276+00:00
date_created = 2021-05-03T22:29:27.716874+00:00
date_fix_committed = None
date_fix_released = None
id = 1927020
importance = undecided
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1927020
milestone = None
owner = cbnorman
owner_name = Chris Norman
private = False
status = invalid
submitter = cbnorman
submitter_name = Chris Norman
tags = []
duplicates = []

Launchpad user Chris Norman(cbnorman) wrote on 2021-05-03T22:29:27.716874+00:00

further background https://discourse.maas.io/t/debian10-fails-on-final-reboot/4486

I'm deploying a debian 10 buster image using MAAS. No errors are reported in MAAS or curtin install but on the final boot cloud-init reports Failed to load metadata and userdata. checking the config for the machine in maas shows all the cloudconfig: to connect to maas but the files with oauth credentials etc dont seem to have been copied to the target.

ubuntu-server-builder commented 1 year ago

Launchpad user Chris Norman(cbnorman) wrote on 2021-05-03T22:29:27.716874+00:00

Launchpad attachments: cloud-init.tar.gz

ubuntu-server-builder commented 1 year ago

Launchpad user James Falcon(falcojr) wrote on 2021-05-05T14:02:29.931267+00:00

Looking at the logs, a few things stand out:

I'm going to set the state of this bug to Incomplete for now. If you can provide more information to show that cloud-init isn't doing what you expected, feel free to set the status back to New.

ubuntu-server-builder commented 1 year ago

Launchpad user Scott Moser(smoser) wrote on 2021-05-05T15:37:29.548163+00:00

When I looked at this, the most obvious thing to me was that curtin did not write a file in /etc/cloud/cloud.cfg.d/ that configured the MAAS datasource.

A quick look at curtin didn't remind me how that file gets written in ubuntu. https://github.com/canonical/curtin/blob/204728a21d1b0a6733a4f229d759b9ab324d9258/curtin/commands/curthooks.py#L1464

The 'cloud-config' entries provided by MAAS (https://discourse.maas.io/t/debian10-fails-on-final-reboot/4486/5) should then get written into /etc/cloud/cloud.cfg.d/ . If they aren't written, then the installed system won't recognize it is configured for maas.

ubuntu-server-builder commented 1 year ago

Launchpad user Chris Norman(cbnorman) wrote on 2021-05-06T10:09:44.048821+00:00

thanks for looking into this and agreed the root problem seems to be the files are not written by curtin. I dont see any errors in the curtin log, do you know why that handle_cloudconfig function would not be run?

ubuntu-server-builder commented 1 year ago

Launchpad user Ryan Harper(raharper) wrote on 2021-05-06T15:50:34+00:00

On Thu, May 6, 2021 at 5:15 AM Chris Norman 1927020@bugs.launchpad.net wrote:

thanks for looking into this and agreed the root problem seems to be the files are not written by curtin. I dont see any errors in the curtin log, do you know why that handle_cloudconfig function would not be run?

MAAS sends this information through the debconf_selections curtin config:

https://curtin.readthedocs.io/en/latest/topics/config.html#debconf-selections

And the Ubuntu cloud-init package has a postinst which parses these values and writes them out:

https://github.com/canonical/cloud-init/blob/ubuntu/daily/devel/debian/cloud-init.postinst#L54

-- You received this bug notification because you are subscribed to cloud- init. https://bugs.launchpad.net/bugs/1927020

Title: cloudconfig not writing maas data source

To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1927020/+subscriptions

ubuntu-server-builder commented 1 year ago

Launchpad user Lee Trager(ltrager) wrote on 2021-05-11T20:51:23.022469+00:00

As per discourse it looks like the bug is in Curtin. /etc/cloud/cloud.cfg.d/90_dpkg_maas.cfg is written then deleted by Curtin.

Applying debconf selections Running command ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmpihv32ogj/target', 'debconf-set-selections'] with allowed return codes [0] (capture=True) Running command ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmpihv32ogj/target', 'dpkg-query', '--list'] with allowed return codes [0] (capture=True) unconfiguring cloud-init cleaning cloud-init config from: ['/tmp/tmpihv32ogj/target/etc/cloud/cloud.cfg.d/90_dpkg_local_cloud_config.cfg', '/tmp/tmpihv32ogj/target/etc/cloud/cloud.cfg.d/90_dpkg_maas.cfg', '/tmp/tmpihv32ogj/target/etc/cloud/cloud.cfg.d/90_dpkg.cfg']

ubuntu-server-builder commented 1 year ago

Launchpad user Ryan Harper(raharper) wrote on 2021-05-11T22:31:53.400624+00:00

As per discourse it looks like the bug is in Curtin.

No

The point of removing any files in /etc/cloud/cloud.cfg.d is that dpkg-reconfigure WILL write out whatever values it needs from the debconf_selections.

So the two questions are:

1) in the maas config sent to this image does it include debconf_selections (I assume yes) 2) does the cloud-init package in the target filesystem include the cloudinit.postinstall from upstream cloud-init which handles the maas preseed ?

ubuntu-server-builder commented 1 year ago

Launchpad user Ryan Harper(raharper) wrote on 2021-05-11T22:36:39.397232+00:00

Let's confirm whether the debian buster cloud-init package in the image contains a postinstall like upstream cloud-init. If not, then that's the bug. If it's present, can we confirm the MAAS curtin config included the debconf_selections.

ubuntu-server-builder commented 1 year ago

Launchpad user Chris Norman(cbnorman) wrote on 2021-05-11T23:01:42.102051+00:00

i believe the maas config is sent to the image. here is the output from the maas server:

#maas admin machine get-curtin-config ykfy3r 
Success.
Machine-readable output follows:
apt:
  preserve_sources_list: false
  proxy: http://192.168.50.10:8000/
  sources_list: 'deb http://deb.debian.org/debian buster main

    deb-src http://deb.debian.org/debian buster main

    deb http://security.debian.org/debian-security buster/updates main

    deb-src http://security.debian.org/debian-security buster/updates main

    '
cloudconfig:
  maas-cloud-config:
    content: "#cloud-config\ndatasource:\n  MAAS:\n    consumer_key: X7HqXFtzruufCcfjnF\n\
      \    metadata_url: http://localhost:5240/MAAS/metadata/\n    token_key: M7VUBVr5KQwgVH3QZZ\n\
      \    token_secret: 8tJ8wUJNp6HxKSEtMwA8YyYtFzhpkGHL\n"
    path: /etc/cloud/cloud.cfg.d/90_maas_cloud_config.cfg
  maas-datasource:
    content: 'datasource_list: [ MAAS ]'
    path: /etc/cloud/cloud.cfg.d/90_maas_datasource.cfg
  maas-reporting:
    content: "#cloud-config\nreporting:\n  maas:\n    consumer_key: X7HqXFtzruufCcfjnF\n\
      \    endpoint: http://localhost:5240/MAAS/metadata/status/ykfy3r\n    token_key:\
      \ M7VUBVr5KQwgVH3QZZ\n    token_secret: 8tJ8wUJNp6HxKSEtMwA8YyYtFzhpkGHL\n \
      \   type: webhook\n"
    path: /etc/cloud/cloud.cfg.d/90_maas_cloud_init_reporting.cfg
  maas-ubuntu-sso:
    content: "#cloud-config\nsnappy:\n  email: admin@admin.com\n"
    path: /etc/cloud/cloud.cfg.d/90_maas_ubuntu_sso.cfg
debconf_selections:
  grub2: grub2   grub2/update_nvram  boolean false
  maas: 'cloud-init   cloud-init/datasources  multiselect MAAS

    cloud-init   cloud-init/maas-metadata-url  string http://localhost:5240/MAAS/metadata/

    cloud-init   cloud-init/maas-metadata-credentials  string oauth_consumer_key=X7HqXFtzruufCcfjnF&oauth_token_key=M7VUBVr5KQwgVH3QZZ&oauth_token_secret=8tJ8wUJNp6HxKSEtMwA8YyYtFzhpkGHL

    cloud-init   cloud-init/local-cloud-config  string manage_etc_hosts: true\nmanual_cache_clean:
    true\nreporting:\n  maas:\n    consumer_key: X7HqXFtzruufCcfjnF\n    endpoint:
    http://localhost:5240/MAAS/metadata/status/ykfy3r\n    token_key: M7VUBVr5KQwgVH3QZZ\n    token_secret:
    8tJ8wUJNp6HxKSEtMwA8YyYtFzhpkGHL\n    type: webhook\n

    '
install:
  error_tarfile: /tmp/curtin-logs.tar
  log_file: /tmp/install.log
  post_files:
  - /tmp/install.log
  - /tmp/curtin-logs.tar
kernel:
  fallback-package: linux-image-amd64
  package: linux-image-amd64
late_commands:
  datasource:
  - curtin
  - in-target
  - --
  - sh
  - -c
  - 'echo "datasource_list: [ MAAS ]" > /etc/cloud/cloud.cfg.d/90_dpkg.cfg'
  maas:
  - wget
  - --no-proxy
  - http://localhost:5240/MAAS/metadata/latest/by-id/ykfy3r/
  - --post-data
  - op=netboot_off
  - -O
  - /dev/null
network:
  config:
  - id: ens33
    mac_address: 00:0c:29:d7:99:89
    mtu: 1500
    name: ens33
    subnets:
    - address: 192.168.50.34/24
      dns_nameservers:
      - 192.168.50.10
      - 8.8.8.8
      dns_search: &id001
      - maas
      type: static
    type: physical
  - address:
    - 192.168.50.10
    search: *id001
    type: nameserver
  version: 1
network_commands:
  builtin:
  - curtin
  - net-meta
  - custom
partitioning_commands:
  builtin:
  - curtin
  - block-meta
  - custom
reporting:
  maas:
    consumer_key: X7HqXFtzruufCcfjnF
    endpoint: http://localhost:5240/MAAS/metadata/status/ykfy3r
    token_key: M7VUBVr5KQwgVH3QZZ
    token_secret: 8tJ8wUJNp6HxKSEtMwA8YyYtFzhpkGHL
    type: webhook
showtrace: true
storage:
  config:
  - grub_device: true
    id: sda
    name: sda
    path: /dev/sda
    ptable: gpt
    type: disk
    wipe: superblock
  - device: sda
    flag: bios_grub
    id: sda-part1
    number: 1
    offset: 4194304B
    size: 1048576B
    type: partition
    wipe: zero
  - device: sda
    flag: boot
    id: sda-part2
    name: sda-part2
    number: 2
    size: 536870912B
    type: partition
    uuid: 09527817-1706-415d-9c5e-b0b4aafdfe8f
    wipe: superblock
  - device: sda
    id: sda-part3
    name: sda-part3
    number: 3
    size: 20929576960B
    type: partition
    uuid: 6408531a-4fed-44d5-a7b0-9494e11c4ecb
    wipe: superblock
  - fstype: fat32
    id: sda-part2_format
    label: efi
    type: format
    uuid: c57cc8ab-50db-4345-b07d-6d3024e29a43
    volume: sda-part2
  - fstype: ext4
    id: sda-part3_format
    label: root
    type: format
    uuid: 06244900-d7a9-455a-a167-40845148af3c
    volume: sda-part3
  - device: sda-part3_format
    id: sda-part3_mount
    path: /
    type: mount
  - device: sda-part2_format
    id: sda-part2_mount
    path: /boot/efi
    type: mount
  version: 1
verbosity: 3
write_files:
  userconfig:
    content: "groups:\n  - docker\nusers:\n  - default\n  - name: buster\n    gecos:\
      \ Service Operator\n    primary_group: myuser\n    groups: [docker,sudo]\n \
      \   lock_passwd: false\n    sudo: ALL=(ALL) NOPASSWD:ALL\n    plain_text_passwd:\
      \ buster\n    shell: /bin/bash\n    ssh_authorized_keys:\n      - ecdsa-sha2-nistp521\
      \ AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBADGl/XFwwwDam1MoyQ3vmfJxaJVRx8ut7uOliGeUt/WzPytO2sx2QRsKxLt/tPnRbj4gJIMH2vhM0QwB1FEsribSAGOwd0bQuCGSpN9Vh+wPsp7xVoiOB5MWDgQGagmSY6XubnWP0pVfUd+78dJLg9ZDAHXOJlVzGHJX8jeslpJo3MIbw==\
      \ chrisnorman@C02XC3C7JG5L\n"
    path: /etc/cloud/cloud.cfg.d/00-users.cfg

if i trap the ephemeral boot image (ubuntu). cd into /curtin and run /var/lib/cloud/instance/scripts/part-001 it outputs:

./bin/curtin --install-deps install   --config=configs/config-000.cfg --config=configs/config-001.cfg   --config=configs/config-002.cfg --config=configs/config-003.cfg   --config=configs/config-004.cfg --config=configs/config-005.cfg --config=configs/config-006.cfg http://192.168.50.10:5248/images/custom/amd64/generic/debian1080/uploaded/root-tgz

In the /curtin/config/* files i can see the config from get-curtin-config.

how can i confirm cloudinit.postinstall exists in the target image?

ubuntu-server-builder commented 1 year ago

Launchpad user Chris Norman(cbnorman) wrote on 2021-05-11T23:56:00.744075+00:00

here is from the finished deployment

root@debian:~# cloud-init --version /usr/bin/cloud-init 20.2 root@debian:~# apt list cloud-init Listing... Done cloud-init/stable 20.2-2~deb10u2 all [upgradable from: 20.2-2~deb10u1] N: There is 1 additional version. Please use the '-a' switch to see it

ubuntu-server-builder commented 1 year ago

Launchpad user Ryan Harper(raharper) wrote on 2021-05-12T14:26:39.594557+00:00

@Chris

Thanks for the info. I see maas is sending the debconf selections I would expect.
For the postinst, you can view it on an installed system with cloud-init here:

/var/lib/dpkg/info/cloud-init.postinst

And if this node was deployed by maas, you can also query the debconf database to see what's set:

debconf-show cloud-init

ubuntu-server-builder commented 1 year ago

Launchpad user Chris Norman(cbnorman) wrote on 2021-05-12T17:05:05.807846+00:00

both seem to exist on the deployed machine

buster@debian:~$ ls /var/lib/dpkg/info/cloud-init.postinst /var/lib/dpkg/info/cloud-init.postinst buster@debian:~$ debconf-show cloud-init debconf: DbDriver "passwords" warning: could not open /var/cache/debconf/passwords.dat: Permission denied

ubuntu-server-builder commented 1 year ago

Launchpad user Ryan Harper(raharper) wrote on 2021-05-12T19:28:28.709880+00:00

What happens when you do:

sudo dpkg-reconfigure cloud-init --frontend=noninteractive

do you see /etc/cloud/cloud.cfg.d/ populated?

if not you can modify /var/lib/dpkg/info/cloud-init.postinst shebang to include execution trace:

!/bin/sh -x

Then re-run the command and capture the output. Maybe we can see why that's not rendering the maas cloud config files.

FWIW, I've extracted the debconf_selections into a file, populated debconf, then re-run dpkg-reconfigure cloud-init and see that it does write out these files in /etc/cloud/cloud.cfg.d/

https://paste.ubuntu.com/p/9yt8nyqBdR/

ubuntu-server-builder commented 1 year ago

Launchpad user Chris Norman(cbnorman) wrote on 2021-05-12T20:01:17.228960+00:00

not sure this is helpful?

root@debian:~# dpkg-reconfigure cloud-init --frontend=noninteractive root@debian:~# ls /etc/cloud/cloud.cfg.d/ 00_debian.cfg 00-users.cfg 05_logging.cfg 50-curtin-networking.cfg curtin-preserve-sources.cfg README root@debian:~# vi /var/lib/dpkg/info/cloud-init.postinst

changed shebang to #!/bin/sh -x

root@debian:~# dpkg-reconfigure cloud-init --frontend=noninteractive

ubuntu-server-builder commented 1 year ago

Launchpad user Ryan Harper(raharper) wrote on 2021-05-12T22:53:13.697897+00:00

Yeah, that sure looks like the maas preseed code in postinst isn't in the debian package, can you compare your file with:

https://github.com/canonical/cloud-init/blob/ubuntu/daily/devel/debian/cloud-init.postinst#L54

specifically, does it have the handle_preseed_maas() function? I suspect not, in which case you'd need that in your cloud-init package in debian for the image to be deployable via maas.

Alternatively, if maas does not send the debconf_selections, then the clean-up and dpkg-reconfigure path won't be taken in curtin.

ubuntu-server-builder commented 1 year ago

Launchpad user Chris Norman(cbnorman) wrote on 2021-05-13T13:18:44.526632+00:00

ok this is definetly the problem. the handle_preseed_maas() is there but a lot shorter so i just replaced the whole file and it worked perfectly. I use http://cdimage.debian.org/cdimage/openstack/archive/10.8.0/debian-10.8.0-openstack-amd64.raw to create the image. i had previously tried apt upgrade cloud-init and that didnt help, what package contains the /var/lib/dpkg/info/cloud-init.postinst file?

ubuntu-server-builder commented 1 year ago

Launchpad user Ryan Harper(raharper) wrote on 2021-05-13T15:24:23.453462+00:00

ok this is definetly the problem. the handle_preseed_maas() is there but a lot shorter so i just replaced the whole file and it worked perfectly.

\o/

what package contains the /var/lib/dpkg/info/cloud-init.postinst file

cloud-init itself. It looks like you should file a bug against debian's cloud-init package and mention that it's not tracking what upstream cloud-init (upstream cloud-init uses release branches to keep their debian/ directory changes for packaging).

I use http://cdimage.debian.org/cdimage/openstack/archive/10.8.0/debian-10.8.0-openstack-amd64.raw to create the image.

You could install cloud-init from Ubuntu, there's a daily PPA from which you could pull a deb

https://launchpad.net/~cloud-init-dev/+archive/ubuntu/daily

ubuntu-server-builder commented 1 year ago

Launchpad user Ryan Harper(raharper) wrote on 2021-05-13T15:26:10.050962+00:00

I'm marking the curtin task invalid; after getting config and logs we confirmed that the cloud-init package in the Debian image does not contain a recent handle_maas_preseed() that Ubuntu uses and curtin/MAAS rely upon to install the MAAS datasource correctly.

ubuntu-server-builder commented 1 year ago

Launchpad user Ryan Harper(raharper) wrote on 2021-05-13T15:26:51.546215+00:00

I'm marking the cloud-init task invalid. After getting config and logs we confirmed that the cloud-init package in the Debian image does not contain a recent handle_maas_preseed() that Ubuntu uses and curtin/MAAS rely upon to install the MAAS datasource correctly.

ubuntu-server-builder commented 1 year ago

Launchpad user Scott Moser(smoser) wrote on 2021-05-13T16:03:43.015451+00:00

what package contains the /var/lib/dpkg/info/cloud-init.postinst file

cloud-init itself. It looks like you should file a bug against debian's cloud-init package and mention that it's not tracking what upstream cloud-init (upstream cloud-init uses release branches to keep their debian/ directory changes for packaging).

That really doesn't seem like the right option to me. Using dpkg preseed "should work", but that seems like an arcane mechanism for providing configuration data. It was a solution that was put in place in ubuntu very early on (probably 12.04) when there wasn't a better option.

There now is a better option: the cloud-config files.

The better path forward would be to change curtin to write the cloud-config files itself as is done in Centos or Ubuntu core.

I suggest that the bug here is in curtin (curtin-hooks). They needed to be aware of this and write the files.

ubuntu-server-builder commented 1 year ago

Launchpad user Ryan Harper(raharper) wrote on 2021-05-14T14:53:36.294532+00:00

I suggest that the bug here is in curtin (curtin-hooks). They needed to be aware of this and write the files.

This is really orchestrated by maas...

MAAS can choose to send debconf_selections OR cloud-config; but it sends both. It's been sending both for quite some time. I'd like to see MAAS decide to no longer send debconf_selections and curtin could write out cloudconfig if present in the default curtin-hooks. In fact the maas images created by maas-image-builder include a curtin-hook which already writes out cloudconfig if present.

The decision to not render cloudconfig from curtin's default curthook has always been controlled by MAAS. I don't recall exactly why but I suspect that MAAS might not decide what config to send on a per image basis; ie they send a superset of config which works for all images (old an new) and for a range of curtin releases which don't render cloud-config for ubuntu but expect debconf_selections to do so.

ubuntu-server-builder commented 1 year ago

Launchpad user Chris Norman(cbnorman) wrote on 2021-05-14T15:09:15.379621+00:00

i believe that when you set apt: preserve_sources_list: true in the curtin_userdata then debconf is not sent by maas, this is where i initially started. I'm not sure how to confirm this though?

ubuntu-server-builder commented 1 year ago

Launchpad user Ryan Harper(raharper) wrote on 2021-05-14T17:12:34+00:00

On Fri, May 14, 2021 at 10:16 AM Chris Norman 1927020@bugs.launchpad.net wrote:

i believe that when you set apt: preserve_sources_list: true in the curtin_userdata then debconf is not sent by maas, this is where i initially started. I'm not sure how to confirm this though?

I don't think so, maybe some MAAS folks can confirm this. Looking at the preseeds, it's in the template.

https://github.com/maas/maas/blob/31a95206afe563012c48a20bcd8492fa32c7f405/contrib/preseeds_v2/curtin_userdata

MAAS always sends debconf_selections for cloud-init as that originally was the only way MAAS had to configure a cloud-init datasource. Over time curtin added support for writing out cloudconfig into the target image. For various reasons MAAS hasn't stopped sending debconf_selections to configure cloud-init in the target image and the Ubuntu cloud-init package has always had support for setting up the MAAS datasource via that method.

https://github.com/maas/maas/blob/1aa6276c0b6d7d702f1ed3036fd11d30e879c285/src/maasserver/compose_preseed.py#L399

There's not much logic on versions or anything; I think it's just been part of the template.

-- You received this bug notification because you are subscribed to the bug report. https://bugs.launchpad.net/bugs/1927020

Title: cloudconfig not writing maas data source

To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1927020/+subscriptions

ubuntu-server-builder commented 1 year ago

Launchpad user Albert Valiev(artscout) wrote on 2022-10-26T11:23:26.453406+00:00

Just in case anyone looking for answer, I managed to find solution, not elegant one but it works:

I defined node_distribution earlier in code:

{{py: node_distribution = node.get_distro_series() }}

  1. cloud-init should be already installed in the image used for custom install. Both buster/bullseye
  2. in case of bullseye python2 should be installed and made to provide /usr/bin/python, also pyyaml should be installed for python2. I did that with this late commands:

{{if node_distribution == "bullseye"}} 27_install_python2: ["curtin", "in-target", "--", "sh", "-c", "DEBIAN_FRONTEND=noninteractive apt-get -y install python2 python-is-python2"] 28_install_python2_yaml: ["curtin", "in-target", "--", "sh", "-c", "wget https://bootstrap.pypa.io/pip/2.7/get-pip.py && python2 get-pip.py && python2 -m pip install pyyaml"] {{endif}}

  1. netplan.io should be installed, while it's not used MaaS expects one to be installed on host so we install it:

    34_add_neplan: ["curtin", "in-target", "--", "sh", "-c", "DEBIAN_FRONTEND=noninteractive apt-get -y install netplan.io"]

  2. postinstall action for cloud-init package should be fired in late commands, no other way (like dpkg-reconfigure or new install of cloud-init works normally, latter adding maas datasource but deleting network config sadly). And as datasource for maas defined in debconf we have to fire it so handle_maas function do it's magic:

{{if node_distribution == "bullseye" or node_distribution == "buster"}} 35_reconfigure_cloud-init: ["curtin", "in-target", "--", "sh", "-c", "/var/lib/dpkg/info/cloud-init.postinst configure || true"] {{endif}}

with this steps debian buster and bullseye installed successfully with MaaS 3.2.4