add support for cloud-init drive as created by libvirt_cloudinit_disk

[rgl]

I would like to use the cloud-init disk generated by https://github.com/dmacvicar/terraform-provider-libvirt/blob/master/website/docs/r/cloudinit.html.markdown. I have an ubuntu regular cloud-init example usage at https://github.com/rgl/terraform-libvirt-ubuntu-example which I'm trying to port to windows.

Essentially, terraform-provider-libvirt creates a iso which is attached as a cd-rom to the VM. The cd-rom label is cidata and the root directory has the following files:

• meta-data
• network-config
• user-data

As displayed by Windows Explorer:

I've tried to follow the cloudbase-init codebase and ended up at https://github.com/cloudbase/cloudbase-init/blob/cd8eca0f91aed04690ed7272485185ab6a1a8e18/cloudbaseinit/metadata/services/osconfigdrive/windows.py#L55-L58 which checks for the label and some files. Is this the best place to add support for the cloud-init? or is it better to create an entire new metadata service class?

[ader1990]

Hello,

First of all, cloudbase-init bugs / code requests are still managed on https://launchpad.net/cloudbase-init and on opendev https://review.opendev.org/#/q/status:open+project:x/cloudbase-init. But we are planning to move to GitHub in the close future, so the issues and PRs on GitHub will take precedence over the launchpad / opendev ones.

There is an old patch under review to add support for your request, the NoCloud metadata provider: https://review.opendev.org/#/c/488182 https://review.opendev.org/#/c/478515

Blueprint for the feature: https://blueprints.launchpad.net/cloudbase-init/+spec/nocloud-metadata-support

Thank you, Adrian Vladu

[ader1990]

@rgl I have rebased the above nocloud patches, you can safely apply them now on the master tree: https://review.opendev.org/#/c/488182 https://review.opendev.org/#/c/478515

[ader1990]

@rgl can you share an archive with the metadata from your environment? It will be helpful to properly test the patch.

Thank you, Adrian Vladu

[rgl]

Thank You for the information! I didn't notice the development was being made outside of github.

I've tried the patches, and they seem to work! a simple user-data with set_hostname works :-)

Though, why did it automatically worked when the NoCloudConfigDriveService service is not defined in the metadata_service property of cloudbase-init-unattend.conf?

My user-data is defined at:

https://github.com/rgl/terraform-libvirt-windows-example/blob/master/main.tf#L36-L45

My goal is to have this plugin do the same as the example at:

https://github.com/rgl/terraform-libvirt-ubuntu-example/blob/master/main.tf#L24-L54

Which is:

• Automatically expand the root disk if there is space available.
• Change a user password
• Format and initialize disks
• Set IP configuration
• Run commands

Do you known if all of this is/will-be implemented?

In the mean time, I'll browse https://launchpad.net/cloudbase-init :-)

---

For reference, I've added the patches to https://github.com/rgl/cloudbase-init/tree/add-no-cloud and tried it by installing cloudbase-init in windows with this convoluted script (it will essentially, stop the cloudbase-init service, replace the cloudbaseinit python package with the one from that repository):

Set-StrictMode -Version Latest
$ProgressPreference = 'SilentlyContinue'
$ErrorActionPreference = 'Stop'
trap {
    Write-Host
    Write-Host "ERROR: $_"
    Write-Host (($_.ScriptStackTrace -split '\r?\n') -replace '^(.*)$','ERROR: $1')
    Write-Host (($_.Exception.ToString() -split '\r?\n') -replace '^(.*)$','ERROR EXCEPTION: $1')
    Write-Host
    Write-Host 'Sleeping for 60m to give you time to look around the virtual machine before self-destruction...'
    Start-Sleep -Seconds (60*60)
    Exit 1
}

$artifactUrl = 'https://www.cloudbase.it/downloads/CloudbaseInitSetup_x64.msi'
$artifactPath = "$env:TEMP\$(Split-Path -Leaf $artifactUrl)"
$artifactLogPath = "$artifactPath.log"

Write-Host 'Downloading the cloudbase-init setup...'
(New-Object System.Net.WebClient).DownloadFile($artifactUrl, $artifactPath)

Write-Host 'Installing cloudbase-init...'
msiexec /i $artifactPath /qn /l*v $artifactLogPath | Out-String -Stream
if ($LASTEXITCODE) {
    throw "Failed with Exit Code $LASTEXITCODE"
}

Stop-Service cloudbase-init

$artifactUrl = 'https://github.com/rgl/cloudbase-init/archive/add-no-cloud.zip'
$artifactPath = "$env:TEMP\$(Split-Path -Leaf $artifactUrl)"

Write-Host 'Downloading the no-cloud setup...'
(New-Object System.Net.WebClient).DownloadFile($artifactUrl, $artifactPath)

Write-Host 'Replacing the cloudbaseinit package...'
Expand-Archive $artifactPath "$artifactPath-extracted"
$cloudbaseInitTmpPackage = Resolve-Path "$artifactPath-extracted\*\cloudbaseinit"
$cloudbaseInitHome = 'C:\Program Files\Cloudbase Solutions\Cloudbase-Init'
$cloudbaseInitPackage = "$cloudbaseInitHome\Python\Lib\site-packages\cloudbaseinit"
$cloudbaseInitPackageBackup = "$cloudbaseInitPackage-$((Get-Date -Format s) -replace '[:-]','')-backup"
Rename-Item $cloudbaseInitPackage $cloudbaseInitPackageBackup
Copy-Item -Recurse $cloudbaseInitTmpPackage $cloudbaseInitPackage

Start-Service cloudbase-init

[ader1990]

Cloudbase-init works on a plugin based framework. Available plugin docs: https://cloudbase-init.readthedocs.io/en/latest/plugins.html

As for the cloud-config example, cloudbase-init supports only setting timezone, hostname and write_file: https://cloudbase-init.readthedocs.io/en/latest/userdata.html#cloud-config

Your requirements:

Automatically expand the root disk if there is space available. -- plugin available: https://cloudbase-init.readthedocs.io/en/latest/plugins.html#volume-expanding-main

Change a user password: available for more advanced services, your options using config drive are limited to creating a cmd/powershell userdata that does that. For this feature there is (again) an older patch that adds cloud-config support, which you can use: https://review.opendev.org/#/c/379354/

Format and initialize disks: not implemented, but you can fallback to the aforementioned userdata

Set IP configuration: not supported for NoCloudConfigDrive, you need to implement the network metadata parsing, as the plugin exists. For example, the config drive implementation: https://github.com/cloudbase/cloudbase-init/blob/master/cloudbaseinit/metadata/services/baseopenstackservice.py#L240 You need to implement for the NoCloudConfigDrive metadata service the get_network_details or get_network_details _v2 method.

Run commands: localscripts plugin or userdata.

Keep in mind that the userdata can be in MIME format: https://cloudbase-init.readthedocs.io/en/latest/userdata.html#multi-part-content

[ader1990]

On another topic, you can easily generate custom Windows images using https://github.com/cloudbase/windows-openstack-imaging-tools/, with support for custom cloudbase-init config files, custom installers and much more.

[rgl]

@ader1990 I'm using the (latest) patch 27 of https://review.opendev.org/#/c/478515 at https://github.com/rgl/cloudbase-init/commits/add-no-cloud but metadata was changed from:

    "public-keys": ["content of ~/.ssh/id_rsa.pub"]

to:

    "public-keys": {
      "key name/id": {"openssh-key": "content of ~/.ssh/id_rsa.pub"},
    }

Was this intended?

Can you also add support for setting the admin-username and password as I've done at https://github.com/rgl/cloudbase-init/commit/7306111b5096844b65fe2db9cf16f46c6fa227c1?

Or if you are accepting patches, where should I sent them to? :-)

Or is this admin-username now deprecated and we should use the new users feature added recently? If so, will it work with the NoCloud metadata service?

[ader1990]

@rgl The change is good, as here is an example of the ec2 compatible metadata.json: https://github.com/cloudbase/cloudbase-init-test-resources/blob/master/openstack/cloudbase-init-metadata/ec2/latest/meta-data.json

Also, the already existing ec2 service implementation is similar: https://github.com/cloudbase/cloudbase-init/blob/master/cloudbaseinit/metadata/services/ec2service.py#L62

You can provide patches now using the git review process, on https://review.opendev.org/. You will need a launchpad account (Ubuntu account) to add new patches.

The same OpenStack rules apply when creating your first patch: https://docs.openstack.org/zaqar/latest/contributor/first_patch.html

Let me know if you encounter any issues and I will gladly help you solve them.

Thank you, Adrian Vladu

[rgl]

@ader1990 I was able to clone the gerrit repo, but how do I access the branch from https://review.opendev.org/#/c/478515? its not appearing as a remote branch of ssh://rgl@review.opendev.org:29418/x/cloudbase-init.git

[ader1990]

@rgl Let's consider you want to change the patch or add a new patch on top of it: https://review.opendev.org/#/c/478515

First of all, you need to clone the upstream cloudbase-init. You can clone the mirror or the review one:

git clone https://github.com/cloudbase/cloudbase-init or git clone https://opendev.org/x/cloudbase-init

Then you can see on the patch review page the download button, where is a list of checkouts / cherry-picks, etc. I prefer cherry-picking most of the time.

Create a work branch: git checkout -b "work_branch"

Then cherry-pick using directly the downloads link for cherry-pick:

git fetch ssh://avladu@review.opendev.org:29418/x/cloudbase-init refs/changes/15/478515/29 && git cherry-pick FETCH_HEAD

Note the /29, which is the last patch set.

After the cherry pick you can change the code, amend it or create a new commit.

After the amend / new commit, install git-review python package (pip install git-review) then run : git review

You might need to run the first time you do this the init of git review, for git review initialization:

git review -S

Note the 'Change-Id: Ib434cf2b2b21bf9faa58e05ba40eb0135385c9ea' at the end of the commit message. If you change the hash, then on git review, it will create a new patch. Leaving the hash there will create a new patchset (revision) of that patch, reaching 30th patchset in this case.

Let me know how it goes :)

Thank you, Adrian Vladu

[ader1990]

@ader1990 I'm using the (latest) patch 27 of https://review.opendev.org/#/c/478515 at https://github.com/rgl/cloudbase-init/commits/add-no-cloud but metadata was changed from:

    "public-keys": ["content of ~/.ssh/id_rsa.pub"]

to:

    "public-keys": {
      "key name/id": {"openssh-key": "content of ~/.ssh/id_rsa.pub"},
    }

Was this intended?

Can you also add support for setting the admin-username and password as I've done at rgl@7306111?

Or if you are accepting patches, where should I sent them to? :-)

Or is this admin-username now deprecated and we should use the new users feature added recently? If so, will it work with the NoCloud metadata service?

On the keys that you are adding for username / password support, they are not permitted as per cloud-init's docs: https://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html

If you can use groups/users/runcmd from userdata, that would be great.