[virt-sysprep] RFE: Add an operation to clean cloud-init data

[eskultety]

When cloud-init runs in an instance, it'll populate /var/lib/cloud with various data. If such an instance is then re-configured to later become an enhanced template image, one has to make sure --delete /var/lib/cloud is passed to virt-sysprep, otherwise the next instance boot provisioned out of such an image may behave unexpectedly, because of the presence of some data in the aforementioned directory - many cloud-init modules have the execution policy set to _once_perboot, e.g. the ssh module which means that the host keys will be generated exactly once, i.e. when an instance is booted from the original vendor cloud image, which in turn means that if one enhances the vendor provided cloud-init image and then runs sysprep, the SSH keys won't be regenerated in the new instance during boot because ssh-keygen service is disabled when cloud-init service is enabled which runs ssh-keygen on demand.

[rwmjones]

Here's an example of a virt-sysprep operation that just rm -rf 's a directory: https://github.com/rwmjones/guestfs-tools/blob/master/sysprep/sysprep_operation_mail_spool.ml

[rwmjones]

I'm a little confused by the initial report. Is it sufficient to simply rm -rf /var/lib/cloud or are there other actions that have to be taken?

[eskultety]

I'm a little confused by the initial report. Is it sufficient to simply rm -rf /var/lib/cloud or are there other actions that have to be taken?

@rwmjones well, yes, /etc/machine-id needs to be reset on systemd platforms if you're not cleaning that one yet.

A few notes on the proposed approach itself:

• on RHEL-like systems an empty /var/lib/cloud directory is created when cloud-init is installed and enabled
• on Debian, FreeBSD and several other distros /var/lib/cloud isn't even created when the cloud-init service is installed and enabled
• during boot cloud-init tries to perform the system configuration using various data sources (first that applies wins):

  __init__.py[DEBUG]: Looking for data source in: ['NoCl
  oud', 'ConfigDrive', 'Azure', 'OpenStack', 'Ec2']

• each of the datasource plugins looks for information including previous cloud-init runs inside /var/lib/cloud:

  2023-05-23 07:05:14,284 - handlers.py[DEBUG]: start: init-local/search-NoCloud: searching for local data from DataSourceNoCloud
  2023-05-23 07:05:14,284 - __init__.py[DEBUG]: Seeing if we can get any data from <class 'cloudinit.sources.DataSourceNoCloud.DataSourceNoCloud'>
  2023-05-23 07:05:14,285 - __init__.py[DEBUG]: Update datasource metadata and network config due to events: boot-new-instance
  2023-05-23 07:05:14,285 - __init__.py[DEBUG]: Machine is running on DataSourceNoCloud [seed=None][dsmode=net]

To summarize the above, removing the /var/lib/cloud directory on Fedora, OpenSUSE, Debian, and FreeBSD is perfectly sufficient in order to reset cloud-init to its desired default behaviour for next boot. However, despite having been doing the above for some time myself, I went over the cloud-init docs and I found this: https://cloudinit.readthedocs.io/en/latest/reference/cli.html#clean. So, in order to be 100% compliant with the docs, which is probably what we actually want to do, we should use the clean subcommand.

[eskultety]

I just stumbled on this: https://gitlab.alpinelinux.org/alpine/cloud/tiny-cloud#main-phase. TL;DR Alpine added experimental implementation of some low memory footprint cloud-init implementation. The difference is that because this is a custom implementation with limited functionality and limited compatibility with cloud-init, there isn't any cloud-init command available. What I'm trying to say is that removal of the /var/lib/cloud directory should be ultimately considered as well when this RFE gets implemented because it'll get us more platform coverage.