UUID based `instance-id` for cloud-init (reset on changes)

[blackboxsw]

Issue description

Feature request: Requirements:

Currently LXD publishes a static metadata.instance-id which matches the hostname via the dev LXD API.

Cloud-init would like the instance-id to change at /dev/lxd/sock /1.0/meta-data anytime the underlying container or vm configuration changes which tells cloud-init to reconfigure the system on next boot:

• 1.0/devices # new device added/removed to the machine (side-effect of a lxc config device add|remove call)
• 1.0/config/cloud-init.* # any cloud-init config keys
• ?? 1.0/config/user.meta-data # if still supported by LXD

• ??Any additional container migration/profile change that would need a require an update of network config within the container/VM

  Not a requirement: Do not need any updates to NoCloud datasource seed files on a running container/VM.

Steps to reproduce (once feature is implemented)

1. lxc launch ubuntu-daily:jammy test-instance-id-updates
2. lxc exec test-instance-id-updates -- cloud-init status --wait --long
3. lxc exec test-instance-id-updates -- curl --unix-socket /dev/lxd/sock http://x/1.0/meta-data # check current instance-id
4. lxc config set test-instance-id-updates cloud-init.user-data
5. lxc exec test-instance-id-updates -- curl --unix-socket /dev/lxd/sock http://x/1.0/meta-data # assert instance-id changed
6. lxc config set test-instance-id-updates cloud-init.network-config
7. lxc exec test-instance-id-updates -- curl --unix-socket /dev/lxd/sock http://x/1.0/meta-data # assert instance-id changed
8. lxc config set test-instance-id-updates cloud-init.vendor-data
9. lxc exec test-instance-id-updates -- curl --unix-socket /dev/lxd/sock http://x/1.0/meta-data # assert instance-id changed

background

On many clouds, cloud-init treats changes to meta-data.instance-id as an dirty-cache check from the platform that cloud-init needs to re-run initial system configuration because network config, user-data or meta-data on the system has changed which requires cloud-init to re-run as if it were a new deployment. Many clouds instrument meta-data.instance-id as a UUID that changes when underlying network, meta-data, vendor-data or user-data change for a given VM.

The new LXDDatasource in cloud-init consumes updated system instance-data from the LXD dev API @ /dev/lxd/sock and it will check forinstance-id changes across reboot in order to determine of a config update is needed.

Required information

• Distribution: Looking to generically support *nix distributions this Dev LXD API behavior in cloud-init
• The output of "lxc info" or if that fails:

  csmith@uptown:~$ lxc info
  config: {}
  api_extensions:
  - storage_zfs_remove_snapshots
  - container_host_shutdown_timeout
  - container_stop_priority
  - container_syscall_filtering
  - auth_pki
  - container_last_used_at
  - etag
  - patch
  - usb_devices
  - https_allowed_credentials
  - image_compression_algorithm
  - directory_manipulation
  - container_cpu_time
  - storage_zfs_use_refquota
  - storage_lvm_mount_options
  - network
  - profile_usedby
  - container_push
  - container_exec_recording
  - certificate_update
  - container_exec_signal_handling
  - gpu_devices
  - container_image_properties
  - migration_progress
  - id_map
  - network_firewall_filtering
  - network_routes
  - storage
  - file_delete
  - file_append
  - network_dhcp_expiry
  - storage_lvm_vg_rename
  - storage_lvm_thinpool_rename
  - network_vlan
  - image_create_aliases
  - container_stateless_copy
  - container_only_migration
  - storage_zfs_clone_copy
  - unix_device_rename
  - storage_lvm_use_thinpool
  - storage_rsync_bwlimit
  - network_vxlan_interface
  - storage_btrfs_mount_options
  - entity_description
  - image_force_refresh
  - storage_lvm_lv_resizing
  - id_map_base
  - file_symlinks
  - container_push_target
  - network_vlan_physical
  - storage_images_delete
  - container_edit_metadata
  - container_snapshot_stateful_migration
  - storage_driver_ceph
  - storage_ceph_user_name
  - resource_limits
  - storage_volatile_initial_source
  - storage_ceph_force_osd_reuse
  - storage_block_filesystem_btrfs
  - resources
  - kernel_limits
  - storage_api_volume_rename
  - macaroon_authentication
  - network_sriov
  - console
  - restrict_devlxd
  - migration_pre_copy
  - infiniband
  - maas_network
  - devlxd_events
  - proxy
  - network_dhcp_gateway
  - file_get_symlink
  - network_leases
  - unix_device_hotplug
  - storage_api_local_volume_handling
  - operation_description
  - clustering
  - event_lifecycle
  - storage_api_remote_volume_handling
  - nvidia_runtime
  - container_mount_propagation
  - container_backup
  - devlxd_images
  - container_local_cross_pool_handling
  - proxy_unix
  - proxy_udp
  - clustering_join
  - proxy_tcp_udp_multi_port_handling
  - network_state
  - proxy_unix_dac_properties
  - container_protection_delete
  - unix_priv_drop
  - pprof_http
  - proxy_haproxy_protocol
  - network_hwaddr
  - proxy_nat
  - network_nat_order
  - container_full
  - candid_authentication
  - backup_compression
  - candid_config
  - nvidia_runtime_config
  - storage_api_volume_snapshots
  - storage_unmapped
  - projects
  - candid_config_key
  - network_vxlan_ttl
  - container_incremental_copy
  - usb_optional_vendorid
  - snapshot_scheduling
  - snapshot_schedule_aliases
  - container_copy_project
  - clustering_server_address
  - clustering_image_replication
  - container_protection_shift
  - snapshot_expiry
  - container_backup_override_pool
  - snapshot_expiry_creation
  - network_leases_location
  - resources_cpu_socket
  - resources_gpu
  - resources_numa
  - kernel_features
  - id_map_current
  - event_location
  - storage_api_remote_volume_snapshots
  - network_nat_address
  - container_nic_routes
  - rbac
  - cluster_internal_copy
  - seccomp_notify
  - lxc_features
  - container_nic_ipvlan
  - network_vlan_sriov
  - storage_cephfs
  - container_nic_ipfilter
  - resources_v2
  - container_exec_user_group_cwd
  - container_syscall_intercept
  - container_disk_shift
  - storage_shifted
  - resources_infiniband
  - daemon_storage
  - instances
  - image_types
  - resources_disk_sata
  - clustering_roles
  - images_expiry
  - resources_network_firmware
  - backup_compression_algorithm
  - ceph_data_pool_name
  - container_syscall_intercept_mount
  - compression_squashfs
  - container_raw_mount
  - container_nic_routed
  - container_syscall_intercept_mount_fuse
  - container_disk_ceph
  - virtual-machines
  - image_profiles
  - clustering_architecture
  - resources_disk_id
  - storage_lvm_stripes
  - vm_boot_priority
  - unix_hotplug_devices
  - api_filtering
  - instance_nic_network
  - clustering_sizing
  - firewall_driver
  - projects_limits
  - container_syscall_intercept_hugetlbfs
  - limits_hugepages
  - container_nic_routed_gateway
  - projects_restrictions
  - custom_volume_snapshot_expiry
  - volume_snapshot_scheduling
  - trust_ca_certificates
  - snapshot_disk_usage
  - clustering_edit_roles
  - container_nic_routed_host_address
  - container_nic_ipvlan_gateway
  - resources_usb_pci
  - resources_cpu_threads_numa
  - resources_cpu_core_die
  - api_os
  - container_nic_routed_host_table
  - container_nic_ipvlan_host_table
  - container_nic_ipvlan_mode
  - resources_system
  - images_push_relay
  - network_dns_search
  - container_nic_routed_limits
  - instance_nic_bridged_vlan
  - network_state_bond_bridge
  - usedby_consistency
  - custom_block_volumes
  - clustering_failure_domains
  - resources_gpu_mdev
  - console_vga_type
  - projects_limits_disk
  - network_type_macvlan
  - network_type_sriov
  - container_syscall_intercept_bpf_devices
  - network_type_ovn
  - projects_networks
  - projects_networks_restricted_uplinks
  - custom_volume_backup
  - backup_override_name
  - storage_rsync_compression
  - network_type_physical
  - network_ovn_external_subnets
  - network_ovn_nat
  - network_ovn_external_routes_remove
  - tpm_device_type
  - storage_zfs_clone_copy_rebase
  - gpu_mdev
  - resources_pci_iommu
  - resources_network_usb
  - resources_disk_address
  - network_physical_ovn_ingress_mode
  - network_ovn_dhcp
  - network_physical_routes_anycast
  - projects_limits_instances
  - network_state_vlan
  - instance_nic_bridged_port_isolation
  - instance_bulk_state_change
  - network_gvrp
  - instance_pool_move
  - gpu_sriov
  - pci_device_type
  - storage_volume_state
  - network_acl
  - migration_stateful
  - disk_state_quota
  - storage_ceph_features
  - projects_compression
  - projects_images_remote_cache_expiry
  - certificate_project
  - network_ovn_acl
  - projects_images_auto_update
  - projects_restricted_cluster_target
  - images_default_architecture
  - network_ovn_acl_defaults
  - gpu_mig
  - project_usage
  - network_bridge_acl
  - warnings
  - projects_restricted_backups_and_snapshots
  - clustering_join_token
  - clustering_description
  - server_trusted_proxy
  - clustering_update_cert
  - storage_api_project
  - server_instance_driver_operational
  - server_supported_storage_drivers
  - event_lifecycle_requestor_address
  - resources_gpu_usb
  - clustering_evacuation
  - network_ovn_nat_address
  - network_bgp
  - network_forward
  - custom_volume_refresh
  - network_counters_errors_dropped
  - metrics
  - image_source_project
  - clustering_config
  - network_peer
  - linux_sysctl
  - network_dns
  - ovn_nic_acceleration
  - certificate_self_renewal
  - instance_project_move
  - storage_volume_project_move
  - cloud_init
  - network_dns_nat
  - database_leader
  - instance_all_projects
  - clustering_groups
  - ceph_rbd_du
  - instance_get_full
  - qemu_metrics
  - gpu_mig_uuid
  - event_project
  - clustering_evacuation_live
  - instance_allow_inconsistent_copy
  api_status: stable
  api_version: "1.0"
  auth: trusted
  public: false
  auth_methods:
  - tls
  environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
  -----BEGIN CERTIFICATE-----
  MIIB9zCCAX2gAwIBAgIQU3bmUn7ogbX2pOjlALWf7TAKBggqhkjOPQQDAzA0MRww
  GgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QHVwdG93
  bjAeFw0xOTEwMjEyMDMzMzRaFw0yOTEwMTgyMDMzMzRaMDQxHDAaBgNVBAoTE2xp
  bnV4Y29udGFpbmVycy5vcmcxFDASBgNVBAMMC3Jvb3RAdXB0b3duMHYwEAYHKoZI
  zj0CAQYFK4EEACIDYgAEzZhp2YHDbTeZIhdxUaKj6JZ5Cpqi2K4qzAdZ243TIo6r
  Ojzh9Q+7Ux9ZUBAXDuzDQEjpN2rSMzFLXxjy6PUUqn+ukSzNGERQYKKv3Rx2qKFs
  3zgt3WGTP/CPvf0fHTY6o1QwUjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYI
  KwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHREEFjAUggZ1cHRvd26HBMCoAguH
  BAqpKAEwCgYIKoZIzj0EAwMDaAAwZQIwDBSN4Xu1B8zP0oU3qDnvdltOmf95EBLf
  s9CJwiovmwxAuyaFwIYJyGC/pWYH35uHAjEAwlIPBWQzs2qjP0L3Z+JI54al2N1N
  iACdzozxVE9eHgGFNOTMppNbBpeKo4tudOab
  -----END CERTIFICATE-----
  certificate_fingerprint: 6aaceb8d2e522b490ee1d95bf2a31be73f1ed8bcf503709e55f630f1b7be0c58
  driver: lxc | qemu
  driver_version: 4.0.0 (devel) | 6.2.0
  firewall: nftables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
  netnsid_getifaddrs: "true"
  seccomp_listener: "true"
  seccomp_listener_continue: "true"
  shiftfs: "false"
  uevent_injection: "true"
  unpriv_fscaps: "true"
  kernel_version: 5.4.0-71-generic
  lxc_features:
  cgroup2: "true"
  core_scheduling: "true"
  devpts_fd: "true"
  idmapped_mounts_v2: "true"
  mount_injection_file: "true"
  network_gateway_device_route: "true"
  network_ipvlan: "true"
  network_l2proxy: "true"
  network_phys_macvlan_mtu: "true"
  network_veth_router: "true"
  pidfd: "true"
  seccomp_allow_deny_syntax: "true"
  seccomp_notify: "true"
  seccomp_proxy_send_notify_fd: "true"
  os_name: Ubuntu
  os_version: "20.04"
  project: default
  server: lxd
  server_clustered: false
  server_name: uptown
  server_pid: 14127
  server_version: "4.22"
  storage: zfs | lvm
  storage_version: 0.8.3-1ubuntu12.6 | 2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30)
  / 4.41.0
  storage_supported_drivers:
  - name: ceph
  version: 15.2.14
  remote: true
  - name: btrfs
  version: 5.4.1
  remote: false
  - name: cephfs
  version: 15.2.14
  remote: true
  - name: dir
  version: "1"
  remote: false
  - name: lvm
  version: 2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.41.0
  remote: false
  - name: zfs
  version: 0.8.3-1ubuntu12.6
  remote: false

Information to attach

• [ ] Any relevant kernel output (dmesg)
• [ ] Container log (lxc info NAME --show-log)
• [ ] Container configuration (lxc config show NAME --expanded)
• [ ] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
• [ ] Output of the client with --debug
• [ ] Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)

[stgraber]

Okay, so having a UUID which then changes if:

• Name is changed (copy or rename)
• User/Vendor data has changed (cloud-init.vendor-data, cloud-init.user-data, user.vendor-data, user.user-data)
• Network config have changed (cloud-init.network-config, user.network-config)

That part is pretty uncontroversial and shouldn't be too hard to achieve. I'm not sure that we hit Update() when the name changes so we may need to do some special casing on that one to properly reset on copy and renames.

Where it gets a bit more hairy is with the network devices especially when profiles get mixed in. We don't want someone doing lxc profile device set default eth0 user.foo blah to trigger a new UUID for every single instance LXD-wide.

So what exactly does cloud-init care about there? We can only do it on network device being added or remove, but then is an existing device being changed to a different MAC and different name inside of the instance going to cause problems if it doesn't reset the UUID too?

[blackboxsw]

Agreed on your header of 3 use-cases:

• [✔] Name change (copy or rename): (as cloud-init needs to minimally reset hostname)
• [✔] Change of cloud-init.vendor-data, cloud-init.user-data, user.vendor-data, user.user-data changes (direct impact to what cloud-init config modules are invoked on a system)
• [✔] Change of cloud-init.network-config, user.network-config: (cclean need for cloud-init to re-render network config)

--- Additional instance-id update requests per your last comment

• [✔] Network device add or remove (cloud-init would then query new device names in 1.0/endpoints and update net config based only on present type=nic device names)

and different name inside of the instance going to cause problems if it doesn't reset the UUID too

Yes this would break cloud-init's generated network match as it is based on NIC name exposed to the instance.

• [✔] An existing network device name change (cloud-init will only query 1.0/devices at boot when instance-id changes across boot/or if opted-into hot-plug events. It will render full network config based on all type=nic device names present)

• [✗ No] An existing device being changed to a different MAC (cloud-init doesn't render network config which specifically matches on MAC address. cloud-init's opt-in hotplug can also watch udev events to see matching expected device names via a udev rule like:

  # Installed by cloud-init due to network hotplug userdata                       
  ACTION!="add|remove", GOTO="cloudinit_end"                                      
  LABEL="cloudinit_hook"         
  SUBSYSTEM=="net", KERNEL="eth0", RUN+="{libexecdir}/hook-hotplug"
  LABEL="cloudinit_end"   

From LXD dev API 1.0/devices output[1] we don't get MAC addr currently cloud-init will rely only on device name of type: nic to render a generic network config on the system without a match on MAC. As long as we know the desired device name we should be in good shape here.

[1] LXD API devices output

$ lxc exec test-it -- curl --unix-socket /dev/lxd/sock http://x/1.0/devices
{"eth0":{"name":"eth0","nictype":"bridged","parent":"lxdbr0","type":"nic"},"root":{"path":"/","pool":"default","type":"disk"}}
lxc exec test-it -- cat /etc/netplan/50-cloud-init.yaml
$ lxc exec test-it -- cat /etc/netplan/50-cloud-init.yaml
# This file is generated from information provided by the datasource.  Changes
# to it will not persist across an instance reboot.  To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    version: 2
    ethernets:
        eth0:
            dhcp4: true

At some point (cloud-init needs to file a separate issue) cloud-init would like client-side visibility to dhcp6 vs dhcp4 needs somehow via LXDAPI but maybe that comes in 1.0/networks/{name}/leases?

[stgraber]

Ok, so we introduce a new volatile.cloud-init.machine-id which gets auto-reset if:

• Name is changed (copy or rename)
• User/Vendor data has changed (cloud-init.vendor-data, cloud-init.user-data, user.vendor-data, user.user-data)
• Network config have changed (cloud-init.network-config, user.network-config)
• The set of nic type devices exposed to the container (ExpandedDevices) has changed in a way that would affect the list of interface names in the container

@blackboxsw does that cover it?

To clarify that last one, it means that adding or removing an interface would trigger it, as would changing the name property on an existing device. However if someone is to remove a device and introduce another one at the same time and using the same instance-visible name, we won't trigger it. That should line up with what cloud-init cares about.

[blackboxsw]

Yes the internal accounting of volatile.cloud-init.machine-id and the trigger cases you mention should suffice for all cloud-init needs. As long as the LXD dev API 1.0/meta-data will reflect instance-id based on the value of volatile.cloud-init.machine-id without caching getting in the way we should be good here.