Failed to launch nested lxd instance in focal using latest/edge

[sergiocazzolato]

Hi, I see an error trying to launch lxd inside another lxd

This started failing 2 days ago using ubuntu focal in GCP instances:

Steps to reproduce:

$ spread -shell google:ubuntu-20.04-64:tasks/google/common/start-instance 2022-06-01 11:45:04 Project content is packed for delivery (1.44MB). 2022-06-01 11:45:04 If killed, discard servers with: spread -reuse-pid=515199 -discard 2022-06-01 11:45:04 Allocating google:ubuntu-20.04-64... 2022-06-01 11:45:14 Waiting for google:ubuntu-20.04-64 (jun011445-439307) to boot at 35.229.125.43... 2022-06-01 11:46:11 Allocated google:ubuntu-20.04-64 (jun011445-439307). 2022-06-01 11:46:11 Connecting to google:ubuntu-20.04-64 (jun011445-439307)... 2022-06-01 11:46:14 Connected to google:ubuntu-20.04-64 (jun011445-439307) at 35.229.125.43. 2022-06-01 11:46:14 Sending project content to google:ubuntu-20.04-64 (jun011445-439307)... 2022-06-01 11:47:17 Preparing google:ubuntu-20.04-64:tasks/google/common/ (jun011445-439307)... 2022-06-01 11:47:33 Executing google:ubuntu-20.04-64:tasks/google/common/start-instance (jun011445-439307) (1/1)... 2022-06-01 11:47:33 Starting shell instead of executing google:ubuntu-20.04-64:tasks/google/common/start-instance...

google:ubuntu-20.04-64 .../tasks/google/common/start-instance# LXD_SNAP_CHANNEL=latest/edge google:ubuntu-20.04-64 .../tasks/google/common/start-instance# snap install lxd --channel="$LXD_SNAP_CHANNEL" lxd (edge) git-528f6aa from Canonical✓ installed google:ubuntu-20.04-64 .../tasks/google/common/start-instance# snap set lxd waitready.timeout=240 google:ubuntu-20.04-64 .../tasks/google/common/start-instance# lxd waitready google:ubuntu-20.04-64 .../tasks/google/common/start-instance# lxd init --auto google:ubuntu-20.04-64 .../tasks/google/common/start-instance# VERSION_ID="focal" google:ubuntu-20.04-64 .../tasks/google/common/start-instance# lxd.lxc launch --quiet "ubuntu:$VERSION_ID" my-nesting-ubuntu -c security.nesting=true google:ubuntu-20.04-64 .../tasks/google/common/start-instance# lxd.lxc list +-------------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +-------------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+ | my-nesting-ubuntu | RUNNING | 10.237.172.179 (eth0) | fd42:98a2:f883:50ed:216:3eff:fef7:3bdd (eth0) | CONTAINER | 0 | +-------------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+ google:ubuntu-20.04-64 .../tasks/google/common/start-instance# lxd.lxc exec my-nesting-ubuntu -- apt autoremove -y lxd Reading package lists... Done Building dependency tree
Reading state information... Done Package 'lxd' is not installed, so not removed The following packages will be REMOVED: libfreetype6 0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded. After this operation, 856 kB disk space will be freed. (Reading database ... 31898 files and directories currently installed.) Removing libfreetype6:amd64 (2.10.1-2ubuntu0.1) ... Processing triggers for libc-bin (2.31-0ubuntu9.7) ... google:ubuntu-20.04-64 .../tasks/google/common/start-instance# lxd.lxc exec my-nesting-ubuntu -- snap install lxd --channel="$LXD_SNAP_CHANNEL" snap "lxd" is already installed, see 'snap help refresh' google:ubuntu-20.04-64 .../tasks/google/common/start-instance# lxd.lxc exec my-nesting-ubuntu -- snap list lxd Name Version Rev Tracking Publisher Notes lxd 4.0.9-8e2046b 22753 4.0/stable/… canonical✓ - google:ubuntu-20.04-64 .../tasks/google/common/start-instance# lxd.lxc exec my-nesting-ubuntu -- snap refresh lxd --channel latest/edge lxd (edge) git-528f6aa from Canonical✓ refreshed google:ubuntu-20.04-64 .../tasks/google/common/start-instance# lxd.lxc exec my-nesting-ubuntu -- snap set lxd waitready.timeout=240 google:ubuntu-20.04-64 .../tasks/google/common/start-instance# lxd.lxc exec my-nesting-ubuntu -- lxd waitready google:ubuntu-20.04-64 .../tasks/google/common/start-instance# lxd.lxc exec my-nesting-ubuntu -- lxd init --auto google:ubuntu-20.04-64 .../tasks/google/common/start-instance# lxd.lxc exec my-nesting-ubuntu -- lxd.lxc launch "ubuntu:${VERSION_ID:-}" my-inner-ubuntu Creating my-inner-ubuntu Error: Failed instance creation: Failed creating instance from image: Process exited with non-zero value 1

Useful output:

lxc info

config: {} api_extensions:

• storage_zfs_remove_snapshots
• container_host_shutdown_timeout
• container_stop_priority
• container_syscall_filtering
• auth_pki
• container_last_used_at
• etag
• patch
• usb_devices
• https_allowed_credentials
• image_compression_algorithm
• directory_manipulation
• container_cpu_time
• storage_zfs_use_refquota
• storage_lvm_mount_options
• network
• profile_usedby
• container_push
• container_exec_recording
• certificate_update
• container_exec_signal_handling
• gpu_devices
• container_image_properties
• migration_progress
• id_map
• network_firewall_filtering
• network_routes
• storage
• file_delete
• file_append
• network_dhcp_expiry
• storage_lvm_vg_rename
• storage_lvm_thinpool_rename
• network_vlan
• image_create_aliases
• container_stateless_copy
• container_only_migration
• storage_zfs_clone_copy
• unix_device_rename
• storage_lvm_use_thinpool
• storage_rsync_bwlimit
• network_vxlan_interface
• storage_btrfs_mount_options
• entity_description
• image_force_refresh
• storage_lvm_lv_resizing
• id_map_base
• file_symlinks
• container_push_target
• network_vlan_physical
• storage_images_delete
• container_edit_metadata
• container_snapshot_stateful_migration
• storage_driver_ceph
• storage_ceph_user_name
• resource_limits
• storage_volatile_initial_source
• storage_ceph_force_osd_reuse
• storage_block_filesystem_btrfs
• resources
• kernel_limits
• storage_api_volume_rename
• macaroon_authentication
• network_sriov
• console
• restrict_devlxd
• migration_pre_copy
• infiniband
• maas_network
• devlxd_events
• proxy
• network_dhcp_gateway
• file_get_symlink
• network_leases
• unix_device_hotplug
• storage_api_local_volume_handling
• operation_description
• clustering
• event_lifecycle
• storage_api_remote_volume_handling
• nvidia_runtime
• container_mount_propagation
• container_backup
• devlxd_images
• container_local_cross_pool_handling
• proxy_unix
• proxy_udp
• clustering_join
• proxy_tcp_udp_multi_port_handling
• network_state
• proxy_unix_dac_properties
• container_protection_delete
• unix_priv_drop
• pprof_http
• proxy_haproxy_protocol
• network_hwaddr
• proxy_nat
• network_nat_order
• container_full
• candid_authentication
• backup_compression
• candid_config
• nvidia_runtime_config
• storage_api_volume_snapshots
• storage_unmapped
• projects
• candid_config_key
• network_vxlan_ttl
• container_incremental_copy
• usb_optional_vendorid
• snapshot_scheduling
• snapshot_schedule_aliases
• container_copy_project
• clustering_server_address
• clustering_image_replication
• container_protection_shift
• snapshot_expiry
• container_backup_override_pool
• snapshot_expiry_creation
• network_leases_location
• resources_cpu_socket
• resources_gpu
• resources_numa
• kernel_features
• id_map_current
• event_location
• storage_api_remote_volume_snapshots
• network_nat_address
• container_nic_routes
• rbac
• cluster_internal_copy
• seccomp_notify
• lxc_features
• container_nic_ipvlan
• network_vlan_sriov
• storage_cephfs
• container_nic_ipfilter
• resources_v2
• container_exec_user_group_cwd
• container_syscall_intercept
• container_disk_shift
• storage_shifted
• resources_infiniband
• daemon_storage
• instances
• image_types
• resources_disk_sata
• clustering_roles
• images_expiry
• resources_network_firmware
• backup_compression_algorithm
• ceph_data_pool_name
• container_syscall_intercept_mount
• compression_squashfs
• container_raw_mount
• container_nic_routed
• container_syscall_intercept_mount_fuse
• container_disk_ceph
• virtual-machines
• image_profiles
• clustering_architecture
• resources_disk_id
• storage_lvm_stripes
• vm_boot_priority
• unix_hotplug_devices
• api_filtering
• instance_nic_network
• clustering_sizing
• firewall_driver
• projects_limits
• container_syscall_intercept_hugetlbfs
• limits_hugepages
• container_nic_routed_gateway
• projects_restrictions
• custom_volume_snapshot_expiry
• volume_snapshot_scheduling
• trust_ca_certificates
• snapshot_disk_usage
• clustering_edit_roles
• container_nic_routed_host_address
• container_nic_ipvlan_gateway
• resources_usb_pci
• resources_cpu_threads_numa
• resources_cpu_core_die
• api_os
• container_nic_routed_host_table
• container_nic_ipvlan_host_table
• container_nic_ipvlan_mode
• resources_system
• images_push_relay
• network_dns_search
• container_nic_routed_limits
• instance_nic_bridged_vlan
• network_state_bond_bridge
• usedby_consistency
• custom_block_volumes
• clustering_failure_domains
• resources_gpu_mdev
• console_vga_type
• projects_limits_disk
• network_type_macvlan
• network_type_sriov
• container_syscall_intercept_bpf_devices
• network_type_ovn
• projects_networks
• projects_networks_restricted_uplinks
• custom_volume_backup
• backup_override_name
• storage_rsync_compression
• network_type_physical
• network_ovn_external_subnets
• network_ovn_nat
• network_ovn_external_routes_remove
• tpm_device_type
• storage_zfs_clone_copy_rebase
• gpu_mdev
• resources_pci_iommu
• resources_network_usb
• resources_disk_address
• network_physical_ovn_ingress_mode
• network_ovn_dhcp
• network_physical_routes_anycast
• projects_limits_instances
• network_state_vlan
• instance_nic_bridged_port_isolation
• instance_bulk_state_change
• network_gvrp
• instance_pool_move
• gpu_sriov
• pci_device_type
• storage_volume_state
• network_acl
• migration_stateful
• disk_state_quota
• storage_ceph_features
• projects_compression
• projects_images_remote_cache_expiry
• certificate_project
• network_ovn_acl
• projects_images_auto_update
• projects_restricted_cluster_target
• images_default_architecture
• network_ovn_acl_defaults
• gpu_mig
• project_usage
• network_bridge_acl
• warnings
• projects_restricted_backups_and_snapshots
• clustering_join_token
• clustering_description
• server_trusted_proxy
• clustering_update_cert
• storage_api_project
• server_instance_driver_operational
• server_supported_storage_drivers
• event_lifecycle_requestor_address
• resources_gpu_usb
• clustering_evacuation
• network_ovn_nat_address
• network_bgp
• network_forward
• custom_volume_refresh
• network_counters_errors_dropped
• metrics
• image_source_project
• clustering_config
• network_peer
• linux_sysctl
• network_dns
• ovn_nic_acceleration
• certificate_self_renewal
• instance_project_move
• storage_volume_project_move
• cloud_init
• network_dns_nat
• database_leader
• instance_all_projects
• clustering_groups
• ceph_rbd_du
• instance_get_full
• qemu_metrics
• gpu_mig_uuid
• event_project
• clustering_evacuation_live
• instance_allow_inconsistent_copy
• network_state_ovn
• storage_volume_api_filtering
• image_restrictions
• storage_zfs_export
• network_dns_records
• storage_zfs_reserve_space
• network_acl_log
• storage_zfs_blocksize
• metrics_cpu_seconds
• instance_snapshot_never
• certificate_token
• instance_nic_routed_neighbor_probe
• event_hub
• agent_nic_config
• projects_restricted_intercept
• metrics_authentication
• images_target_project
• cluster_migration_inconsistent_copy
• cluster_ovn_chassis
• container_syscall_intercept_sched_setscheduler
• storage_lvm_thinpool_metadata_size
• storage_volume_state_total
• instance_file_head
• instances_nic_host_name
• image_copy_profile
• container_syscall_intercept_sysinfo
• clustering_evacuation_mode
• resources_pci_vpd api_status: stable api_version: "1.0" auth: trusted public: false auth_methods:
• tls environment: addresses: [] architectures:
  • x86_64
  • i686 certificate: | -----BEGIN CERTIFICATE----- MIICITCCAaegAwIBAgIQDO/c4cgY1LZ7HcEDSHMrijAKBggqhkjOPQQDAzA+MRww GgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMR4wHAYDVQQDDBVyb290QGp1bjAx MTQ0NS00MzkzMDcwHhcNMjIwNjAxMTQ1MTA5WhcNMzIwNTI5MTQ1MTA5WjA+MRww GgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMR4wHAYDVQQDDBVyb290QGp1bjAx MTQ0NS00MzkzMDcwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASZ5zSF08+jQR9n7gBs HqlRUUCVGpQq5Ic06CMQKqruIoywjlNzKXoDkWJCJKBp/FdaKNS03SUSNNeNSuTd Go2uErsIh0DKRKVEFeQkbtEZ8VU4fG0yTILQLQD+qOT9gK+jajBoMA4GA1UdDwEB /wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMDMGA1Ud EQQsMCqCEGp1bjAxMTQ0NS00MzkzMDeHBH8AAAGHEAAAAAAAAAAAAAAAAAAAAAEw CgYIKoZIzj0EAwMDaAAwZQIxAMDQgRlHDZ9L5UbyKb6tm8PcN97nKluR6icTxUbP 1JMUrtYXSzZoLTTwpwh44oF25AIwVqOhwjAeOrqdgtEAt8kvyGkG7kRtZfn4RE1s 84lSw6iIprpjfLZKIezAzQxlXdc6 -----END CERTIFICATE----- certificate_fingerprint: 9a3191f6bc0d5b88ccf38a6ad5633701c84193fba607a16fc3c3024aa836286d driver: lxc driver_version: 4.0.0 (devel) firewall: xtables kernel: Linux kernel_architecture: x86_64 kernel_features: idmapped_mounts: "true" netnsid_getifaddrs: "true" seccomp_listener: "true" seccomp_listener_continue: "true" shiftfs: "false" uevent_injection: "true" unpriv_fscaps: "true" kernel_version: 5.13.0-1024-gcp lxc_features: cgroup2: "true" core_scheduling: "true" devpts_fd: "true" idmapped_mounts_v2: "true" mount_injection_file: "true" network_gateway_device_route: "true" network_ipvlan: "true" network_l2proxy: "true" network_phys_macvlan_mtu: "true" network_veth_router: "true" pidfd: "true" seccomp_allow_deny_syntax: "true" seccomp_notify: "true" seccomp_proxy_send_notify_fd: "true" os_name: Ubuntu os_version: "20.04" project: default server: lxd server_clustered: false server_event_mode: full-mesh server_name: jun011445-439307 server_pid: 3001 server_version: "5.2" storage: dir storage_version: "1" storage_supported_drivers:
  • name: btrfs version: 5.4.1 remote: false
  • name: cephfs version: 15.2.16 remote: true
  • name: dir version: "1" remote: false
  • name: lvm version: 2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0 remote: false
  • name: zfs version: 2.0.6-1ubuntu2.1 remote: false
  • name: ceph version: 15.2.16 remote: true

[tomponline]

Please can we see the output from the diagnostics steps from @stgraber here https://github.com/lxc/lxd/issues/10449#issuecomment-1135342849

Thanks

[tomponline]

Does using images:ubuntu/focal work?

[stgraber]

My gut feeling is that this is an apparmor thing. Can you show the sudo dmesg output from the host?

[sergiocazzolato]

Does using images:ubuntu/focal work?

I reproduce the error using focal instead of 20.04

My gut feeling is that this is an apparmor thing. Can you show the sudo dmesg output from the host?

this is the output: https://pastebin.pl/view/9e24ab44

[sergiocazzolato]

Please can we see the output from the diagnostics steps from @stgraber here #10449 (comment)

Thanks

This is the output from the host https://pastebin.pl/view/84a8c11c

[stgraber]

stgraber@dakara:~/data/code/lxc/lxd (lxc/master)$ lxc launch images:ubuntu/focal nesting -c security.nesting=true
Creating nesting
Starting nesting
stgraber@dakara:~/data/code/lxc/lxd (lxc/master)$ lxc exec nesting -- apt update
Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease                                     
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]                    
Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1490 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1833 kB]
Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [257 kB]
Get:7 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [948 kB]       
Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [135 kB]
Get:9 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [705 kB]       
Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [337 kB]              
Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [926 kB]
Fetched 6857 kB in 2s (3396 kB/s)                         
Reading package lists... Done
Building dependency tree       
Reading state information... Done
All packages are up to date.
stgraber@dakara:~/data/code/lxc/lxd (lxc/master)$ lxc exec nesting -- apt install snapd --yes
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  apparmor dbus-user-session dirmngr gnupg gnupg-l10n gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server gpgconf gpgsm libasn1-8-heimdal libassuan0
  libgssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal libheimntlm0-heimdal libhx509-5-heimdal libkrb5-26-heimdal libksba8 libldap-2.4-2
  libldap-common liblzo2-2 libnpth0 libroken18-heimdal libsasl2-2 libsasl2-modules libsasl2-modules-db libwind0-heimdal pinentry-curses squashfs-tools
Suggested packages:
  apparmor-profiles-extra apparmor-utils pinentry-gnome3 tor parcimonie xloadimage scdaemon libsasl2-modules-gssapi-mit | libsasl2-modules-gssapi-heimdal
  libsasl2-modules-ldap libsasl2-modules-otp libsasl2-modules-sql pinentry-doc zenity | kdialog
The following NEW packages will be installed:
  apparmor dbus-user-session dirmngr gnupg gnupg-l10n gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server gpgconf gpgsm libasn1-8-heimdal libassuan0
  libgssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal libheimntlm0-heimdal libhx509-5-heimdal libkrb5-26-heimdal libksba8 libldap-2.4-2
  libldap-common liblzo2-2 libnpth0 libroken18-heimdal libsasl2-2 libsasl2-modules libsasl2-modules-db libwind0-heimdal pinentry-curses snapd squashfs-tools
0 upgraded, 33 newly installed, 0 to remove and 0 not upgraded.
Need to get 38.6 MB of archives.
After this operation, 161 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 apparmor amd64 2.13.3-7ubuntu5.1 [494 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 dbus-user-session amd64 1.12.16-2ubuntu2.2 [9416 B]
Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libassuan0 amd64 2.5.3-7ubuntu2 [35.7 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 gpgconf amd64 2.2.19-3ubuntu2.1 [124 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libksba8 amd64 1.3.5-2 [92.6 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 libroken18-heimdal amd64 7.7.0+dfsg-1ubuntu1 [41.8 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libasn1-8-heimdal amd64 7.7.0+dfsg-1ubuntu1 [181 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libheimbase1-heimdal amd64 7.7.0+dfsg-1ubuntu1 [29.7 kB]
Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libhcrypto4-heimdal amd64 7.7.0+dfsg-1ubuntu1 [87.9 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libwind0-heimdal amd64 7.7.0+dfsg-1ubuntu1 [48.0 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 libhx509-5-heimdal amd64 7.7.0+dfsg-1ubuntu1 [107 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 libkrb5-26-heimdal amd64 7.7.0+dfsg-1ubuntu1 [208 kB]
Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 libheimntlm0-heimdal amd64 7.7.0+dfsg-1ubuntu1 [15.1 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 libgssapi3-heimdal amd64 7.7.0+dfsg-1ubuntu1 [96.1 kB]
Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libsasl2-modules-db amd64 2.1.27+dfsg-2ubuntu0.1 [14.7 kB]
Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libsasl2-2 amd64 2.1.27+dfsg-2ubuntu0.1 [49.3 kB]
Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libldap-common all 2.4.49+dfsg-2ubuntu1.9 [16.6 kB]
Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libldap-2.4-2 amd64 2.4.49+dfsg-2ubuntu1.9 [155 kB]
Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 libnpth0 amd64 1.6-1 [7736 B]
Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 dirmngr amd64 2.2.19-3ubuntu2.1 [329 kB]
Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 gnupg-l10n all 2.2.19-3ubuntu2.1 [51.7 kB]
Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 gnupg-utils amd64 2.2.19-3ubuntu2.1 [480 kB]
Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 gpg amd64 2.2.19-3ubuntu2.1 [483 kB]
Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 pinentry-curses amd64 1.1.0-3build1 [36.3 kB]
Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 gpg-agent amd64 2.2.19-3ubuntu2.1 [232 kB]
Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 gpg-wks-client amd64 2.2.19-3ubuntu2.1 [97.6 kB]
Get:27 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 gpg-wks-server amd64 2.2.19-3ubuntu2.1 [90.3 kB]
Get:28 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 gpgsm amd64 2.2.19-3ubuntu2.1 [217 kB]
Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 gnupg all 2.2.19-3ubuntu2.1 [259 kB]
Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB]
Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libsasl2-modules amd64 2.1.27+dfsg-2ubuntu0.1 [48.8 kB]
Get:32 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 squashfs-tools amd64 1:4.4-1ubuntu0.3 [117 kB]
Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 snapd amd64 2.54.3+20.04.1ubuntu0.3 [34.3 MB]
Fetched 38.6 MB in 2s (17.7 MB/s) 
Extracting templates from packages: 100%
Preconfiguring packages ...
Selecting previously unselected package apparmor.
(Reading database ... 15955 files and directories currently installed.)
Preparing to unpack .../00-apparmor_2.13.3-7ubuntu5.1_amd64.deb ...
Unpacking apparmor (2.13.3-7ubuntu5.1) ...
Selecting previously unselected package dbus-user-session.
Preparing to unpack .../01-dbus-user-session_1.12.16-2ubuntu2.2_amd64.deb ...
Unpacking dbus-user-session (1.12.16-2ubuntu2.2) ...
Selecting previously unselected package libassuan0:amd64.
Preparing to unpack .../02-libassuan0_2.5.3-7ubuntu2_amd64.deb ...
Unpacking libassuan0:amd64 (2.5.3-7ubuntu2) ...
Selecting previously unselected package gpgconf.
Preparing to unpack .../03-gpgconf_2.2.19-3ubuntu2.1_amd64.deb ...
Unpacking gpgconf (2.2.19-3ubuntu2.1) ...
Selecting previously unselected package libksba8:amd64.
Preparing to unpack .../04-libksba8_1.3.5-2_amd64.deb ...
Unpacking libksba8:amd64 (1.3.5-2) ...
Selecting previously unselected package libroken18-heimdal:amd64.
Preparing to unpack .../05-libroken18-heimdal_7.7.0+dfsg-1ubuntu1_amd64.deb ...
Unpacking libroken18-heimdal:amd64 (7.7.0+dfsg-1ubuntu1) ...
Selecting previously unselected package libasn1-8-heimdal:amd64.
Preparing to unpack .../06-libasn1-8-heimdal_7.7.0+dfsg-1ubuntu1_amd64.deb ...
Unpacking libasn1-8-heimdal:amd64 (7.7.0+dfsg-1ubuntu1) ...
Selecting previously unselected package libheimbase1-heimdal:amd64.
Preparing to unpack .../07-libheimbase1-heimdal_7.7.0+dfsg-1ubuntu1_amd64.deb ...
Unpacking libheimbase1-heimdal:amd64 (7.7.0+dfsg-1ubuntu1) ...
Selecting previously unselected package libhcrypto4-heimdal:amd64.
Preparing to unpack .../08-libhcrypto4-heimdal_7.7.0+dfsg-1ubuntu1_amd64.deb ...
Unpacking libhcrypto4-heimdal:amd64 (7.7.0+dfsg-1ubuntu1) ...
Selecting previously unselected package libwind0-heimdal:amd64.
Preparing to unpack .../09-libwind0-heimdal_7.7.0+dfsg-1ubuntu1_amd64.deb ...
Unpacking libwind0-heimdal:amd64 (7.7.0+dfsg-1ubuntu1) ...
Selecting previously unselected package libhx509-5-heimdal:amd64.
Preparing to unpack .../10-libhx509-5-heimdal_7.7.0+dfsg-1ubuntu1_amd64.deb ...
Unpacking libhx509-5-heimdal:amd64 (7.7.0+dfsg-1ubuntu1) ...
Selecting previously unselected package libkrb5-26-heimdal:amd64.
Preparing to unpack .../11-libkrb5-26-heimdal_7.7.0+dfsg-1ubuntu1_amd64.deb ...
Unpacking libkrb5-26-heimdal:amd64 (7.7.0+dfsg-1ubuntu1) ...
Selecting previously unselected package libheimntlm0-heimdal:amd64.
Preparing to unpack .../12-libheimntlm0-heimdal_7.7.0+dfsg-1ubuntu1_amd64.deb ...
Unpacking libheimntlm0-heimdal:amd64 (7.7.0+dfsg-1ubuntu1) ...
Selecting previously unselected package libgssapi3-heimdal:amd64.
Preparing to unpack .../13-libgssapi3-heimdal_7.7.0+dfsg-1ubuntu1_amd64.deb ...
Unpacking libgssapi3-heimdal:amd64 (7.7.0+dfsg-1ubuntu1) ...
Selecting previously unselected package libsasl2-modules-db:amd64.
Preparing to unpack .../14-libsasl2-modules-db_2.1.27+dfsg-2ubuntu0.1_amd64.deb ...
Unpacking libsasl2-modules-db:amd64 (2.1.27+dfsg-2ubuntu0.1) ...
Selecting previously unselected package libsasl2-2:amd64.
Preparing to unpack .../15-libsasl2-2_2.1.27+dfsg-2ubuntu0.1_amd64.deb ...
Unpacking libsasl2-2:amd64 (2.1.27+dfsg-2ubuntu0.1) ...
Selecting previously unselected package libldap-common.
Preparing to unpack .../16-libldap-common_2.4.49+dfsg-2ubuntu1.9_all.deb ...
Unpacking libldap-common (2.4.49+dfsg-2ubuntu1.9) ...
Selecting previously unselected package libldap-2.4-2:amd64.
Preparing to unpack .../17-libldap-2.4-2_2.4.49+dfsg-2ubuntu1.9_amd64.deb ...
Unpacking libldap-2.4-2:amd64 (2.4.49+dfsg-2ubuntu1.9) ...
Selecting previously unselected package libnpth0:amd64.
Preparing to unpack .../18-libnpth0_1.6-1_amd64.deb ...
Unpacking libnpth0:amd64 (1.6-1) ...
Selecting previously unselected package dirmngr.
Preparing to unpack .../19-dirmngr_2.2.19-3ubuntu2.1_amd64.deb ...
Unpacking dirmngr (2.2.19-3ubuntu2.1) ...
Selecting previously unselected package gnupg-l10n.
Preparing to unpack .../20-gnupg-l10n_2.2.19-3ubuntu2.1_all.deb ...
Unpacking gnupg-l10n (2.2.19-3ubuntu2.1) ...
Selecting previously unselected package gnupg-utils.
Preparing to unpack .../21-gnupg-utils_2.2.19-3ubuntu2.1_amd64.deb ...
Unpacking gnupg-utils (2.2.19-3ubuntu2.1) ...
Selecting previously unselected package gpg.
Preparing to unpack .../22-gpg_2.2.19-3ubuntu2.1_amd64.deb ...
Unpacking gpg (2.2.19-3ubuntu2.1) ...
Selecting previously unselected package pinentry-curses.
Preparing to unpack .../23-pinentry-curses_1.1.0-3build1_amd64.deb ...
Unpacking pinentry-curses (1.1.0-3build1) ...
Selecting previously unselected package gpg-agent.
Preparing to unpack .../24-gpg-agent_2.2.19-3ubuntu2.1_amd64.deb ...
Unpacking gpg-agent (2.2.19-3ubuntu2.1) ...
Selecting previously unselected package gpg-wks-client.
Preparing to unpack .../25-gpg-wks-client_2.2.19-3ubuntu2.1_amd64.deb ...
Unpacking gpg-wks-client (2.2.19-3ubuntu2.1) ...
Selecting previously unselected package gpg-wks-server.
Preparing to unpack .../26-gpg-wks-server_2.2.19-3ubuntu2.1_amd64.deb ...
Unpacking gpg-wks-server (2.2.19-3ubuntu2.1) ...
Selecting previously unselected package gpgsm.
Preparing to unpack .../27-gpgsm_2.2.19-3ubuntu2.1_amd64.deb ...
Unpacking gpgsm (2.2.19-3ubuntu2.1) ...
Selecting previously unselected package gnupg.
Preparing to unpack .../28-gnupg_2.2.19-3ubuntu2.1_all.deb ...
Unpacking gnupg (2.2.19-3ubuntu2.1) ...
Selecting previously unselected package liblzo2-2:amd64.
Preparing to unpack .../29-liblzo2-2_2.10-2_amd64.deb ...
Unpacking liblzo2-2:amd64 (2.10-2) ...
Selecting previously unselected package libsasl2-modules:amd64.
Preparing to unpack .../30-libsasl2-modules_2.1.27+dfsg-2ubuntu0.1_amd64.deb ...
Unpacking libsasl2-modules:amd64 (2.1.27+dfsg-2ubuntu0.1) ...
Selecting previously unselected package squashfs-tools.
Preparing to unpack .../31-squashfs-tools_1%3a4.4-1ubuntu0.3_amd64.deb ...
Unpacking squashfs-tools (1:4.4-1ubuntu0.3) ...
Selecting previously unselected package snapd.
Preparing to unpack .../32-snapd_2.54.3+20.04.1ubuntu0.3_amd64.deb ...
Unpacking snapd (2.54.3+20.04.1ubuntu0.3) ...
Setting up libksba8:amd64 (1.3.5-2) ...
Setting up libsasl2-modules:amd64 (2.1.27+dfsg-2ubuntu0.1) ...
Setting up liblzo2-2:amd64 (2.10-2) ...
Setting up libnpth0:amd64 (1.6-1) ...
Setting up libassuan0:amd64 (2.5.3-7ubuntu2) ...
Setting up libldap-common (2.4.49+dfsg-2ubuntu1.9) ...
Setting up libsasl2-modules-db:amd64 (2.1.27+dfsg-2ubuntu0.1) ...
Setting up apparmor (2.13.3-7ubuntu5.1) ...
Created symlink /etc/systemd/system/sysinit.target.wants/apparmor.service → /lib/systemd/system/apparmor.service.
Reloading AppArmor profiles 
Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
Setting up dbus-user-session (1.12.16-2ubuntu2.2) ...
Setting up gnupg-l10n (2.2.19-3ubuntu2.1) ...
Setting up libsasl2-2:amd64 (2.1.27+dfsg-2ubuntu0.1) ...
Setting up libroken18-heimdal:amd64 (7.7.0+dfsg-1ubuntu1) ...
Setting up gpgconf (2.2.19-3ubuntu2.1) ...
Setting up gpg (2.2.19-3ubuntu2.1) ...
Setting up libheimbase1-heimdal:amd64 (7.7.0+dfsg-1ubuntu1) ...
Setting up gnupg-utils (2.2.19-3ubuntu2.1) ...
Setting up pinentry-curses (1.1.0-3build1) ...
Setting up gpg-agent (2.2.19-3ubuntu2.1) ...
Created symlink /etc/systemd/user/sockets.target.wants/gpg-agent-browser.socket → /usr/lib/systemd/user/gpg-agent-browser.socket.
Created symlink /etc/systemd/user/sockets.target.wants/gpg-agent-extra.socket → /usr/lib/systemd/user/gpg-agent-extra.socket.
Created symlink /etc/systemd/user/sockets.target.wants/gpg-agent-ssh.socket → /usr/lib/systemd/user/gpg-agent-ssh.socket.
Created symlink /etc/systemd/user/sockets.target.wants/gpg-agent.socket → /usr/lib/systemd/user/gpg-agent.socket.
Setting up squashfs-tools (1:4.4-1ubuntu0.3) ...
Setting up libasn1-8-heimdal:amd64 (7.7.0+dfsg-1ubuntu1) ...
Setting up gpgsm (2.2.19-3ubuntu2.1) ...
Setting up libhcrypto4-heimdal:amd64 (7.7.0+dfsg-1ubuntu1) ...
Setting up libwind0-heimdal:amd64 (7.7.0+dfsg-1ubuntu1) ...
Setting up gpg-wks-server (2.2.19-3ubuntu2.1) ...
Setting up snapd (2.54.3+20.04.1ubuntu0.3) ...
Created symlink /etc/systemd/system/multi-user.target.wants/snapd.apparmor.service → /lib/systemd/system/snapd.apparmor.service.
Created symlink /etc/systemd/system/multi-user.target.wants/snapd.autoimport.service → /lib/systemd/system/snapd.autoimport.service.
Created symlink /etc/systemd/system/multi-user.target.wants/snapd.core-fixup.service → /lib/systemd/system/snapd.core-fixup.service.
Created symlink /etc/systemd/system/multi-user.target.wants/snapd.recovery-chooser-trigger.service → /lib/systemd/system/snapd.recovery-chooser-trigger.service
.
Created symlink /etc/systemd/system/multi-user.target.wants/snapd.seeded.service → /lib/systemd/system/snapd.seeded.service.
Created symlink /etc/systemd/system/cloud-final.service.wants/snapd.seeded.service → /lib/systemd/system/snapd.seeded.service.
Created symlink /etc/systemd/system/multi-user.target.wants/snapd.service → /lib/systemd/system/snapd.service.
Created symlink /etc/systemd/system/timers.target.wants/snapd.snap-repair.timer → /lib/systemd/system/snapd.snap-repair.timer.
Created symlink /etc/systemd/system/sockets.target.wants/snapd.socket → /lib/systemd/system/snapd.socket.
Created symlink /etc/systemd/system/final.target.wants/snapd.system-shutdown.service → /lib/systemd/system/snapd.system-shutdown.service.
snapd.failure.service is a disabled or a static unit, not starting it.
snapd.snap-repair.service is a disabled or a static unit, not starting it.
Setting up libhx509-5-heimdal:amd64 (7.7.0+dfsg-1ubuntu1) ...
Setting up libkrb5-26-heimdal:amd64 (7.7.0+dfsg-1ubuntu1) ...
Setting up libheimntlm0-heimdal:amd64 (7.7.0+dfsg-1ubuntu1) ...
Setting up libgssapi3-heimdal:amd64 (7.7.0+dfsg-1ubuntu1) ...
Setting up libldap-2.4-2:amd64 (2.4.49+dfsg-2ubuntu1.9) ...
Setting up dirmngr (2.2.19-3ubuntu2.1) ...
Created symlink /etc/systemd/user/sockets.target.wants/dirmngr.socket → /usr/lib/systemd/user/dirmngr.socket.
Setting up gpg-wks-client (2.2.19-3ubuntu2.1) ...
Setting up gnupg (2.2.19-3ubuntu2.1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...
Processing triggers for systemd (245.4-4ubuntu3.17) ...
Processing triggers for dbus (1.12.16-2ubuntu2.2) ...
Processing triggers for mime-support (3.64ubuntu1) ...
stgraber@dakara:~/data/code/lxc/lxd (lxc/master)$ lxc exec nesting -- snap install lxd --edge
2022-06-01T17:51:56Z INFO Waiting for automatic snapd restart...
lxd (edge) git-528f6aa from Canonical✓ installed
stgraber@dakara:~/data/code/lxc/lxd (lxc/master)$ lxc exec nesting -- lxd init --auto
stgraber@dakara:~/data/code/lxc/lxd (lxc/master)$ lxc exec nesting -- lxc launch images:ubuntu/focal inner
Creating inner
Starting inner                             

[stgraber]

stgraber@dakara:~/data/code/lxc/lxd (lxc/master)$ lxc launch ubuntu:focal nesting -c security.nesting=true
Creating nesting
Starting nesting                            
stgraber@dakara:~/data/code/lxc/lxd (lxc/master)$ lxc exec nesting -- snap remove lxd
lxd removed
stgraber@dakara:~/data/code/lxc/lxd (lxc/master)$ lxc exec nesting -- snap install lxd --edge
lxd (edge) git-528f6aa from Canonical✓ installed
stgraber@dakara:~/data/code/lxc/lxd (lxc/master)$ lxc exec nesting -- lxd init --auto
stgraber@dakara:~/data/code/lxc/lxd (lxc/master)$ lxc exec nesting -- lxc launch images:ubuntu/focal inner
Creating inner
Starting inner       

[sergiocazzolato]

stgraber@dakara:~/data/code/lxc/lxd (lxc/master)$ lxc launch ubuntu:focal nesting -c security.nesting=true
Creating nesting
Starting nesting                            
stgraber@dakara:~/data/code/lxc/lxd (lxc/master)$ lxc exec nesting -- snap remove lxd
lxd removed
stgraber@dakara:~/data/code/lxc/lxd (lxc/master)$ lxc exec nesting -- snap install lxd --edge
lxd (edge) git-528f6aa from Canonical✓ installed
stgraber@dakara:~/data/code/lxc/lxd (lxc/master)$ lxc exec nesting -- lxd init --auto
stgraber@dakara:~/data/code/lxc/lxd (lxc/master)$ lxc exec nesting -- lxc launch images:ubuntu/focal inner
Creating inner
Starting inner       

Is it a cloud image? I am reproducing that issue in google instances, I could provide access if you want

[stgraber]

Did the whole thing again using a clean 20.04 VM as the host and got it to fail then.

[stgraber]

Okay, I figured it out. It's an image issue for the inner container.

root@nesting:~# lxc launch images:alpine/edge a1
Creating a1
Starting a1                                 
root@nesting:~# lxc launch images:ubuntu/20.04 u1
Creating u1
Starting u1                                 
root@nesting:~# lxc launch ubuntu:20.04 inner
Creating inner
Error: Failed instance creation: Failed creating instance from image: Process exited with non-zero value 1

[stgraber]

The problem is that Ubuntu cloud images have recently started including a pre-populated /dev which cannot be unpacked inside of a container as the creation of device nodes isn't allowed.

[stgraber]

create_inode: failed to create character device squashfs-root/dev/console, because Operation not permitted

create_inode: failed to create character device squashfs-root/dev/full, because Operation not permitted

create_inode: failed to create character device squashfs-root/dev/mapper/control, because Operation not permitted

create_inode: failed to create character device squashfs-root/dev/null, because Operation not permitted

create_inode: failed to create character device squashfs-root/dev/ptmx, because Operation not permitted

create_inode: failed to create character device squashfs-root/dev/random, because Operation not permitted

create_inode: failed to create character device squashfs-root/dev/tty, because Operation not permitted

create_inode: failed to create character device squashfs-root/dev/urandom, because Operation not permitted

create_inode: failed to create character device squashfs-root/dev/zero, because Operation not permitted

[stgraber]

https://bugs.launchpad.net/cloud-images/+bug/1976552

[sergiocazzolato]

https://bugs.launchpad.net/cloud-images/+bug/1976552

Nice, thanks

[stgraber]

https://github.com/lxc/lxd/pull/10495