canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.33k stars 926 forks source link

Virtual machine in RUNNING state after systemclt halt #12024

Closed yhaliaw closed 1 year ago

yhaliaw commented 1 year ago

Required information

Issue description

Using systemctl halt -i within a virtual machine to shutdown will still result in Running status for the VM. In past versions this used result in Stopped status.

Using systemctl poweroff -i would still result in Stopped status. Only halt suffer from this issue.

Steps to reproduce

  1. Create a VM with: lxc launch ubuntu:22.04 --vm test
  2. Execute halt within the VM to shutdown: lxc exec test -- systemctl halt -i
  3. Verify the status of the test VM after a while with: lxc info test

Changing step 2, to lxc exec test -- systemctl poweroff -i should result in the machine in Stopped state.

Information to attach

Resources: Processes: -1 Network usage: eth0: Type: broadcast State: UP Host interface: tap7b283096 MAC address: 00:16:3e:6d:58:e7 MTU: 1500 Bytes received: 295.72kB Bytes sent: 20.14kB Packets received: 259 Packets sent: 177 IP addresses: inet: 10.176.39.138/24 (global) inet6: fd42:e589:cb5a:4d:216:3eff:fe6d:58e7/64 (global) inet6: fe80::216:3eff:fe6d:58e7/64 (link)

Log:

 - [X] Container configuration (`lxc config show NAME --expanded`)

architecture: x86_64 config: image.architecture: amd64 image.description: ubuntu 22.04 LTS amd64 (release) (20230712) image.label: release image.os: ubuntu image.release: jammy image.serial: "20230712" image.type: disk-kvm.img image.version: "22.04" volatile.base_image: 5cec7f1c12c99b73b42886b878e3c5927b9b3e0e1a181465ecaeadc529c618fc volatile.cloud-init.instance-id: 132dad1e-d8a8-42d3-bcb2-0499a9cedcd1 volatile.eth0.host_name: tap7b283096 volatile.eth0.hwaddr: 00:16:3e:6d:58:e7 volatile.last_state.power: RUNNING volatile.uuid: e10324bd-2a34-4203-9f63-155af1b1dce5 volatile.uuid.generation: e10324bd-2a34-4203-9f63-155af1b1dce5 volatile.vsock_id: "2399186832" devices: eth0: name: eth0 network: lxdbr0 type: nic root: path: / pool: default type: disk ephemeral: false profiles:

 - [X] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)

time="2023-07-16T16:44:05+08:00" level=warning msg=" - Couldn't find the CGroup network priority controller, network priority will be ignored"

 - [X] Output of the client with --debug

DEBUG [2023-07-17T12:44:17+08:00] Connecting to a local LXD over a Unix socket DEBUG [2023-07-17T12:44:17+08:00] Sending request to LXD etag= method=GET url="http://unix.socket/1.0" DEBUG [2023-07-17T12:44:17+08:00] Got response struct from LXD
DEBUG [2023-07-17T12:44:17+08:00] { "config": {}, "api_extensions": [ "storage_zfs_remove_snapshots", "container_host_shutdown_timeout", "container_stop_priority", "container_syscall_filtering", "auth_pki", "container_last_used_at", "etag", "patch", "usb_devices", "https_allowed_credentials", "image_compression_algorithm", "directory_manipulation", "container_cpu_time", "storage_zfs_use_refquota", "storage_lvm_mount_options", "network", "profile_usedby", "container_push", "container_exec_recording", "certificate_update", "container_exec_signal_handling", "gpu_devices", "container_image_properties", "migration_progress", "id_map", "network_firewall_filtering", "network_routes", "storage", "file_delete", "file_append", "network_dhcp_expiry", "storage_lvm_vg_rename", "storage_lvm_thinpool_rename", "network_vlan", "image_create_aliases", "container_stateless_copy", "container_only_migration", "storage_zfs_clone_copy", "unix_device_rename", "storage_lvm_use_thinpool", "storage_rsync_bwlimit", "network_vxlan_interface", "storage_btrfs_mount_options", "entity_description", "image_force_refresh", "storage_lvm_lv_resizing", "id_map_base", "file_symlinks", "container_push_target", "network_vlan_physical", "storage_images_delete", "container_edit_metadata", "container_snapshot_stateful_migration", "storage_driver_ceph", "storage_ceph_user_name", "resource_limits", "storage_volatile_initial_source", "storage_ceph_force_osd_reuse", "storage_block_filesystem_btrfs", "resources", "kernel_limits", "storage_api_volume_rename", "macaroon_authentication", "network_sriov", "console", "restrict_devlxd", "migration_pre_copy", "infiniband", "maas_network", "devlxd_events", "proxy", "network_dhcp_gateway", "file_get_symlink", "network_leases", "unix_device_hotplug", "storage_api_local_volume_handling", "operation_description", "clustering", "event_lifecycle", "storage_api_remote_volume_handling", "nvidia_runtime", "container_mount_propagation", "container_backup", "devlxd_images", "container_local_cross_pool_handling", "proxy_unix", "proxy_udp", "clustering_join", "proxy_tcp_udp_multi_port_handling", "network_state", "proxy_unix_dac_properties", "container_protection_delete", "unix_priv_drop", "pprof_http", "proxy_haproxy_protocol", "network_hwaddr", "proxy_nat", "network_nat_order", "container_full", "candid_authentication", "backup_compression", "candid_config", "nvidia_runtime_config", "storage_api_volume_snapshots", "storage_unmapped", "projects", "candid_config_key", "network_vxlan_ttl", "container_incremental_copy", "usb_optional_vendorid", "snapshot_scheduling", "snapshot_schedule_aliases", "container_copy_project", "clustering_server_address", "clustering_image_replication", "container_protection_shift", "snapshot_expiry", "container_backup_override_pool", "snapshot_expiry_creation", "network_leases_location", "resources_cpu_socket", "resources_gpu", "resources_numa", "kernel_features", "id_map_current", "event_location", "storage_api_remote_volume_snapshots", "network_nat_address", "container_nic_routes", "rbac", "cluster_internal_copy", "seccomp_notify", "lxc_features", "container_nic_ipvlan", "network_vlan_sriov", "storage_cephfs", "container_nic_ipfilter", "resources_v2", "container_exec_user_group_cwd", "container_syscall_intercept", "container_disk_shift", "storage_shifted", "resources_infiniband", "daemon_storage", "instances", "image_types", "resources_disk_sata", "clustering_roles", "images_expiry", "resources_network_firmware", "backup_compression_algorithm", "ceph_data_pool_name", "container_syscall_intercept_mount", "compression_squashfs", "container_raw_mount", "container_nic_routed", "container_syscall_intercept_mount_fuse", "container_disk_ceph", "virtual-machines", "image_profiles", "clustering_architecture", "resources_disk_id", "storage_lvm_stripes", "vm_boot_priority", "unix_hotplug_devices", "api_filtering", "instance_nic_network", "clustering_sizing", "firewall_driver", "projects_limits", "container_syscall_intercept_hugetlbfs", "limits_hugepages", "container_nic_routed_gateway", "projects_restrictions", "custom_volume_snapshot_expiry", "volume_snapshot_scheduling", "trust_ca_certificates", "snapshot_disk_usage", "clustering_edit_roles", "container_nic_routed_host_address", "container_nic_ipvlan_gateway", "resources_usb_pci", "resources_cpu_threads_numa", "resources_cpu_core_die", "api_os", "container_nic_routed_host_table", "container_nic_ipvlan_host_table", "container_nic_ipvlan_mode", "resources_system", "images_push_relay", "network_dns_search", "container_nic_routed_limits", "instance_nic_bridged_vlan", "network_state_bond_bridge", "usedby_consistency", "custom_block_volumes", "clustering_failure_domains", "resources_gpu_mdev", "console_vga_type", "projects_limits_disk", "network_type_macvlan", "network_type_sriov", "container_syscall_intercept_bpf_devices", "network_type_ovn", "projects_networks", "projects_networks_restricted_uplinks", "custom_volume_backup", "backup_override_name", "storage_rsync_compression", "network_type_physical", "network_ovn_external_subnets", "network_ovn_nat", "network_ovn_external_routes_remove", "tpm_device_type", "storage_zfs_clone_copy_rebase", "gpu_mdev", "resources_pci_iommu", "resources_network_usb", "resources_disk_address", "network_physical_ovn_ingress_mode", "network_ovn_dhcp", "network_physical_routes_anycast", "projects_limits_instances", "network_state_vlan", "instance_nic_bridged_port_isolation", "instance_bulk_state_change", "network_gvrp", "instance_pool_move", "gpu_sriov", "pci_device_type", "storage_volume_state", "network_acl", "migration_stateful", "disk_state_quota", "storage_ceph_features", "projects_compression", "projects_images_remote_cache_expiry", "certificate_project", "network_ovn_acl", "projects_images_auto_update", "projects_restricted_cluster_target", "images_default_architecture", "network_ovn_acl_defaults", "gpu_mig", "project_usage", "network_bridge_acl", "warnings", "projects_restricted_backups_and_snapshots", "clustering_join_token", "clustering_description", "server_trusted_proxy", "clustering_update_cert", "storage_api_project", "server_instance_driver_operational", "server_supported_storage_drivers", "event_lifecycle_requestor_address", "resources_gpu_usb", "clustering_evacuation", "network_ovn_nat_address", "network_bgp", "network_forward", "custom_volume_refresh", "network_counters_errors_dropped", "metrics", "image_source_project", "clustering_config", "network_peer", "linux_sysctl", "network_dns", "ovn_nic_acceleration", "certificate_self_renewal", "instance_project_move", "storage_volume_project_move", "cloud_init", "network_dns_nat", "database_leader", "instance_all_projects", "clustering_groups", "ceph_rbd_du", "instance_get_full", "qemu_metrics", "gpu_mig_uuid", "event_project", "clustering_evacuation_live", "instance_allow_inconsistent_copy", "network_state_ovn", "storage_volume_api_filtering", "image_restrictions", "storage_zfs_export", "network_dns_records", "storage_zfs_reserve_space", "network_acl_log", "storage_zfs_blocksize", "metrics_cpu_seconds", "instance_snapshot_never", "certificate_token", "instance_nic_routed_neighbor_probe", "event_hub", "agent_nic_config", "projects_restricted_intercept", "metrics_authentication", "images_target_project", "cluster_migration_inconsistent_copy", "cluster_ovn_chassis", "container_syscall_intercept_sched_setscheduler", "storage_lvm_thinpool_metadata_size", "storage_volume_state_total", "instance_file_head", "instances_nic_host_name", "image_copy_profile", "container_syscall_intercept_sysinfo", "clustering_evacuation_mode", "resources_pci_vpd", "qemu_raw_conf", "storage_cephfs_fscache", "network_load_balancer", "vsock_api", "instance_ready_state", "network_bgp_holdtime", "storage_volumes_all_projects", "metrics_memory_oom_total", "storage_buckets", "storage_buckets_create_credentials", "metrics_cpu_effective_total", "projects_networks_restricted_access", "storage_buckets_local", "loki", "acme", "internal_metrics", "cluster_join_token_expiry", "remote_token_expiry", "init_preseed", "storage_volumes_created_at", "cpu_hotplug", "projects_networks_zones", "network_txqueuelen", "cluster_member_state", "instances_placement_scriptlet", "storage_pool_source_wipe", "zfs_block_mode", "instance_generation_id", "disk_io_cache", "amd_sev", "storage_pool_loop_resize", "migration_vm_live", "ovn_nic_nesting", "oidc", "network_ovn_l3only", "ovn_nic_acceleration_vdpa", "cluster_healing", "instances_state_total", "auth_user", "security_csm", "instances_rebuild", "numa_cpu_placement" ], "api_status": "stable", "api_version": "1.0", "auth": "trusted", "public": false, "auth_methods": [ "tls" ], "auth_user_name": "yhaliaw", "auth_user_method": "unix", "environment": { "addresses": [], "architectures": [ "x86_64", "i686" ], "certificate": "-----BEGIN CERTIFICATE-----\nMIICBjCCAYygAwIBAgIQZD1S3AZ+ZITdDIwOy7WfxDAKBggqhkjOPQQDAzA1MRww\nGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRUwEwYDVQQDDAxyb290QGRlc2t0\nb3AwHhcNMjMwNjI2MDYxMjI1WhcNMzMwNjIzMDYxMjI1WjA1MRwwGgYDVQQKExNs\naW51eGNvbnRhaW5lcnMub3JnMRUwEwYDVQQDDAxyb290QGRlc2t0b3AwdjAQBgcq\nhkjOPQIBBgUrgQQAIgNiAARGNIDC1owm7HpvcfTHcvN54n/ftxM8QDf5cvn6jvgG\n0PMIMeVmdQZE2/81mc6jI2XwthQ4VbfOXQHTvGuL6COmMe2rpyFaCJddqi10gEls\n8A1TlE79ALH9C54806D4DX2jYTBfMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAK\nBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMCoGA1UdEQQjMCGCB2Rlc2t0b3CHBH8A\nAAGHEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDaAAwZQIwCE0a9zPRuW4T\nrZBzBfzSQXyWdkXFUdykhppJKpeuVAaVxf7OtiMs4b3f7/2pIQ1wAjEA6u3XeSpI\nX2ZIjsFT8LEvNkm5RqaZGOygCPbu0jM+j66PgYlAcABMuIyq8MertI2Z\n-----END CERTIFICATE-----\n", "certificate_fingerprint": "a09ea47f5c8b4892458a5c9f55b88620df5b48ef0669b36a848e8802b41882d4", "driver": "lxc | qemu", "driver_version": "5.0.2 | 8.0.0", "firewall": "nftables", "kernel": "Linux", "kernel_architecture": "x86_64", "kernel_features": { "idmapped_mounts": "true", "netnsid_getifaddrs": "true", "seccomp_listener": "true", "seccomp_listener_continue": "true", "shiftfs": "false", "uevent_injection": "true", "unpriv_fscaps": "true" }, "kernel_version": "5.19.0-46-generic", "lxc_features": { "cgroup2": "true", "core_scheduling": "true", "devpts_fd": "true", "idmapped_mounts_v2": "true", "mount_injection_file": "true", "network_gateway_device_route": "true", "network_ipvlan": "true", "network_l2proxy": "true", "network_phys_macvlan_mtu": "true", "network_veth_router": "true", "pidfd": "true", "seccomp_allow_deny_syntax": "true", "seccomp_notify": "true", "seccomp_proxy_send_notify_fd": "true" }, "os_name": "Ubuntu", "os_version": "22.04", "project": "default", "server": "lxd", "server_clustered": false, "server_event_mode": "full-mesh", "server_name": "desktop", "server_pid": 2693, "server_version": "5.15", "storage": "dir", "storage_version": "1", "storage_supported_drivers": [ { "Name": "cephobject", "Version": "17.2.5", "Remote": true }, { "Name": "dir", "Version": "1", "Remote": false }, { "Name": "lvm", "Version": "2.03.11(2) (2021-01-08) / 1.02.175 (2021-01-08) / 4.47.0", "Remote": false }, { "Name": "zfs", "Version": "2.1.5-1ubuntu6", "Remote": false }, { "Name": "btrfs", "Version": "5.16.2", "Remote": false }, { "Name": "ceph", "Version": "17.2.5", "Remote": true }, { "Name": "cephfs", "Version": "17.2.5", "Remote": true } ] } } DEBUG [2023-07-17T12:44:17+08:00] Connected to the websocket: ws://unix.socket/1.0/events DEBUG [2023-07-17T12:44:17+08:00] Sending request to LXD etag= method=POST url="http://unix.socket/1.0/instances/test/exec" DEBUG [2023-07-17T12:44:17+08:00] { "command": [ "systemctl", "halt", "-i" ], "wait-for-websocket": true, "interactive": true, "environment": { "TERM": "xterm-256color" }, "width": 270, "height": 52, "record-output": false, "user": 0, "group": 0, "cwd": "" } DEBUG [2023-07-17T12:44:17+08:00] Got operation from LXD
DEBUG [2023-07-17T12:44:17+08:00] { "id": "917275a4-5243-4229-9d84-11a562137cb5", "class": "websocket", "description": "Executing command", "created_at": "2023-07-17T12:44:17.011298334+08:00", "updated_at": "2023-07-17T12:44:17.011298334+08:00", "status": "Running", "status_code": 103, "resources": { "instances": [ "/1.0/instances/test" ] }, "metadata": { "command": [ "systemctl", "halt", "-i" ], "environment": { "HOME": "/root", "LANG": "C.UTF-8", "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "TERM": "xterm-256color", "USER": "root" }, "fds": { "0": "d83c232c80e8605910d432e5c67d06e2f624e4745cd1c6092cc76e326b734bc3", "control": "240c112f9a3c26d34aac07f734288d82fd47354d9640410517b8574f964fb98d" }, "interactive": true }, "may_cancel": false, "err": "", "location": "none" } DEBUG [2023-07-17T12:44:17+08:00] Connected to the websocket: ws://unix.socket/1.0/operations/917275a4-5243-4229-9d84-11a562137cb5/websocket?secret=240c112f9a3c26d34aac07f734288d82fd47354d9640410517b8574f964fb98d DEBUG [2023-07-17T12:44:17+08:00] Connected to the websocket: ws://unix.socket/1.0/operations/917275a4-5243-4229-9d84-11a562137cb5/websocket?secret=d83c232c80e8605910d432e5c67d06e2f624e4745cd1c6092cc76e326b734bc3 DEBUG [2023-07-17T12:44:17+08:00] Sending request to LXD etag= method=GET url="http://unix.socket/1.0/operations/917275a4-5243-4229-9d84-11a562137cb5" DEBUG [2023-07-17T12:44:17+08:00] Websocket: Started read mirror address=/var/snap/lxd/common/lxd/unix.socket DEBUG [2023-07-17T12:44:17+08:00] Websocket: Started write mirror address=/var/snap/lxd/common/lxd/unix.socket DEBUG [2023-07-17T12:44:17+08:00] Got response struct from LXD
DEBUG [2023-07-17T12:44:17+08:00] { "id": "917275a4-5243-4229-9d84-11a562137cb5", "class": "websocket", "description": "Executing command", "created_at": "2023-07-17T12:44:17.011298334+08:00", "updated_at": "2023-07-17T12:44:17.011298334+08:00", "status": "Running", "status_code": 103, "resources": { "instances": [ "/1.0/instances/test" ] }, "metadata": { "command": [ "systemctl", "halt", "-i" ], "environment": { "HOME": "/root", "LANG": "C.UTF-8", "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "TERM": "xterm-256color", "USER": "root" }, "fds": { "0": "d83c232c80e8605910d432e5c67d06e2f624e4745cd1c6092cc76e326b734bc3", "control": "240c112f9a3c26d34aac07f734288d82fd47354d9640410517b8574f964fb98d" }, "interactive": true }, "may_cancel": false, "err": "", "location": "none" } DEBUG [2023-07-17T12:44:17+08:00] Websocket: Got barrier message address=/var/snap/lxd/common/lxd/unix.socket DEBUG [2023-07-17T12:44:17+08:00] Websocket: Stopped write mirror address=/var/snap/lxd/common/lxd/unix.socket

 - [X] Output of the daemon with --debug (alternatively output of `lxc monitor` while reproducing the issue)

ocation: none metadata: context: id: 512bb3cc-83e5-4163-aaed-86662afc4fc7 local: /var/snap/lxd/common/lxd/unix.socket remote: '@' level: debug message: Event listener server handler started timestamp: "2023-07-17T12:47:15.467013287+08:00" type: logging

location: none metadata: context: instance: test instanceType: virtual-machine project: default level: debug message: Instance agent started timestamp: "2023-07-17T12:47:16.543928402+08:00" type: logging

location: none metadata: context: etag: "" method: GET url: https://custom.socket/1.0 level: debug message: Sending request to LXD timestamp: "2023-07-17T12:47:16.546479757+08:00" type: logging

location: none metadata: context: {} level: debug message: Connecting to a VM agent over a VM socket timestamp: "2023-07-17T12:47:16.546460801+08:00" type: logging

location: none metadata: context: {} level: debug message: Got response struct from LXD timestamp: "2023-07-17T12:47:16.552068903+08:00" type: logging

location: none metadata: context: {} level: debug message: "\n\t{\n\t\t\"cid\": 2,\n\t\t\"port\": 35154,\n\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\nMIICBjCCAYygAwIBAgIQZD1S3AZ+ZITdDIwOy7WfxDAKBggqhkjOPQQDAzA1MRww\nGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRUwEwYDVQQDDAxyb290QGRlc2t0\nb3AwHhcNMjMwNjI2MDYxMjI1WhcNMzMwNjIzMDYxMjI1WjA1MRwwGgYDVQQKExNs\naW51eGNvbnRhaW5lcnMub3JnMRUwEwYDVQQDDAxyb290QGRlc2t0b3AwdjAQBgcq\nhkjOPQIBBgUrgQQAIgNiAARGNIDC1owm7HpvcfTHcvN54n/ftxM8QDf5cvn6jvgG\n0PMIMeVmdQZE2/81mc6jI2XwthQ4VbfOXQHTvGuL6COmMe2rpyFaCJddqi10gEls\n8A1TlE79ALH9C54806D4DX2jYTBfMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAK\nBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMCoGA1UdEQQjMCGCB2Rlc2t0b3CHBH8A\nAAGHEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDaAAwZQIwCE0a9zPRuW4T\nrZBzBfzSQXyWdkXFUdykhppJKpeuVAaVxf7OtiMs4b3f7/2pIQ1wAjEA6u3XeSpI\nX2ZIjsFT8LEvNkm5RqaZGOygCPbu0jM+j66PgYlAcABMuIyq8MertI2Z\n-----END CERTIFICATE-----\n\",\n\t\t\"devlxd\": true\n\t}" timestamp: "2023-07-17T12:47:16.552380636+08:00" type: logging

location: none metadata: context: etag: "" method: PUT url: https://custom.socket/1.0 level: debug message: Sending request to LXD timestamp: "2023-07-17T12:47:16.552351912+08:00" type: logging

location: none metadata: context: {} level: debug message: "\n\t{\n\t\t\"config\": null,\n\t\t\"api_extensions\": [\n\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\"container_stop_priority\",\n\t\t\t\"container_syscall_filtering\",\n\t\t\t\"auth_pki\",\n\t\t\t\"container_last_used_at\",\n\t\t\t\"etag\",\n\t\t\t\"patch\",\n\t\t\t\"usb_devices\",\n\t\t\t\"https_allowed_credentials\",\n\t\t\t\"image_compression_algorithm\",\n\t\t\t\"directory_manipulation\",\n\t\t\t\"container_cpu_time\",\n\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\"network\",\n\t\t\t\"profile_usedby\",\n\t\t\t\"container_push\",\n\t\t\t\"container_exec_recording\",\n\t\t\t\"certificate_update\",\n\t\t\t\"container_exec_signal_handling\",\n\t\t\t\"gpu_devices\",\n\t\t\t\"container_image_properties\",\n\t\t\t\"migration_progress\",\n\t\t\t\"id_map\",\n\t\t\t\"network_firewall_filtering\",\n\t\t\t\"network_routes\",\n\t\t\t\"storage\",\n\t\t\t\"file_delete\",\n\t\t\t\"file_append\",\n\t\t\t\"network_dhcp_expiry\",\n\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\"network_vlan\",\n\t\t\t\"image_create_aliases\",\n\t\t\t\"container_stateless_copy\",\n\t\t\t\"container_only_migration\",\n\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\"unix_device_rename\",\n\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\"network_vxlan_interface\",\n\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\"entity_description\",\n\t\t\t\"image_force_refresh\",\n\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\"id_map_base\",\n\t\t\t\"file_symlinks\",\n\t\t\t\"container_push_target\",\n\t\t\t\"network_vlan_physical\",\n\t\t\t\"storage_images_delete\",\n\t\t\t\"container_edit_metadata\",\n\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\"storage_driver_ceph\",\n\t\t\t\"storage_ceph_user_name\",\n\t\t\t\"resource_limits\",\n\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\"resources\",\n\t\t\t\"kernel_limits\",\n\t\t\t\"storage_api_volume_rename\",\n\t\t\t\"macaroon_authentication\",\n\t\t\t\"network_sriov\",\n\t\t\t\"console\",\n\t\t\t\"restrict_devlxd\",\n\t\t\t\"migration_pre_copy\",\n\t\t\t\"infiniband\",\n\t\t\t\"maas_network\",\n\t\t\t\"devlxd_events\",\n\t\t\t\"proxy\",\n\t\t\t\"network_dhcp_gateway\",\n\t\t\t\"file_get_symlink\",\n\t\t\t\"network_leases\",\n\t\t\t\"unix_device_hotplug\",\n\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\"operation_description\",\n\t\t\t\"clustering\",\n\t\t\t\"event_lifecycle\",\n\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\"nvidia_runtime\",\n\t\t\t\"container_mount_propagation\",\n\t\t\t\"container_backup\",\n\t\t\t\"devlxd_images\",\n\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\"proxy_unix\",\n\t\t\t\"proxy_udp\",\n\t\t\t\"clustering_join\",\n\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\"network_state\",\n\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\"container_protection_delete\",\n\t\t\t\"unix_priv_drop\",\n\t\t\t\"pprof_http\",\n\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\"network_hwaddr\",\n\t\t\t\"proxy_nat\",\n\t\t\t\"network_nat_order\",\n\t\t\t\"container_full\",\n\t\t\t\"candid_authentication\",\n\t\t\t\"backup_compression\",\n\t\t\t\"candid_config\",\n\t\t\t\"nvidia_runtime_config\",\n\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\"storage_unmapped\",\n\t\t\t\"projects\",\n\t\t\t\"candid_config_key\",\n\t\t\t\"network_vxlan_ttl\",\n\t\t\t\"container_incremental_copy\",\n\t\t\t\"usb_optional_vendorid\",\n\t\t\t\"snapshot_scheduling\",\n\t\t\t\"snapshot_schedule_aliases\",\n\t\t\t\"container_copy_project\",\n\t\t\t\"clustering_server_address\",\n\t\t\t\"clustering_image_replication\",\n\t\t\t\"container_protection_shift\",\n\t\t\t\"snapshot_expiry\",\n\t\t\t\"container_backup_override_pool\",\n\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\"network_leases_location\",\n\t\t\t\"resources_cpu_socket\",\n\t\t\t\"resources_gpu\",\n\t\t\t\"resources_numa\",\n\t\t\t\"kernel_features\",\n\t\t\t\"id_map_current\",\n\t\t\t\"event_location\",\n\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\"network_nat_address\",\n\t\t\t\"container_nic_routes\",\n\t\t\t\"rbac\",\n\t\t\t\"cluster_internal_copy\",\n\t\t\t\"seccomp_notify\",\n\t\t\t\"lxc_features\",\n\t\t\t\"container_nic_ipvlan\",\n\t\t\t\"network_vlan_sriov\",\n\t\t\t\"storage_cephfs\",\n\t\t\t\"container_nic_ipfilter\",\n\t\t\t\"resources_v2\",\n\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\"container_syscall_intercept\",\n\t\t\t\"container_disk_shift\",\n\t\t\t\"storage_shifted\",\n\t\t\t\"resources_infiniband\",\n\t\t\t\"daemon_storage\",\n\t\t\t\"instances\",\n\t\t\t\"image_types\",\n\t\t\t\"resources_disk_sata\",\n\t\t\t\"clustering_roles\",\n\t\t\t\"images_expiry\",\n\t\t\t\"resources_network_firmware\",\n\t\t\t\"backup_compression_algorithm\",\n\t\t\t\"ceph_data_pool_name\",\n\t\t\t\"container_syscall_intercept_mount\",\n\t\t\t\"compression_squashfs\",\n\t\t\t\"container_raw_mount\",\n\t\t\t\"container_nic_routed\",\n\t\t\t\"container_syscall_intercept_mount_fuse\",\n\t\t\t\"container_disk_ceph\",\n\t\t\t\"virtual-machines\",\n\t\t\t\"image_profiles\",\n\t\t\t\"clustering_architecture\",\n\t\t\t\"resources_disk_id\",\n\t\t\t\"storage_lvm_stripes\",\n\t\t\t\"vm_boot_priority\",\n\t\t\t\"unix_hotplug_devices\",\n\t\t\t\"api_filtering\",\n\t\t\t\"instance_nic_network\",\n\t\t\t\"clustering_sizing\",\n\t\t\t\"firewall_driver\",\n\t\t\t\"projects_limits\",\n\t\t\t\"container_syscall_intercept_hugetlbfs\",\n\t\t\t\"limits_hugepages\",\n\t\t\t\"container_nic_routed_gateway\",\n\t\t\t\"projects_restrictions\",\n\t\t\t\"custom_volume_snapshot_expiry\",\n\t\t\t\"volume_snapshot_scheduling\",\n\t\t\t\"trust_ca_certificates\",\n\t\t\t\"snapshot_disk_usage\",\n\t\t\t\"clustering_edit_roles\",\n\t\t\t\"container_nic_routed_host_address\",\n\t\t\t\"container_nic_ipvlan_gateway\",\n\t\t\t\"resources_usb_pci\",\n\t\t\t\"resources_cpu_threads_numa\",\n\t\t\t\"resources_cpu_core_die\",\n\t\t\t\"api_os\",\n\t\t\t\"container_nic_routed_host_table\",\n\t\t\t\"container_nic_ipvlan_host_table\",\n\t\t\t\"container_nic_ipvlan_mode\",\n\t\t\t\"resources_system\",\n\t\t\t\"images_push_relay\",\n\t\t\t\"network_dns_search\",\n\t\t\t\"container_nic_routed_limits\",\n\t\t\t\"instance_nic_bridged_vlan\",\n\t\t\t\"network_state_bond_bridge\",\n\t\t\t\"usedby_consistency\",\n\t\t\t\"custom_block_volumes\",\n\t\t\t\"clustering_failure_domains\",\n\t\t\t\"resources_gpu_mdev\",\n\t\t\t\"console_vga_type\",\n\t\t\t\"projects_limits_disk\",\n\t\t\t\"network_type_macvlan\",\n\t\t\t\"network_type_sriov\",\n\t\t\t\"container_syscall_intercept_bpf_devices\",\n\t\t\t\"network_type_ovn\",\n\t\t\t\"projects_networks\",\n\t\t\t\"projects_networks_restricted_uplinks\",\n\t\t\t\"custom_volume_backup\",\n\t\t\t\"backup_override_name\",\n\t\t\t\"storage_rsync_compression\",\n\t\t\t\"network_type_physical\",\n\t\t\t\"network_ovn_external_subnets\",\n\t\t\t\"network_ovn_nat\",\n\t\t\t\"network_ovn_external_routes_remove\",\n\t\t\t\"tpm_device_type\",\n\t\t\t\"storage_zfs_clone_copy_rebase\",\n\t\t\t\"gpu_mdev\",\n\t\t\t\"resources_pci_iommu\",\n\t\t\t\"resources_network_usb\",\n\t\t\t\"resources_disk_address\",\n\t\t\t\"network_physical_ovn_ingress_mode\",\n\t\t\t\"network_ovn_dhcp\",\n\t\t\t\"network_physical_routes_anycast\",\n\t\t\t\"projects_limits_instances\",\n\t\t\t\"network_state_vlan\",\n\t\t\t\"instance_nic_bridged_port_isolation\",\n\t\t\t\"instance_bulk_state_change\",\n\t\t\t\"network_gvrp\",\n\t\t\t\"instance_pool_move\",\n\t\t\t\"gpu_sriov\",\n\t\t\t\"pci_device_type\",\n\t\t\t\"storage_volume_state\",\n\t\t\t\"network_acl\",\n\t\t\t\"migration_stateful\",\n\t\t\t\"disk_state_quota\",\n\t\t\t\"storage_ceph_features\",\n\t\t\t\"projects_compression\",\n\t\t\t\"projects_images_remote_cache_expiry\",\n\t\t\t\"certificate_project\",\n\t\t\t\"network_ovn_acl\",\n\t\t\t\"projects_images_auto_update\",\n\t\t\t\"projects_restricted_cluster_target\",\n\t\t\t\"images_default_architecture\",\n\t\t\t\"network_ovn_acl_defaults\",\n\t\t\t\"gpu_mig\",\n\t\t\t\"project_usage\",\n\t\t\t\"network_bridge_acl\",\n\t\t\t\"warnings\",\n\t\t\t\"projects_restricted_backups_and_snapshots\",\n\t\t\t\"clustering_join_token\",\n\t\t\t\"clustering_description\",\n\t\t\t\"server_trusted_proxy\",\n\t\t\t\"clustering_update_cert\",\n\t\t\t\"storage_api_project\",\n\t\t\t\"server_instance_driver_operational\",\n\t\t\t\"server_supported_storage_drivers\",\n\t\t\t\"event_lifecycle_requestor_address\",\n\t\t\t\"resources_gpu_usb\",\n\t\t\t\"clustering_evacuation\",\n\t\t\t\"network_ovn_nat_address\",\n\t\t\t\"network_bgp\",\n\t\t\t\"network_forward\",\n\t\t\t\"custom_volume_refresh\",\n\t\t\t\"network_counters_errors_dropped\",\n\t\t\t\"metrics\",\n\t\t\t\"image_source_project\",\n\t\t\t\"clustering_config\",\n\t\t\t\"network_peer\",\n\t\t\t\"linux_sysctl\",\n\t\t\t\"network_dns\",\n\t\t\t\"ovn_nic_acceleration\",\n\t\t\t\"certificate_self_renewal\",\n\t\t\t\"instance_project_move\",\n\t\t\t\"storage_volume_project_move\",\n\t\t\t\"cloud_init\",\n\t\t\t\"network_dns_nat\",\n\t\t\t\"database_leader\",\n\t\t\t\"instance_all_projects\",\n\t\t\t\"clustering_groups\",\n\t\t\t\"ceph_rbd_du\",\n\t\t\t\"instance_get_full\",\n\t\t\t\"qemu_metrics\",\n\t\t\t\"gpu_mig_uuid\",\n\t\t\t\"event_project\",\n\t\t\t\"clustering_evacuation_live\",\n\t\t\t\"instance_allow_inconsistent_copy\",\n\t\t\t\"network_state_ovn\",\n\t\t\t\"storage_volume_api_filtering\",\n\t\t\t\"image_restrictions\",\n\t\t\t\"storage_zfs_export\",\n\t\t\t\"network_dns_records\",\n\t\t\t\"storage_zfs_reserve_space\",\n\t\t\t\"network_acl_log\",\n\t\t\t\"storage_zfs_blocksize\",\n\t\t\t\"metrics_cpu_seconds\",\n\t\t\t\"instance_snapshot_never\",\n\t\t\t\"certificate_token\",\n\t\t\t\"instance_nic_routed_neighbor_probe\",\n\t\t\t\"event_hub\",\n\t\t\t\"agent_nic_config\",\n\t\t\t\"projects_restricted_intercept\",\n\t\t\t\"metrics_authentication\",\n\t\t\t\"images_target_project\",\n\t\t\t\"cluster_migration_inconsistent_copy\",\n\t\t\t\"cluster_ovn_chassis\",\n\t\t\t\"container_syscall_intercept_sched_setscheduler\",\n\t\t\t\"storage_lvm_thinpool_metadata_size\",\n\t\t\t\"storage_volume_state_total\",\n\t\t\t\"instance_file_head\",\n\t\t\t\"instances_nic_host_name\",\n\t\t\t\"image_copy_profile\",\n\t\t\t\"container_syscall_intercept_sysinfo\",\n\t\t\t\"clustering_evacuation_mode\",\n\t\t\t\"resources_pci_vpd\",\n\t\t\t\"qemu_raw_conf\",\n\t\t\t\"storage_cephfs_fscache\",\n\t\t\t\"network_load_balancer\",\n\t\t\t\"vsock_api\",\n\t\t\t\"instance_ready_state\",\n\t\t\t\"network_bgp_holdtime\",\n\t\t\t\"storage_volumes_all_projects\",\n\t\t\t\"metrics_memory_oom_total\",\n\t\t\t\"storage_buckets\",\n\t\t\t\"storage_buckets_create_credentials\",\n\t\t\t\"metrics_cpu_effective_total\",\n\t\t\t\"projects_networks_restricted_access\",\n\t\t\t\"storage_buckets_local\",\n\t\t\t\"loki\",\n\t\t\t\"acme\",\n\t\t\t\"internal_metrics\",\n\t\t\t\"cluster_join_token_expiry\",\n\t\t\t\"remote_token_expiry\",\n\t\t\t\"init_preseed\",\n\t\t\t\"storage_volumes_created_at\",\n\t\t\t\"cpu_hotplug\",\n\t\t\t\"projects_networks_zones\",\n\t\t\t\"network_txqueuelen\",\n\t\t\t\"cluster_member_state\",\n\t\t\t\"instances_placement_scriptlet\",\n\t\t\t\"storage_pool_source_wipe\",\n\t\t\t\"zfs_block_mode\",\n\t\t\t\"instance_generation_id\",\n\t\t\t\"disk_io_cache\",\n\t\t\t\"amd_sev\",\n\t\t\t\"storage_pool_loop_resize\",\n\t\t\t\"migration_vm_live\",\n\t\t\t\"ovn_nic_nesting\",\n\t\t\t\"oidc\",\n\t\t\t\"network_ovn_l3only\",\n\t\t\t\"ovn_nic_acceleration_vdpa\",\n\t\t\t\"cluster_healing\",\n\t\t\t\"instances_state_total\",\n\t\t\t\"auth_user\",\n\t\t\t\"security_csm\",\n\t\t\t\"instances_rebuild\",\n\t\t\t\"numa_cpu_placement\"\n\t\t],\n\t\t\"api_status\": \"stable\",\n\t\t\"api_version\": \"1.0\",\n\t\t\"auth\": \"trusted\",\n\t\t\"public\": false,\n\t\t\"auth_methods\": [\n\t\t\t\"tls\"\n\t\t],\n\t\t\"auth_user_name\": \"\",\n\t\t\"auth_user_method\": \"\",\n\t\t\"environment\": {\n\t\t\t\"addresses\": null,\n\t\t\t\"architectures\": null,\n\t\t\t\"certificate\": \"\",\n\t\t\t\"certificate_fingerprint\": \"\",\n\t\t\t\"driver\": \"\",\n\t\t\t\"driver_version\": \"\",\n\t\t\t\"firewall\": \"\",\n\t\t\t\"kernel\": \"Linux\",\n\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\"kernel_features\": null,\n\t\t\t\"kernel_version\": \"5.15.0-1037-kvm\",\n\t\t\t\"lxc_features\": null,\n\t\t\t\"os_name\": \"\",\n\t\t\t\"os_version\": \"\",\n\t\t\t\"project\": \"\",\n\t\t\t\"server\": \"lxd-agent\",\n\t\t\t\"server_clustered\": false,\n\t\t\t\"server_event_mode\": \"\",\n\t\t\t\"server_name\": \"test\",\n\t\t\t\"server_pid\": 221,\n\t\t\t\"server_version\": \"5.15\",\n\t\t\t\"storage\": \"\",\n\t\t\t\"storage_version\": \"\",\n\t\t\t\"storage_supported_drivers\": null\n\t\t}\n\t}" timestamp: "2023-07-17T12:47:16.552224785+08:00" type: logging

location: none metadata: context: fingerprint: "0" subject: CN=root@desktop,O=linuxcontainers.org level: debug message: Matched trusted cert timestamp: "2023-07-17T12:47:16.598990115+08:00" type: logging

location: none metadata: context: fingerprint: "0" subject: CN=root@desktop,O=linuxcontainers.org level: debug message: Matched trusted cert timestamp: "2023-07-17T12:47:17.811917614+08:00" type: logging

location: none metadata: context: fingerprint: "0" subject: CN=root@desktop,O=linuxcontainers.org level: debug message: Matched trusted cert timestamp: "2023-07-17T12:47:17.817537779+08:00" type: logging

location: none metadata: context: fingerprint: "0" subject: CN=root@desktop,O=linuxcontainers.org level: debug message: Matched trusted cert timestamp: "2023-07-17T12:47:17.824405294+08:00" type: logging

location: none metadata: context: fingerprint: "0" subject: CN=root@desktop,O=linuxcontainers.org level: debug message: Matched trusted cert timestamp: "2023-07-17T12:47:17.829934879+08:00" type: logging

location: none metadata: context: fingerprint: "0" subject: CN=root@desktop,O=linuxcontainers.org level: debug message: Matched trusted cert timestamp: "2023-07-17T12:47:17.83648429+08:00" type: logging

location: none metadata: context: fingerprint: "0" subject: CN=root@desktop,O=linuxcontainers.org level: debug message: Matched trusted cert timestamp: "2023-07-17T12:47:17.842092603+08:00" type: logging

location: none metadata: context: ip: '@' method: GET protocol: unix url: /1.0 username: yhaliaw level: debug message: Handling API request timestamp: "2023-07-17T12:47:26.563795041+08:00" type: logging

location: none metadata: context: ip: '@' method: GET protocol: unix url: /1.0/events username: yhaliaw level: debug message: Handling API request timestamp: "2023-07-17T12:47:26.56483896+08:00" type: logging

location: none metadata: context: ip: '@' method: POST protocol: unix url: /1.0/instances/test/exec username: yhaliaw level: debug message: Handling API request timestamp: "2023-07-17T12:47:26.565109245+08:00" type: logging

location: none metadata: context: id: 22422edf-cd0b-4ad1-9426-48e60efece45 local: /var/snap/lxd/common/lxd/unix.socket remote: '@' level: debug message: Event listener server handler started timestamp: "2023-07-17T12:47:26.564965737+08:00" type: logging

location: none metadata: context: class: websocket description: Executing command operation: 9815aed4-d997-4d30-967a-5235542a3067 project: default level: debug message: New operation timestamp: "2023-07-17T12:47:26.567840415+08:00" type: logging

location: none metadata: context: {} level: debug message: Waiting for exec websockets to connect timestamp: "2023-07-17T12:47:26.567945311+08:00" type: logging

location: none metadata: context: class: websocket description: Executing command operation: 9815aed4-d997-4d30-967a-5235542a3067 project: default level: debug message: Started operation timestamp: "2023-07-17T12:47:26.56786952+08:00" type: logging

location: none metadata: class: websocket created_at: "2023-07-17T12:47:26.566106147+08:00" description: Executing command err: "" id: 9815aed4-d997-4d30-967a-5235542a3067 location: none may_cancel: false metadata: command:

location: none metadata: class: websocket created_at: "2023-07-17T12:47:26.566106147+08:00" description: Executing command err: "" id: 9815aed4-d997-4d30-967a-5235542a3067 location: none may_cancel: false metadata: command:

location: none metadata: context: ip: '@' method: GET protocol: unix url: /1.0/operations/9815aed4-d997-4d30-967a-5235542a3067/websocket?secret=33a57d44498551d999e863a6026074d97f25e3916647ca08c70d2673dfca5ba3 username: yhaliaw level: debug message: Handling API request timestamp: "2023-07-17T12:47:26.568257023+08:00" type: logging

location: none metadata: context: class: websocket description: Executing command operation: 9815aed4-d997-4d30-967a-5235542a3067 project: default level: debug message: Connecting to operation timestamp: "2023-07-17T12:47:26.568273153+08:00" type: logging

location: none metadata: context: class: websocket description: Executing command operation: 9815aed4-d997-4d30-967a-5235542a3067 project: default level: debug message: Connected to operation timestamp: "2023-07-17T12:47:26.568322165+08:00" type: logging

location: none metadata: context: ip: '@' method: GET protocol: unix url: /1.0/operations/9815aed4-d997-4d30-967a-5235542a3067/websocket?secret=54bfd0ab8a1ec6cea5f89c0ba38b39ae2ef8468dd19f9ae0720356d28ca27c88 username: yhaliaw level: debug message: Handling API request timestamp: "2023-07-17T12:47:26.56849617+08:00" type: logging

location: none metadata: context: class: websocket description: Executing command operation: 9815aed4-d997-4d30-967a-5235542a3067 project: default level: debug message: Connecting to operation timestamp: "2023-07-17T12:47:26.568512881+08:00" type: logging

location: none metadata: context: class: websocket description: Executing command operation: 9815aed4-d997-4d30-967a-5235542a3067 project: default level: debug message: Connected to operation timestamp: "2023-07-17T12:47:26.568561131+08:00" type: logging

location: none metadata: context: ip: '@' method: GET protocol: unix url: /1.0/operations/9815aed4-d997-4d30-967a-5235542a3067 username: yhaliaw level: debug message: Handling API request timestamp: "2023-07-17T12:47:26.568796761+08:00" type: logging

location: none metadata: context: etag: "" method: GET url: https://custom.socket/1.0 level: debug message: Sending request to LXD timestamp: "2023-07-17T12:47:26.570775516+08:00" type: logging

location: none metadata: context: {} level: debug message: Connecting to a VM agent over a VM socket timestamp: "2023-07-17T12:47:26.570760117+08:00" type: logging

location: none metadata: context: {} level: debug message: Got response struct from LXD timestamp: "2023-07-17T12:47:26.577210723+08:00" type: logging

location: none metadata: context: {} level: debug message: "\n\t{\n\t\t\"config\": null,\n\t\t\"api_extensions\": [\n\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\"container_stop_priority\",\n\t\t\t\"container_syscall_filtering\",\n\t\t\t\"auth_pki\",\n\t\t\t\"container_last_used_at\",\n\t\t\t\"etag\",\n\t\t\t\"patch\",\n\t\t\t\"usb_devices\",\n\t\t\t\"https_allowed_credentials\",\n\t\t\t\"image_compression_algorithm\",\n\t\t\t\"directory_manipulation\",\n\t\t\t\"container_cpu_time\",\n\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\"network\",\n\t\t\t\"profile_usedby\",\n\t\t\t\"container_push\",\n\t\t\t\"container_exec_recording\",\n\t\t\t\"certificate_update\",\n\t\t\t\"container_exec_signal_handling\",\n\t\t\t\"gpu_devices\",\n\t\t\t\"container_image_properties\",\n\t\t\t\"migration_progress\",\n\t\t\t\"id_map\",\n\t\t\t\"network_firewall_filtering\",\n\t\t\t\"network_routes\",\n\t\t\t\"storage\",\n\t\t\t\"file_delete\",\n\t\t\t\"file_append\",\n\t\t\t\"network_dhcp_expiry\",\n\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\"network_vlan\",\n\t\t\t\"image_create_aliases\",\n\t\t\t\"container_stateless_copy\",\n\t\t\t\"container_only_migration\",\n\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\"unix_device_rename\",\n\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\"network_vxlan_interface\",\n\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\"entity_description\",\n\t\t\t\"image_force_refresh\",\n\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\"id_map_base\",\n\t\t\t\"file_symlinks\",\n\t\t\t\"container_push_target\",\n\t\t\t\"network_vlan_physical\",\n\t\t\t\"storage_images_delete\",\n\t\t\t\"container_edit_metadata\",\n\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\"storage_driver_ceph\",\n\t\t\t\"storage_ceph_user_name\",\n\t\t\t\"resource_limits\",\n\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\"resources\",\n\t\t\t\"kernel_limits\",\n\t\t\t\"storage_api_volume_rename\",\n\t\t\t\"macaroon_authentication\",\n\t\t\t\"network_sriov\",\n\t\t\t\"console\",\n\t\t\t\"restrict_devlxd\",\n\t\t\t\"migration_pre_copy\",\n\t\t\t\"infiniband\",\n\t\t\t\"maas_network\",\n\t\t\t\"devlxd_events\",\n\t\t\t\"proxy\",\n\t\t\t\"network_dhcp_gateway\",\n\t\t\t\"file_get_symlink\",\n\t\t\t\"network_leases\",\n\t\t\t\"unix_device_hotplug\",\n\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\"operation_description\",\n\t\t\t\"clustering\",\n\t\t\t\"event_lifecycle\",\n\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\"nvidia_runtime\",\n\t\t\t\"container_mount_propagation\",\n\t\t\t\"container_backup\",\n\t\t\t\"devlxd_images\",\n\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\"proxy_unix\",\n\t\t\t\"proxy_udp\",\n\t\t\t\"clustering_join\",\n\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\"network_state\",\n\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\"container_protection_delete\",\n\t\t\t\"unix_priv_drop\",\n\t\t\t\"pprof_http\",\n\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\"network_hwaddr\",\n\t\t\t\"proxy_nat\",\n\t\t\t\"network_nat_order\",\n\t\t\t\"container_full\",\n\t\t\t\"candid_authentication\",\n\t\t\t\"backup_compression\",\n\t\t\t\"candid_config\",\n\t\t\t\"nvidia_runtime_config\",\n\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\"storage_unmapped\",\n\t\t\t\"projects\",\n\t\t\t\"candid_config_key\",\n\t\t\t\"network_vxlan_ttl\",\n\t\t\t\"container_incremental_copy\",\n\t\t\t\"usb_optional_vendorid\",\n\t\t\t\"snapshot_scheduling\",\n\t\t\t\"snapshot_schedule_aliases\",\n\t\t\t\"container_copy_project\",\n\t\t\t\"clustering_server_address\",\n\t\t\t\"clustering_image_replication\",\n\t\t\t\"container_protection_shift\",\n\t\t\t\"snapshot_expiry\",\n\t\t\t\"container_backup_override_pool\",\n\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\"network_leases_location\",\n\t\t\t\"resources_cpu_socket\",\n\t\t\t\"resources_gpu\",\n\t\t\t\"resources_numa\",\n\t\t\t\"kernel_features\",\n\t\t\t\"id_map_current\",\n\t\t\t\"event_location\",\n\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\"network_nat_address\",\n\t\t\t\"container_nic_routes\",\n\t\t\t\"rbac\",\n\t\t\t\"cluster_internal_copy\",\n\t\t\t\"seccomp_notify\",\n\t\t\t\"lxc_features\",\n\t\t\t\"container_nic_ipvlan\",\n\t\t\t\"network_vlan_sriov\",\n\t\t\t\"storage_cephfs\",\n\t\t\t\"container_nic_ipfilter\",\n\t\t\t\"resources_v2\",\n\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\"container_syscall_intercept\",\n\t\t\t\"container_disk_shift\",\n\t\t\t\"storage_shifted\",\n\t\t\t\"resources_infiniband\",\n\t\t\t\"daemon_storage\",\n\t\t\t\"instances\",\n\t\t\t\"image_types\",\n\t\t\t\"resources_disk_sata\",\n\t\t\t\"clustering_roles\",\n\t\t\t\"images_expiry\",\n\t\t\t\"resources_network_firmware\",\n\t\t\t\"backup_compression_algorithm\",\n\t\t\t\"ceph_data_pool_name\",\n\t\t\t\"container_syscall_intercept_mount\",\n\t\t\t\"compression_squashfs\",\n\t\t\t\"container_raw_mount\",\n\t\t\t\"container_nic_routed\",\n\t\t\t\"container_syscall_intercept_mount_fuse\",\n\t\t\t\"container_disk_ceph\",\n\t\t\t\"virtual-machines\",\n\t\t\t\"image_profiles\",\n\t\t\t\"clustering_architecture\",\n\t\t\t\"resources_disk_id\",\n\t\t\t\"storage_lvm_stripes\",\n\t\t\t\"vm_boot_priority\",\n\t\t\t\"unix_hotplug_devices\",\n\t\t\t\"api_filtering\",\n\t\t\t\"instance_nic_network\",\n\t\t\t\"clustering_sizing\",\n\t\t\t\"firewall_driver\",\n\t\t\t\"projects_limits\",\n\t\t\t\"container_syscall_intercept_hugetlbfs\",\n\t\t\t\"limits_hugepages\",\n\t\t\t\"container_nic_routed_gateway\",\n\t\t\t\"projects_restrictions\",\n\t\t\t\"custom_volume_snapshot_expiry\",\n\t\t\t\"volume_snapshot_scheduling\",\n\t\t\t\"trust_ca_certificates\",\n\t\t\t\"snapshot_disk_usage\",\n\t\t\t\"clustering_edit_roles\",\n\t\t\t\"container_nic_routed_host_address\",\n\t\t\t\"container_nic_ipvlan_gateway\",\n\t\t\t\"resources_usb_pci\",\n\t\t\t\"resources_cpu_threads_numa\",\n\t\t\t\"resources_cpu_core_die\",\n\t\t\t\"api_os\",\n\t\t\t\"container_nic_routed_host_table\",\n\t\t\t\"container_nic_ipvlan_host_table\",\n\t\t\t\"container_nic_ipvlan_mode\",\n\t\t\t\"resources_system\",\n\t\t\t\"images_push_relay\",\n\t\t\t\"network_dns_search\",\n\t\t\t\"container_nic_routed_limits\",\n\t\t\t\"instance_nic_bridged_vlan\",\n\t\t\t\"network_state_bond_bridge\",\n\t\t\t\"usedby_consistency\",\n\t\t\t\"custom_block_volumes\",\n\t\t\t\"clustering_failure_domains\",\n\t\t\t\"resources_gpu_mdev\",\n\t\t\t\"console_vga_type\",\n\t\t\t\"projects_limits_disk\",\n\t\t\t\"network_type_macvlan\",\n\t\t\t\"network_type_sriov\",\n\t\t\t\"container_syscall_intercept_bpf_devices\",\n\t\t\t\"network_type_ovn\",\n\t\t\t\"projects_networks\",\n\t\t\t\"projects_networks_restricted_uplinks\",\n\t\t\t\"custom_volume_backup\",\n\t\t\t\"backup_override_name\",\n\t\t\t\"storage_rsync_compression\",\n\t\t\t\"network_type_physical\",\n\t\t\t\"network_ovn_external_subnets\",\n\t\t\t\"network_ovn_nat\",\n\t\t\t\"network_ovn_external_routes_remove\",\n\t\t\t\"tpm_device_type\",\n\t\t\t\"storage_zfs_clone_copy_rebase\",\n\t\t\t\"gpu_mdev\",\n\t\t\t\"resources_pci_iommu\",\n\t\t\t\"resources_network_usb\",\n\t\t\t\"resources_disk_address\",\n\t\t\t\"network_physical_ovn_ingress_mode\",\n\t\t\t\"network_ovn_dhcp\",\n\t\t\t\"network_physical_routes_anycast\",\n\t\t\t\"projects_limits_instances\",\n\t\t\t\"network_state_vlan\",\n\t\t\t\"instance_nic_bridged_port_isolation\",\n\t\t\t\"instance_bulk_state_change\",\n\t\t\t\"network_gvrp\",\n\t\t\t\"instance_pool_move\",\n\t\t\t\"gpu_sriov\",\n\t\t\t\"pci_device_type\",\n\t\t\t\"storage_volume_state\",\n\t\t\t\"network_acl\",\n\t\t\t\"migration_stateful\",\n\t\t\t\"disk_state_quota\",\n\t\t\t\"storage_ceph_features\",\n\t\t\t\"projects_compression\",\n\t\t\t\"projects_images_remote_cache_expiry\",\n\t\t\t\"certificate_project\",\n\t\t\t\"network_ovn_acl\",\n\t\t\t\"projects_images_auto_update\",\n\t\t\t\"projects_restricted_cluster_target\",\n\t\t\t\"images_default_architecture\",\n\t\t\t\"network_ovn_acl_defaults\",\n\t\t\t\"gpu_mig\",\n\t\t\t\"project_usage\",\n\t\t\t\"network_bridge_acl\",\n\t\t\t\"warnings\",\n\t\t\t\"projects_restricted_backups_and_snapshots\",\n\t\t\t\"clustering_join_token\",\n\t\t\t\"clustering_description\",\n\t\t\t\"server_trusted_proxy\",\n\t\t\t\"clustering_update_cert\",\n\t\t\t\"storage_api_project\",\n\t\t\t\"server_instance_driver_operational\",\n\t\t\t\"server_supported_storage_drivers\",\n\t\t\t\"event_lifecycle_requestor_address\",\n\t\t\t\"resources_gpu_usb\",\n\t\t\t\"clustering_evacuation\",\n\t\t\t\"network_ovn_nat_address\",\n\t\t\t\"network_bgp\",\n\t\t\t\"network_forward\",\n\t\t\t\"custom_volume_refresh\",\n\t\t\t\"network_counters_errors_dropped\",\n\t\t\t\"metrics\",\n\t\t\t\"image_source_project\",\n\t\t\t\"clustering_config\",\n\t\t\t\"network_peer\",\n\t\t\t\"linux_sysctl\",\n\t\t\t\"network_dns\",\n\t\t\t\"ovn_nic_acceleration\",\n\t\t\t\"certificate_self_renewal\",\n\t\t\t\"instance_project_move\",\n\t\t\t\"storage_volume_project_move\",\n\t\t\t\"cloud_init\",\n\t\t\t\"network_dns_nat\",\n\t\t\t\"database_leader\",\n\t\t\t\"instance_all_projects\",\n\t\t\t\"clustering_groups\",\n\t\t\t\"ceph_rbd_du\",\n\t\t\t\"instance_get_full\",\n\t\t\t\"qemu_metrics\",\n\t\t\t\"gpu_mig_uuid\",\n\t\t\t\"event_project\",\n\t\t\t\"clustering_evacuation_live\",\n\t\t\t\"instance_allow_inconsistent_copy\",\n\t\t\t\"network_state_ovn\",\n\t\t\t\"storage_volume_api_filtering\",\n\t\t\t\"image_restrictions\",\n\t\t\t\"storage_zfs_export\",\n\t\t\t\"network_dns_records\",\n\t\t\t\"storage_zfs_reserve_space\",\n\t\t\t\"network_acl_log\",\n\t\t\t\"storage_zfs_blocksize\",\n\t\t\t\"metrics_cpu_seconds\",\n\t\t\t\"instance_snapshot_never\",\n\t\t\t\"certificate_token\",\n\t\t\t\"instance_nic_routed_neighbor_probe\",\n\t\t\t\"event_hub\",\n\t\t\t\"agent_nic_config\",\n\t\t\t\"projects_restricted_intercept\",\n\t\t\t\"metrics_authentication\",\n\t\t\t\"images_target_project\",\n\t\t\t\"cluster_migration_inconsistent_copy\",\n\t\t\t\"cluster_ovn_chassis\",\n\t\t\t\"container_syscall_intercept_sched_setscheduler\",\n\t\t\t\"storage_lvm_thinpool_metadata_size\",\n\t\t\t\"storage_volume_state_total\",\n\t\t\t\"instance_file_head\",\n\t\t\t\"instances_nic_host_name\",\n\t\t\t\"image_copy_profile\",\n\t\t\t\"container_syscall_intercept_sysinfo\",\n\t\t\t\"clustering_evacuation_mode\",\n\t\t\t\"resources_pci_vpd\",\n\t\t\t\"qemu_raw_conf\",\n\t\t\t\"storage_cephfs_fscache\",\n\t\t\t\"network_load_balancer\",\n\t\t\t\"vsock_api\",\n\t\t\t\"instance_ready_state\",\n\t\t\t\"network_bgp_holdtime\",\n\t\t\t\"storage_volumes_all_projects\",\n\t\t\t\"metrics_memory_oom_total\",\n\t\t\t\"storage_buckets\",\n\t\t\t\"storage_buckets_create_credentials\",\n\t\t\t\"metrics_cpu_effective_total\",\n\t\t\t\"projects_networks_restricted_access\",\n\t\t\t\"storage_buckets_local\",\n\t\t\t\"loki\",\n\t\t\t\"acme\",\n\t\t\t\"internal_metrics\",\n\t\t\t\"cluster_join_token_expiry\",\n\t\t\t\"remote_token_expiry\",\n\t\t\t\"init_preseed\",\n\t\t\t\"storage_volumes_created_at\",\n\t\t\t\"cpu_hotplug\",\n\t\t\t\"projects_networks_zones\",\n\t\t\t\"network_txqueuelen\",\n\t\t\t\"cluster_member_state\",\n\t\t\t\"instances_placement_scriptlet\",\n\t\t\t\"storage_pool_source_wipe\",\n\t\t\t\"zfs_block_mode\",\n\t\t\t\"instance_generation_id\",\n\t\t\t\"disk_io_cache\",\n\t\t\t\"amd_sev\",\n\t\t\t\"storage_pool_loop_resize\",\n\t\t\t\"migration_vm_live\",\n\t\t\t\"ovn_nic_nesting\",\n\t\t\t\"oidc\",\n\t\t\t\"network_ovn_l3only\",\n\t\t\t\"ovn_nic_acceleration_vdpa\",\n\t\t\t\"cluster_healing\",\n\t\t\t\"instances_state_total\",\n\t\t\t\"auth_user\",\n\t\t\t\"security_csm\",\n\t\t\t\"instances_rebuild\",\n\t\t\t\"numa_cpu_placement\"\n\t\t],\n\t\t\"api_status\": \"stable\",\n\t\t\"api_version\": \"1.0\",\n\t\t\"auth\": \"trusted\",\n\t\t\"public\": false,\n\t\t\"auth_methods\": [\n\t\t\t\"tls\"\n\t\t],\n\t\t\"auth_user_name\": \"\",\n\t\t\"auth_user_method\": \"\",\n\t\t\"environment\": {\n\t\t\t\"addresses\": null,\n\t\t\t\"architectures\": null,\n\t\t\t\"certificate\": \"\",\n\t\t\t\"certificate_fingerprint\": \"\",\n\t\t\t\"driver\": \"\",\n\t\t\t\"driver_version\": \"\",\n\t\t\t\"firewall\": \"\",\n\t\t\t\"kernel\": \"Linux\",\n\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\"kernel_features\": null,\n\t\t\t\"kernel_version\": \"5.15.0-1037-kvm\",\n\t\t\t\"lxc_features\": null,\n\t\t\t\"os_name\": \"\",\n\t\t\t\"os_version\": \"\",\n\t\t\t\"project\": \"\",\n\t\t\t\"server\": \"lxd-agent\",\n\t\t\t\"server_clustered\": false,\n\t\t\t\"server_event_mode\": \"\",\n\t\t\t\"server_name\": \"test\",\n\t\t\t\"server_pid\": 221,\n\t\t\t\"server_version\": \"5.15\",\n\t\t\t\"storage\": \"\",\n\t\t\t\"storage_version\": \"\",\n\t\t\t\"storage_supported_drivers\": null\n\t\t}\n\t}" timestamp: "2023-07-17T12:47:26.577408402+08:00" type: logging

location: none metadata: context: etag: "" method: POST url: https://custom.socket/1.0/exec level: debug message: Sending request to LXD timestamp: "2023-07-17T12:47:26.582520129+08:00" type: logging

location: none metadata: context: {} level: debug message: 'Connected to the websocket: wss://custom.socket/1.0/events' timestamp: "2023-07-17T12:47:26.582501985+08:00" type: logging

location: none metadata: context: {} level: debug message: "\n\t{\n\t\t\"command\": [\n\t\t\t\"systemctl\",\n\t\t\t\"halt\",\n\t\t\t\"-i\"\n\t\t],\n\t\t\"wait-for-websocket\": true,\n\t\t\"interactive\": true,\n\t\t\"environment\": {\n\t\t\t\"HOME\": \"/root\",\n\t\t\t\"LANG\": \"C.UTF-8\",\n\t\t\t\"PATH\": \"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\n\t\t\t\"TERM\": \"xterm-256color\",\n\t\t\t\"USER\": \"root\"\n\t\t},\n\t\t\"width\": 270,\n\t\t\"height\": 52,\n\t\t\"record-output\": false,\n\t\t\"user\": 0,\n\t\t\"group\": 0,\n\t\t\"cwd\": \"\"\n\t}" timestamp: "2023-07-17T12:47:26.582545366+08:00" type: logging

location: none metadata: context: {} level: debug message: Got operation from LXD timestamp: "2023-07-17T12:47:26.587610795+08:00" type: logging

location: none metadata: context: {} level: debug message: "\n\t{\n\t\t\"id\": \"7a8180c9-1346-4151-a979-1c77e8531cf3\",\n\t\t\"class\": \"websocket\",\n\t\t\"description\": \"Executing command\",\n\t\t\"created_at\": \"2023-07-17T04:47:26.535080887Z\",\n\t\t\"updated_at\": \"2023-07-17T04:47:26.535080887Z\",\n\t\t\"status\": \"Running\",\n\t\t\"status_code\": 103,\n\t\t\"resources\": {},\n\t\t\"metadata\": {\n\t\t\t\"command\": [\n\t\t\t\t\"systemctl\",\n\t\t\t\t\"halt\",\n\t\t\t\t\"-i\"\n\t\t\t],\n\t\t\t\"environment\": {\n\t\t\t\t\"HOME\": \"/root\",\n\t\t\t\t\"LANG\": \"C.UTF-8\",\n\t\t\t\t\"PATH\": \"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin\",\n\t\t\t\t\"TERM\": \"xterm-256color\",\n\t\t\t\t\"USER\": \"root\"\n\t\t\t},\n\t\t\t\"fds\": {\n\t\t\t\t\"0\": \"253e55d1d98e0e94ec6e4bc63d715f4451ed16b368cf8b39e81eae71841dedcf\",\n\t\t\t\t\"control\": \"924eeba240ce24f8cad658b8fc3e0172eed9c768e929cf2b4739d1bdc6598a29\"\n\t\t\t},\n\t\t\t\"interactive\": true\n\t\t},\n\t\t\"may_cancel\": false,\n\t\t\"err\": \"\",\n\t\t\"location\": \"\"\n\t}" timestamp: "2023-07-17T12:47:26.587637054+08:00" type: logging

location: none metadata: context: {} level: debug message: 'Connected to the websocket: wss://custom.socket/1.0/operations/7a8180c9-1346-4151-a979-1c77e8531cf3/websocket?secret=924eeba240ce24f8cad658b8fc3e0172eed9c768e929cf2b4739d1bdc6598a29' timestamp: "2023-07-17T12:47:26.592465712+08:00" type: logging

location: none metadata: action: instance-exec context: command:

location: none metadata: context: {} level: debug message: 'Connected to the websocket: wss://custom.socket/1.0/operations/7a8180c9-1346-4151-a979-1c77e8531cf3/websocket?secret=253e55d1d98e0e94ec6e4bc63d715f4451ed16b368cf8b39e81eae71841dedcf' timestamp: "2023-07-17T12:47:26.597450049+08:00" type: logging

location: none metadata: context: PID: "0" instance: test interactive: "true" project: default level: debug message: Instance process started timestamp: "2023-07-17T12:47:26.597472541+08:00" type: logging

location: none metadata: context: address: vm(3877801998):8443 level: debug message: 'Websocket: Started write mirror' timestamp: "2023-07-17T12:47:26.597516463+08:00" type: logging

location: none metadata: context: etag: "" method: GET url: https://custom.socket/1.0/operations/7a8180c9-1346-4151-a979-1c77e8531cf3 level: debug message: Sending request to LXD timestamp: "2023-07-17T12:47:26.59755243+08:00" type: logging

location: none metadata: context: address: '@' level: debug message: 'Websocket: Started write mirror' timestamp: "2023-07-17T12:47:26.597569823+08:00" type: logging

location: none metadata: context: address: '@' level: debug message: 'Websocket: Started read mirror' timestamp: "2023-07-17T12:47:26.597533355+08:00" type: logging

location: none metadata: context: PID: "0" instance: test interactive: "true" number: "0" project: default level: debug message: Exec mirror websocket started timestamp: "2023-07-17T12:47:26.597517235+08:00" type: logging

location: none metadata: context: address: vm(3877801998):8443 level: debug message: 'Websocket: Started read mirror' timestamp: "2023-07-17T12:47:26.5974987+08:00" type: logging

location: none metadata: context: PID: "0" instance: test interactive: "true" project: default level: debug message: Exec control handler started timestamp: "2023-07-17T12:47:26.597512876+08:00" type: logging

location: none metadata: context: address: vm(3877801998):8443 level: debug message: 'Websocket: Stopped write mirror' timestamp: "2023-07-17T12:47:26.60982026+08:00" type: logging

location: none metadata: context: address: vm(3877801998):8443 level: debug message: 'Websocket: Got barrier message' timestamp: "2023-07-17T12:47:26.609792258+08:00" type: logging

location: none metadata: context: {} level: debug message: Got response struct from LXD timestamp: "2023-07-17T12:47:26.611621523+08:00" type: logging

location: none metadata: context: {} level: debug message: "\n\t{\n\t\t\"id\": \"7a8180c9-1346-4151-a979-1c77e8531cf3\",\n\t\t\"class\": \"websocket\",\n\t\t\"description\": \"Executing command\",\n\t\t\"created_at\": \"2023-07-17T04:47:26.535080887Z\",\n\t\t\"updated_at\": \"2023-07-17T04:47:26.535080887Z\",\n\t\t\"status\": \"Running\",\n\t\t\"status_code\": 103,\n\t\t\"resources\": {},\n\t\t\"metadata\": {\n\t\t\t\"command\": [\n\t\t\t\t\"systemctl\",\n\t\t\t\t\"halt\",\n\t\t\t\t\"-i\"\n\t\t\t],\n\t\t\t\"environment\": {\n\t\t\t\t\"HOME\": \"/root\",\n\t\t\t\t\"LANG\": \"C.UTF-8\",\n\t\t\t\t\"PATH\": \"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin\",\n\t\t\t\t\"TERM\": \"xterm-256color\",\n\t\t\t\t\"USER\": \"root\"\n\t\t\t},\n\t\t\t\"fds\": {\n\t\t\t\t\"0\": \"253e55d1d98e0e94ec6e4bc63d715f4451ed16b368cf8b39e81eae71841dedcf\",\n\t\t\t\t\"control\": \"924eeba240ce24f8cad658b8fc3e0172eed9c768e929cf2b4739d1bdc6598a29\"\n\t\t\t},\n\t\t\t\"interactive\": true\n\t\t},\n\t\t\"may_cancel\": false,\n\t\t\"err\": \"\",\n\t\t\"location\": \"\"\n\t}" timestamp: "2023-07-17T12:47:26.61165712+08:00" type: logging

location: none metadata: context: PID: "0" err: exitStatus: "0" instance: test interactive: "true" project: default level: debug message: Instance process stopped timestamp: "2023-07-17T12:47:26.611774519+08:00" type: logging

location: none metadata: context: address: '@' level: debug message: 'Websocket: Sending barrier message' timestamp: "2023-07-17T12:47:26.611828961+08:00" type: logging

location: none metadata: context: PID: "0" instance: test interactive: "true" project: default level: debug message: Exec control handler finished timestamp: "2023-07-17T12:47:26.611808863+08:00" type: logging

location: none metadata: context: address: '@' level: debug message: 'Websocket: Stopped read mirror' timestamp: "2023-07-17T12:47:26.611866811+08:00" type: logging

location: none metadata: context: address: vm(3877801998):8443 level: debug message: 'Websocket: Sending barrier message' timestamp: "2023-07-17T12:47:26.611811749+08:00" type: logging

location: none metadata: context: address: vm(3877801998):8443 level: debug message: 'Websocket: Stopped read mirror' timestamp: "2023-07-17T12:47:26.611846674+08:00" type: logging

location: none metadata: context: address: '@' level: debug message: 'Websocket: Stopped write mirror' timestamp: "2023-07-17T12:47:26.611986034+08:00" type: logging

location: none metadata: context: address: '@' level: debug message: 'Websocket: Got normal closure' timestamp: "2023-07-17T12:47:26.611969493+08:00" type: logging

location: none metadata: context: PID: "0" instance: test interactive: "true" number: "0" project: default level: debug message: Exec mirror websocket finished timestamp: "2023-07-17T12:47:26.612443228+08:00" type: logging

location: none metadata: context: class: websocket description: Executing command operation: 9815aed4-d997-4d30-967a-5235542a3067 project: default level: debug message: Updated metadata for operation timestamp: "2023-07-17T12:47:26.61246594+08:00" type: logging

location: none metadata: context: class: websocket description: Executing command operation: 9815aed4-d997-4d30-967a-5235542a3067 project: default level: debug message: Success for operation timestamp: "2023-07-17T12:47:26.612490326+08:00" type: logging

location: none metadata: class: websocket created_at: "2023-07-17T12:47:26.566106147+08:00" description: Executing command err: "" id: 9815aed4-d997-4d30-967a-5235542a3067 location: none may_cancel: false metadata: command:

location: none metadata: class: websocket created_at: "2023-07-17T12:47:26.566106147+08:00" description: Executing command err: "" id: 9815aed4-d997-4d30-967a-5235542a3067 location: none may_cancel: false metadata: command:

location: none metadata: context: listener: 22422edf-cd0b-4ad1-9426-48e60efece45 local: /var/snap/lxd/common/lxd/unix.socket remote: '@' level: debug message: Event listener server handler stopped timestamp: "2023-07-17T12:47:26.613540397+08:00" type: logging

tomponline commented 1 year ago

Hi,

Which previous versions are you referring to?

I reproduced this behaviour with LXD 5.15, but I'm not clear on what the correct behaviour should actually be.

The console shows the OS correctly entered halted state:

lxc console test
[  OK  ] Stopped Create Static Device Nodes in /dev.
[  OK  ] Stopped Create System Users.
[  OK  ] Stopped Device-Mapper Multipath Device Controller.
[  OK  ] Stopped Remount Root and Kernel File Systems.
[  OK  ] Stopped File System Check on Root Device.
[  OK  ] Stopped Monitoring of LVM2… dmeventd or progress polling.
[  OK  ] Reached target System Shutdown.
[  OK  ] Reached target Late Shutdown Services.
         Starting System Halt...
reboot: System halted

And from looking at the docs around the meaning of "halt", the halt state seems to be one where it should remain running, so this behaviour sounds correct at first glance.

systemctl poweroff -i results in:

[  OK  ] Stopped Create Static Device Nodes in /dev.
[  OK  ] Stopped Create System Users.
[  OK  ] Stopped Device-Mapper Multipath Device Controller.
[  OK  ] Stopped Remount Root and Kernel File Systems.
[  OK  ] Stopped File System Check on Root Device.
[  OK  ] Stopped Monitoring of LVM2… dmeventd or progress polling.
[  OK  ] Reached target System Shutdown.
[  OK  ] Reached target Late Shutdown Services.
[  OK  ] Finished System Power Off.
[  OK  ] Reached target System Power Off.
reboot: Power down

And systemctl reboot -i results in:

[  OK  ] Stopped Create Static Device Nodes in /dev.
[  OK  ] Stopped Create System Users.
[  OK  ] Stopped Device-Mapper Multipath Device Controller.
[  OK  ] Stopped Remount Root and Kernel File Systems.
[  OK  ] Stopped File System Check on Root Device.
[  OK  ] Stopped Monitoring of LVM2… dmeventd or progress polling.
[  OK  ] Reached target System Shutdown.
[  OK  ] Reached target Late Shutdown Services.
[  OK  ] Finished System Reboot.
[  OK  ] Reached target System Reboot.
reboot: Restarting system

So these 3 different commands seem to be doing the correct thing.

tomponline commented 1 year ago

@yhaliaw are you happy for this to be closed?

tomponline commented 1 year ago

I'll close this for now, but if you have any extra info please let us know and we can reopen.