search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.28k stars 915 forks source link

Multipass (using driver lxd): Could not determine IP address within 120000ms #13683

Closed manztihagi closed 6 days ago

manztihagi commented 2 weeks ago

Hi,

I've suggested from multipass thread in this link : https://github.com/canonical/multipass/issues/3528

The problem is when calling multipass list, the error appears. Several method that guided by multipass contributor has been followed but still cant resolve the problem.

My multipass driver is use lxd driver. For history, may see the link above.

Hopeful from lxd contributor can help to resolve the problem.

Distribution: Ubuntu Distribution version: 22.04.4 LTS (Jammy Jellyfish) Snap list:

chr@chr:/$ snap list --all lxd core20 core22 core 24 snapd
Name    Version         Rev    Tracking       Publisher   Notes
core20  20240227        2264   latest/stable  canonical✓  base,disabled
core20  20240416        2318   latest/stable  canonical✓  base
core22  20240111        1122   latest/stable  canonical✓  base,disabled
core22  20240408        1380   latest/stable  canonical✓  base
lxd     5.21.1-10f4115  28322  latest/stable  canonical✓  disabled
lxd     5.21.1-2d13beb  28463  latest/stable  canonical✓  -
snapd   2.62            21465  latest/stable  canonical✓  snapd,disabled
snapd   2.63            21759  latest/stable  canonical✓  snapd
chr@chr:/$

lxc info:

chr@chr:/$ sudo lxc info
config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- backup_compression
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- instances_nic_host_name
- image_copy_profile
- container_syscall_intercept_sysinfo
- clustering_evacuation_mode
- resources_pci_vpd
- qemu_raw_conf
- storage_cephfs_fscache
- network_load_balancer
- vsock_api
- instance_ready_state
- network_bgp_holdtime
- storage_volumes_all_projects
- metrics_memory_oom_total
- storage_buckets
- storage_buckets_create_credentials
- metrics_cpu_effective_total
- projects_networks_restricted_access
- storage_buckets_local
- loki
- acme
- internal_metrics
- cluster_join_token_expiry
- remote_token_expiry
- init_preseed
- storage_volumes_created_at
- cpu_hotplug
- projects_networks_zones
- network_txqueuelen
- cluster_member_state
- instances_placement_scriptlet
- storage_pool_source_wipe
- zfs_block_mode
- instance_generation_id
- disk_io_cache
- amd_sev
- storage_pool_loop_resize
- migration_vm_live
- ovn_nic_nesting
- oidc
- network_ovn_l3only
- ovn_nic_acceleration_vdpa
- cluster_healing
- instances_state_total
- auth_user
- security_csm
- instances_rebuild
- numa_cpu_placement
- custom_volume_iso
- network_allocations
- storage_api_remote_volume_snapshot_copy
- zfs_delegate
- operations_get_query_all_projects
- metadata_configuration
- syslog_socket
- event_lifecycle_name_and_project
- instances_nic_limits_priority
- disk_initial_volume_configuration
- operation_wait
- cluster_internal_custom_volume_copy
- disk_io_bus
- storage_cephfs_create_missing
- instance_move_config
- ovn_ssl_config
- init_preseed_storage_volumes
- metrics_instances_count
- server_instance_type_info
- resources_disk_mounted
- server_version_lts
- oidc_groups_claim
- loki_config_instance
- storage_volatile_uuid
- import_instance_devices
- instances_uefi_vars
- instances_migration_stateful
- container_syscall_filtering_allow_deny_syntax
- access_management
- vm_disk_io_limits
- storage_volumes_all
- instances_files_modify_permissions
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
auth_user_name: root
auth_user_method: unix
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
....deleted by me...
    -----END CERTIFICATE-----
  certificate_fingerprint: ..deleted by me...
  driver: lxc | qemu
  driver_version: 6.0.0 | 8.2.1
  instance_types:
  - container
  - virtual-machine
  firewall: nftables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    idmapped_mounts: "true"
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    uevent_injection: "true"
    unpriv_fscaps: "true"
  kernel_version: 5.15.0-107-generic
  lxc_features:
    cgroup2: "true"
    core_scheduling: "true"
    devpts_fd: "true"
    idmapped_mounts_v2: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: Ubuntu
  os_version: "22.04"
  project: default
  server: lxd
  server_clustered: false
  server_event_mode: full-mesh
  server_name: chr
  server_pid: 1770
  server_version: 5.21.1
  server_lts: true
  storage: zfs | dir
  storage_version: 2.1.5-1ubuntu6~22.04.3 | 1
  storage_supported_drivers:
  - name: powerflex
    version: 1.16 (nvme-cli)
    remote: true
  - name: zfs
    version: 2.1.5-1ubuntu6~22.04.3
    remote: false
  - name: btrfs
    version: 5.16.2
    remote: false
  - name: ceph
    version: 17.2.7
    remote: true
  - name: cephfs
    version: 17.2.7
    remote: true
  - name: cephobject
    version: 17.2.7
    remote: true
  - name: dir
    version: "1"
    remote: false
  - name: lvm
    version: 2.03.11(2) (2021-01-08) / 1.02.175 (2021-01-08) / 4.45.0
    remote: false
chr@chr:/$ lxc version
Client version: 5.21.1 LTS
Server version: unreachable
chr@chr:/$ lxd version
5.21.1 LTS
chr@chr:/$

Kernel output (dmesg):

[   15.040839] audit: type=1400 audit(1719554973.284:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirtd" pid=940 comm="apparmor_parser"
[   15.040855] audit: type=1400 audit(1719554973.284:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirtd//qemu_bridge_helper" pid=940 comm="apparmor_parser"
[   15.041769] audit: type=1400 audit(1719554973.284:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/man" pid=935 comm="apparmor_parser"
[   15.041780] audit: type=1400 audit(1719554973.284:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_filter" pid=935 comm="apparmor_parser"
[   15.041788] audit: type=1400 audit(1719554973.284:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_groff" pid=935 comm="apparmor_parser"
[   15.043529] audit: type=1400 audit(1719554973.288:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="swtpm" pid=936 comm="apparmor_parser"
[   15.045787] audit: type=1400 audit(1719554973.288:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=930 comm="apparmor_parser"
[   15.047847] audit: type=1400 audit(1719554973.292:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="virt-aa-helper" pid=938 comm="apparmor_parser"
[   15.049956] audit: type=1400 audit(1719554973.292:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=931 comm="apparmor_parser"
[   15.049969] audit: type=1400 audit(1719554973.292:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=931 comm="apparmor_parser"
[  138.737263] eno2 speed is unknown, defaulting to 1000
[  149.047496] kauditd_printk_skb: 56 callbacks suppressed
[  149.047503] audit: type=1400 audit(1719555107.292:68): apparmor="DENIED" operation="capable" profile="/snap/snapd/21759/usr/lib/snapd/snap-confine" pid=1046 comm="snap-confine" capability=12  capname="net_admin"
[  149.047517] audit: type=1400 audit(1719555107.292:69): apparmor="DENIED" operation="capable" profile="/snap/snapd/21759/usr/lib/snapd/snap-confine" pid=1046 comm="snap-confine" capability=38  capname="perfmon"
[  149.065207] audit: type=1400 audit(1719555107.308:70): apparmor="DENIED" operation="capable" profile="/snap/snapd/21759/usr/lib/snapd/snap-confine" pid=1047 comm="snap-confine" capability=12  capname="net_admin"
[  149.065222] audit: type=1400 audit(1719555107.308:71): apparmor="DENIED" operation="capable" profile="/snap/snapd/21759/usr/lib/snapd/snap-confine" pid=1047 comm="snap-confine" capability=38  capname="perfmon"
[  151.771598] loop12: detected capacity change from 0 to 8
[  154.567060] audit: type=1326 audit(1719555112.808:72): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=snap.network-manager.networkmanager pid=1047 comm="NetworkManager" exe="/snap/network-manager/907/usr/sbin/NetworkManager" sig=0 arch=c000003e syscall=314 compat=0 ip=0x7fabb161e88d code=0x50000
[  154.576083] audit: type=1400 audit(1719555112.820:73): apparmor="DENIED" operation="capable" profile="snap.network-manager.networkmanager" pid=1047 comm="NetworkManager" capability=1  capname="dac_override"
[  154.627245] eno2 speed is unknown, defaulting to 1000
[  155.319141] audit: type=1400 audit(1719555113.564:74): apparmor="DENIED" operation="open" profile="snap.multipass.multipassd" name="/usr/lib64/" pid=1046 comm="multipassd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[  155.319157] audit: type=1400 audit(1719555113.564:75): apparmor="DENIED" operation="open" profile="snap.multipass.multipassd" name="/usr/lib64/" pid=1046 comm="multipassd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[  155.321083] audit: type=1400 audit(1719555113.564:76): apparmor="DENIED" operation="open" profile="snap.multipass.multipassd" name="/usr/lib32/" pid=1046 comm="multipassd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[  155.321098] audit: type=1400 audit(1719555113.564:77): apparmor="DENIED" operation="open" profile="snap.multipass.multipassd" name="/usr/lib32/" pid=1046 comm="multipassd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[  155.321799] audit: type=1400 audit(1719555113.564:78): apparmor="DENIED" operation="open" profile="snap.multipass.multipassd" name="/usr/lib64/" pid=1046 comm="multipassd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[  155.323995] audit: type=1400 audit(1719555113.568:79): apparmor="DENIED" operation="open" profile="snap.multipass.multipassd" name="/usr/lib64/" pid=1046 comm="multipassd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[  155.324006] audit: type=1400 audit(1719555113.568:80): apparmor="DENIED" operation="open" profile="snap.multipass.multipassd" name="/usr/lib64/" pid=1046 comm="multipassd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[  155.324014] audit: type=1400 audit(1719555113.568:81): apparmor="DENIED" operation="open" profile="snap.multipass.multipassd" name="/usr/lib32/" pid=1046 comm="multipassd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[  156.374650] NET: Registered PF_VSOCK protocol family
[  156.538370] IPv6: ADDRCONF(NETDEV_CHANGE): eno1: link becomes ready
[  156.548037] eno2 speed is unknown, defaulting to 1000
[  158.559709] eno2 speed is unknown, defaulting to 1000
[  158.768894] L1TF CPU bug present and SMT on, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/l1tf.html for details.
[  161.318282] spl: loading out-of-tree module taints kernel.
[  161.328023] znvpair: module license 'CDDL' taints kernel.
[  161.328026] Disabling lock debugging due to kernel taint
[  161.559911] ZFS: Loaded module v2.1.5-1ubuntu6~22.04.3, ZFS pool version 5000, ZFS filesystem version 5
[  165.287655] kauditd_printk_skb: 2 callbacks suppressed
[  165.287662] audit: type=1400 audit(1719555123.532:84): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd_dnsmasq-lxdbr0_</var/snap/lxd/common/lxd>" pid=2824 comm="apparmor_parser"
[  166.165240] audit: type=1400 audit(1719555124.408:85): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd_dnsmasq-mpbr0_</var/snap/lxd/common/lxd>" pid=2850 comm="apparmor_parser"
[  167.288558] mpbr0: port 1(tap6fe7758c) entered blocking state
[  167.288563] mpbr0: port 1(tap6fe7758c) entered disabled state
[  167.288633] device tap6fe7758c entered promiscuous mode
[  167.461136] audit: type=1400 audit(1719555125.704:86): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-multipass_chr-uat_</var/snap/lxd/common/lxd>" pid=2939 comm="apparmor_parser"
[  167.493463] eno2 speed is unknown, defaulting to 1000
[  167.942572] mpbr0: port 1(tap6fe7758c) entered blocking state
[  167.942584] mpbr0: port 1(tap6fe7758c) entered forwarding state
[  167.942670] IPv6: ADDRCONF(NETDEV_CHANGE): mpbr0: link becomes ready
[  168.989240] mpbr0: port 2(tapf48c5ddb) entered blocking state
[  168.989246] mpbr0: port 2(tapf48c5ddb) entered disabled state
[  168.989329] device tapf48c5ddb entered promiscuous mode
[  169.042441] eno2 speed is unknown, defaulting to 1000
[  169.097648] audit: type=1400 audit(1719555127.399:87): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-multipass_chr-uat2_</var/snap/lxd/common/lxd>" pid=3001 comm="apparmor_parser"
[  169.716428] mpbr0: port 2(tapf48c5ddb) entered blocking state
[  169.716440] mpbr0: port 2(tapf48c5ddb) entered forwarding state
[  176.492006] kvm: vcpu 11: requested 72827 ns lapic timer period limited to 200000 ns
[  176.492238] kvm: vcpu 10: requested 72827 ns lapic timer period limited to 200000 ns
[  176.492437] kvm: vcpu 12: requested 72827 ns lapic timer period limited to 200000 ns
[  176.492441] kvm: vcpu 13: requested 72827 ns lapic timer period limited to 200000 ns
[  176.492456] kvm: vcpu 9: requested 72827 ns lapic timer period limited to 200000 ns
[  176.492473] kvm: vcpu 15: requested 72827 ns lapic timer period limited to 200000 ns
[  176.504792] kvm: vcpu 14: requested 72827 ns lapic timer period limited to 200000 ns
[  273.152728] mpbr0: port 3(tap12162556) entered blocking state
[  273.152741] mpbr0: port 3(tap12162556) entered disabled state
[  273.152929] device tap12162556 entered promiscuous mode
[  273.209581] eno2 speed is unknown, defaulting to 1000
[  273.274288] audit: type=1400 audit(1719555231.884:88): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-multipass_chr-uatweb_</var/snap/lxd/common/lxd>" pid=3475 comm="apparmor_parser"
[  273.644338] mpbr0: port 3(tap12162556) entered blocking state
[  273.644350] mpbr0: port 3(tap12162556) entered forwarding state
[33165.255931] loop12: detected capacity change from 0 to 239856
[33168.181540] audit: type=1400 audit(1719588126.790:89): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/snap/snapd/21759/usr/lib/snapd/snap-confine" pid=14382 comm="apparmor_parser"
[33168.181553] audit: type=1400 audit(1719588126.790:90): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/snap/snapd/21759/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=14382 comm="apparmor_parser"
[33168.294025] audit: type=1400 audit(1719588126.902:91): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap-update-ns.multipass" pid=14384 comm="apparmor_parser"
[33168.334236] audit: type=1400 audit(1719588126.942:92): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.multipass.hook.configure" pid=14386 comm="apparmor_parser"
[33168.336197] audit: type=1400 audit(1719588126.942:93): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.multipass.hook.pre-refresh" pid=14389 comm="apparmor_parser"
[33168.336464] audit: type=1400 audit(1719588126.942:94): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.multipass.hook.install" pid=14387 comm="apparmor_parser"
[33168.336663] audit: type=1400 audit(1719588126.942:95): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.multipass.hook.post-refresh" pid=14388 comm="apparmor_parser"
[33168.336730] audit: type=1400 audit(1719588126.942:96): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.multipass.hook.remove" pid=14390 comm="apparmor_parser"
[33168.369256] audit: type=1400 audit(1719588126.974:97): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.multipass.multipass" pid=14391 comm="apparmor_parser"
[33168.468223] audit: type=1400 audit(1719588127.074:98): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.multipass.multipassd" pid=14392 comm="apparmor_parser"
[157072.124790] perf: interrupt took too long (2529 > 2500), lowering kernel.perf_event_max_sample_rate to 79000
[216630.013630] eno2 speed is unknown, defaulting to 1000
[219952.447690] perf: interrupt took too long (3172 > 3161), lowering kernel.perf_event_max_sample_rate to 63000
[239998.919143] mpbr0: port 1(tap6fe7758c) entered disabled state
[239999.108928] device tap6fe7758c left promiscuous mode
[239999.108962] mpbr0: port 1(tap6fe7758c) entered disabled state
[239999.546608] kauditd_printk_skb: 30 callbacks suppressed
[239999.546616] audit: type=1400 audit(1719794958.165:129): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-multipass_chr-uat_</var/snap/lxd/common/lxd>" pid=73786 comm="apparmor_parser"
[240038.368842] mpbr0: port 1(tap5b5719da) entered blocking state
[240038.368851] mpbr0: port 1(tap5b5719da) entered disabled state
[240038.368990] device tap5b5719da entered promiscuous mode
[240038.476538] eno2 speed is unknown, defaulting to 1000
[240038.523993] audit: type=1400 audit(1719794997.141:130): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-multipass_chr-uat_</var/snap/lxd/common/lxd>" pid=74114 comm="apparmor_parser"
[240038.868602] mpbr0: port 1(tap5b5719da) entered blocking state
[240038.868619] mpbr0: port 1(tap5b5719da) entered forwarding state
chr@chr:/$

Container information:

chr@chr:/$ sudo lxc info chr-uat --show-log
Error: Failed to fetch instance "chr-uat" in project "default": Instance not found
chr@chr:/$ sudo lxc config show chr-uat --expanded
Error: Failed to fetch instance "chr-uat" in project "default": Instance not found
chr@chr:/$

Main daemon-log:

chr@chr:/$ sudo cat /var/snap/lxd/common/lxd/logs/lxd.log
time="2024-06-28T13:11:56+07:00" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2024-06-28T13:13:37+07:00" level=warning msg="The backing filesystem doesn't support quotas, skipping set quota" driver=dir path=/var/snap/lxd/common/lxd/storage-pools/multipass/virtual-machines/multipass_chr-uatweb pool=multipass size=80635494400 volID=2
time="2024-07-01T07:49:43+07:00" level=warning msg="The backing filesystem doesn't support quotas, skipping set quota" driver=dir path=/var/snap/lxd/common/lxd/storage-pools/multipass/virtual-machines/multipass_chr-uat pool=multipass size=107479040000 volID=1
chr@chr:/$

lxc monitor

chr@chr:/$ sudo lxc monitor
location: none
metadata:
  context:
    id: b918c25d-3e88-4206-b492-2b70e227d660
    local: /var/snap/lxd/common/lxd/unix.socket
    remote: '@'
  level: debug
  message: Event listener server handler started
timestamp: "2024-07-01T08:14:46.780016424+07:00"
type: logging
simondeziel commented 2 weeks ago

@manztihagi based on https://github.com/canonical/multipass/issues/3528#issuecomment-2196019823, it seems the problem is that lxc list --project=multipass is able to pull the instance's IP but not multipass list. If so, you could try to check which call multipass does by running lxc monitor --pretty while calling multipass list.

It'd also be interesting to compare the monitor output with that of lxc list --project=multipass as this later one works and would provide a reference point.

simondeziel commented 2 weeks ago

Container information:

In those lxc info/lxc config show, it seems you were missing the needed --project=multipass bit.

manztihagi commented 2 weeks ago

Hi @simondeziel,

result of sudo lxc monitor --pretty

running multiple times with same result log until get an error as per subjected above

DEBUG  [2024-07-03T08:24:28+07:00] GetInstanceUsage started                      driver=dir instance=chr-uat pool=multipass project=multipass
DEBUG  [2024-07-03T08:24:28+07:00] GetInstanceUsage finished                     driver=dir instance=chr-uat pool=multipass project=multipass
DEBUG  [2024-07-03T08:24:28+07:00] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/networks/mpbr0/leases?project=multipass" username=root
DEBUG  [2024-07-03T08:24:29+07:00] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/virtual-machines/chr-uat/state?project=multipass" username=root
DEBUG  [2024-07-03T08:24:29+07:00] Sending request to LXD                        etag= method=GET url="https://custom.socket/1.0"
DEBUG  [2024-07-03T08:24:29+07:00] Connecting to a VM agent over a VM socket
DEBUG  [2024-07-03T08:24:29+07:00] Got response struct from LXD
DEBUG  [2024-07-03T08:24:29+07:00] Sending request to LXD                        etag= method=GET url="https://custom.socket/1.0/state"
DEBUG  [2024-07-03T08:24:29+07:00]
        {
                "config": null,
                "api_extensions": [
                        "storage_zfs_remove_snapshots",
                        "container_host_shutdown_timeout",
                        "container_stop_priority",
                        "container_syscall_filtering",
                        "auth_pki",
                        "container_last_used_at",
                        "etag",
                        "patch",
                        "usb_devices",
                        "https_allowed_credentials",
                        "image_compression_algorithm",
                        "directory_manipulation",
                        "container_cpu_time",
                        "storage_zfs_use_refquota",
                        "storage_lvm_mount_options",
                        "network",
                        "profile_usedby",
                        "container_push",
                        "container_exec_recording",
                        "certificate_update",
                        "container_exec_signal_handling",
                        "gpu_devices",
                        "container_image_properties",
                        "migration_progress",
                        "id_map",
                        "network_firewall_filtering",
                        "network_routes",
                        "storage",
                        "file_delete",
                        "file_append",
                        "network_dhcp_expiry",
                        "storage_lvm_vg_rename",
                        "storage_lvm_thinpool_rename",
                        "network_vlan",
                        "image_create_aliases",
                        "container_stateless_copy",
                        "container_only_migration",
                        "storage_zfs_clone_copy",
                        "unix_device_rename",
                        "storage_lvm_use_thinpool",
                        "storage_rsync_bwlimit",
                        "network_vxlan_interface",
                        "storage_btrfs_mount_options",
                        "entity_description",
                        "image_force_refresh",
                        "storage_lvm_lv_resizing",
                        "id_map_base",
                        "file_symlinks",
                        "container_push_target",
                        "network_vlan_physical",
                        "storage_images_delete",
                        "container_edit_metadata",
                        "container_snapshot_stateful_migration",
                        "storage_driver_ceph",
                        "storage_ceph_user_name",
                        "resource_limits",
                        "storage_volatile_initial_source",
                        "storage_ceph_force_osd_reuse",
                        "storage_block_filesystem_btrfs",
                        "resources",
                        "kernel_limits",
                        "storage_api_volume_rename",
                        "network_sriov",
                        "console",
                        "restrict_devlxd",
                        "migration_pre_copy",
                        "infiniband",
                        "maas_network",
                        "devlxd_events",
                        "proxy",
                        "network_dhcp_gateway",
                        "file_get_symlink",
                        "network_leases",
                        "unix_device_hotplug",
                        "storage_api_local_volume_handling",
                        "operation_description",
                        "clustering",
                        "event_lifecycle",
                        "storage_api_remote_volume_handling",
                        "nvidia_runtime",
                        "container_mount_propagation",
                        "container_backup",
                        "devlxd_images",
                        "container_local_cross_pool_handling",
                        "proxy_unix",
                        "proxy_udp",
                        "clustering_join",
                        "proxy_tcp_udp_multi_port_handling",
                        "network_state",
                        "proxy_unix_dac_properties",
                        "container_protection_delete",
                        "unix_priv_drop",
                        "pprof_http",
                        "proxy_haproxy_protocol",
                        "network_hwaddr",
                        "proxy_nat",
                        "network_nat_order",
                        "container_full",
                        "backup_compression",
                        "nvidia_runtime_config",
                        "storage_api_volume_snapshots",
                        "storage_unmapped",
                        "projects",
                        "network_vxlan_ttl",
                        "container_incremental_copy",
                        "usb_optional_vendorid",
                        "snapshot_scheduling",
                        "snapshot_schedule_aliases",
                        "container_copy_project",
                        "clustering_server_address",
                        "clustering_image_replication",
                        "container_protection_shift",
                        "snapshot_expiry",
                        "container_backup_override_pool",
                        "snapshot_expiry_creation",
                        "network_leases_location",
                        "resources_cpu_socket",
                        "resources_gpu",
                        "resources_numa",
                        "kernel_features",
                        "id_map_current",
                        "event_location",
                        "storage_api_remote_volume_snapshots",
                        "network_nat_address",
                        "container_nic_routes",
                        "cluster_internal_copy",
                        "seccomp_notify",
                        "lxc_features",
                        "container_nic_ipvlan",
                        "network_vlan_sriov",
                        "storage_cephfs",
                        "container_nic_ipfilter",
                        "resources_v2",
                        "container_exec_user_group_cwd",
                        "container_syscall_intercept",
                        "container_disk_shift",
                        "storage_shifted",
                        "resources_infiniband",
                        "daemon_storage",
                        "instances",
                        "image_types",
                        "resources_disk_sata",
                        "clustering_roles",
                        "images_expiry",
                        "resources_network_firmware",
                        "backup_compression_algorithm",
                        "ceph_data_pool_name",
                        "container_syscall_intercept_mount",
                        "compression_squashfs",
                        "container_raw_mount",
                        "container_nic_routed",
                        "container_syscall_intercept_mount_fuse",
                        "container_disk_ceph",
                        "virtual-machines",
                        "image_profiles",
                        "clustering_architecture",
                        "resources_disk_id",
                        "storage_lvm_stripes",
                        "vm_boot_priority",
                        "unix_hotplug_devices",
                        "api_filtering",
                        "instance_nic_network",
                        "clustering_sizing",
                        "firewall_driver",
                        "projects_limits",
                        "container_syscall_intercept_hugetlbfs",
                        "limits_hugepages",
                        "container_nic_routed_gateway",
                        "projects_restrictions",
                        "custom_volume_snapshot_expiry",
                        "volume_snapshot_scheduling",
                        "trust_ca_certificates",
                        "snapshot_disk_usage",
                        "clustering_edit_roles",
                        "container_nic_routed_host_address",
                        "container_nic_ipvlan_gateway",
                        "resources_usb_pci",
                        "resources_cpu_threads_numa",
                        "resources_cpu_core_die",
                        "api_os",
                        "container_nic_routed_host_table",
                        "container_nic_ipvlan_host_table",
                        "container_nic_ipvlan_mode",
                        "resources_system",
                        "images_push_relay",
                        "network_dns_search",
                        "container_nic_routed_limits",
                        "instance_nic_bridged_vlan",
                        "network_state_bond_bridge",
                        "usedby_consistency",
                        "custom_block_volumes",
                        "clustering_failure_domains",
                        "resources_gpu_mdev",
                        "console_vga_type",
                        "projects_limits_disk",
                        "network_type_macvlan",
                        "network_type_sriov",
                        "container_syscall_intercept_bpf_devices",
                        "network_type_ovn",
                        "projects_networks",
                        "projects_networks_restricted_uplinks",
                        "custom_volume_backup",
                        "backup_override_name",
                        "storage_rsync_compression",
                        "network_type_physical",
                        "network_ovn_external_subnets",
                        "network_ovn_nat",
                        "network_ovn_external_routes_remove",
                        "tpm_device_type",
                        "storage_zfs_clone_copy_rebase",
                        "gpu_mdev",
                        "resources_pci_iommu",
                        "resources_network_usb",
                        "resources_disk_address",
                        "network_physical_ovn_ingress_mode",
                        "network_ovn_dhcp",
                        "network_physical_routes_anycast",
                        "projects_limits_instances",
                        "network_state_vlan",
                        "instance_nic_bridged_port_isolation",
                        "instance_bulk_state_change",
                        "network_gvrp",
                        "instance_pool_move",
                        "gpu_sriov",
                        "pci_device_type",
                        "storage_volume_state",
                        "network_acl",
                        "migration_stateful",
                        "disk_state_quota",
                        "storage_ceph_features",
                        "projects_compression",
                        "projects_images_remote_cache_expiry",
                        "certificate_project",
                        "network_ovn_acl",
                        "projects_images_auto_update",
                        "projects_restricted_cluster_target",
                        "images_default_architecture",
                        "network_ovn_acl_defaults",
                        "gpu_mig",
                        "project_usage",
                        "network_bridge_acl",
                        "warnings",
                        "projects_restricted_backups_and_snapshots",
                        "clustering_join_token",
                        "clustering_description",
                        "server_trusted_proxy",
                        "clustering_update_cert",
                        "storage_api_project",
                        "server_instance_driver_operational",
                        "server_supported_storage_drivers",
                        "event_lifecycle_requestor_address",
                        "resources_gpu_usb",
                        "clustering_evacuation",
                        "network_ovn_nat_address",
                        "network_bgp",
                        "network_forward",
                        "custom_volume_refresh",
                        "network_counters_errors_dropped",
                        "metrics",
                        "image_source_project",
                        "clustering_config",
                        "network_peer",
                        "linux_sysctl",
                        "network_dns",
                        "ovn_nic_acceleration",
                        "certificate_self_renewal",
                        "instance_project_move",
                        "storage_volume_project_move",
                        "cloud_init",
                        "network_dns_nat",
                        "database_leader",
                        "instance_all_projects",
                        "clustering_groups",
                        "ceph_rbd_du",
                        "instance_get_full",
                        "qemu_metrics",
                        "gpu_mig_uuid",
                        "event_project",
                        "clustering_evacuation_live",
                        "instance_allow_inconsistent_copy",
                        "network_state_ovn",
                        "storage_volume_api_filtering",
                        "image_restrictions",
                        "storage_zfs_export",
                        "network_dns_records",
                        "storage_zfs_reserve_space",
                        "network_acl_log",
                        "storage_zfs_blocksize",
                        "metrics_cpu_seconds",
                        "instance_snapshot_never",
                        "certificate_token",
                        "instance_nic_routed_neighbor_probe",
                        "event_hub",
                        "agent_nic_config",
                        "projects_restricted_intercept",
                        "metrics_authentication",
                        "images_target_project",
                        "cluster_migration_inconsistent_copy",
                        "cluster_ovn_chassis",
                        "container_syscall_intercept_sched_setscheduler",
                        "storage_lvm_thinpool_metadata_size",
                        "storage_volume_state_total",
                        "instance_file_head",
                        "instances_nic_host_name",
                        "image_copy_profile",
                        "container_syscall_intercept_sysinfo",
                        "clustering_evacuation_mode",
                        "resources_pci_vpd",
                        "qemu_raw_conf",
                        "storage_cephfs_fscache",
                        "network_load_balancer",
                        "vsock_api",
                        "instance_ready_state",
                        "network_bgp_holdtime",
                        "storage_volumes_all_projects",
                        "metrics_memory_oom_total",
                        "storage_buckets",
                        "storage_buckets_create_credentials",
                        "metrics_cpu_effective_total",
                        "projects_networks_restricted_access",
                        "storage_buckets_local",
                        "loki",
                        "acme",
                        "internal_metrics",
                        "cluster_join_token_expiry",
                        "remote_token_expiry",
                        "init_preseed",
                        "storage_volumes_created_at",
                        "cpu_hotplug",
                        "projects_networks_zones",
                        "network_txqueuelen",
                        "cluster_member_state",
                        "instances_placement_scriptlet",
                        "storage_pool_source_wipe",
                        "zfs_block_mode",
                        "instance_generation_id",
                        "disk_io_cache",
                        "amd_sev",
                        "storage_pool_loop_resize",
                        "migration_vm_live",
                        "ovn_nic_nesting",
                        "oidc",
                        "network_ovn_l3only",
                        "ovn_nic_acceleration_vdpa",
                        "cluster_healing",
                        "instances_state_total",
                        "auth_user",
                        "security_csm",
                        "instances_rebuild",
                        "numa_cpu_placement",
                        "custom_volume_iso",
                        "network_allocations",
                        "storage_api_remote_volume_snapshot_copy",
                        "zfs_delegate",
                        "operations_get_query_all_projects",
                        "metadata_configuration",
                        "syslog_socket",
                        "event_lifecycle_name_and_project",
                        "instances_nic_limits_priority",
                        "disk_initial_volume_configuration",
                        "operation_wait",
                        "cluster_internal_custom_volume_copy",
                        "disk_io_bus",
                        "storage_cephfs_create_missing",
                        "instance_move_config",
                        "ovn_ssl_config",
                        "init_preseed_storage_volumes",
                        "metrics_instances_count",
                        "server_instance_type_info",
                        "resources_disk_mounted",
                        "server_version_lts",
                        "oidc_groups_claim",
                        "loki_config_instance",
                        "storage_volatile_uuid",
                        "import_instance_devices",
                        "instances_uefi_vars",
                        "instances_migration_stateful",
                        "container_syscall_filtering_allow_deny_syntax",
                        "access_management",
                        "vm_disk_io_limits",
                        "storage_volumes_all",
                        "instances_files_modify_permissions"
                ],
                "api_status": "stable",
                "api_version": "1.0",
                "auth": "trusted",
                "public": false,
                "auth_methods": [
                        "tls"
                ],
                "auth_user_name": "",
                "auth_user_method": "",
                "environment": {
                        "addresses": null,
                        "architectures": null,
                        "certificate": "",
                        "certificate_fingerprint": "",
                        "driver": "",
                        "driver_version": "",
                        "instance_types": null,
                        "firewall": "",
                        "kernel": "Linux",
                        "kernel_architecture": "x86_64",
                        "kernel_features": null,
                        "kernel_version": "5.15.0-112-generic",
                        "lxc_features": null,
                        "os_name": "",
                        "os_version": "",
                        "project": "",
                        "server": "lxd-agent",
                        "server_clustered": false,
                        "server_event_mode": "",
                        "server_name": "chr-uat",
                        "server_pid": 478,
                        "server_version": "5.21.1",
                        "server_lts": false,
                        "storage": "",
                        "storage_version": "",
                        "storage_supported_drivers": null
                }
        }
DEBUG  [2024-07-03T08:24:29+07:00] Got response struct from LXD
DEBUG  [2024-07-03T08:24:29+07:00]
        {
                "status": "",
                "status_code": 0,
                "disk": null,
                "memory": {
                        "usage": 1788071936,
                        "usage_peak": 0,
                        "total": 8314474496,
                        "swap_usage": 0,
                        "swap_usage_peak": 0
                },
                "network": {
                        "enp5s0": {
                                "addresses": [
                                        {
                                                "family": "inet",
                                                "address": "10.67.90.39",
                                                "netmask": "24",
                                                "scope": "global"
                                        },
                                        {
                                                "family": "inet6",
                                                "address": "fd42:d158:353c:6900:5054:ff:fe11:13bf",
                                                "netmask": "64",
                                                "scope": "global"
                                        },
                                        {
                                                "family": "inet6",
                                                "address": "fe80::5054:ff:fe11:13bf",
                                                "netmask": "64",
                                                "scope": "link"
                                        }
                                ],
                                "counters": {
                                        "bytes_received": 4957856,
                                        "bytes_sent": 5088355,
                                        "packets_received": 35233,
                                        "packets_sent": 45180,
                                        "errors_received": 0,
                                        "errors_sent": 0,
                                        "packets_dropped_outbound": 0,
                                        "packets_dropped_inbound": 0
                                },
                                "hwaddr": "52:54:00:11:13:bf",
                                "host_name": "",
                                "mtu": 1500,
                                "state": "up",
                                "type": "broadcast"
                        },
                        "lo": {
                                "addresses": [
                                        {
                                                "family": "inet",
                                                "address": "127.0.0.1",
                                                "netmask": "8",
                                                "scope": "local"
                                        },
                                        {
                                                "family": "inet6",
                                                "address": "::1",
                                                "netmask": "128",
                                                "scope": "local"
                                        }
                                ],
                                "counters": {
                                        "bytes_received": 0,
                                        "bytes_sent": 0,
                                        "packets_received": 0,
                                        "packets_sent": 0,
                                        "errors_received": 0,
                                        "errors_sent": 0,
                                        "packets_dropped_outbound": 0,
                                        "packets_dropped_inbound": 0
                                },
                                "hwaddr": "",
                                "host_name": "",
                                "mtu": 65536,
                                "state": "up",
                                "type": "loopback"
                        }
                },
                "pid": 1,
                "processes": 44,
                "cpu": {
                        "usage": 337016359000
                }
        }
DEBUG  [2024-07-03T08:24:29+07:00] GetInstanceUsage started                      driver=dir instance=chr-uat pool=multipass project=multipass
DEBUG  [2024-07-03T08:24:29+07:00] GetInstanceUsage finished                     driver=dir instance=chr-uat pool=multipass project=multipass
DEBUG  [2024-07-03T08:24:29+07:00] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/networks/mpbr0/leases?project=multipass" username=root

sudo lxc list --project=multipass

logs:

DEBUG  [2024-07-03T08:31:10+07:00] Event listener server handler started         id=55a0711d-8c99-49c9-8c67-81a572bce689 local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2024-07-03T08:31:12+07:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=root
DEBUG  [2024-07-03T08:31:12+07:00] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/instances?filter=&project=multipass&recursion=2" username=root
DEBUG  [2024-07-03T08:31:12+07:00] Connecting to a VM agent over a VM socket
DEBUG  [2024-07-03T08:31:12+07:00] Sending request to LXD                        etag= method=GET url="https://custom.socket/1.0"
DEBUG  [2024-07-03T08:31:12+07:00] Connecting to a VM agent over a VM socket
DEBUG  [2024-07-03T08:31:12+07:00] Connecting to a VM agent over a VM socket
DEBUG  [2024-07-03T08:31:12+07:00] Sending request to LXD                        etag= method=GET url="https://custom.socket/1.0"
DEBUG  [2024-07-03T08:31:12+07:00] Sending request to LXD                        etag= method=GET url="https://custom.socket/1.0"
DEBUG  [2024-07-03T08:31:12+07:00] Got response struct from LXD
DEBUG  [2024-07-03T08:31:12+07:00] Sending request to LXD                        etag= method=GET url="https://custom.socket/1.0/state"
DEBUG  [2024-07-03T08:31:12+07:00]
        {
                "config": null,
                "api_extensions": [
                        "storage_zfs_remove_snapshots",
                        "container_host_shutdown_timeout",
                        "container_stop_priority",
                        "container_syscall_filtering",
                        "auth_pki",
                        "container_last_used_at",
                        "etag",
                        "patch",
                        "usb_devices",
                        "https_allowed_credentials",
                        "image_compression_algorithm",
                        "directory_manipulation",
                        "container_cpu_time",
                        "storage_zfs_use_refquota",
                        "storage_lvm_mount_options",
                        "network",
                        "profile_usedby",
                        "container_push",
                        "container_exec_recording",
                        "certificate_update",
                        "container_exec_signal_handling",
                        "gpu_devices",
                        "container_image_properties",
                        "migration_progress",
                        "id_map",
                        "network_firewall_filtering",
                        "network_routes",
                        "storage",
                        "file_delete",
                        "file_append",
                        "network_dhcp_expiry",
                        "storage_lvm_vg_rename",
                        "storage_lvm_thinpool_rename",
                        "network_vlan",
                        "image_create_aliases",
                        "container_stateless_copy",
                        "container_only_migration",
                        "storage_zfs_clone_copy",
                        "unix_device_rename",
                        "storage_lvm_use_thinpool",
                        "storage_rsync_bwlimit",
                        "network_vxlan_interface",
                        "storage_btrfs_mount_options",
                        "entity_description",
                        "image_force_refresh",
                        "storage_lvm_lv_resizing",
                        "id_map_base",
                        "file_symlinks",
                        "container_push_target",
                        "network_vlan_physical",
                        "storage_images_delete",
                        "container_edit_metadata",
                        "container_snapshot_stateful_migration",
                        "storage_driver_ceph",
                        "storage_ceph_user_name",
                        "resource_limits",
                        "storage_volatile_initial_source",
                        "storage_ceph_force_osd_reuse",
                        "storage_block_filesystem_btrfs",
                        "resources",
                        "kernel_limits",
                        "storage_api_volume_rename",
                        "network_sriov",
                        "console",
                        "restrict_devlxd",
                        "migration_pre_copy",
                        "infiniband",
                        "maas_network",
                        "devlxd_events",
                        "proxy",
                        "network_dhcp_gateway",
                        "file_get_symlink",
                        "network_leases",
                        "unix_device_hotplug",
                        "storage_api_local_volume_handling",
                        "operation_description",
                        "clustering",
                        "event_lifecycle",
                        "storage_api_remote_volume_handling",
                        "nvidia_runtime",
                        "container_mount_propagation",
                        "container_backup",
                        "devlxd_images",
                        "container_local_cross_pool_handling",
                        "proxy_unix",
                        "proxy_udp",
                        "clustering_join",
                        "proxy_tcp_udp_multi_port_handling",
                        "network_state",
                        "proxy_unix_dac_properties",
                        "container_protection_delete",
                        "unix_priv_drop",
                        "pprof_http",
                        "proxy_haproxy_protocol",
                        "network_hwaddr",
                        "proxy_nat",
                        "network_nat_order",
                        "container_full",
                        "backup_compression",
                        "nvidia_runtime_config",
                        "storage_api_volume_snapshots",
                        "storage_unmapped",
                        "projects",
                        "network_vxlan_ttl",
                        "container_incremental_copy",
                        "usb_optional_vendorid",
                        "snapshot_scheduling",
                        "snapshot_schedule_aliases",
                        "container_copy_project",
                        "clustering_server_address",
                        "clustering_image_replication",
                        "container_protection_shift",
                        "snapshot_expiry",
                        "container_backup_override_pool",
                        "snapshot_expiry_creation",
                        "network_leases_location",
                        "resources_cpu_socket",
                        "resources_gpu",
                        "resources_numa",
                        "kernel_features",
                        "id_map_current",
                        "event_location",
                        "storage_api_remote_volume_snapshots",
                        "network_nat_address",
                        "container_nic_routes",
                        "cluster_internal_copy",
                        "seccomp_notify",
                        "lxc_features",
                        "container_nic_ipvlan",
                        "network_vlan_sriov",
                        "storage_cephfs",
                        "container_nic_ipfilter",
                        "resources_v2",
                        "container_exec_user_group_cwd",
                        "container_syscall_intercept",
                        "container_disk_shift",
                        "storage_shifted",
                        "resources_infiniband",
                        "daemon_storage",
                        "instances",
                        "image_types",
                        "resources_disk_sata",
                        "clustering_roles",
                        "images_expiry",
                        "resources_network_firmware",
                        "backup_compression_algorithm",
                        "ceph_data_pool_name",
                        "container_syscall_intercept_mount",
                        "compression_squashfs",
                        "container_raw_mount",
                        "container_nic_routed",
                        "container_syscall_intercept_mount_fuse",
                        "container_disk_ceph",
                        "virtual-machines",
                        "image_profiles",
                        "clustering_architecture",
                        "resources_disk_id",
                        "storage_lvm_stripes",
                        "vm_boot_priority",
                        "unix_hotplug_devices",
                        "api_filtering",
                        "instance_nic_network",
                        "clustering_sizing",
                        "firewall_driver",
                        "projects_limits",
                        "container_syscall_intercept_hugetlbfs",
                        "limits_hugepages",
                        "container_nic_routed_gateway",
                        "projects_restrictions",
                        "custom_volume_snapshot_expiry",
                        "volume_snapshot_scheduling",
                        "trust_ca_certificates",
                        "snapshot_disk_usage",
                        "clustering_edit_roles",
                        "container_nic_routed_host_address",
                        "container_nic_ipvlan_gateway",
                        "resources_usb_pci",
                        "resources_cpu_threads_numa",
                        "resources_cpu_core_die",
                        "api_os",
                        "container_nic_routed_host_table",
                        "container_nic_ipvlan_host_table",
                        "container_nic_ipvlan_mode",
                        "resources_system",
                        "images_push_relay",
                        "network_dns_search",
                        "container_nic_routed_limits",
                        "instance_nic_bridged_vlan",
                        "network_state_bond_bridge",
                        "usedby_consistency",
                        "custom_block_volumes",
                        "clustering_failure_domains",
                        "resources_gpu_mdev",
                        "console_vga_type",
                        "projects_limits_disk",
                        "network_type_macvlan",
                        "network_type_sriov",
                        "container_syscall_intercept_bpf_devices",
                        "network_type_ovn",
                        "projects_networks",
                        "projects_networks_restricted_uplinks",
                        "custom_volume_backup",
                        "backup_override_name",
                        "storage_rsync_compression",
                        "network_type_physical",
                        "network_ovn_external_subnets",
                        "network_ovn_nat",
                        "network_ovn_external_routes_remove",
                        "tpm_device_type",
                        "storage_zfs_clone_copy_rebase",
                        "gpu_mdev",
                        "resources_pci_iommu",
                        "resources_network_usb",
                        "resources_disk_address",
                        "network_physical_ovn_ingress_mode",
                        "network_ovn_dhcp",
                        "network_physical_routes_anycast",
                        "projects_limits_instances",
                        "network_state_vlan",
                        "instance_nic_bridged_port_isolation",
                        "instance_bulk_state_change",
                        "network_gvrp",
                        "instance_pool_move",
                        "gpu_sriov",
                        "pci_device_type",
                        "storage_volume_state",
                        "network_acl",
                        "migration_stateful",
                        "disk_state_quota",
                        "storage_ceph_features",
                        "projects_compression",
                        "projects_images_remote_cache_expiry",
                        "certificate_project",
                        "network_ovn_acl",
                        "projects_images_auto_update",
                        "projects_restricted_cluster_target",
                        "images_default_architecture",
                        "network_ovn_acl_defaults",
                        "gpu_mig",
                        "project_usage",
                        "network_bridge_acl",
                        "warnings",
                        "projects_restricted_backups_and_snapshots",
                        "clustering_join_token",
                        "clustering_description",
                        "server_trusted_proxy",
                        "clustering_update_cert",
                        "storage_api_project",
                        "server_instance_driver_operational",
                        "server_supported_storage_drivers",
                        "event_lifecycle_requestor_address",
                        "resources_gpu_usb",
                        "clustering_evacuation",
                        "network_ovn_nat_address",
                        "network_bgp",
                        "network_forward",
                        "custom_volume_refresh",
                        "network_counters_errors_dropped",
                        "metrics",
                        "image_source_project",
                        "clustering_config",
                        "network_peer",
                        "linux_sysctl",
                        "network_dns",
                        "ovn_nic_acceleration",
                        "certificate_self_renewal",
                        "instance_project_move",
                        "storage_volume_project_move",
                        "cloud_init",
                        "network_dns_nat",
                        "database_leader",
                        "instance_all_projects",
                        "clustering_groups",
                        "ceph_rbd_du",
                        "instance_get_full",
                        "qemu_metrics",
                        "gpu_mig_uuid",
                        "event_project",
                        "clustering_evacuation_live",
                        "instance_allow_inconsistent_copy",
                        "network_state_ovn",
                        "storage_volume_api_filtering",
                        "image_restrictions",
                        "storage_zfs_export",
                        "network_dns_records",
                        "storage_zfs_reserve_space",
                        "network_acl_log",
                        "storage_zfs_blocksize",
                        "metrics_cpu_seconds",
                        "instance_snapshot_never",
                        "certificate_token",
                        "instance_nic_routed_neighbor_probe",
                        "event_hub",
                        "agent_nic_config",
                        "projects_restricted_intercept",
                        "metrics_authentication",
                        "images_target_project",
                        "cluster_migration_inconsistent_copy",
                        "cluster_ovn_chassis",
                        "container_syscall_intercept_sched_setscheduler",
                        "storage_lvm_thinpool_metadata_size",
                        "storage_volume_state_total",
                        "instance_file_head",
                        "instances_nic_host_name",
                        "image_copy_profile",
                        "container_syscall_intercept_sysinfo",
                        "clustering_evacuation_mode",
                        "resources_pci_vpd",
                        "qemu_raw_conf",
                        "storage_cephfs_fscache",
                        "network_load_balancer",
                        "vsock_api",
                        "instance_ready_state",
                        "network_bgp_holdtime",
                        "storage_volumes_all_projects",
                        "metrics_memory_oom_total",
                        "storage_buckets",
                        "storage_buckets_create_credentials",
                        "metrics_cpu_effective_total",
                        "projects_networks_restricted_access",
                        "storage_buckets_local",
                        "loki",
                        "acme",
                        "internal_metrics",
                        "cluster_join_token_expiry",
                        "remote_token_expiry",
                        "init_preseed",
                        "storage_volumes_created_at",
                        "cpu_hotplug",
                        "projects_networks_zones",
                        "network_txqueuelen",
                        "cluster_member_state",
                        "instances_placement_scriptlet",
                        "storage_pool_source_wipe",
                        "zfs_block_mode",
                        "instance_generation_id",
                        "disk_io_cache",
                        "amd_sev",
                        "storage_pool_loop_resize",
                        "migration_vm_live",
                        "ovn_nic_nesting",
                        "oidc",
                        "network_ovn_l3only",
                        "ovn_nic_acceleration_vdpa",
                        "cluster_healing",
                        "instances_state_total",
                        "auth_user",
                        "security_csm",
                        "instances_rebuild",
                        "numa_cpu_placement",
                        "custom_volume_iso",
                        "network_allocations",
                        "storage_api_remote_volume_snapshot_copy",
                        "zfs_delegate",
                        "operations_get_query_all_projects",
                        "metadata_configuration",
                        "syslog_socket",
                        "event_lifecycle_name_and_project",
                        "instances_nic_limits_priority",
                        "disk_initial_volume_configuration",
                        "operation_wait",
                        "cluster_internal_custom_volume_copy",
                        "disk_io_bus",
                        "storage_cephfs_create_missing",
                        "instance_move_config",
                        "ovn_ssl_config",
                        "init_preseed_storage_volumes",
                        "metrics_instances_count",
                        "server_instance_type_info",
                        "resources_disk_mounted",
                        "server_version_lts",
                        "oidc_groups_claim",
                        "loki_config_instance",
                        "storage_volatile_uuid",
                        "import_instance_devices",
                        "instances_uefi_vars",
                        "instances_migration_stateful",
                        "container_syscall_filtering_allow_deny_syntax",
                        "access_management",
                        "vm_disk_io_limits",
                        "storage_volumes_all",
                        "instances_files_modify_permissions"
                ],
                "api_status": "stable",
                "api_version": "1.0",
                "auth": "trusted",
                "public": false,
                "auth_methods": [
                        "tls"
                ],
                "auth_user_name": "",
                "auth_user_method": "",
                "environment": {
                        "addresses": null,
                        "architectures": null,
                        "certificate": "",
                        "certificate_fingerprint": "",
                        "driver": "",
                        "driver_version": "",
                        "instance_types": null,
                        "firewall": "",
                        "kernel": "Linux",
                        "kernel_architecture": "x86_64",
                        "kernel_features": null,
                        "kernel_version": "5.15.0-112-generic",
                        "lxc_features": null,
                        "os_name": "",
                        "os_version": "",
                        "project": "",
                        "server": "lxd-agent",
                        "server_clustered": false,
                        "server_event_mode": "",
                        "server_name": "chr-uat",
                        "server_pid": 478,
                        "server_version": "5.21.1",
                        "server_lts": false,
                        "storage": "",
                        "storage_version": "",
                        "storage_supported_drivers": null
                }
        }
DEBUG  [2024-07-03T08:31:12+07:00] Got response struct from LXD
DEBUG  [2024-07-03T08:31:12+07:00] Sending request to LXD                        etag= method=GET url="https://custom.socket/1.0/state"
DEBUG  [2024-07-03T08:31:12+07:00]
        {
                "config": null,
                "api_extensions": [
                        "storage_zfs_remove_snapshots",
                        "container_host_shutdown_timeout",
                        "container_stop_priority",
                        "container_syscall_filtering",
                        "auth_pki",
                        "container_last_used_at",
                        "etag",
                        "patch",
                        "usb_devices",
                        "https_allowed_credentials",
                        "image_compression_algorithm",
                        "directory_manipulation",
                        "container_cpu_time",
                        "storage_zfs_use_refquota",
                        "storage_lvm_mount_options",
                        "network",
                        "profile_usedby",
                        "container_push",
                        "container_exec_recording",
                        "certificate_update",
                        "container_exec_signal_handling",
                        "gpu_devices",
                        "container_image_properties",
                        "migration_progress",
                        "id_map",
                        "network_firewall_filtering",
                        "network_routes",
                        "storage",
                        "file_delete",
                        "file_append",
                        "network_dhcp_expiry",
                        "storage_lvm_vg_rename",
                        "storage_lvm_thinpool_rename",
                        "network_vlan",
                        "image_create_aliases",
                        "container_stateless_copy",
                        "container_only_migration",
                        "storage_zfs_clone_copy",
                        "unix_device_rename",
                        "storage_lvm_use_thinpool",
                        "storage_rsync_bwlimit",
                        "network_vxlan_interface",
                        "storage_btrfs_mount_options",
                        "entity_description",
                        "image_force_refresh",
                        "storage_lvm_lv_resizing",
                        "id_map_base",
                        "file_symlinks",
                        "container_push_target",
                        "network_vlan_physical",
                        "storage_images_delete",
                        "container_edit_metadata",
                        "container_snapshot_stateful_migration",
                        "storage_driver_ceph",
                        "storage_ceph_user_name",
                        "resource_limits",
                        "storage_volatile_initial_source",
                        "storage_ceph_force_osd_reuse",
                        "storage_block_filesystem_btrfs",
                        "resources",
                        "kernel_limits",
                        "storage_api_volume_rename",
                        "network_sriov",
                        "console",
                        "restrict_devlxd",
                        "migration_pre_copy",
                        "infiniband",
                        "maas_network",
                        "devlxd_events",
                        "proxy",
                        "network_dhcp_gateway",
                        "file_get_symlink",
                        "network_leases",
                        "unix_device_hotplug",
                        "storage_api_local_volume_handling",
                        "operation_description",
                        "clustering",
                        "event_lifecycle",
                        "storage_api_remote_volume_handling",
                        "nvidia_runtime",
                        "container_mount_propagation",
                        "container_backup",
                        "devlxd_images",
                        "container_local_cross_pool_handling",
                        "proxy_unix",
                        "proxy_udp",
                        "clustering_join",
                        "proxy_tcp_udp_multi_port_handling",
                        "network_state",
                        "proxy_unix_dac_properties",
                        "container_protection_delete",
                        "unix_priv_drop",
                        "pprof_http",
                        "proxy_haproxy_protocol",
                        "network_hwaddr",
                        "proxy_nat",
                        "network_nat_order",
                        "container_full",
                        "backup_compression",
                        "nvidia_runtime_config",
                        "storage_api_volume_snapshots",
                        "storage_unmapped",
                        "projects",
                        "network_vxlan_ttl",
                        "container_incremental_copy",
                        "usb_optional_vendorid",
                        "snapshot_scheduling",
                        "snapshot_schedule_aliases",
                        "container_copy_project",
                        "clustering_server_address",
                        "clustering_image_replication",
                        "container_protection_shift",
                        "snapshot_expiry",
                        "container_backup_override_pool",
                        "snapshot_expiry_creation",
                        "network_leases_location",
                        "resources_cpu_socket",
                        "resources_gpu",
                        "resources_numa",
                        "kernel_features",
                        "id_map_current",
                        "event_location",
                        "storage_api_remote_volume_snapshots",
                        "network_nat_address",
                        "container_nic_routes",
                        "cluster_internal_copy",
                        "seccomp_notify",
                        "lxc_features",
                        "container_nic_ipvlan",
                        "network_vlan_sriov",
                        "storage_cephfs",
                        "container_nic_ipfilter",
                        "resources_v2",
                        "container_exec_user_group_cwd",
                        "container_syscall_intercept",
                        "container_disk_shift",
                        "storage_shifted",
                        "resources_infiniband",
                        "daemon_storage",
                        "instances",
                        "image_types",
                        "resources_disk_sata",
                        "clustering_roles",
                        "images_expiry",
                        "resources_network_firmware",
                        "backup_compression_algorithm",
                        "ceph_data_pool_name",
                        "container_syscall_intercept_mount",
                        "compression_squashfs",
                        "container_raw_mount",
                        "container_nic_routed",
                        "container_syscall_intercept_mount_fuse",
                        "container_disk_ceph",
                        "virtual-machines",
                        "image_profiles",
                        "clustering_architecture",
                        "resources_disk_id",
                        "storage_lvm_stripes",
                        "vm_boot_priority",
                        "unix_hotplug_devices",
                        "api_filtering",
                        "instance_nic_network",
                        "clustering_sizing",
                        "firewall_driver",
                        "projects_limits",
                        "container_syscall_intercept_hugetlbfs",
                        "limits_hugepages",
                        "container_nic_routed_gateway",
                        "projects_restrictions",
                        "custom_volume_snapshot_expiry",
                        "volume_snapshot_scheduling",
                        "trust_ca_certificates",
                        "snapshot_disk_usage",
                        "clustering_edit_roles",
                        "container_nic_routed_host_address",
                        "container_nic_ipvlan_gateway",
                        "resources_usb_pci",
                        "resources_cpu_threads_numa",
                        "resources_cpu_core_die",
                        "api_os",
                        "container_nic_routed_host_table",
                        "container_nic_ipvlan_host_table",
                        "container_nic_ipvlan_mode",
                        "resources_system",
                        "images_push_relay",
                        "network_dns_search",
                        "container_nic_routed_limits",
                        "instance_nic_bridged_vlan",
                        "network_state_bond_bridge",
                        "usedby_consistency",
                        "custom_block_volumes",
                        "clustering_failure_domains",
                        "resources_gpu_mdev",
                        "console_vga_type",
                        "projects_limits_disk",
                        "network_type_macvlan",
                        "network_type_sriov",
                        "container_syscall_intercept_bpf_devices",
                        "network_type_ovn",
                        "projects_networks",
                        "projects_networks_restricted_uplinks",
                        "custom_volume_backup",
                        "backup_override_name",
                        "storage_rsync_compression",
                        "network_type_physical",
                        "network_ovn_external_subnets",
                        "network_ovn_nat",
                        "network_ovn_external_routes_remove",
                        "tpm_device_type",
                        "storage_zfs_clone_copy_rebase",
                        "gpu_mdev",
                        "resources_pci_iommu",
                        "resources_network_usb",
                        "resources_disk_address",
                        "network_physical_ovn_ingress_mode",
                        "network_ovn_dhcp",
                        "network_physical_routes_anycast",
                        "projects_limits_instances",
                        "network_state_vlan",
                        "instance_nic_bridged_port_isolation",
                        "instance_bulk_state_change",
                        "network_gvrp",
                        "instance_pool_move",
                        "gpu_sriov",
                        "pci_device_type",
                        "storage_volume_state",
                        "network_acl",
                        "migration_stateful",
                        "disk_state_quota",
                        "storage_ceph_features",
                        "projects_compression",
                        "projects_images_remote_cache_expiry",
                        "certificate_project",
                        "network_ovn_acl",
                        "projects_images_auto_update",
                        "projects_restricted_cluster_target",
                        "images_default_architecture",
                        "network_ovn_acl_defaults",
                        "gpu_mig",
                        "project_usage",
                        "network_bridge_acl",
                        "warnings",
                        "projects_restricted_backups_and_snapshots",
                        "clustering_join_token",
                        "clustering_description",
                        "server_trusted_proxy",
                        "clustering_update_cert",
                        "storage_api_project",
                        "server_instance_driver_operational",
                        "server_supported_storage_drivers",
                        "event_lifecycle_requestor_address",
                        "resources_gpu_usb",
                        "clustering_evacuation",
                        "network_ovn_nat_address",
                        "network_bgp",
                        "network_forward",
                        "custom_volume_refresh",
                        "network_counters_errors_dropped",
                        "metrics",
                        "image_source_project",
                        "clustering_config",
                        "network_peer",
                        "linux_sysctl",
                        "network_dns",
                        "ovn_nic_acceleration",
                        "certificate_self_renewal",
                        "instance_project_move",
                        "storage_volume_project_move",
                        "cloud_init",
                        "network_dns_nat",
                        "database_leader",
                        "instance_all_projects",
                        "clustering_groups",
                        "ceph_rbd_du",
                        "instance_get_full",
                        "qemu_metrics",
                        "gpu_mig_uuid",
                        "event_project",
                        "clustering_evacuation_live",
                        "instance_allow_inconsistent_copy",
                        "network_state_ovn",
                        "storage_volume_api_filtering",
                        "image_restrictions",
                        "storage_zfs_export",
                        "network_dns_records",
                        "storage_zfs_reserve_space",
                        "network_acl_log",
                        "storage_zfs_blocksize",
                        "metrics_cpu_seconds",
                        "instance_snapshot_never",
                        "certificate_token",
                        "instance_nic_routed_neighbor_probe",
                        "event_hub",
                        "agent_nic_config",
                        "projects_restricted_intercept",
                        "metrics_authentication",
                        "images_target_project",
                        "cluster_migration_inconsistent_copy",
                        "cluster_ovn_chassis",
                        "container_syscall_intercept_sched_setscheduler",
                        "storage_lvm_thinpool_metadata_size",
                        "storage_volume_state_total",
                        "instance_file_head",
                        "instances_nic_host_name",
                        "image_copy_profile",
                        "container_syscall_intercept_sysinfo",
                        "clustering_evacuation_mode",
                        "resources_pci_vpd",
                        "qemu_raw_conf",
                        "storage_cephfs_fscache",
                        "network_load_balancer",
                        "vsock_api",
                        "instance_ready_state",
                        "network_bgp_holdtime",
                        "storage_volumes_all_projects",
                        "metrics_memory_oom_total",
                        "storage_buckets",
                        "storage_buckets_create_credentials",
                        "metrics_cpu_effective_total",
                        "projects_networks_restricted_access",
                        "storage_buckets_local",
                        "loki",
                        "acme",
                        "internal_metrics",
                        "cluster_join_token_expiry",
                        "remote_token_expiry",
                        "init_preseed",
                        "storage_volumes_created_at",
                        "cpu_hotplug",
                        "projects_networks_zones",
                        "network_txqueuelen",
                        "cluster_member_state",
                        "instances_placement_scriptlet",
                        "storage_pool_source_wipe",
                        "zfs_block_mode",
                        "instance_generation_id",
                        "disk_io_cache",
                        "amd_sev",
                        "storage_pool_loop_resize",
                        "migration_vm_live",
                        "ovn_nic_nesting",
                        "oidc",
                        "network_ovn_l3only",
                        "ovn_nic_acceleration_vdpa",
                        "cluster_healing",
                        "instances_state_total",
                        "auth_user",
                        "security_csm",
                        "instances_rebuild",
                        "numa_cpu_placement",
                        "custom_volume_iso",
                        "network_allocations",
                        "storage_api_remote_volume_snapshot_copy",
                        "zfs_delegate",
                        "operations_get_query_all_projects",
                        "metadata_configuration",
                        "syslog_socket",
                        "event_lifecycle_name_and_project",
                        "instances_nic_limits_priority",
                        "disk_initial_volume_configuration",
                        "operation_wait",
                        "cluster_internal_custom_volume_copy",
                        "disk_io_bus",
                        "storage_cephfs_create_missing",
                        "instance_move_config",
                        "ovn_ssl_config",
                        "init_preseed_storage_volumes",
                        "metrics_instances_count",
                        "server_instance_type_info",
                        "resources_disk_mounted",
                        "server_version_lts",
                        "oidc_groups_claim",
                        "loki_config_instance",
                        "storage_volatile_uuid",
                        "import_instance_devices",
                        "instances_uefi_vars",
                        "instances_migration_stateful",
                        "container_syscall_filtering_allow_deny_syntax",
                        "access_management",
                        "vm_disk_io_limits",
                        "storage_volumes_all",
                        "instances_files_modify_permissions"
                ],
                "api_status": "stable",
                "api_version": "1.0",
                "auth": "trusted",
                "public": false,
                "auth_methods": [
                        "tls"
                ],
                "auth_user_name": "",
                "auth_user_method": "",
                "environment": {
                        "addresses": null,
                        "architectures": null,
                        "certificate": "",
                        "certificate_fingerprint": "",
                        "driver": "",
                        "driver_version": "",
                        "instance_types": null,
                        "firewall": "",
                        "kernel": "Linux",
                        "kernel_architecture": "x86_64",
                        "kernel_features": null,
                        "kernel_version": "6.8.0-36-generic",
                        "lxc_features": null,
                        "os_name": "",
                        "os_version": "",
                        "project": "",
                        "server": "lxd-agent",
                        "server_clustered": false,
                        "server_event_mode": "",
                        "server_name": "chr-uat2",
                        "server_pid": 571,
                        "server_version": "5.21.1",
                        "server_lts": false,
                        "storage": "",
                        "storage_version": "",
                        "storage_supported_drivers": null
                }
        }
DEBUG  [2024-07-03T08:31:12+07:00] Got response struct from LXD
DEBUG  [2024-07-03T08:31:12+07:00] Sending request to LXD                        etag= method=GET url="https://custom.socket/1.0/state"
DEBUG  [2024-07-03T08:31:12+07:00]
        {
                "config": null,
                "api_extensions": [
                        "storage_zfs_remove_snapshots",
                        "container_host_shutdown_timeout",
                        "container_stop_priority",
                        "container_syscall_filtering",
                        "auth_pki",
                        "container_last_used_at",
                        "etag",
                        "patch",
                        "usb_devices",
                        "https_allowed_credentials",
                        "image_compression_algorithm",
                        "directory_manipulation",
                        "container_cpu_time",
                        "storage_zfs_use_refquota",
                        "storage_lvm_mount_options",
                        "network",
                        "profile_usedby",
                        "container_push",
                        "container_exec_recording",
                        "certificate_update",
                        "container_exec_signal_handling",
                        "gpu_devices",
                        "container_image_properties",
                        "migration_progress",
                        "id_map",
                        "network_firewall_filtering",
                        "network_routes",
                        "storage",
                        "file_delete",
                        "file_append",
                        "network_dhcp_expiry",
                        "storage_lvm_vg_rename",
                        "storage_lvm_thinpool_rename",
                        "network_vlan",
                        "image_create_aliases",
                        "container_stateless_copy",
                        "container_only_migration",
                        "storage_zfs_clone_copy",
                        "unix_device_rename",
                        "storage_lvm_use_thinpool",
                        "storage_rsync_bwlimit",
                        "network_vxlan_interface",
                        "storage_btrfs_mount_options",
                        "entity_description",
                        "image_force_refresh",
                        "storage_lvm_lv_resizing",
                        "id_map_base",
                        "file_symlinks",
                        "container_push_target",
                        "network_vlan_physical",
                        "storage_images_delete",
                        "container_edit_metadata",
                        "container_snapshot_stateful_migration",
                        "storage_driver_ceph",
                        "storage_ceph_user_name",
                        "resource_limits",
                        "storage_volatile_initial_source",
                        "storage_ceph_force_osd_reuse",
                        "storage_block_filesystem_btrfs",
                        "resources",
                        "kernel_limits",
                        "storage_api_volume_rename",
                        "network_sriov",
                        "console",
                        "restrict_devlxd",
                        "migration_pre_copy",
                        "infiniband",
                        "maas_network",
                        "devlxd_events",
                        "proxy",
                        "network_dhcp_gateway",
                        "file_get_symlink",
                        "network_leases",
                        "unix_device_hotplug",
                        "storage_api_local_volume_handling",
                        "operation_description",
                        "clustering",
                        "event_lifecycle",
                        "storage_api_remote_volume_handling",
                        "nvidia_runtime",
                        "container_mount_propagation",
                        "container_backup",
                        "devlxd_images",
                        "container_local_cross_pool_handling",
                        "proxy_unix",
                        "proxy_udp",
                        "clustering_join",
                        "proxy_tcp_udp_multi_port_handling",
                        "network_state",
                        "proxy_unix_dac_properties",
                        "container_protection_delete",
                        "unix_priv_drop",
                        "pprof_http",
                        "proxy_haproxy_protocol",
                        "network_hwaddr",
                        "proxy_nat",
                        "network_nat_order",
                        "container_full",
                        "backup_compression",
                        "nvidia_runtime_config",
                        "storage_api_volume_snapshots",
                        "storage_unmapped",
                        "projects",
                        "network_vxlan_ttl",
                        "container_incremental_copy",
                        "usb_optional_vendorid",
                        "snapshot_scheduling",
                        "snapshot_schedule_aliases",
                        "container_copy_project",
                        "clustering_server_address",
                        "clustering_image_replication",
                        "container_protection_shift",
                        "snapshot_expiry",
                        "container_backup_override_pool",
                        "snapshot_expiry_creation",
                        "network_leases_location",
                        "resources_cpu_socket",
                        "resources_gpu",
                        "resources_numa",
                        "kernel_features",
                        "id_map_current",
                        "event_location",
                        "storage_api_remote_volume_snapshots",
                        "network_nat_address",
                        "container_nic_routes",
                        "cluster_internal_copy",
                        "seccomp_notify",
                        "lxc_features",
                        "container_nic_ipvlan",
                        "network_vlan_sriov",
                        "storage_cephfs",
                        "container_nic_ipfilter",
                        "resources_v2",
                        "container_exec_user_group_cwd",
                        "container_syscall_intercept",
                        "container_disk_shift",
                        "storage_shifted",
                        "resources_infiniband",
                        "daemon_storage",
                        "instances",
                        "image_types",
                        "resources_disk_sata",
                        "clustering_roles",
                        "images_expiry",
                        "resources_network_firmware",
                        "backup_compression_algorithm",
                        "ceph_data_pool_name",
                        "container_syscall_intercept_mount",
                        "compression_squashfs",
                        "container_raw_mount",
                        "container_nic_routed",
                        "container_syscall_intercept_mount_fuse",
                        "container_disk_ceph",
                        "virtual-machines",
                        "image_profiles",
                        "clustering_architecture",
                        "resources_disk_id",
                        "storage_lvm_stripes",
                        "vm_boot_priority",
                        "unix_hotplug_devices",
                        "api_filtering",
                        "instance_nic_network",
                        "clustering_sizing",
                        "firewall_driver",
                        "projects_limits",
                        "container_syscall_intercept_hugetlbfs",
                        "limits_hugepages",
                        "container_nic_routed_gateway",
                        "projects_restrictions",
                        "custom_volume_snapshot_expiry",
                        "volume_snapshot_scheduling",
                        "trust_ca_certificates",
                        "snapshot_disk_usage",
                        "clustering_edit_roles",
                        "container_nic_routed_host_address",
                        "container_nic_ipvlan_gateway",
                        "resources_usb_pci",
                        "resources_cpu_threads_numa",
                        "resources_cpu_core_die",
                        "api_os",
                        "container_nic_routed_host_table",
                        "container_nic_ipvlan_host_table",
                        "container_nic_ipvlan_mode",
                        "resources_system",
                        "images_push_relay",
                        "network_dns_search",
                        "container_nic_routed_limits",
                        "instance_nic_bridged_vlan",
                        "network_state_bond_bridge",
                        "usedby_consistency",
                        "custom_block_volumes",
                        "clustering_failure_domains",
                        "resources_gpu_mdev",
                        "console_vga_type",
                        "projects_limits_disk",
                        "network_type_macvlan",
                        "network_type_sriov",
                        "container_syscall_intercept_bpf_devices",
                        "network_type_ovn",
                        "projects_networks",
                        "projects_networks_restricted_uplinks",
                        "custom_volume_backup",
                        "backup_override_name",
                        "storage_rsync_compression",
                        "network_type_physical",
                        "network_ovn_external_subnets",
                        "network_ovn_nat",
                        "network_ovn_external_routes_remove",
                        "tpm_device_type",
                        "storage_zfs_clone_copy_rebase",
                        "gpu_mdev",
                        "resources_pci_iommu",
                        "resources_network_usb",
                        "resources_disk_address",
                        "network_physical_ovn_ingress_mode",
                        "network_ovn_dhcp",
                        "network_physical_routes_anycast",
                        "projects_limits_instances",
                        "network_state_vlan",
                        "instance_nic_bridged_port_isolation",
                        "instance_bulk_state_change",
                        "network_gvrp",
                        "instance_pool_move",
                        "gpu_sriov",
                        "pci_device_type",
                        "storage_volume_state",
                        "network_acl",
                        "migration_stateful",
                        "disk_state_quota",
                        "storage_ceph_features",
                        "projects_compression",
                        "projects_images_remote_cache_expiry",
                        "certificate_project",
                        "network_ovn_acl",
                        "projects_images_auto_update",
                        "projects_restricted_cluster_target",
                        "images_default_architecture",
                        "network_ovn_acl_defaults",
                        "gpu_mig",
                        "project_usage",
                        "network_bridge_acl",
                        "warnings",
                        "projects_restricted_backups_and_snapshots",
                        "clustering_join_token",
                        "clustering_description",
                        "server_trusted_proxy",
                        "clustering_update_cert",
                        "storage_api_project",
                        "server_instance_driver_operational",
                        "server_supported_storage_drivers",
                        "event_lifecycle_requestor_address",
                        "resources_gpu_usb",
                        "clustering_evacuation",
                        "network_ovn_nat_address",
                        "network_bgp",
                        "network_forward",
                        "custom_volume_refresh",
                        "network_counters_errors_dropped",
                        "metrics",
                        "image_source_project",
                        "clustering_config",
                        "network_peer",
                        "linux_sysctl",
                        "network_dns",
                        "ovn_nic_acceleration",
                        "certificate_self_renewal",
                        "instance_project_move",
                        "storage_volume_project_move",
                        "cloud_init",
                        "network_dns_nat",
                        "database_leader",
                        "instance_all_projects",
                        "clustering_groups",
                        "ceph_rbd_du",
                        "instance_get_full",
                        "qemu_metrics",
                        "gpu_mig_uuid",
                        "event_project",
                        "clustering_evacuation_live",
                        "instance_allow_inconsistent_copy",
                        "network_state_ovn",
                        "storage_volume_api_filtering",
                        "image_restrictions",
                        "storage_zfs_export",
                        "network_dns_records",
                        "storage_zfs_reserve_space",
                        "network_acl_log",
                        "storage_zfs_blocksize",
                        "metrics_cpu_seconds",
                        "instance_snapshot_never",
                        "certificate_token",
                        "instance_nic_routed_neighbor_probe",
                        "event_hub",
                        "agent_nic_config",
                        "projects_restricted_intercept",
                        "metrics_authentication",
                        "images_target_project",
                        "cluster_migration_inconsistent_copy",
                        "cluster_ovn_chassis",
                        "container_syscall_intercept_sched_setscheduler",
                        "storage_lvm_thinpool_metadata_size",
                        "storage_volume_state_total",
                        "instance_file_head",
                        "instances_nic_host_name",
                        "image_copy_profile",
                        "container_syscall_intercept_sysinfo",
                        "clustering_evacuation_mode",
                        "resources_pci_vpd",
                        "qemu_raw_conf",
                        "storage_cephfs_fscache",
                        "network_load_balancer",
                        "vsock_api",
                        "instance_ready_state",
                        "network_bgp_holdtime",
                        "storage_volumes_all_projects",
                        "metrics_memory_oom_total",
                        "storage_buckets",
                        "storage_buckets_create_credentials",
                        "metrics_cpu_effective_total",
                        "projects_networks_restricted_access",
                        "storage_buckets_local",
                        "loki",
                        "acme",
                        "internal_metrics",
                        "cluster_join_token_expiry",
                        "remote_token_expiry",
                        "init_preseed",
                        "storage_volumes_created_at",
                        "cpu_hotplug",
                        "projects_networks_zones",
                        "network_txqueuelen",
                        "cluster_member_state",
                        "instances_placement_scriptlet",
                        "storage_pool_source_wipe",
                        "zfs_block_mode",
                        "instance_generation_id",
                        "disk_io_cache",
                        "amd_sev",
                        "storage_pool_loop_resize",
                        "migration_vm_live",
                        "ovn_nic_nesting",
                        "oidc",
                        "network_ovn_l3only",
                        "ovn_nic_acceleration_vdpa",
                        "cluster_healing",
                        "instances_state_total",
                        "auth_user",
                        "security_csm",
                        "instances_rebuild",
                        "numa_cpu_placement",
                        "custom_volume_iso",
                        "network_allocations",
                        "storage_api_remote_volume_snapshot_copy",
                        "zfs_delegate",
                        "operations_get_query_all_projects",
                        "metadata_configuration",
                        "syslog_socket",
                        "event_lifecycle_name_and_project",
                        "instances_nic_limits_priority",
                        "disk_initial_volume_configuration",
                        "operation_wait",
                        "cluster_internal_custom_volume_copy",
                        "disk_io_bus",
                        "storage_cephfs_create_missing",
                        "instance_move_config",
                        "ovn_ssl_config",
                        "init_preseed_storage_volumes",
                        "metrics_instances_count",
                        "server_instance_type_info",
                        "resources_disk_mounted",
                        "server_version_lts",
                        "oidc_groups_claim",
                        "loki_config_instance",
                        "storage_volatile_uuid",
                        "import_instance_devices",
                        "instances_uefi_vars",
                        "instances_migration_stateful",
                        "container_syscall_filtering_allow_deny_syntax",
                        "access_management",
                        "vm_disk_io_limits",
                        "storage_volumes_all",
                        "instances_files_modify_permissions"
                ],
                "api_status": "stable",
                "api_version": "1.0",
                "auth": "trusted",
                "public": false,
                "auth_methods": [
                        "tls"
                ],
                "auth_user_name": "",
                "auth_user_method": "",
                "environment": {
                        "addresses": null,
                        "architectures": null,
                        "certificate": "",
                        "certificate_fingerprint": "",
                        "driver": "",
                        "driver_version": "",
                        "instance_types": null,
                        "firewall": "",
                        "kernel": "Linux",
                        "kernel_architecture": "x86_64",
                        "kernel_features": null,
                        "kernel_version": "6.8.0-36-generic",
                        "lxc_features": null,
                        "os_name": "",
                        "os_version": "",
                        "project": "",
                        "server": "lxd-agent",
                        "server_clustered": false,
                        "server_event_mode": "",
                        "server_name": "chr-uatweb",
                        "server_pid": 582,
                        "server_version": "5.21.1",
                        "server_lts": false,
                        "storage": "",
                        "storage_version": "",
                        "storage_supported_drivers": null
                }
        }
DEBUG  [2024-07-03T08:31:12+07:00] Got response struct from LXD
DEBUG  [2024-07-03T08:31:12+07:00]
        {
                "status": "",
                "status_code": 0,
                "disk": null,
                "memory": {
                        "usage": 1794646016,
                        "usage_peak": 0,
                        "total": 8314474496,
                        "swap_usage": 0,
                        "swap_usage_peak": 0
                },
                "network": {
                        "enp5s0": {
                                "addresses": [
                                        {
                                                "family": "inet",
                                                "address": "10.67.90.39",
                                                "netmask": "24",
                                                "scope": "global"
                                        },
                                        {
                                                "family": "inet6",
                                                "address": "fd42:d158:353c:6900:5054:ff:fe11:13bf",
                                                "netmask": "64",
                                                "scope": "global"
                                        },
                                        {
                                                "family": "inet6",
                                                "address": "fe80::5054:ff:fe11:13bf",
                                                "netmask": "64",
                                                "scope": "link"
                                        }
                                ],
                                "counters": {
                                        "bytes_received": 5036751,
                                        "bytes_sent": 5401143,
                                        "packets_received": 35595,
                                        "packets_sent": 45622,
                                        "errors_received": 0,
                                        "errors_sent": 0,
                                        "packets_dropped_outbound": 0,
                                        "packets_dropped_inbound": 0
                                },
                                "hwaddr": "52:54:00:11:13:bf",
                                "host_name": "",
                                "mtu": 1500,
                                "state": "up",
                                "type": "broadcast"
                        },
                        "lo": {
                                "addresses": [
                                        {
                                                "family": "inet",
                                                "address": "127.0.0.1",
                                                "netmask": "8",
                                                "scope": "local"
                                        },
                                        {
                                                "family": "inet6",
                                                "address": "::1",
                                                "netmask": "128",
                                                "scope": "local"
                                        }
                                ],
                                "counters": {
                                        "bytes_received": 0,
                                        "bytes_sent": 0,
                                        "packets_received": 0,
                                        "packets_sent": 0,
                                        "errors_received": 0,
                                        "errors_sent": 0,
                                        "packets_dropped_outbound": 0,
                                        "packets_dropped_inbound": 0
                                },
                                "hwaddr": "",
                                "host_name": "",
                                "mtu": 65536,
                                "state": "up",
                                "type": "loopback"
                        }
                },
                "pid": 1,
                "processes": 44,
                "cpu": {
                        "usage": 339610896000
                }
        }
DEBUG  [2024-07-03T08:31:12+07:00] GetInstanceUsage started                      driver=dir instance=chr-uat pool=multipass project=multipass
DEBUG  [2024-07-03T08:31:12+07:00] GetInstanceUsage finished                     driver=dir instance=chr-uat pool=multipass project=multipass
DEBUG  [2024-07-03T08:31:12+07:00] Got response struct from LXD
DEBUG  [2024-07-03T08:31:12+07:00]
        {
                "status": "",
                "status_code": 0,
                "disk": null,
                "memory": {
                        "usage": 943697920,
                        "usage_peak": 0,
                        "total": 8317427712,
                        "swap_usage": 0,
                        "swap_usage_peak": 0
                },
                "network": {
                        "enp5s0": {
                                "addresses": [
                                        {
                                                "family": "inet",
                                                "address": "10.67.90.103",
                                                "netmask": "24",
                                                "scope": "global"
                                        },
                                        {
                                                "family": "inet6",
                                                "address": "fd42:d158:353c:6900:5054:ff:fec8:1c57",
                                                "netmask": "64",
                                                "scope": "global"
                                        },
                                        {
                                                "family": "inet6",
                                                "address": "fe80::5054:ff:fec8:1c57",
                                                "netmask": "64",
                                                "scope": "link"
                                        }
                                ],
                                "counters": {
                                        "bytes_received": 3002032,
                                        "bytes_sent": 228295,
                                        "packets_received": 3598,
                                        "packets_sent": 2367,
                                        "errors_received": 0,
                                        "errors_sent": 0,
                                        "packets_dropped_outbound": 0,
                                        "packets_dropped_inbound": 0
                                },
                                "hwaddr": "52:54:00:c8:1c:57",
                                "host_name": "",
                                "mtu": 1500,
                                "state": "up",
                                "type": "broadcast"
                        },
                        "lo": {
                                "addresses": [
                                        {
                                                "family": "inet",
                                                "address": "127.0.0.1",
                                                "netmask": "8",
                                                "scope": "local"
                                        },
                                        {
                                                "family": "inet6",
                                                "address": "::1",
                                                "netmask": "128",
                                                "scope": "local"
                                        }
                                ],
                                "counters": {
                                        "bytes_received": 12744,
                                        "bytes_sent": 12744,
                                        "packets_received": 136,
                                        "packets_sent": 136,
                                        "errors_received": 0,
                                        "errors_sent": 0,
                                        "packets_dropped_outbound": 0,
                                        "packets_dropped_inbound": 0
                                },
                                "hwaddr": "",
                                "host_name": "",
                                "mtu": 65536,
                                "state": "up",
                                "type": "loopback"
                        }
                },
                "pid": 1,
                "processes": 26,
                "cpu": {
                        "usage": 180807716000
                }
        }
DEBUG  [2024-07-03T08:31:12+07:00] Got response struct from LXD
DEBUG  [2024-07-03T08:31:12+07:00]
        {
                "status": "",
                "status_code": 0,
                "disk": null,
                "memory": {
                        "usage": 844226560,
                        "usage_peak": 0,
                        "total": 8317411328,
                        "swap_usage": 0,
                        "swap_usage_peak": 0
                },
                "network": {
                        "enp5s0": {
                                "addresses": [
                                        {
                                                "family": "inet",
                                                "address": "10.67.90.144",
                                                "netmask": "24",
                                                "scope": "global"
                                        },
                                        {
                                                "family": "inet6",
                                                "address": "fd42:d158:353c:6900:5054:ff:fefc:8737",
                                                "netmask": "64",
                                                "scope": "global"
                                        },
                                        {
                                                "family": "inet6",
                                                "address": "fe80::5054:ff:fefc:8737",
                                                "netmask": "64",
                                                "scope": "link"
                                        }
                                ],
                                "counters": {
                                        "bytes_received": 472960,
                                        "bytes_sent": 98390,
                                        "packets_received": 1410,
                                        "packets_sent": 880,
                                        "errors_received": 0,
                                        "errors_sent": 0,
                                        "packets_dropped_outbound": 0,
                                        "packets_dropped_inbound": 0
                                },
                                "hwaddr": "52:54:00:fc:87:37",
                                "host_name": "",
                                "mtu": 1500,
                                "state": "up",
                                "type": "broadcast"
                        },
                        "lo": {
                                "addresses": [
                                        {
                                                "family": "inet",
                                                "address": "127.0.0.1",
                                                "netmask": "8",
                                                "scope": "local"
                                        },
                                        {
                                                "family": "inet6",
                                                "address": "::1",
                                                "netmask": "128",
                                                "scope": "local"
                                        }
                                ],
                                "counters": {
                                        "bytes_received": 8960,
                                        "bytes_sent": 8960,
                                        "packets_received": 108,
                                        "packets_sent": 108,
                                        "errors_received": 0,
                                        "errors_sent": 0,
                                        "packets_dropped_outbound": 0,
                                        "packets_dropped_inbound": 0
                                },
                                "hwaddr": "",
                                "host_name": "",
                                "mtu": 65536,
                                "state": "up",
                                "type": "loopback"
                        }
                },
                "pid": 1,
                "processes": 24,
                "cpu": {
                        "usage": 160556149000
                }
        }
DEBUG  [2024-07-03T08:31:12+07:00] GetInstanceUsage started                      driver=dir instance=chr-uatweb pool=multipass project=multipass
DEBUG  [2024-07-03T08:31:12+07:00] GetInstanceUsage started                      driver=dir instance=chr-uat2 pool=multipass project=multipass
DEBUG  [2024-07-03T08:31:12+07:00] GetInstanceUsage finished                     driver=dir instance=chr-uatweb pool=multipass project=multipass
DEBUG  [2024-07-03T08:31:12+07:00] GetInstanceUsage finished                     driver=dir instance=chr-uat2 pool=multipass project=multipass

result:

chr@chr:~$ sudo lxc list --project=multipass
+------------+---------+-----------------------+------------------------------------------------+-----------------+-----------+
|    NAME    |  STATE  |         IPV4          |                      IPV6                      |      TYPE       | SNAPSHOTS |
+------------+---------+-----------------------+------------------------------------------------+-----------------+-----------+
| chr-uat    | RUNNING | 10.67.90.39 (enp5s0)  | fd42:d158:353c:6900:5054:ff:fe11:13bf (enp5s0) | VIRTUAL-MACHINE | 0         |
+------------+---------+-----------------------+------------------------------------------------+-----------------+-----------+
| chr-uat2   | RUNNING | 10.67.90.103 (enp5s0) | fd42:d158:353c:6900:5054:ff:fec8:1c57 (enp5s0) | VIRTUAL-MACHINE | 0         |
+------------+---------+-----------------------+------------------------------------------------+-----------------+-----------+
| chr-uatweb | RUNNING | 10.67.90.144 (enp5s0) | fd42:d158:353c:6900:5054:ff:fefc:8737 (enp5s0) | VIRTUAL-MACHINE | 0         |
+------------+---------+-----------------------+------------------------------------------------+-----------------+-----------+
chr@chr:~$

lxc info --project=multipass

result:

chr@chr:~$ sudo lxc info --project=multipass
config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- backup_compression
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- instances_nic_host_name
- image_copy_profile
- container_syscall_intercept_sysinfo
- clustering_evacuation_mode
- resources_pci_vpd
- qemu_raw_conf
- storage_cephfs_fscache
- network_load_balancer
- vsock_api
- instance_ready_state
- network_bgp_holdtime
- storage_volumes_all_projects
- metrics_memory_oom_total
- storage_buckets
- storage_buckets_create_credentials
- metrics_cpu_effective_total
- projects_networks_restricted_access
- storage_buckets_local
- loki
- acme
- internal_metrics
- cluster_join_token_expiry
- remote_token_expiry
- init_preseed
- storage_volumes_created_at
- cpu_hotplug
- projects_networks_zones
- network_txqueuelen
- cluster_member_state
- instances_placement_scriptlet
- storage_pool_source_wipe
- zfs_block_mode
- instance_generation_id
- disk_io_cache
- amd_sev
- storage_pool_loop_resize
- migration_vm_live
- ovn_nic_nesting
- oidc
- network_ovn_l3only
- ovn_nic_acceleration_vdpa
- cluster_healing
- instances_state_total
- auth_user
- security_csm
- instances_rebuild
- numa_cpu_placement
- custom_volume_iso
- network_allocations
- storage_api_remote_volume_snapshot_copy
- zfs_delegate
- operations_get_query_all_projects
- metadata_configuration
- syslog_socket
- event_lifecycle_name_and_project
- instances_nic_limits_priority
- disk_initial_volume_configuration
- operation_wait
- cluster_internal_custom_volume_copy
- disk_io_bus
- storage_cephfs_create_missing
- instance_move_config
- ovn_ssl_config
- init_preseed_storage_volumes
- metrics_instances_count
- server_instance_type_info
- resources_disk_mounted
- server_version_lts
- oidc_groups_claim
- loki_config_instance
- storage_volatile_uuid
- import_instance_devices
- instances_uefi_vars
- instances_migration_stateful
- container_syscall_filtering_allow_deny_syntax
- access_management
- vm_disk_io_limits
- storage_volumes_all
- instances_files_modify_permissions
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
auth_user_name: root
auth_user_method: unix
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    ..deleted by me..
    -----END CERTIFICATE-----
  certificate_fingerprint: ..deleted by me..
  driver: lxc | qemu
  driver_version: 6.0.0 | 8.2.1
  instance_types:
  - container
  - virtual-machine
  firewall: nftables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    idmapped_mounts: "true"
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    uevent_injection: "true"
    unpriv_fscaps: "true"
  kernel_version: 5.15.0-107-generic
  lxc_features:
    cgroup2: "true"
    core_scheduling: "true"
    devpts_fd: "true"
    idmapped_mounts_v2: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: Ubuntu
  os_version: "22.04"
  project: multipass
  server: lxd
  server_clustered: false
  server_event_mode: full-mesh
  server_name: chr
  server_pid: 1766
  server_version: 5.21.1
  server_lts: true
  storage: dir | zfs
  storage_version: 1 | 2.1.5-1ubuntu6~22.04.3
  storage_supported_drivers:
  - name: cephfs
    version: 17.2.7
    remote: true
  - name: cephobject
    version: 17.2.7
    remote: true
  - name: dir
    version: "1"
    remote: false
  - name: lvm
    version: 2.03.11(2) (2021-01-08) / 1.02.175 (2021-01-08) / 4.45.0
    remote: false
  - name: powerflex
    version: 1.16 (nvme-cli)
    remote: true
  - name: zfs
    version: 2.1.5-1ubuntu6~22.04.3
    remote: false
  - name: btrfs
    version: 5.16.2
    remote: false
  - name: ceph
    version: 17.2.7
    remote: true
chr@chr:~$

lxc config show / lxc config show --project=multipass

result:

chr@chr:~$ sudo lxc config show
config: {}
chr@chr:~$ sudo lxc config show --project=multipass
config: {}
chr@chr:~$
simondeziel commented 2 weeks ago

I compared the 2 logs and nothing obvious jumped at me. If you have simple reproducing steps, please provide them in here. Please note that I've never used multipass so don't skip anything even if it seems obvious to you, thanks :)

manztihagi commented 1 week ago

Hi @simondeziel, I just run the command:

multipass list on the shell, then showing the message as per subjected above. and I with @ricab already long discussion until open the case in here.

ricab commented 1 week ago

Hi @simondeziel, I suggested that @manztihagi come to you because, for some reason, LXD does not return a lease with list-leases in this case, even though list does. Multipass uses the corresponding API to obtain the lease, which is what is failing. Given that docker is/was involved, and since it is known to interfere, I suspected it could be related.

Copying here a screenshot from this comment:

image

simondeziel commented 1 week ago

Indeed Docker is known to prevent IP allocation. However the lxc list --project=multipass output shows the VM has an IPv4, maybe it's statically assigned?

@manztihagi please report what you get from sudo lxc exec --project=multipass chr-uat -- sh -c 'ip address; ip route; ping -c3 1.1.1.1'.

manztihagi commented 1 week ago

@simondeziel, here we are:

chr@chr:~$ sudo lxc exec --project=multipass chr-uat -- sh -c 'ip address; ip route; ping -c3 1.1.1.1'
[sudo] password for chr:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 52:54:00:11:13:bf brd ff:ff:ff:ff:ff:ff
    inet 10.67.90.39/24 brd 10.67.90.255 scope global enp5s0
       valid_lft forever preferred_lft forever
    inet6 fd42:d158:353c:6900:5054:ff:fe11:13bf/64 scope global mngtmpaddr noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe11:13bf/64 scope link
       valid_lft forever preferred_lft forever
default via 10.67.90.1 dev enp5s0 proto static
10.67.90.0/24 dev enp5s0 proto kernel scope link src 10.67.90.39
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=48 time=36.0 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=48 time=35.7 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=48 time=32.7 ms

--- 1.1.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 32.696/34.792/35.957/1.485 ms
chr@chr:~$
simondeziel commented 6 days ago

@manztihagi looks like this chr-uat instance is configured to use a static IP, not one assigned via DHCP. As such, it's expected that there won't be any IPv4 lease showing in list-leases. As such, I'll close this issue as not a bug but if that's a wrong analysis, please comment and/or re-open and we'll take another look. Thanks

manztihagi commented 6 days ago

@simondeziel the chr-uat wasn't use the static IP, the chr-uat IP was assigned at multipass that using DHCP.

this result from lxc --project=multipass network list-leases mpbr0 and result of IP Address type is Dynamic.

chr@chr:~$ sudo lxc --project=multipass network list-leases mpbr0
+------------+-------------------+---------------------------------------+---------+
|  HOSTNAME  |    MAC ADDRESS    |              IP ADDRESS               |  TYPE   |
+------------+-------------------+---------------------------------------+---------+
| chr-uat    | 52:54:00:11:13:bf | fd42:d158:353c:6900:5054:ff:fe11:13bf | DYNAMIC |
+------------+-------------------+---------------------------------------+---------+
| chr-uat2   | 52:54:00:c8:1c:57 | 10.67.90.103                          | DYNAMIC |
+------------+-------------------+---------------------------------------+---------+
| chr-uat2   | 52:54:00:c8:1c:57 | fd42:d158:353c:6900:5054:ff:fec8:1c57 | DYNAMIC |
+------------+-------------------+---------------------------------------+---------+
| chr-uatweb | 52:54:00:fc:87:37 | fd42:d158:353c:6900:5054:ff:fefc:8737 | DYNAMIC |
+------------+-------------------+---------------------------------------+---------+
chr@chr:~$
simondeziel commented 6 days ago

The above shows that chr-uat as a no lease and the ip address and ip route output indicate the IP and associated routes were not obtained via DHCP.

Please provide the network configuration from inside the instance chr-uat please. This would be the netplan or ifupdown or systemd-networkd or NetworkManager config. This way, we can verify it.

manztihagi commented 20 hours ago

@simondeziel ..

Look's like you're analyze was correct, the chr-uat was configured by static IP not the dhcp.

I'm replying on https://github.com/canonical/multipass/issues/3528#issuecomment-2232268182