monstermunchkin
commented
1 year ago I just gave it a try, and it worked as expected.
@diabolusss what's the output of lxc version?
Closed diabolusss closed 1 year ago
monstermunchkin
commented
1 year ago I just gave it a try, and it worked as expected.
@diabolusss what's the output of lxc version?
diabolusss
commented
1 year ago I just gave it a try, and it worked as expected.
@diabolusss what's the output of
lxc version?
@monstermunchkin i suppose there was a client side cache issue. After client restart, the command worked without problems.
(server)$ ssh remotevm snap info lxd
installed: git-9a64f64 (25118) 181MB -
$ lxc version remotevm:
Client version: 5.15
Server version: 5.15
(client) $ snap info lxd
installed: git-0f3b413 (25269) 183MB -
$ lxc version
Client version: 5.15
Server version: 5.15Could you, please, clarify about "Backups" listing in lxc storage volume info pool_name volume_name?
I can't find any commands to get information about them or manually remove obsolete ones.
I see that they have an expiration time, but expired ones are not removed.
Backups:
+---------+-----------------------+-----------------------+-------------+-------------------+
| NAME | TAKEN AT | EXPIRES AT | VOLUME ONLY | OPTIMIZED STORAGE |
+---------+-----------------------+-----------------------+-------------+-------------------+
| backup0 | 2023/07/14 09:44 EEST | 2023/07/15 09:44 EEST | NO | YES |
+---------+-----------------------+-----------------------+-------------+-------------------+
| backup1 | 2023/07/14 17:44 EEST | 2023/07/15 17:44 EEST | NO | YES |
+---------+-----------------------+-----------------------+-------------+-------------------+
| backup2 | 2023/07/14 20:11 EEST | 2023/07/15 20:11 EEST | YES | NO |
+---------+-----------------------+-----------------------+-------------+-------------------+
| backup3 | 2023/07/14 23:02 EEST | 2023/07/15 23:02 EEST | YES | NO |
+---------+-----------------------+-----------------------+-------------+-------------------+
| backup4 | 2023/07/14 23:21 EEST | 2023/07/15 23:21 EEST | YES | YES |
+---------+-----------------------+-----------------------+-------------+-------------------+
| backup5 | 2023/07/17 14:03 EEST | 2023/07/18 14:03 EEST | YES | YES |
+---------+-----------------------+-----------------------+-------------+-------------------+
Could you, please, clarify about "Backups" listing in lxc storage volume info pool_name volume_name?
Backups aren't listed.
I see that they have an expiration time, but expired ones are not removed.
You're right. LXD is missing a task which removes expired storage volume backups. It only removes expired instance backups. So, that's a bug.
tomponline
commented
1 year ago @monstermunchkin good spot, can you open a separate issue for that please?
tomponline
commented
1 year ago Can this be closed now?
monstermunchkin
commented
1 year ago I'm closing this as the export works as expected. Howver, I've created https://github.com/canonical/lxd/issues/12028 to track the removal of expired storage volume backups.
@diabolusss thanks for pointing out that bug :+1:
diabolusss
commented
1 year ago Backups aren't listed.
@monstermunchkin You are welcome, however, what do you mean by that? I clearly see a list of them or it's just a log?
$ lxc storage volume info btrfs_pool01 sm
Name: sm
Type: custom
Content type: filesystem
Usage: 292.00KiB
Created: 2023/07/08 20:43 EEST
Snapshots:
+-------------------+-------------+-----------------------+
| NAME | DESCRIPTION | EXPIRES AT |
+-------------------+-------------+-----------------------+
| auto--20230715-14 | | 2023/07/17 15:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230715-15 | | 2023/07/17 16:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230715-16 | | 2023/07/17 17:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230715-17 | | 2023/07/17 18:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230715-18 | | 2023/07/17 19:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230715-19 | | 2023/07/17 20:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230715-20 | | 2023/07/17 21:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230715-21 | | 2023/07/17 22:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230715-22 | | 2023/07/17 23:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230715-23 | | 2023/07/18 00:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-0 | | 2023/07/18 01:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-1 | | 2023/07/18 02:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-2 | | 2023/07/18 03:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-3 | | 2023/07/18 04:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-4 | | 2023/07/18 05:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-5 | | 2023/07/18 06:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-6 | | 2023/07/18 07:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-7 | | 2023/07/18 08:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-8 | | 2023/07/18 09:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-9 | | 2023/07/18 10:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-10 | | 2023/07/18 11:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-11 | | 2023/07/18 12:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-12 | | 2023/07/18 13:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-13 | | 2023/07/18 14:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-14 | | 2023/07/18 15:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-15 | | 2023/07/18 16:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-16 | | 2023/07/18 17:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-17 | | 2023/07/18 18:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-18 | | 2023/07/18 19:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-19 | | 2023/07/18 20:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-20 | | 2023/07/18 21:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-21 | | 2023/07/18 22:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-22 | | 2023/07/18 23:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230716-23 | | 2023/07/19 00:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230717-0 | | 2023/07/19 01:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230717-1 | | 2023/07/19 02:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230717-2 | | 2023/07/19 03:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230717-3 | | 2023/07/19 04:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230717-4 | | 2023/07/19 05:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230717-5 | | 2023/07/19 06:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230717-6 | | 2023/07/19 07:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230717-7 | | 2023/07/19 08:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230717-8 | | 2023/07/19 09:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230717-9 | | 2023/07/19 10:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230717-10 | | 2023/07/19 11:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230717-11 | | 2023/07/19 12:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230717-12 | | 2023/07/19 13:04 EEST |
+-------------------+-------------+-----------------------+
| auto--20230717-13 | | 2023/07/19 14:04 EEST |
+-------------------+-------------+-----------------------+
Backups:
+---------+-----------------------+-----------------------+-------------+-------------------+
| NAME | TAKEN AT | EXPIRES AT | VOLUME ONLY | OPTIMIZED STORAGE |
+---------+-----------------------+-----------------------+-------------+-------------------+
| backup0 | 2023/07/14 09:44 EEST | 2023/07/15 09:44 EEST | NO | YES |
+---------+-----------------------+-----------------------+-------------+-------------------+
| backup1 | 2023/07/14 17:44 EEST | 2023/07/15 17:44 EEST | NO | YES |
+---------+-----------------------+-----------------------+-------------+-------------------+
| backup2 | 2023/07/14 20:11 EEST | 2023/07/15 20:11 EEST | YES | NO |
+---------+-----------------------+-----------------------+-------------+-------------------+
| backup3 | 2023/07/14 23:02 EEST | 2023/07/15 23:02 EEST | YES | NO |
+---------+-----------------------+-----------------------+-------------+-------------------+
| backup4 | 2023/07/14 23:21 EEST | 2023/07/15 23:21 EEST | YES | YES |
+---------+-----------------------+-----------------------+-------------+-------------------+
| backup5 | 2023/07/17 14:03 EEST | 2023/07/18 14:03 EEST | YES | YES |
+---------+-----------------------+-----------------------+-------------+-------------------+
however, what do you mean by that? I clearly see a list of them or it's just a log?
You're right, I misread your sentence. I thought you were talking about lxc storage volume ls <pool> which doesn't list backups.
What exactly do you want to have clarified in lxc storage volume info?
diabolusss
commented
1 year ago You're right, I misread your sentence. I thought you were talking about
lxc storage volume ls <pool>which doesn't list backups.What exactly do you want to have clarified in
lxc storage volume info?
@monstermunchkin I don't quite understand the purpose of this list, because:
Required information
lxc info
driver: lxc driver_version: 5.0.0 firewall: nftables kernel: Linux kernel_architecture: x86_64 kernel_version: 5.4.0-144-generic
os_name: Ubuntu os_version: "20.04" project: default server: lxd server_clustered: false server_event_mode: full-mesh server_name: x server_version: "5.15" storage: btrfs | lvm storage_version: 5.16.2 | 2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.41.0 storage_supported_drivers:
Issue description
lxc storage volume exportfor btrfs pool volume fails when trying to export backup file from remote client. And failed attempt doesn't clean up consumed space.lxc image exportin the same situation works without issues.Steps to reproduce
lxc storage volume export remotevm:btrfs_pool01 sm -v --volume-only --debug --optimized-storage &> volume_export_remotely.loglxc storage volume export btrfs_pool01 sm -v --volume-only --debug --optimized-storage &> volume_export_on_server.logInformation to attach
Fail: lxc storage volume export remotevm:btrfs_pool01 sm -v --volume-only --debug --optimized-storage &> volume_export_remotely.log
time="2023-07-14T23:21:48+03:00" level=debug msg="Connecting to a remote LXD over HTTPS" url="https://10.160.195.143:8443" time="2023-07-14T23:21:48+03:00" level=debug msg="Sending request to LXD" etag= method=GET url="https://10.160.195.143:8443/1.0" time="2023-07-14T23:21:48+03:00" level=debug msg="Got response struct from LXD" time="2023-07-14T23:21:48+03:00" level=debug msg="\n\t{\n\t\t\"config\": {\n\t\t\t\"core.https_address\": \"192.168.10.225\",\n\t\t\t\"core.trust_password\": true,\n\t\t\t\"images.auto_update_interval\": \"0\"\n\t\t},\n\t\t\"api_extensions\": [\n\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\"container_stop_priority\",\n\t\t\t\"container_syscall_filtering\",\n\t\t\t\"auth_pki\",\n\t\t\t\"container_last_used_at\",\n\t\t\t\"etag\",\n\t\t\t\"patch\",\n\t\t\t\"usb_devices\",\n\t\t\t\"https_allowed_credentials\",\n\t\t\t\"image_compression_algorithm\",\n\t\t\t\"directory_manipulation\",\n\t\t\t\"container_cpu_time\",\n\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\"network\",\n\t\t\t\"profile_usedby\",\n\t\t\t\"container_push\",\n\t\t\t\"container_exec_recording\",\n\t\t\t\"certificate_update\",\n\t\t\t\"container_exec_signal_handling\",\n\t\t\t\"gpu_devices\",\n\t\t\t\"container_image_properties\",\n\t\t\t\"migration_progress\",\n\t\t\t\"id_map\",\n\t\t\t\"network_firewall_filtering\",\n\t\t\t\"network_routes\",\n\t\t\t\"storage\",\n\t\t\t\"file_delete\",\n\t\t\t\"file_append\",\n\t\t\t\"network_dhcp_expiry\",\n\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\"network_vlan\",\n\t\t\t\"image_create_aliases\",\n\t\t\t\"container_stateless_copy\",\n\t\t\t\"container_only_migration\",\n\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\"unix_device_rename\",\n\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\"network_vxlan_interface\",\n\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\"entity_description\",\n\t\t\t\"image_force_refresh\",\n\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\"id_map_base\",\n\t\t\t\"file_symlinks\",\n\t\t\t\"container_push_target\",\n\t\t\t\"network_vlan_physical\",\n\t\t\t\"storage_images_delete\",\n\t\t\t\"container_edit_metadata\",\n\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\"storage_driver_ceph\",\n\t\t\t\"storage_ceph_user_name\",\n\t\t\t\"resource_limits\",\n\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\"resources\",\n\t\t\t\"kernel_limits\",\n\t\t\t\"storage_api_volume_rename\",\n\t\t\t\"macaroon_authentication\",\n\t\t\t\"network_sriov\",\n\t\t\t\"console\",\n\t\t\t\"restrict_devlxd\",\n\t\t\t\"migration_pre_copy\",\n\t\t\t\"infiniband\",\n\t\t\t\"maas_network\",\n\t\t\t\"devlxd_events\",\n\t\t\t\"proxy\",\n\t\t\t\"network_dhcp_gateway\",\n\t\t\t\"file_get_symlink\",\n\t\t\t\"network_leases\",\n\t\t\t\"unix_device_hotplug\",\n\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\"operation_description\",\n\t\t\t\"clustering\",\n\t\t\t\"event_lifecycle\",\n\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\"nvidia_runtime\",\n\t\t\t\"container_mount_propagation\",\n\t\t\t\"container_backup\",\n\t\t\t\"devlxd_images\",\n\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\"proxy_unix\",\n\t\t\t\"proxy_udp\",\n\t\t\t\"clustering_join\",\n\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\"network_state\",\n\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\"container_protection_delete\",\n\t\t\t\"unix_priv_drop\",\n\t\t\t\"pprof_http\",\n\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\"network_hwaddr\",\n\t\t\t\"proxy_nat\",\n\t\t\t\"network_nat_order\",\n\t\t\t\"container_full\",\n\t\t\t\"candid_authentication\",\n\t\t\t\"backup_compression\",\n\t\t\t\"candid_config\",\n\t\t\t\"nvidia_runtime_config\",\n\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\"storage_unmapped\",\n\t\t\t\"projects\",\n\t\t\t\"candid_config_key\",\n\t\t\t\"network_vxlan_ttl\",\n\t\t\t\"container_incremental_copy\",\n\t\t\t\"usb_optional_vendorid\",\n\t\t\t\"snapshot_scheduling\",\n\t\t\t\"snapshot_schedule_aliases\",\n\t\t\t\"container_copy_project\",\n\t\t\t\"clustering_server_address\",\n\t\t\t\"clustering_image_replication\",\n\t\t\t\"container_protection_shift\",\n\t\t\t\"snapshot_expiry\",\n\t\t\t\"container_backup_override_pool\",\n\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\"network_leases_location\",\n\t\t\t\"resources_cpu_socket\",\n\t\t\t\"resources_gpu\",\n\t\t\t\"resources_numa\",\n\t\t\t\"kernel_features\",\n\t\t\t\"id_map_current\",\n\t\t\t\"event_location\",\n\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\"network_nat_address\",\n\t\t\t\"container_nic_routes\",\n\t\t\t\"rbac\",\n\t\t\t\"cluster_internal_copy\",\n\t\t\t\"seccomp_notify\",\n\t\t\t\"lxc_features\",\n\t\t\t\"container_nic_ipvlan\",\n\t\t\t\"network_vlan_sriov\",\n\t\t\t\"storage_cephfs\",\n\t\t\t\"container_nic_ipfilter\",\n\t\t\t\"resources_v2\",\n\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\"container_syscall_intercept\",\n\t\t\t\"container_disk_shift\",\n\t\t\t\"storage_shifted\",\n\t\t\t\"resources_infiniband\",\n\t\t\t\"daemon_storage\",\n\t\t\t\"instances\",\n\t\t\t\"image_types\",\n\t\t\t\"resources_disk_sata\",\n\t\t\t\"clustering_roles\",\n\t\t\t\"images_expiry\",\n\t\t\t\"resources_network_firmware\",\n\t\t\t\"backup_compression_algorithm\",\n\t\t\t\"ceph_data_pool_name\",\n\t\t\t\"container_syscall_intercept_mount\",\n\t\t\t\"compression_squashfs\",\n\t\t\t\"container_raw_mount\",\n\t\t\t\"container_nic_routed\",\n\t\t\t\"container_syscall_intercept_mount_fuse\",\n\t\t\t\"container_disk_ceph\",\n\t\t\t\"virtual-machines\",\n\t\t\t\"image_profiles\",\n\t\t\t\"clustering_architecture\",\n\t\t\t\"resources_disk_id\",\n\t\t\t\"storage_lvm_stripes\",\n\t\t\t\"vm_boot_priority\",\n\t\t\t\"unix_hotplug_devices\",\n\t\t\t\"api_filtering\",\n\t\t\t\"instance_nic_network\",\n\t\t\t\"clustering_sizing\",\n\t\t\t\"firewall_driver\",\n\t\t\t\"projects_limits\",\n\t\t\t\"container_syscall_intercept_hugetlbfs\",\n\t\t\t\"limits_hugepages\",\n\t\t\t\"container_nic_routed_gateway\",\n\t\t\t\"projects_restrictions\",\n\t\t\t\"custom_volume_snapshot_expiry\",\n\t\t\t\"volume_snapshot_scheduling\",\n\t\t\t\"trust_ca_certificates\",\n\t\t\t\"snapshot_disk_usage\",\n\t\t\t\"clustering_edit_roles\",\n\t\t\t\"container_nic_routed_host_address\",\n\t\t\t\"container_nic_ipvlan_gateway\",\n\t\t\t\"resources_usb_pci\",\n\t\t\t\"resources_cpu_threads_numa\",\n\t\t\t\"resources_cpu_core_die\",\n\t\t\t\"api_os\",\n\t\t\t\"container_nic_routed_host_table\",\n\t\t\t\"container_nic_ipvlan_host_table\",\n\t\t\t\"container_nic_ipvlan_mode\",\n\t\t\t\"resources_system\",\n\t\t\t\"images_push_relay\",\n\t\t\t\"network_dns_search\",\n\t\t\t\"container_nic_routed_limits\",\n\t\t\t\"instance_nic_bridged_vlan\",\n\t\t\t\"network_state_bond_bridge\",\n\t\t\t\"usedby_consistency\",\n\t\t\t\"custom_block_volumes\",\n\t\t\t\"clustering_failure_domains\",\n\t\t\t\"resources_gpu_mdev\",\n\t\t\t\"console_vga_type\",\n\t\t\t\"projects_limits_disk\",\n\t\t\t\"network_type_macvlan\",\n\t\t\t\"network_type_sriov\",\n\t\t\t\"container_syscall_intercept_bpf_devices\",\n\t\t\t\"network_type_ovn\",\n\t\t\t\"projects_networks\",\n\t\t\t\"projects_networks_restricted_uplinks\",\n\t\t\t\"custom_volume_backup\",\n\t\t\t\"backup_override_name\",\n\t\t\t\"storage_rsync_compression\",\n\t\t\t\"network_type_physical\",\n\t\t\t\"network_ovn_external_subnets\",\n\t\t\t\"network_ovn_nat\",\n\t\t\t\"network_ovn_external_routes_remove\",\n\t\t\t\"tpm_device_type\",\n\t\t\t\"storage_zfs_clone_copy_rebase\",\n\t\t\t\"gpu_mdev\",\n\t\t\t\"resources_pci_iommu\",\n\t\t\t\"resources_network_usb\",\n\t\t\t\"resources_disk_address\",\n\t\t\t\"network_physical_ovn_ingress_mode\",\n\t\t\t\"network_ovn_dhcp\",\n\t\t\t\"network_physical_routes_anycast\",\n\t\t\t\"projects_limits_instances\",\n\t\t\t\"network_state_vlan\",\n\t\t\t\"instance_nic_bridged_port_isolation\",\n\t\t\t\"instance_bulk_state_change\",\n\t\t\t\"network_gvrp\",\n\t\t\t\"instance_pool_move\",\n\t\t\t\"gpu_sriov\",\n\t\t\t\"pci_device_type\",\n\t\t\t\"storage_volume_state\",\n\t\t\t\"network_acl\",\n\t\t\t\"migration_stateful\",\n\t\t\t\"disk_state_quota\",\n\t\t\t\"storage_ceph_features\",\n\t\t\t\"projects_compression\",\n\t\t\t\"projects_images_remote_cache_expiry\",\n\t\t\t\"certificate_project\",\n\t\t\t\"network_ovn_acl\",\n\t\t\t\"projects_images_auto_update\",\n\t\t\t\"projects_restricted_cluster_target\",\n\t\t\t\"images_default_architecture\",\n\t\t\t\"network_ovn_acl_defaults\",\n\t\t\t\"gpu_mig\",\n\t\t\t\"project_usage\",\n\t\t\t\"network_bridge_acl\",\n\t\t\t\"warnings\",\n\t\t\t\"projects_restricted_backups_and_snapshots\",\n\t\t\t\"clustering_join_token\",\n\t\t\t\"clustering_description\",\n\t\t\t\"server_trusted_proxy\",\n\t\t\t\"clustering_update_cert\",\n\t\t\t\"storage_api_project\",\n\t\t\t\"server_instance_driver_operational\",\n\t\t\t\"server_supported_storage_drivers\",\n\t\t\t\"event_lifecycle_requestor_address\",\n\t\t\t\"resources_gpu_usb\",\n\t\t\t\"clustering_evacuation\",\n\t\t\t\"network_ovn_nat_address\",\n\t\t\t\"network_bgp\",\n\t\t\t\"network_forward\",\n\t\t\t\"custom_volume_refresh\",\n\t\t\t\"network_counters_errors_dropped\",\n\t\t\t\"metrics\",\n\t\t\t\"image_source_project\",\n\t\t\t\"clustering_config\",\n\t\t\t\"network_peer\",\n\t\t\t\"linux_sysctl\",\n\t\t\t\"network_dns\",\n\t\t\t\"ovn_nic_acceleration\",\n\t\t\t\"certificate_self_renewal\",\n\t\t\t\"instance_project_move\",\n\t\t\t\"storage_volume_project_move\",\n\t\t\t\"cloud_init\",\n\t\t\t\"network_dns_nat\",\n\t\t\t\"database_leader\",\n\t\t\t\"instance_all_projects\",\n\t\t\t\"clustering_groups\",\n\t\t\t\"ceph_rbd_du\",\n\t\t\t\"instance_get_full\",\n\t\t\t\"qemu_metrics\",\n\t\t\t\"gpu_mig_uuid\",\n\t\t\t\"event_project\",\n\t\t\t\"clustering_evacuation_live\",\n\t\t\t\"instance_allow_inconsistent_copy\",\n\t\t\t\"network_state_ovn\",\n\t\t\t\"storage_volume_api_filtering\",\n\t\t\t\"image_restrictions\",\n\t\t\t\"storage_zfs_export\",\n\t\t\t\"network_dns_records\",\n\t\t\t\"storage_zfs_reserve_space\",\n\t\t\t\"network_acl_log\",\n\t\t\t\"storage_zfs_blocksize\",\n\t\t\t\"metrics_cpu_seconds\",\n\t\t\t\"instance_snapshot_never\",\n\t\t\t\"certificate_token\",\n\t\t\t\"instance_nic_routed_neighbor_probe\",\n\t\t\t\"event_hub\",\n\t\t\t\"agent_nic_config\",\n\t\t\t\"projects_restricted_intercept\",\n\t\t\t\"metrics_authentication\",\n\t\t\t\"images_target_project\",\n\t\t\t\"cluster_migration_inconsistent_copy\",\n\t\t\t\"cluster_ovn_chassis\",\n\t\t\t\"container_syscall_intercept_sched_setscheduler\",\n\t\t\t\"storage_lvm_thinpool_metadata_size\",\n\t\t\t\"storage_volume_state_total\",\n\t\t\t\"instance_file_head\",\n\t\t\t\"instances_nic_host_name\",\n\t\t\t\"image_copy_profile\",\n\t\t\t\"container_syscall_intercept_sysinfo\",\n\t\t\t\"clustering_evacuation_mode\",\n\t\t\t\"resources_pci_vpd\",\n\t\t\t\"qemu_raw_conf\",\n\t\t\t\"storage_cephfs_fscache\",\n\t\t\t\"network_load_balancer\",\n\t\t\t\"vsock_api\",\n\t\t\t\"instance_ready_state\",\n\t\t\t\"network_bgp_holdtime\",\n\t\t\t\"storage_volumes_all_projects\",\n\t\t\t\"metrics_memory_oom_total\",\n\t\t\t\"storage_buckets\",\n\t\t\t\"storage_buckets_create_credentials\",\n\t\t\t\"metrics_cpu_effective_total\",\n\t\t\t\"projects_networks_restricted_access\",\n\t\t\t\"storage_buckets_local\",\n\t\t\t\"loki\",\n\t\t\t\"acme\",\n\t\t\t\"internal_metrics\",\n\t\t\t\"cluster_join_token_expiry\",\n\t\t\t\"remote_token_expiry\",\n\t\t\t\"init_preseed\",\n\t\t\t\"storage_volumes_created_at\",\n\t\t\t\"cpu_hotplug\",\n\t\t\t\"projects_networks_zones\",\n\t\t\t\"network_txqueuelen\",\n\t\t\t\"cluster_member_state\",\n\t\t\t\"instances_placement_scriptlet\",\n\t\t\t\"storage_pool_source_wipe\",\n\t\t\t\"zfs_block_mode\",\n\t\t\t\"instance_generation_id\",\n\t\t\t\"disk_io_cache\",\n\t\t\t\"amd_sev\",\n\t\t\t\"storage_pool_loop_resize\",\n\t\t\t\"migration_vm_live\",\n\t\t\t\"ovn_nic_nesting\",\n\t\t\t\"oidc\",\n\t\t\t\"network_ovn_l3only\",\n\t\t\t\"ovn_nic_acceleration_vdpa\",\n\t\t\t\"cluster_healing\",\n\t\t\t\"instances_state_total\",\n\t\t\t\"auth_user\",\n\t\t\t\"security_csm\",\n\t\t\t\"instances_rebuild\",\n\t\t\t\"numa_cpu_placement\"\n\t\t],\n\t\t\"api_status\": \"stable\",\n\t\t\"api_version\": \"1.0\",\n\t\t\"auth\": \"trusted\",\n\t\t\"public\": false,\n\t\t\"auth_methods\": [\n\t\t\t\"tls\"\n\t\t],\n\t\t\"environment\": {\n\t\t\t\"addresses\": [\n\t\t\t\t\"192.168.10.225:8443\"\n\t\t\t],\n\t\t\t\"architectures\": [\n\t\t\t\t\"x86_64\",\n\t\t\t\t\"i686\"\n\t\t\t],\n\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nMIICBzCCAY2gAwIBAgIRAKIrYljXGEzphSCmSSTu0jYwCgYIKoZIzj0EAwMwNTEc\\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEVMBMGA1UEAwwMcm9vdEB2aG9k\\naWtvMB4XDTIwMDQyNzIwMDM0M1oXDTMwMDQyNTIwMDM0M1owNTEcMBoGA1UEChMT\\nbGludXhjb250YWluZXJzLm9yZzEVMBMGA1UEAwwMcm9vdEB2aG9kaWtvMHYwEAYH\\nKoZIzj0CAQYFK4EEACIDYgAEgN5kNCMEL43oV+D845W1ofk1wnHmMQTUqkJvYr5s\\npmjW+51BSeDWllRQEqeDfFeHo629cvidRYQPE1llxpbXf8xLUbvpS5MAHMnsh2GI\\nuvypasOY/G9iof4eQXFl+t16o2EwXzAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\\nCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAqBgNVHREEIzAhggd2aG9kaWtvhwR/\\nAAABhxAAAAAAAAAAAAAAAAAAAAABMAoGCCqGSM49BAMDA2gAMGUCMCf+AvToMrRX\\nDkKgLkA8uGfyI37McTHJfcuZiNKHM5dKqp8E4YE9uXFbijCETGGmtQIxAMBQGEn7\\nK4ZiF1arE4NVYnjGbJEY2UZvGWBRJ1qhn6cXsgxyJFGeirNv6D5h4Az4qQ==\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\"certificate_fingerprint\": \"c0abcbe6f8d27d7551eb3bf3e3385e73ec24068dcaea977036bd84db204ffb5d\",\n\t\t\t\"driver\": \"lxc\",\n\t\t\t\"driver_version\": \"5.0.0\",\n\t\t\t\"firewall\": \"nftables\",\n\t\t\t\"kernel\": \"Linux\",\n\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\"kernel_features\": {\n\t\t\t\t\"idmapped_mounts\": \"false\",\n\t\t\t\t\"netnsid_getifaddrs\": \"true\",\n\t\t\t\t\"seccomp_listener\": \"true\",\n\t\t\t\t\"seccomp_listener_continue\": \"true\",\n\t\t\t\t\"shiftfs\": \"false\",\n\t\t\t\t\"uevent_injection\": \"true\",\n\t\t\t\t\"unpriv_fscaps\": \"true\"\n\t\t\t},\n\t\t\t\"kernel_version\": \"5.4.0-144-generic\",\n\t\t\t\"lxc_features\": {\n\t\t\t\t\"cgroup2\": \"true\",\n\t\t\t\t\"core_scheduling\": \"true\",\n\t\t\t\t\"devpts_fd\": \"true\",\n\t\t\t\t\"idmapped_mounts_v2\": \"true\",\n\t\t\t\t\"mount_injection_file\": \"true\",\n\t\t\t\t\"network_gateway_device_route\": \"true\",\n\t\t\t\t\"network_ipvlan\": \"true\",\n\t\t\t\t\"network_l2proxy\": \"true\",\n\t\t\t\t\"network_phys_macvlan_mtu\": \"true\",\n\t\t\t\t\"network_veth_router\": \"true\",\n\t\t\t\t\"pidfd\": \"true\",\n\t\t\t\t\"seccomp_allow_deny_syntax\": \"true\",\n\t\t\t\t\"seccomp_notify\": \"true\",\n\t\t\t\t\"seccomp_proxy_send_notify_fd\": \"true\"\n\t\t\t},\n\t\t\t\"os_name\": \"Ubuntu\",\n\t\t\t\"os_version\": \"20.04\",\n\t\t\t\"project\": \"default\",\n\t\t\t\"server\": \"lxd\",\n\t\t\t\"server_clustered\": false,\n\t\t\t\"server_event_mode\": \"full-mesh\",\n\t\t\t\"server_name\": \"vhodiko\",\n\t\t\t\"server_pid\": 112798,\n\t\t\t\"server_version\": \"5.15\",\n\t\t\t\"storage\": \"lvm | btrfs\",\n\t\t\t\"storage_version\": \"2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.41.0 | 5.16.2\",\n\t\t\t\"storage_supported_drivers\": [\n\t\t\t\t{\n\t\t\t\t\t\"Name\": \"zfs\",\n\t\t\t\t\t\"Version\": \"0.8.3-1ubuntu12.14\",\n\t\t\t\t\t\"Remote\": false\n\t\t\t\t},\n\t\t\t\t{\n\t\t\t\t\t\"Name\": \"btrfs\",\n\t\t\t\t\t\"Version\": \"5.16.2\",\n\t\t\t\t\t\"Remote\": false\n\t\t\t\t},\n\t\t\t\t{\n\t\t\t\t\t\"Name\": \"ceph\",\n\t\t\t\t\t\"Version\": \"17.2.5\",\n\t\t\t\t\t\"Remote\": true\n\t\t\t\t},\n\t\t\t\t{\n\t\t\t\t\t\"Name\": \"cephfs\",\n\t\t\t\t\t\"Version\": \"17.2.5\",\n\t\t\t\t\t\"Remote\": true\n\t\t\t\t},\n\t\t\t\t{\n\t\t\t\t\t\"Name\": \"cephobject\",\n\t\t\t\t\t\"Version\": \"17.2.5\",\n\t\t\t\t\t\"Remote\": true\n\t\t\t\t},\n\t\t\t\t{\n\t\t\t\t\t\"Name\": \"dir\",\n\t\t\t\t\t\"Version\": \"1\",\n\t\t\t\t\t\"Remote\": false\n\t\t\t\t},\n\t\t\t\t{\n\t\t\t\t\t\"Name\": \"lvm\",\n\t\t\t\t\t\"Version\": \"2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.41.0\",\n\t\t\t\t\t\"Remote\": false\n\t\t\t\t}\n\t\t\t]\n\t\t}\n\t}" time="2023-07-14T23:21:49+03:00" level=debug msg="Connected to the websocket: wss://10.160.195.143:8443/1.0/events" time="2023-07-14T23:21:49+03:00" level=debug msg="Sending request to LXD" etag= method=POST url="https://10.160.195.143:8443/1.0/storage-pools/btrfs_pool01/volumes/custom/sm/backups" time="2023-07-14T23:21:49+03:00" level=debug msg="\n\t{\n\t\t\"name\": \"\",\n\t\t\"expires_at\": \"2023-07-15T23:21:48.84894603+03:00\",\n\t\t\"volume_only\": true,\n\t\t\"optimized_storage\": true,\n\t\t\"compression_algorithm\": \"\"\n\t}" time="2023-07-14T23:21:49+03:00" level=debug msg="Got operation from LXD" time="2023-07-14T23:21:49+03:00" level=debug msg="\n\t{\n\t\t\"id\": \"639b89b9-4305-4cf5-9793-fc6bd2c25d82\",\n\t\t\"class\": \"task\",\n\t\t\"description\": \"Creating custom volume backup\",\n\t\t\"created_at\": \"2023-07-14T22:21:49.527582798+02:00\",\n\t\t\"updated_at\": \"2023-07-14T22:21:49.527582798+02:00\",\n\t\t\"status\": \"Running\",\n\t\t\"status_code\": 103,\n\t\t\"resources\": {\n\t\t\t\"backups\": [\n\t\t\t\t\"/1.0/storage-pools/btrfs_pool01/volumes/custom/sm/backups/backup4\"\n\t\t\t],\n\t\t\t\"storage_volumes\": [\n\t\t\t\t\"/1.0/storage-pools/btrfs_pool01/volumes/custom/sm\"\n\t\t\t]\n\t\t},\n\t\t\"metadata\": null,\n\t\t\"may_cancel\": false,\n\t\t\"err\": \"\",\n\t\t\"location\": \"none\"\n\t}" time="2023-07-14T23:21:49+03:00" level=debug msg="Sending request to LXD" etag= method=GET url="https://10.160.195.143:8443/1.0/operations/639b89b9-4305-4cf5-9793-fc6bd2c25d82" time="2023-07-14T23:21:50+03:00" level=debug msg="Got response struct from LXD" time="2023-07-14T23:21:50+03:00" level=debug msg="\n\t{\n\t\t\"id\": \"639b89b9-4305-4cf5-9793-fc6bd2c25d82\",\n\t\t\"class\": \"task\",\n\t\t\"description\": \"Creating custom volume backup\",\n\t\t\"created_at\": \"2023-07-14T22:21:49.527582798+02:00\",\n\t\t\"updated_at\": \"2023-07-14T22:21:49.527582798+02:00\",\n\t\t\"status\": \"Running\",\n\t\t\"status_code\": 103,\n\t\t\"resources\": {\n\t\t\t\"backups\": [\n\t\t\t\t\"/1.0/storage-pools/btrfs_pool01/volumes/custom/sm/backups/backup4\"\n\t\t\t],\n\t\t\t\"storage_volumes\": [\n\t\t\t\t\"/1.0/storage-pools/btrfs_pool01/volumes/custom/sm\"\n\t\t\t]\n\t\t},\n\t\t\"metadata\": null,\n\t\t\"may_cancel\": false,\n\t\t\"err\": \"\",\n\t\t\"location\": \"none\"\n\t}" time="2023-07-14T23:27:14+03:00" level=debug msg="Sending request to LXD" etag= method=DELETE url="https://10.160.195.143:8443/1.0/storage-pools/btrfs_pool01/volumes/custom/sm/backups/%2F1.0%2Fstorage-pools%2Fbtrfs_pool01%2Fvolumes%2Fcustom%2Fsm%2Fbackups%2Fbackup4" Error: Failed to fetch storage volume backup file: Storage volume backup not foundFree space on lxd server after this attempt (available was ~17.5GB):
Non removed temp backup:
It seems that lxc tries to find that folder locally on client instead of server.
SUCCESS: lxc storage volume export btrfs_pool01 sm -v --volume-only --debug --optimized-storage &> volume_export_on_server.log
time="2023-07-14T22:45:45+02:00" level=debug msg="Connecting to a local LXD over a Unix socket" time="2023-07-14T22:45:45+02:00" level=debug msg="Sending request to LXD" etag= method=GET url="http://unix.socket/1.0" time="2023-07-14T22:45:45+02:00" level=debug msg="Got response struct from LXD" time="2023-07-14T22:45:45+02:00" level=debug msg="\n\t{\n\t\t\"config\": {\n\t\t\t\"core.https_address\": \"192.168.10.225\",\n\t\t\t\"core.trust_password\": true,\n\t\t\t\"images.auto_update_interval\": \"0\"\n\t\t},\n\t\t\"api_extensions\": [\n\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\"container_stop_priority\",\n\t\t\t\"container_syscall_filtering\",\n\t\t\t\"auth_pki\",\n\t\t\t\"container_last_used_at\",\n\t\t\t\"etag\",\n\t\t\t\"patch\",\n\t\t\t\"usb_devices\",\n\t\t\t\"https_allowed_credentials\",\n\t\t\t\"image_compression_algorithm\",\n\t\t\t\"directory_manipulation\",\n\t\t\t\"container_cpu_time\",\n\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\"network\",\n\t\t\t\"profile_usedby\",\n\t\t\t\"container_push\",\n\t\t\t\"container_exec_recording\",\n\t\t\t\"certificate_update\",\n\t\t\t\"container_exec_signal_handling\",\n\t\t\t\"gpu_devices\",\n\t\t\t\"container_image_properties\",\n\t\t\t\"migration_progress\",\n\t\t\t\"id_map\",\n\t\t\t\"network_firewall_filtering\",\n\t\t\t\"network_routes\",\n\t\t\t\"storage\",\n\t\t\t\"file_delete\",\n\t\t\t\"file_append\",\n\t\t\t\"network_dhcp_expiry\",\n\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\"network_vlan\",\n\t\t\t\"image_create_aliases\",\n\t\t\t\"container_stateless_copy\",\n\t\t\t\"container_only_migration\",\n\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\"unix_device_rename\",\n\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\"network_vxlan_interface\",\n\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\"entity_description\",\n\t\t\t\"image_force_refresh\",\n\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\"id_map_base\",\n\t\t\t\"file_symlinks\",\n\t\t\t\"container_push_target\",\n\t\t\t\"network_vlan_physical\",\n\t\t\t\"storage_images_delete\",\n\t\t\t\"container_edit_metadata\",\n\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\"storage_driver_ceph\",\n\t\t\t\"storage_ceph_user_name\",\n\t\t\t\"resource_limits\",\n\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\"resources\",\n\t\t\t\"kernel_limits\",\n\t\t\t\"storage_api_volume_rename\",\n\t\t\t\"macaroon_authentication\",\n\t\t\t\"network_sriov\",\n\t\t\t\"console\",\n\t\t\t\"restrict_devlxd\",\n\t\t\t\"migration_pre_copy\",\n\t\t\t\"infiniband\",\n\t\t\t\"maas_network\",\n\t\t\t\"devlxd_events\",\n\t\t\t\"proxy\",\n\t\t\t\"network_dhcp_gateway\",\n\t\t\t\"file_get_symlink\",\n\t\t\t\"network_leases\",\n\t\t\t\"unix_device_hotplug\",\n\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\"operation_description\",\n\t\t\t\"clustering\",\n\t\t\t\"event_lifecycle\",\n\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\"nvidia_runtime\",\n\t\t\t\"container_mount_propagation\",\n\t\t\t\"container_backup\",\n\t\t\t\"devlxd_images\",\n\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\"proxy_unix\",\n\t\t\t\"proxy_udp\",\n\t\t\t\"clustering_join\",\n\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\"network_state\",\n\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\"container_protection_delete\",\n\t\t\t\"unix_priv_drop\",\n\t\t\t\"pprof_http\",\n\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\"network_hwaddr\",\n\t\t\t\"proxy_nat\",\n\t\t\t\"network_nat_order\",\n\t\t\t\"container_full\",\n\t\t\t\"candid_authentication\",\n\t\t\t\"backup_compression\",\n\t\t\t\"candid_config\",\n\t\t\t\"nvidia_runtime_config\",\n\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\"storage_unmapped\",\n\t\t\t\"projects\",\n\t\t\t\"candid_config_key\",\n\t\t\t\"network_vxlan_ttl\",\n\t\t\t\"container_incremental_copy\",\n\t\t\t\"usb_optional_vendorid\",\n\t\t\t\"snapshot_scheduling\",\n\t\t\t\"snapshot_schedule_aliases\",\n\t\t\t\"container_copy_project\",\n\t\t\t\"clustering_server_address\",\n\t\t\t\"clustering_image_replication\",\n\t\t\t\"container_protection_shift\",\n\t\t\t\"snapshot_expiry\",\n\t\t\t\"container_backup_override_pool\",\n\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\"network_leases_location\",\n\t\t\t\"resources_cpu_socket\",\n\t\t\t\"resources_gpu\",\n\t\t\t\"resources_numa\",\n\t\t\t\"kernel_features\",\n\t\t\t\"id_map_current\",\n\t\t\t\"event_location\",\n\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\"network_nat_address\",\n\t\t\t\"container_nic_routes\",\n\t\t\t\"rbac\",\n\t\t\t\"cluster_internal_copy\",\n\t\t\t\"seccomp_notify\",\n\t\t\t\"lxc_features\",\n\t\t\t\"container_nic_ipvlan\",\n\t\t\t\"network_vlan_sriov\",\n\t\t\t\"storage_cephfs\",\n\t\t\t\"container_nic_ipfilter\",\n\t\t\t\"resources_v2\",\n\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\"container_syscall_intercept\",\n\t\t\t\"container_disk_shift\",\n\t\t\t\"storage_shifted\",\n\t\t\t\"resources_infiniband\",\n\t\t\t\"daemon_storage\",\n\t\t\t\"instances\",\n\t\t\t\"image_types\",\n\t\t\t\"resources_disk_sata\",\n\t\t\t\"clustering_roles\",\n\t\t\t\"images_expiry\",\n\t\t\t\"resources_network_firmware\",\n\t\t\t\"backup_compression_algorithm\",\n\t\t\t\"ceph_data_pool_name\",\n\t\t\t\"container_syscall_intercept_mount\",\n\t\t\t\"compression_squashfs\",\n\t\t\t\"container_raw_mount\",\n\t\t\t\"container_nic_routed\",\n\t\t\t\"container_syscall_intercept_mount_fuse\",\n\t\t\t\"container_disk_ceph\",\n\t\t\t\"virtual-machines\",\n\t\t\t\"image_profiles\",\n\t\t\t\"clustering_architecture\",\n\t\t\t\"resources_disk_id\",\n\t\t\t\"storage_lvm_stripes\",\n\t\t\t\"vm_boot_priority\",\n\t\t\t\"unix_hotplug_devices\",\n\t\t\t\"api_filtering\",\n\t\t\t\"instance_nic_network\",\n\t\t\t\"clustering_sizing\",\n\t\t\t\"firewall_driver\",\n\t\t\t\"projects_limits\",\n\t\t\t\"container_syscall_intercept_hugetlbfs\",\n\t\t\t\"limits_hugepages\",\n\t\t\t\"container_nic_routed_gateway\",\n\t\t\t\"projects_restrictions\",\n\t\t\t\"custom_volume_snapshot_expiry\",\n\t\t\t\"volume_snapshot_scheduling\",\n\t\t\t\"trust_ca_certificates\",\n\t\t\t\"snapshot_disk_usage\",\n\t\t\t\"clustering_edit_roles\",\n\t\t\t\"container_nic_routed_host_address\",\n\t\t\t\"container_nic_ipvlan_gateway\",\n\t\t\t\"resources_usb_pci\",\n\t\t\t\"resources_cpu_threads_numa\",\n\t\t\t\"resources_cpu_core_die\",\n\t\t\t\"api_os\",\n\t\t\t\"container_nic_routed_host_table\",\n\t\t\t\"container_nic_ipvlan_host_table\",\n\t\t\t\"container_nic_ipvlan_mode\",\n\t\t\t\"resources_system\",\n\t\t\t\"images_push_relay\",\n\t\t\t\"network_dns_search\",\n\t\t\t\"container_nic_routed_limits\",\n\t\t\t\"instance_nic_bridged_vlan\",\n\t\t\t\"network_state_bond_bridge\",\n\t\t\t\"usedby_consistency\",\n\t\t\t\"custom_block_volumes\",\n\t\t\t\"clustering_failure_domains\",\n\t\t\t\"resources_gpu_mdev\",\n\t\t\t\"console_vga_type\",\n\t\t\t\"projects_limits_disk\",\n\t\t\t\"network_type_macvlan\",\n\t\t\t\"network_type_sriov\",\n\t\t\t\"container_syscall_intercept_bpf_devices\",\n\t\t\t\"network_type_ovn\",\n\t\t\t\"projects_networks\",\n\t\t\t\"projects_networks_restricted_uplinks\",\n\t\t\t\"custom_volume_backup\",\n\t\t\t\"backup_override_name\",\n\t\t\t\"storage_rsync_compression\",\n\t\t\t\"network_type_physical\",\n\t\t\t\"network_ovn_external_subnets\",\n\t\t\t\"network_ovn_nat\",\n\t\t\t\"network_ovn_external_routes_remove\",\n\t\t\t\"tpm_device_type\",\n\t\t\t\"storage_zfs_clone_copy_rebase\",\n\t\t\t\"gpu_mdev\",\n\t\t\t\"resources_pci_iommu\",\n\t\t\t\"resources_network_usb\",\n\t\t\t\"resources_disk_address\",\n\t\t\t\"network_physical_ovn_ingress_mode\",\n\t\t\t\"network_ovn_dhcp\",\n\t\t\t\"network_physical_routes_anycast\",\n\t\t\t\"projects_limits_instances\",\n\t\t\t\"network_state_vlan\",\n\t\t\t\"instance_nic_bridged_port_isolation\",\n\t\t\t\"instance_bulk_state_change\",\n\t\t\t\"network_gvrp\",\n\t\t\t\"instance_pool_move\",\n\t\t\t\"gpu_sriov\",\n\t\t\t\"pci_device_type\",\n\t\t\t\"storage_volume_state\",\n\t\t\t\"network_acl\",\n\t\t\t\"migration_stateful\",\n\t\t\t\"disk_state_quota\",\n\t\t\t\"storage_ceph_features\",\n\t\t\t\"projects_compression\",\n\t\t\t\"projects_images_remote_cache_expiry\",\n\t\t\t\"certificate_project\",\n\t\t\t\"network_ovn_acl\",\n\t\t\t\"projects_images_auto_update\",\n\t\t\t\"projects_restricted_cluster_target\",\n\t\t\t\"images_default_architecture\",\n\t\t\t\"network_ovn_acl_defaults\",\n\t\t\t\"gpu_mig\",\n\t\t\t\"project_usage\",\n\t\t\t\"network_bridge_acl\",\n\t\t\t\"warnings\",\n\t\t\t\"projects_restricted_backups_and_snapshots\",\n\t\t\t\"clustering_join_token\",\n\t\t\t\"clustering_description\",\n\t\t\t\"server_trusted_proxy\",\n\t\t\t\"clustering_update_cert\",\n\t\t\t\"storage_api_project\",\n\t\t\t\"server_instance_driver_operational\",\n\t\t\t\"server_supported_storage_drivers\",\n\t\t\t\"event_lifecycle_requestor_address\",\n\t\t\t\"resources_gpu_usb\",\n\t\t\t\"clustering_evacuation\",\n\t\t\t\"network_ovn_nat_address\",\n\t\t\t\"network_bgp\",\n\t\t\t\"network_forward\",\n\t\t\t\"custom_volume_refresh\",\n\t\t\t\"network_counters_errors_dropped\",\n\t\t\t\"metrics\",\n\t\t\t\"image_source_project\",\n\t\t\t\"clustering_config\",\n\t\t\t\"network_peer\",\n\t\t\t\"linux_sysctl\",\n\t\t\t\"network_dns\",\n\t\t\t\"ovn_nic_acceleration\",\n\t\t\t\"certificate_self_renewal\",\n\t\t\t\"instance_project_move\",\n\t\t\t\"storage_volume_project_move\",\n\t\t\t\"cloud_init\",\n\t\t\t\"network_dns_nat\",\n\t\t\t\"database_leader\",\n\t\t\t\"instance_all_projects\",\n\t\t\t\"clustering_groups\",\n\t\t\t\"ceph_rbd_du\",\n\t\t\t\"instance_get_full\",\n\t\t\t\"qemu_metrics\",\n\t\t\t\"gpu_mig_uuid\",\n\t\t\t\"event_project\",\n\t\t\t\"clustering_evacuation_live\",\n\t\t\t\"instance_allow_inconsistent_copy\",\n\t\t\t\"network_state_ovn\",\n\t\t\t\"storage_volume_api_filtering\",\n\t\t\t\"image_restrictions\",\n\t\t\t\"storage_zfs_export\",\n\t\t\t\"network_dns_records\",\n\t\t\t\"storage_zfs_reserve_space\",\n\t\t\t\"network_acl_log\",\n\t\t\t\"storage_zfs_blocksize\",\n\t\t\t\"metrics_cpu_seconds\",\n\t\t\t\"instance_snapshot_never\",\n\t\t\t\"certificate_token\",\n\t\t\t\"instance_nic_routed_neighbor_probe\",\n\t\t\t\"event_hub\",\n\t\t\t\"agent_nic_config\",\n\t\t\t\"projects_restricted_intercept\",\n\t\t\t\"metrics_authentication\",\n\t\t\t\"images_target_project\",\n\t\t\t\"cluster_migration_inconsistent_copy\",\n\t\t\t\"cluster_ovn_chassis\",\n\t\t\t\"container_syscall_intercept_sched_setscheduler\",\n\t\t\t\"storage_lvm_thinpool_metadata_size\",\n\t\t\t\"storage_volume_state_total\",\n\t\t\t\"instance_file_head\",\n\t\t\t\"instances_nic_host_name\",\n\t\t\t\"image_copy_profile\",\n\t\t\t\"container_syscall_intercept_sysinfo\",\n\t\t\t\"clustering_evacuation_mode\",\n\t\t\t\"resources_pci_vpd\",\n\t\t\t\"qemu_raw_conf\",\n\t\t\t\"storage_cephfs_fscache\",\n\t\t\t\"network_load_balancer\",\n\t\t\t\"vsock_api\",\n\t\t\t\"instance_ready_state\",\n\t\t\t\"network_bgp_holdtime\",\n\t\t\t\"storage_volumes_all_projects\",\n\t\t\t\"metrics_memory_oom_total\",\n\t\t\t\"storage_buckets\",\n\t\t\t\"storage_buckets_create_credentials\",\n\t\t\t\"metrics_cpu_effective_total\",\n\t\t\t\"projects_networks_restricted_access\",\n\t\t\t\"storage_buckets_local\",\n\t\t\t\"loki\",\n\t\t\t\"acme\",\n\t\t\t\"internal_metrics\",\n\t\t\t\"cluster_join_token_expiry\",\n\t\t\t\"remote_token_expiry\",\n\t\t\t\"init_preseed\",\n\t\t\t\"storage_volumes_created_at\",\n\t\t\t\"cpu_hotplug\",\n\t\t\t\"projects_networks_zones\",\n\t\t\t\"network_txqueuelen\",\n\t\t\t\"cluster_member_state\",\n\t\t\t\"instances_placement_scriptlet\",\n\t\t\t\"storage_pool_source_wipe\",\n\t\t\t\"zfs_block_mode\",\n\t\t\t\"instance_generation_id\",\n\t\t\t\"disk_io_cache\",\n\t\t\t\"amd_sev\",\n\t\t\t\"storage_pool_loop_resize\",\n\t\t\t\"migration_vm_live\",\n\t\t\t\"ovn_nic_nesting\",\n\t\t\t\"oidc\",\n\t\t\t\"network_ovn_l3only\",\n\t\t\t\"ovn_nic_acceleration_vdpa\",\n\t\t\t\"cluster_healing\",\n\t\t\t\"instances_state_total\",\n\t\t\t\"auth_user\",\n\t\t\t\"security_csm\",\n\t\t\t\"instances_rebuild\",\n\t\t\t\"numa_cpu_placement\"\n\t\t],\n\t\t\"api_status\": \"stable\",\n\t\t\"api_version\": \"1.0\",\n\t\t\"auth\": \"trusted\",\n\t\t\"public\": false,\n\t\t\"auth_methods\": [\n\t\t\t\"tls\"\n\t\t],\n\t\t\"auth_user_name\": \"vitaly\",\n\t\t\"auth_user_method\": \"unix\",\n\t\t\"environment\": {\n\t\t\t\"addresses\": [\n\t\t\t\t\"192.168.10.225:8443\"\n\t\t\t],\n\t\t\t\"architectures\": [\n\t\t\t\t\"x86_64\",\n\t\t\t\t\"i686\"\n\t\t\t],\n\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nMIICBzCCAY2gAwIBAgIRAKIrYljXGEzphSCmSSTu0jYwCgYIKoZIzj0EAwMwNTEc\\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEVMBMGA1UEAwwMcm9vdEB2aG9k\\naWtvMB4XDTIwMDQyNzIwMDM0M1oXDTMwMDQyNTIwMDM0M1owNTEcMBoGA1UEChMT\\nbGludXhjb250YWluZXJzLm9yZzEVMBMGA1UEAwwMcm9vdEB2aG9kaWtvMHYwEAYH\\nKoZIzj0CAQYFK4EEACIDYgAEgN5kNCMEL43oV+D845W1ofk1wnHmMQTUqkJvYr5s\\npmjW+51BSeDWllRQEqeDfFeHo629cvidRYQPE1llxpbXf8xLUbvpS5MAHMnsh2GI\\nuvypasOY/G9iof4eQXFl+t16o2EwXzAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\\nCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAqBgNVHREEIzAhggd2aG9kaWtvhwR/\\nAAABhxAAAAAAAAAAAAAAAAAAAAABMAoGCCqGSM49BAMDA2gAMGUCMCf+AvToMrRX\\nDkKgLkA8uGfyI37McTHJfcuZiNKHM5dKqp8E4YE9uXFbijCETGGmtQIxAMBQGEn7\\nK4ZiF1arE4NVYnjGbJEY2UZvGWBRJ1qhn6cXsgxyJFGeirNv6D5h4Az4qQ==\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\"certificate_fingerprint\": \"c0abcbe6f8d27d7551eb3bf3e3385e73ec24068dcaea977036bd84db204ffb5d\",\n\t\t\t\"driver\": \"lxc\",\n\t\t\t\"driver_version\": \"5.0.0\",\n\t\t\t\"firewall\": \"nftables\",\n\t\t\t\"kernel\": \"Linux\",\n\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\"kernel_features\": {\n\t\t\t\t\"idmapped_mounts\": \"false\",\n\t\t\t\t\"netnsid_getifaddrs\": \"true\",\n\t\t\t\t\"seccomp_listener\": \"true\",\n\t\t\t\t\"seccomp_listener_continue\": \"true\",\n\t\t\t\t\"shiftfs\": \"false\",\n\t\t\t\t\"uevent_injection\": \"true\",\n\t\t\t\t\"unpriv_fscaps\": \"true\"\n\t\t\t},\n\t\t\t\"kernel_version\": \"5.4.0-144-generic\",\n\t\t\t\"lxc_features\": {\n\t\t\t\t\"cgroup2\": \"true\",\n\t\t\t\t\"core_scheduling\": \"true\",\n\t\t\t\t\"devpts_fd\": \"true\",\n\t\t\t\t\"idmapped_mounts_v2\": \"true\",\n\t\t\t\t\"mount_injection_file\": \"true\",\n\t\t\t\t\"network_gateway_device_route\": \"true\",\n\t\t\t\t\"network_ipvlan\": \"true\",\n\t\t\t\t\"network_l2proxy\": \"true\",\n\t\t\t\t\"network_phys_macvlan_mtu\": \"true\",\n\t\t\t\t\"network_veth_router\": \"true\",\n\t\t\t\t\"pidfd\": \"true\",\n\t\t\t\t\"seccomp_allow_deny_syntax\": \"true\",\n\t\t\t\t\"seccomp_notify\": \"true\",\n\t\t\t\t\"seccomp_proxy_send_notify_fd\": \"true\"\n\t\t\t},\n\t\t\t\"os_name\": \"Ubuntu\",\n\t\t\t\"os_version\": \"20.04\",\n\t\t\t\"project\": \"default\",\n\t\t\t\"server\": \"lxd\",\n\t\t\t\"server_clustered\": false,\n\t\t\t\"server_event_mode\": \"full-mesh\",\n\t\t\t\"server_name\": \"v\",\n\t\t\t\"server_pid\": 112798,\n\t\t\t\"server_version\": \"5.15\",\n\t\t\t\"storage\": \"btrfs | lvm\",\n\t\t\t\"storage_version\": \"5.16.2 | 2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.41.0\",\n\t\t\t\"storage_supported_drivers\": [\n\t\t\t\t{\n\t\t\t\t\t\"Name\": \"zfs\",\n\t\t\t\t\t\"Version\": \"0.8.3-1ubuntu12.14\",\n\t\t\t\t\t\"Remote\": false\n\t\t\t\t},\n\t\t\t\t{\n\t\t\t\t\t\"Name\": \"btrfs\",\n\t\t\t\t\t\"Version\": \"5.16.2\",\n\t\t\t\t\t\"Remote\": false\n\t\t\t\t},\n\t\t\t\t{\n\t\t\t\t\t\"Name\": \"ceph\",\n\t\t\t\t\t\"Version\": \"17.2.5\",\n\t\t\t\t\t\"Remote\": true\n\t\t\t\t},\n\t\t\t\t{\n\t\t\t\t\t\"Name\": \"cephfs\",\n\t\t\t\t\t\"Version\": \"17.2.5\",\n\t\t\t\t\t\"Remote\": true\n\t\t\t\t},\n\t\t\t\t{\n\t\t\t\t\t\"Name\": \"cephobject\",\n\t\t\t\t\t\"Version\": \"17.2.5\",\n\t\t\t\t\t\"Remote\": true\n\t\t\t\t},\n\t\t\t\t{\n\t\t\t\t\t\"Name\": \"dir\",\n\t\t\t\t\t\"Version\": \"1\",\n\t\t\t\t\t\"Remote\": false\n\t\t\t\t},\n\t\t\t\t{\n\t\t\t\t\t\"Name\": \"lvm\",\n\t\t\t\t\t\"Version\": \"2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.41.0\",\n\t\t\t\t\t\"Remote\": false\n\t\t\t\t}\n\t\t\t]\n\t\t}\n\t}" time="2023-07-14T22:45:45+02:00" level=debug msg="Connected to the websocket: ws://unix.socket/1.0/events" time="2023-07-14T22:45:45+02:00" level=debug msg="Sending request to LXD" etag= method=POST url="http://unix.socket/1.0/storage-pools/btrfs_pool01/volumes/custom/sm/backups" time="2023-07-14T22:45:45+02:00" level=debug msg="\n\t{\n\t\t\"name\": \"\",\n\t\t\"expires_at\": \"2023-07-15T22:45:45.920431096+02:00\",\n\t\t\"volume_only\": true,\n\t\t\"optimized_storage\": true,\n\t\t\"compression_algorithm\": \"\"\n\t}" time="2023-07-14T22:45:45+02:00" level=debug msg="Got operation from LXD" time="2023-07-14T22:45:45+02:00" level=debug msg="\n\t{\n\t\t\"id\": \"59076afb-022f-447d-90f8-999bf3c8a3ce\",\n\t\t\"class\": \"task\",\n\t\t\"description\": \"Creating custom volume backup\",\n\t\t\"created_at\": \"2023-07-14T22:45:45.926680871+02:00\",\n\t\t\"updated_at\": \"2023-07-14T22:45:45.926680871+02:00\",\n\t\t\"status\": \"Running\",\n\t\t\"status_code\": 103,\n\t\t\"resources\": {\n\t\t\t\"backups\": [\n\t\t\t\t\"/1.0/storage-pools/btrfs_pool01/volumes/custom/sm/backups/backup5\"\n\t\t\t],\n\t\t\t\"storage_volumes\": [\n\t\t\t\t\"/1.0/storage-pools/btrfs_pool01/volumes/custom/sm\"\n\t\t\t]\n\t\t},\n\t\t\"metadata\": null,\n\t\t\"may_cancel\": false,\n\t\t\"err\": \"\",\n\t\t\"location\": \"none\"\n\t}" time="2023-07-14T22:45:45+02:00" level=debug msg="Sending request to LXD" etag= method=GET url="http://unix.socket/1.0/operations/59076afb-022f-447d-90f8-999bf3c8a3ce" time="2023-07-14T22:45:45+02:00" level=debug msg="Got response struct from LXD" time="2023-07-14T22:45:45+02:00" level=debug msg="\n\t{\n\t\t\"id\": \"59076afb-022f-447d-90f8-999bf3c8a3ce\",\n\t\t\"class\": \"task\",\n\t\t\"description\": \"Creating custom volume backup\",\n\t\t\"created_at\": \"2023-07-14T22:45:45.926680871+02:00\",\n\t\t\"updated_at\": \"2023-07-14T22:45:45.926680871+02:00\",\n\t\t\"status\": \"Running\",\n\t\t\"status_code\": 103,\n\t\t\"resources\": {\n\t\t\t\"backups\": [\n\t\t\t\t\"/1.0/storage-pools/btrfs_pool01/volumes/custom/sm/backups/backup5\"\n\t\t\t],\n\t\t\t\"storage_volumes\": [\n\t\t\t\t\"/1.0/storage-pools/btrfs_pool01/volumes/custom/sm\"\n\t\t\t]\n\t\t},\n\t\t\"metadata\": null,\n\t\t\"may_cancel\": false,\n\t\t\"err\": \"\",\n\t\t\"location\": \"none\"\n\t}" Exporting the backup: 100% (748.24MB/s) Backup exported successfully! time="2023-07-14T22:51:15+02:00" level=debug msg="Sending request to LXD" etag= method=DELETE url="http://unix.socket/1.0/storage-pools/btrfs_pool01/volumes/custom/sm/backups/backup5" time="2023-07-14T22:51:15+02:00" level=debug msg="Got operation from LXD" time="2023-07-14T22:51:15+02:00" level=debug msg="\n\t{\n\t\t\"id\": \"56071968-2a50-410d-9755-087898edcae1\",\n\t\t\"class\": \"task\",\n\t\t\"description\": \"Deleting custom volume backup\",\n\t\t\"created_at\": \"2023-07-14T22:51:15.409704238+02:00\",\n\t\t\"updated_at\": \"2023-07-14T22:51:15.409704238+02:00\",\n\t\t\"status\": \"Running\",\n\t\t\"status_code\": 103,\n\t\t\"resources\": {\n\t\t\t\"backups\": [\n\t\t\t\t\"/1.0/storage-pools/btrfs_pool01/volumes/custom/sm/backups/backup5\"\n\t\t\t],\n\t\t\t\"storage_volumes\": [\n\t\t\t\t\"/1.0/storage-pools/btrfs_pool01/volumes/custom/sm\"\n\t\t\t]\n\t\t},\n\t\t\"metadata\": null,\n\t\t\"may_cancel\": false,\n\t\t\"err\": \"\",\n\t\t\"location\": \"none\"\n\t}" time="2023-07-14T22:51:15+02:00" level=debug msg="Sending request to LXD" etag= method=GET url="http://unix.socket/1.0/operations/56071968-2a50-410d-9755-087898edcae1" time="2023-07-14T22:51:15+02:00" level=debug msg="Got response struct from LXD" time="2023-07-14T22:51:15+02:00" level=debug msg="\n\t{\n\t\t\"id\": \"56071968-2a50-410d-9755-087898edcae1\",\n\t\t\"class\": \"task\",\n\t\t\"description\": \"Deleting custom volume backup\",\n\t\t\"created_at\": \"2023-07-14T22:51:15.409704238+02:00\",\n\t\t\"updated_at\": \"2023-07-14T22:51:15.409704238+02:00\",\n\t\t\"status\": \"Running\",\n\t\t\"status_code\": 103,\n\t\t\"resources\": {\n\t\t\t\"backups\": [\n\t\t\t\t\"/1.0/storage-pools/btrfs_pool01/volumes/custom/sm/backups/backup5\"\n\t\t\t],\n\t\t\t\"storage_volumes\": [\n\t\t\t\t\"/1.0/storage-pools/btrfs_pool01/volumes/custom/sm\"\n\t\t\t]\n\t\t},\n\t\t\"metadata\": null,\n\t\t\"may_cancel\": false,\n\t\t\"err\": \"\",\n\t\t\"location\": \"none\"\n\t}"