search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.38k stars 931 forks source link

golang client net/http: timeout awaiting response headers #10377

Closed julio641742 closed 2 years ago

julio641742 commented 2 years ago

Required information

Issue description

A brief description of the problem. Should include what you were attempting to do, what you did, what happened and what you expected to see happen.

I developend an application with pylxd before and did not run into any issues with it. Now I am migrating my codebase to go and using the lxd client library.

I ran into in issue with timeouts

Post "https://127.0.0.1:8443/1.0/networks?project=sample-project": net/http: timeout awaiting response headers

In this case, the python code uses a timeout of None which means that it will wait indefinitely until it gets a response back.

https://github.com/lxc/pylxd/blob/a44deea30cb2dce3b1fd34240155c5af6479f1a2/pylxd/client.py#L333

On network creation, the function is blocked until it was created

https://github.com/lxc/pylxd/blob/a44deea30cb2dce3b1fd34240155c5af6479f1a2/pylxd/models/network.py#L112

In the go codebase, there is a timeout of 30 seconds for response headers

https://github.com/lxc/lxd/blob/2ed60cd252b2d3daf64e8ad0413d5097568f2a1e/client/util.go#L29

That is the problem, my network creation request takes more than 30 seconds to complete.

Possible fixes are to increase the timeout or provide a way for the programmer to increase it manually.

Note: The network gets created in the background even after the timeout, but I want a way to block my application until the request is completed successfully. I do not want to write a loop checking the state of the network until it gets created :(

Steps to reproduce

  1. Step one Deploy a slow LXD server
  2. Step two Write a go application to create a new OVN network
    1. Step three After 30 seconds, the CreateNetwork function thorws a timeout error

Information to attach

tomponline commented 2 years ago

What is the contents and url of your request?

tomponline commented 2 years ago

We added timeouts recently (as all network programming should have timeouts) but we may need to increase it to cover all common workload types (although our ovn tests are completing in under that time) so would be good to understand why its taking so long in your case.

tomponline commented 2 years ago

How long does it take with your pylxd app?

julio641742 commented 2 years ago 
import time
from pylxd import Client

client = Client(
    verify=False,
    cert=("lxd.crt", "lxd.key"),
    endpoint="https://remote-ip:8443",
    project="sample-project",
)

tic = time.perf_counter()
client.networks.create(
    "ovn-network",
    type="ovn",
    config={
        "ipv4.nat": "true",
        "ipv6.address": "none",
        "network": "parent-network",
    },
)
toc = time.perf_counter()

print(f"Request took {toc - tic:0.4f} seconds")

Request took 103.0310 seconds

My LXD server is running dozens of containers/vms/networks, that might be why it takes almost 2 minutes to create that network

tomponline commented 2 years ago

Please can you run 'lxc monitor --type=logging --pretty' on the server while trying to create the network so we can see which steps take time.

itoffshore commented 2 years ago

I recently starting getting this timeout error too copying an image from a local to a remote (my ISP upload is around 18-20mbps & the lxd nodes are connected with wireguard) - the image size is 179mb

lxc image copy -v --copy-aliases --mode=push local:ubuntu-minio-22.04 myremote:

the image copies but the alias does not get applied

tomponline commented 2 years ago

Please can you run the command with --debug output as I'm surprised that an operation like copying an image isn't done with an async operation.

tomponline commented 2 years ago

Please can you also provide the exact error. We did not limit request timeout. But rather just the initial header response timeout. So I suspect something else aside from bandwidth limitations is at play here (it may be firewall or multi ip issue).

itoffshore commented 2 years ago 
lxc image copy --debug -v --copy-aliases --mode=push local:ubuntu-minio-22.04 iacimg:
DEBUG  [2022-05-08T06:46:57+01:00] Connecting to a local LXD over a Unix socket 
DEBUG  [2022-05-08T06:46:57+01:00] Sending request to LXD                        etag= method=GET url="http://unix.socket/1.0"
DEBUG  [2022-05-08T06:46:57+01:00] Got response struct from LXD                 
DEBUG  [2022-05-08T06:46:57+01:00] 
        {
                "config": {
                        "core.https_address": "10.201.0.1"
                },
                "api_extensions": [
                        "storage_zfs_remove_snapshots",
                        "container_host_shutdown_timeout",
                        "container_stop_priority",
                        "container_syscall_filtering",
                        "auth_pki",
                        "container_last_used_at",
                        "etag",
                        "patch",
                        "usb_devices",
                        "https_allowed_credentials",
                        "image_compression_algorithm",
                        "directory_manipulation",
                        "container_cpu_time",
                        "storage_zfs_use_refquota",
                        "storage_lvm_mount_options",
                        "network",
                        "profile_usedby",
                        "container_push",
                        "container_exec_recording",
                        "certificate_update",
                        "container_exec_signal_handling",
                        "gpu_devices",
                        "container_image_properties",
                        "migration_progress",
                        "id_map",
                        "network_firewall_filtering",
                        "network_routes",
                        "storage",
                        "file_delete",
                        "file_append",
                        "network_dhcp_expiry",
                        "storage_lvm_vg_rename",
                        "storage_lvm_thinpool_rename",
                        "network_vlan",
                        "image_create_aliases",
                        "container_stateless_copy",
                        "container_only_migration",
                        "storage_zfs_clone_copy",
                        "unix_device_rename",
                        "storage_lvm_use_thinpool",
                        "storage_rsync_bwlimit",
                        "network_vxlan_interface",
                        "storage_btrfs_mount_options",
                        "entity_description",
                        "image_force_refresh",
                        "storage_lvm_lv_resizing",
                        "id_map_base",
                        "file_symlinks",
                        "container_push_target",
                        "network_vlan_physical",
                        "storage_images_delete",
                        "container_edit_metadata",
                        "container_snapshot_stateful_migration",
                        "storage_driver_ceph",
                        "storage_ceph_user_name",
                        "resource_limits",
                        "storage_volatile_initial_source",
                        "storage_ceph_force_osd_reuse",
                        "storage_block_filesystem_btrfs",
                        "resources",
                        "kernel_limits",
                        "storage_api_volume_rename",
                        "macaroon_authentication",
                        "network_sriov",
                        "console",
                        "restrict_devlxd",
                        "migration_pre_copy",
                        "infiniband",
                        "maas_network",
                        "devlxd_events",
                        "proxy",
                        "network_dhcp_gateway",
                        "file_get_symlink",
                        "network_leases",
                        "unix_device_hotplug",
                        "storage_api_local_volume_handling",
                        "operation_description",
                        "clustering",
                        "event_lifecycle",
                        "storage_api_remote_volume_handling",
                        "nvidia_runtime",
                        "container_mount_propagation",
                        "container_backup",
                        "devlxd_images",
                        "container_local_cross_pool_handling",
                        "proxy_unix",
                        "proxy_udp",
                        "clustering_join",
                        "proxy_tcp_udp_multi_port_handling",
                        "network_state",
                        "proxy_unix_dac_properties",
                        "container_protection_delete",
                        "unix_priv_drop",
                        "pprof_http",
                        "proxy_haproxy_protocol",
                        "network_hwaddr",
                        "proxy_nat",
                        "network_nat_order",
                        "container_full",
                        "candid_authentication",
                        "backup_compression",
                        "candid_config",
                        "nvidia_runtime_config",
                        "storage_api_volume_snapshots",
                        "storage_unmapped",
                        "projects",
                        "candid_config_key",
                        "network_vxlan_ttl",
                        "container_incremental_copy",
                        "usb_optional_vendorid",
                        "snapshot_scheduling",
                        "snapshot_schedule_aliases",
                        "container_copy_project",
                        "clustering_server_address",
                        "clustering_image_replication",
                        "container_protection_shift",
                        "snapshot_expiry",
                        "container_backup_override_pool",
                        "snapshot_expiry_creation",
                        "network_leases_location",
                        "resources_cpu_socket",
                        "resources_gpu",
                        "resources_numa",
                        "kernel_features",
                        "id_map_current",
                        "event_location",
                        "storage_api_remote_volume_snapshots",
                        "network_nat_address",
                        "container_nic_routes",
                        "rbac",
                        "cluster_internal_copy",
                        "seccomp_notify",
                        "lxc_features",
                        "container_nic_ipvlan",
                        "network_vlan_sriov",
                        "storage_cephfs",
                        "container_nic_ipfilter",
                        "resources_v2",
                        "container_exec_user_group_cwd",
                        "container_syscall_intercept",
                        "container_disk_shift",
                        "storage_shifted",
                        "resources_infiniband",
                        "daemon_storage",
                        "instances",
                        "image_types",
                        "resources_disk_sata",
                        "clustering_roles",
                        "images_expiry",
                        "resources_network_firmware",
                        "backup_compression_algorithm",
                        "ceph_data_pool_name",
                        "container_syscall_intercept_mount",
                        "compression_squashfs",
                        "container_raw_mount",
                        "container_nic_routed",
                        "container_syscall_intercept_mount_fuse",
                        "container_disk_ceph",
                        "virtual-machines",
                        "image_profiles",
                        "clustering_architecture",
                        "resources_disk_id",
                        "storage_lvm_stripes",
                        "vm_boot_priority",
                        "unix_hotplug_devices",
                        "api_filtering",
                        "instance_nic_network",
                        "clustering_sizing",
                        "firewall_driver",
                        "projects_limits",
                        "container_syscall_intercept_hugetlbfs",
                        "limits_hugepages",
                        "container_nic_routed_gateway",
                        "projects_restrictions",
                        "custom_volume_snapshot_expiry",
                        "volume_snapshot_scheduling",
                        "trust_ca_certificates",
                        "snapshot_disk_usage",
                        "clustering_edit_roles",
                        "container_nic_routed_host_address",
                        "container_nic_ipvlan_gateway",
                        "resources_usb_pci",
                        "resources_cpu_threads_numa",
                        "resources_cpu_core_die",
                        "api_os",
                        "container_nic_routed_host_table",
                        "container_nic_ipvlan_host_table",
                        "container_nic_ipvlan_mode",
                        "resources_system",
                        "images_push_relay",
                        "network_dns_search",
                        "container_nic_routed_limits",
                        "instance_nic_bridged_vlan",
                        "network_state_bond_bridge",
                        "usedby_consistency",
                        "custom_block_volumes",
                        "clustering_failure_domains",
                        "resources_gpu_mdev",
                        "console_vga_type",
                        "projects_limits_disk",
                        "network_type_macvlan",
                        "network_type_sriov",
                        "container_syscall_intercept_bpf_devices",
                        "network_type_ovn",
                        "projects_networks",
                        "projects_networks_restricted_uplinks",
                        "custom_volume_backup",
                        "backup_override_name",
                        "storage_rsync_compression",
                        "network_type_physical",
                        "network_ovn_external_subnets",
                        "network_ovn_nat",
                        "network_ovn_external_routes_remove",
                        "tpm_device_type",
                        "storage_zfs_clone_copy_rebase",
                        "gpu_mdev",
                        "resources_pci_iommu",
                        "resources_network_usb",
                        "resources_disk_address",
                        "network_physical_ovn_ingress_mode",
                        "network_ovn_dhcp",
                        "network_physical_routes_anycast",
                        "projects_limits_instances",
                        "network_state_vlan",
                        "instance_nic_bridged_port_isolation",
                        "instance_bulk_state_change",
                        "network_gvrp",
                        "instance_pool_move",
                        "gpu_sriov",
                        "pci_device_type",
                        "storage_volume_state",
                        "network_acl",
                        "migration_stateful",
                        "disk_state_quota",
                        "storage_ceph_features",
                        "projects_compression",
                        "projects_images_remote_cache_expiry",
                        "certificate_project",
                        "network_ovn_acl",
                        "projects_images_auto_update",
                        "projects_restricted_cluster_target",
                        "images_default_architecture",
                        "network_ovn_acl_defaults",
                        "gpu_mig",
                        "project_usage",
                        "network_bridge_acl",
                        "warnings",
                        "projects_restricted_backups_and_snapshots",
                        "clustering_join_token",
                        "clustering_description",
                        "server_trusted_proxy",
                        "clustering_update_cert",
                        "storage_api_project",
                        "server_instance_driver_operational",
                        "server_supported_storage_drivers",
                        "event_lifecycle_requestor_address",
                        "resources_gpu_usb",
                        "clustering_evacuation",
                        "network_ovn_nat_address",
                        "network_bgp",
                        "network_forward",
                        "custom_volume_refresh",
                        "network_counters_errors_dropped",
                        "metrics",
                        "image_source_project",
                        "clustering_config",
                        "network_peer",
                        "linux_sysctl",
                        "network_dns",
                        "ovn_nic_acceleration",
                        "certificate_self_renewal",
                        "instance_project_move",
                        "storage_volume_project_move",
                        "cloud_init",
                        "network_dns_nat",
                        "database_leader",
                        "instance_all_projects",
                        "clustering_groups",
                        "ceph_rbd_du",
                        "instance_get_full",
                        "qemu_metrics",
                        "gpu_mig_uuid",
                        "event_project",
                        "clustering_evacuation_live",
                        "instance_allow_inconsistent_copy",
                        "network_state_ovn",
                        "storage_volume_api_filtering",
                        "image_restrictions",
                        "storage_zfs_export",
                        "network_dns_records",
                        "storage_zfs_reserve_space",
                        "network_acl_log",
                        "storage_zfs_blocksize",
                        "metrics_cpu_seconds",
                        "instance_snapshot_never",
                        "certificate_token",
                        "instance_nic_routed_neighbor_probe",
                        "event_hub",
                        "agent_nic_config",
                        "projects_restricted_intercept",
                        "metrics_authentication",
                        "images_target_project",
                        "cluster_migration_inconsistent_copy",
                        "cluster_ovn_chassis",
                        "container_syscall_intercept_sched_setscheduler",
                        "storage_lvm_thinpool_metadata_size",
                        "storage_volume_state_total",
                        "instance_file_head",
                        "instances_nic_host_name",
                        "image_copy_profile",
                        "container_syscall_intercept_sysinfo",
                        "clustering_evacuation_mode"
                ],
                "api_status": "stable",
                "api_version": "1.0",
                "auth": "trusted",
                "public": false,
                "auth_methods": [
                        "tls"
                ],
                "environment": {
                        "addresses": [
                                "10.201.0.1:8443"
                        ],
                        "architectures": [
                                "x86_64",
                                "i686"
                        ],
                        "certificate": "-----BEGIN CERTIFICATE-----\nMIICCzCCAZKgAwIBAgIQUI0swBDs6vJwp2PlwqkJazAKBggqhkjOPQQDAzA3MRww\nGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRcwFQYDVQQDDA5yb290QGVuZGVh\ndm91cjAeFw0yMjA0MjUxNTU0NDlaFw0zMjA0MjIxNTU0NDlaMDcxHDAaBgNVBAoT\nE2xpbnV4Y29udGFpbmVycy5vcmcxFzAVBgNVBAMMDnJvb3RAZW5kZWF2b3VyMHYw\nEAYHKoZIzj0CAQYFK4EEACIDYgAEgjrXWznR94gEKBONGs4smRBLo7qqyaMwkiTq\n/2J7iMKF5nsZgqwNv1UJnw1AlJnLbx+M5bkwA0AtCTyeSbtNRnIgFgs8pjmR6n7q\nHuV428RLPl4JMlrdsQ9SIdQx3dSVo2MwYTAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\nBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAsBgNVHREEJTAjggllbmRlYXZv\ndXKHBH8AAAGHEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDZwAwZAIwQvRC\nwVy+aWeg0rUYSOboOVy7rYzQEdDNlfgJWe2HNwBHhywtpab7+ooZgZfU4mXMAjBW\n8w9kCHwLh3/bBwO7TNmV7t/k2ZHB+GpBZiRCNpEHUpdfio20Wjm+7954uyd6g6Y=\n-----END CERTIFICATE-----\n",
                        "certificate_fingerprint": "0532a6edbf8bdd51b97faeb9b2669191ed7f5aa1ece9234a4d1e4c7abf3b3c34",
                        "driver": "lxc | qemu",
                        "driver_version": "4.0.12 | 6.1.1",
                        "firewall": "nftables",
                        "kernel": "Linux",
                        "kernel_architecture": "x86_64",
                        "kernel_features": {
                                "idmapped_mounts": "true",
                                "netnsid_getifaddrs": "true",
                                "seccomp_listener": "true",
                                "seccomp_listener_continue": "true",
                                "shiftfs": "false",
                                "uevent_injection": "true",
                                "unpriv_fscaps": "true"
                        },
                        "kernel_version": "5.17.5-hardened1-1-hardened",
                        "lxc_features": {
                                "cgroup2": "true",
                                "core_scheduling": "true",
                                "devpts_fd": "true",
                                "idmapped_mounts_v2": "true",
                                "mount_injection_file": "true",
                                "network_gateway_device_route": "true",
                                "network_ipvlan": "true",
                                "network_l2proxy": "true",
                                "network_phys_macvlan_mtu": "true",
                                "network_veth_router": "true",
                                "pidfd": "true",
                                "seccomp_allow_deny_syntax": "true",
                                "seccomp_notify": "true",
                                "seccomp_proxy_send_notify_fd": "true"
                        },
                        "os_name": "EndeavourOS",
                        "os_version": "",
                        "project": "default",
                        "server": "lxd",
                        "server_clustered": false,
                        "server_event_mode": "full-mesh",
                        "server_name": "endeavour",
                        "server_pid": 13740,
                        "server_version": "5.1",
                        "storage": "zfs",
                        "storage_version": "2.1.4-1",
                        "storage_supported_drivers": [
                                {
                                        "Name": "zfs",
                                        "Version": "2.1.4-1",
                                        "Remote": false
                                },
                                {
                                        "Name": "ceph",
                                        "Version": "15.2.14",
                                        "Remote": true
                                },
                                {
                                        "Name": "btrfs",
                                        "Version": "5.4.1",
                                        "Remote": false
                                },
                                {
                                        "Name": "cephfs",
                                        "Version": "15.2.14",
                                        "Remote": true
                                },
                                {
                                        "Name": "dir",
                                        "Version": "1",
                                        "Remote": false
                                },
                                {
                                        "Name": "lvm",
                                        "Version": "2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0",
                                        "Remote": false
                                }
                        ]
                }
        } 
DEBUG  [2022-05-08T06:46:57+01:00] Connecting to a remote LXD over HTTPS        
DEBUG  [2022-05-08T06:46:57+01:00] Sending request to LXD                        etag= method=GET url="https://10.203.0.1:8443/1.0"
DEBUG  [2022-05-08T06:46:57+01:00] Got response struct from LXD                 
DEBUG  [2022-05-08T06:46:57+01:00] 
        {
                "config": {
                        "core.https_address": "10.203.0.1"
                },
                "api_extensions": [
                        "storage_zfs_remove_snapshots",
                        "container_host_shutdown_timeout",
                        "container_stop_priority",
                        "container_syscall_filtering",
                        "auth_pki",
                        "container_last_used_at",
                        "etag",
                        "patch",
                        "usb_devices",
                        "https_allowed_credentials",
                        "image_compression_algorithm",
                        "directory_manipulation",
                        "container_cpu_time",
                        "storage_zfs_use_refquota",
                        "storage_lvm_mount_options",
                        "network",
                        "profile_usedby",
                        "container_push",
                        "container_exec_recording",
                        "certificate_update",
                        "container_exec_signal_handling",
                        "gpu_devices",
                        "container_image_properties",
                        "migration_progress",
                        "id_map",
                        "network_firewall_filtering",
                        "network_routes",
                        "storage",
                        "file_delete",
                        "file_append",
                        "network_dhcp_expiry",
                        "storage_lvm_vg_rename",
                        "storage_lvm_thinpool_rename",
                        "network_vlan",
                        "image_create_aliases",
                        "container_stateless_copy",
                        "container_only_migration",
                        "storage_zfs_clone_copy",
                        "unix_device_rename",
                        "storage_lvm_use_thinpool",
                        "storage_rsync_bwlimit",
                        "network_vxlan_interface",
                        "storage_btrfs_mount_options",
                        "entity_description",
                        "image_force_refresh",
                        "storage_lvm_lv_resizing",
                        "id_map_base",
                        "file_symlinks",
                        "container_push_target",
                        "network_vlan_physical",
                        "storage_images_delete",
                        "container_edit_metadata",
                        "container_snapshot_stateful_migration",
                        "storage_driver_ceph",
                        "storage_ceph_user_name",
                        "resource_limits",
                        "storage_volatile_initial_source",
                        "storage_ceph_force_osd_reuse",
                        "storage_block_filesystem_btrfs",
                        "resources",
                        "kernel_limits",
                        "storage_api_volume_rename",
                        "macaroon_authentication",
                        "network_sriov",
                        "console",
                        "restrict_devlxd",
                        "migration_pre_copy",
                        "infiniband",
                        "maas_network",
                        "devlxd_events",
                        "proxy",
                        "network_dhcp_gateway",
                        "file_get_symlink",
                        "network_leases",
                        "unix_device_hotplug",
                        "storage_api_local_volume_handling",
                        "operation_description",
                        "clustering",
                        "event_lifecycle",
                        "storage_api_remote_volume_handling",
                        "nvidia_runtime",
                        "container_mount_propagation",
                        "container_backup",
                        "devlxd_images",
                        "container_local_cross_pool_handling",
                        "proxy_unix",
                        "proxy_udp",
                        "clustering_join",
                        "proxy_tcp_udp_multi_port_handling",
                        "network_state",
                        "proxy_unix_dac_properties",
                        "container_protection_delete",
                        "unix_priv_drop",
                        "pprof_http",
                        "proxy_haproxy_protocol",
                        "network_hwaddr",
                        "proxy_nat",
                        "network_nat_order",
                        "container_full",
                        "candid_authentication",
                        "backup_compression",
                        "candid_config",
                        "nvidia_runtime_config",
                        "storage_api_volume_snapshots",
                        "storage_unmapped",
                        "projects",
                        "candid_config_key",
                        "network_vxlan_ttl",
                        "container_incremental_copy",
                        "usb_optional_vendorid",
                        "snapshot_scheduling",
                        "snapshot_schedule_aliases",
                        "container_copy_project",
                        "clustering_server_address",
                        "clustering_image_replication",
                        "container_protection_shift",
                        "snapshot_expiry",
                        "container_backup_override_pool",
                        "snapshot_expiry_creation",
                        "network_leases_location",
                        "resources_cpu_socket",
                        "resources_gpu",
                        "resources_numa",
                        "kernel_features",
                        "id_map_current",
                        "event_location",
                        "storage_api_remote_volume_snapshots",
                        "network_nat_address",
                        "container_nic_routes",
                        "rbac",
                        "cluster_internal_copy",
                        "seccomp_notify",
                        "lxc_features",
                        "container_nic_ipvlan",
                        "network_vlan_sriov",
                        "storage_cephfs",
                        "container_nic_ipfilter",
                        "resources_v2",
                        "container_exec_user_group_cwd",
                        "container_syscall_intercept",
                        "container_disk_shift",
                        "storage_shifted",
                        "resources_infiniband",
                        "daemon_storage",
                        "instances",
                        "image_types",
                        "resources_disk_sata",
                        "clustering_roles",
                        "images_expiry",
                        "resources_network_firmware",
                        "backup_compression_algorithm",
                        "ceph_data_pool_name",
                        "container_syscall_intercept_mount",
                        "compression_squashfs",
                        "container_raw_mount",
                        "container_nic_routed",
                        "container_syscall_intercept_mount_fuse",
                        "container_disk_ceph",
                        "virtual-machines",
                        "image_profiles",
                        "clustering_architecture",
                        "resources_disk_id",
                        "storage_lvm_stripes",
                        "vm_boot_priority",
                        "unix_hotplug_devices",
                        "api_filtering",
                        "instance_nic_network",
                        "clustering_sizing",
                        "firewall_driver",
                        "projects_limits",
                        "container_syscall_intercept_hugetlbfs",
                        "limits_hugepages",
                        "container_nic_routed_gateway",
                        "projects_restrictions",
                        "custom_volume_snapshot_expiry",
                        "volume_snapshot_scheduling",
                        "trust_ca_certificates",
                        "snapshot_disk_usage",
                        "clustering_edit_roles",
                        "container_nic_routed_host_address",
                        "container_nic_ipvlan_gateway",
                        "resources_usb_pci",
                        "resources_cpu_threads_numa",
                        "resources_cpu_core_die",
                        "api_os",
                        "container_nic_routed_host_table",
                        "container_nic_ipvlan_host_table",
                        "container_nic_ipvlan_mode",
                        "resources_system",
                        "images_push_relay",
                        "network_dns_search",
                        "container_nic_routed_limits",
                        "instance_nic_bridged_vlan",
                        "network_state_bond_bridge",
                        "usedby_consistency",
                        "custom_block_volumes",
                        "clustering_failure_domains",
                        "resources_gpu_mdev",
                        "console_vga_type",
                        "projects_limits_disk",
                        "network_type_macvlan",
                        "network_type_sriov",
                        "container_syscall_intercept_bpf_devices",
                        "network_type_ovn",
                        "projects_networks",
                        "projects_networks_restricted_uplinks",
                        "custom_volume_backup",
                        "backup_override_name",
                        "storage_rsync_compression",
                        "network_type_physical",
                        "network_ovn_external_subnets",
                        "network_ovn_nat",
                        "network_ovn_external_routes_remove",
                        "tpm_device_type",
                        "storage_zfs_clone_copy_rebase",
                        "gpu_mdev",
                        "resources_pci_iommu",
                        "resources_network_usb",
                        "resources_disk_address",
                        "network_physical_ovn_ingress_mode",
                        "network_ovn_dhcp",
                        "network_physical_routes_anycast",
                        "projects_limits_instances",
                        "network_state_vlan",
                        "instance_nic_bridged_port_isolation",
                        "instance_bulk_state_change",
                        "network_gvrp",
                        "instance_pool_move",
                        "gpu_sriov",
                        "pci_device_type",
                        "storage_volume_state",
                        "network_acl",
                        "migration_stateful",
                        "disk_state_quota",
                        "storage_ceph_features",
                        "projects_compression",
                        "projects_images_remote_cache_expiry",
                        "certificate_project",
                        "network_ovn_acl",
                        "projects_images_auto_update",
                        "projects_restricted_cluster_target",
                        "images_default_architecture",
                        "network_ovn_acl_defaults",
                        "gpu_mig",
                        "project_usage",
                        "network_bridge_acl",
                        "warnings",
                        "projects_restricted_backups_and_snapshots",
                        "clustering_join_token",
                        "clustering_description",
                        "server_trusted_proxy",
                        "clustering_update_cert",
                        "storage_api_project",
                        "server_instance_driver_operational",
                        "server_supported_storage_drivers",
                        "event_lifecycle_requestor_address",
                        "resources_gpu_usb",
                        "clustering_evacuation",
                        "network_ovn_nat_address",
                        "network_bgp",
                        "network_forward",
                        "custom_volume_refresh",
                        "network_counters_errors_dropped",
                        "metrics",
                        "image_source_project",
                        "clustering_config",
                        "network_peer",
                        "linux_sysctl",
                        "network_dns",
                        "ovn_nic_acceleration",
                        "certificate_self_renewal",
                        "instance_project_move",
                        "storage_volume_project_move",
                        "cloud_init",
                        "network_dns_nat",
                        "database_leader",
                        "instance_all_projects",
                        "clustering_groups",
                        "ceph_rbd_du",
                        "instance_get_full",
                        "qemu_metrics",
                        "gpu_mig_uuid",
                        "event_project",
                        "clustering_evacuation_live",
                        "instance_allow_inconsistent_copy",
                        "network_state_ovn",
                        "storage_volume_api_filtering",
                        "image_restrictions",
                        "storage_zfs_export",
                        "network_dns_records",
                        "storage_zfs_reserve_space",
                        "network_acl_log",
                        "storage_zfs_blocksize",
                        "metrics_cpu_seconds",
                        "instance_snapshot_never",
                        "certificate_token",
                        "instance_nic_routed_neighbor_probe",
                        "event_hub",
                        "agent_nic_config",
                        "projects_restricted_intercept",
                        "metrics_authentication",
                        "images_target_project",
                        "cluster_migration_inconsistent_copy",
                        "cluster_ovn_chassis",
                        "container_syscall_intercept_sched_setscheduler",
                        "storage_lvm_thinpool_metadata_size",
                        "storage_volume_state_total",
                        "instance_file_head",
                        "instances_nic_host_name",
                        "image_copy_profile",
                        "container_syscall_intercept_sysinfo",
                        "clustering_evacuation_mode"
                ],
                "api_status": "stable",
                "api_version": "1.0",
                "auth": "trusted",
                "public": false,
                "auth_methods": [
                        "tls"
                ],
                "environment": {
                        "addresses": [
                                "10.203.0.1:8443"
                        ],
                        "architectures": [
                                "x86_64",
                                "i686"
                        ],
                        "certificate": "-----BEGIN CERTIFICATE-----\nMIICAjCCAYmgAwIBAgIQN4N01EkRTWt3KmQdy46muDAKBggqhkjOPQQDAzA0MRww\nGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QGJhY2t1\ncDAeFw0yMjA1MDcyMjI2MzNaFw0zMjA1MDQyMjI2MzNaMDQxHDAaBgNVBAoTE2xp\nbnV4Y29udGFpbmVycy5vcmcxFDASBgNVBAMMC3Jvb3RAYmFja3VwMHYwEAYHKoZI\nzj0CAQYFK4EEACIDYgAERizSl/gpUEExmNc27xXKpHIlqQOOWNh0oNhj+4Bgf0Sf\nHV6/ggxBIKGxzZvwGp/shDU2PONlwnATOt2OUUGVVVlKcZLleP5DV9Ku82ty1Y8k\ntbRYE4nGq55NRLBBXMvno2AwXjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYI\nKwYBBQUHAwEwDAYDVR0TAQH/BAIwADApBgNVHREEIjAgggZiYWNrdXCHBH8AAAGH\nEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDZwAwZAIwUZ1NZCLyKl6mfWjg\nZdRl6Ga7UwXVxzaw+YVmHFvZdDDslJWDODpRfHktYOcwmarWAjAmHKRMdv6Z2Ro3\nQgYeHZ1DkVsK7XX9HKHmHuUNmmafeeVmyyuKiztkKSl1h/8ty5U=\n-----END CERTIFICATE-----\n",
                        "certificate_fingerprint": "2aca9ad8acf0160e3bc16302ae31515d1aeff4a6d3ad030bff619a49c58562ac",
                        "driver": "qemu | lxc",
                        "driver_version": "6.1.1 | 4.0.12",
                        "firewall": "nftables",
                        "kernel": "Linux",
                        "kernel_architecture": "x86_64",
                        "kernel_features": {
                                "idmapped_mounts": "true",
                                "netnsid_getifaddrs": "true",
                                "seccomp_listener": "true",
                                "seccomp_listener_continue": "true",
                                "shiftfs": "false",
                                "uevent_injection": "true",
                                "unpriv_fscaps": "true"
                        },
                        "kernel_version": "5.15.0-27-generic",
                        "lxc_features": {
                                "cgroup2": "true",
                                "core_scheduling": "true",
                                "devpts_fd": "true",
                                "idmapped_mounts_v2": "true",
                                "mount_injection_file": "true",
                                "network_gateway_device_route": "true",
                                "network_ipvlan": "true",
                                "network_l2proxy": "true",
                                "network_phys_macvlan_mtu": "true",
                                "network_veth_router": "true",
                                "pidfd": "true",
                                "seccomp_allow_deny_syntax": "true",
                                "seccomp_notify": "true",
                                "seccomp_proxy_send_notify_fd": "true"
                        },
                        "os_name": "Ubuntu",
                        "os_version": "22.04",
                        "project": "default",
                        "server": "lxd",
                        "server_clustered": false,
                        "server_event_mode": "full-mesh",
                        "server_name": "backup",
                        "server_pid": 1154,
                        "server_version": "5.1",
                        "storage": "zfs",
                        "storage_version": "2.1.2-1ubuntu3",
                        "storage_supported_drivers": [
                                {
                                        "Name": "btrfs",
                                        "Version": "5.4.1",
                                        "Remote": false
                                },
                                {
                                        "Name": "cephfs",
                                        "Version": "15.2.14",
                                        "Remote": true
                                },
                                {
                                        "Name": "dir",
                                        "Version": "1",
                                        "Remote": false
                                },
                                {
                                        "Name": "lvm",
                                        "Version": "2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0",
                                        "Remote": false
                                },
                                {
                                        "Name": "zfs",
                                        "Version": "2.1.2-1ubuntu3",
                                        "Remote": false
                                },
                                {
                                        "Name": "ceph",
                                        "Version": "15.2.14",
                                        "Remote": true
                                }
                        ]
                }
        } 
DEBUG  [2022-05-08T06:46:57+01:00] Sending request to LXD                        etag= method=GET url="http://unix.socket/1.0/images/aliases/ubuntu-minio-22.04"
DEBUG  [2022-05-08T06:46:57+01:00] Got response struct from LXD                 
DEBUG  [2022-05-08T06:46:57+01:00] 
        {
                "description": "",
                "target": "e53eb2fa269ee79775b38d6207796dd427826cd03babc3fd667b28cc6c3e14c0",
                "name": "ubuntu-minio-22.04",
                "type": "container"
        } 
DEBUG  [2022-05-08T06:46:57+01:00] Sending request to LXD                        etag= method=GET url="http://unix.socket/1.0/images/e53eb2fa269ee79775b38d6207796dd427826cd03babc3fd667b28cc6c3e14c0"
DEBUG  [2022-05-08T06:46:57+01:00] Got response struct from LXD                 
DEBUG  [2022-05-08T06:46:57+01:00] 
        {
                "auto_update": false,
                "properties": {
                        "description": "IAC image: ubuntu-minio-22.04"
                },
                "public": false,
                "expires_at": "2022-05-26T08:51:17+01:00",
                "profiles": [
                        "default"
                ],
                "aliases": [
                        {
                                "name": "ubuntu-minio-22.04",
                                "description": ""
                        }
                ],
                "architecture": "x86_64",
                "cached": false,
                "filename": "",
                "fingerprint": "e53eb2fa269ee79775b38d6207796dd427826cd03babc3fd667b28cc6c3e14c0",
                "size": 188141771,
                "type": "container",
                "created_at": "2022-05-07T17:34:21.514701392Z",
                "last_used_at": "2022-05-07T17:34:45.49451231Z",
                "uploaded_at": "2022-05-07T17:34:21.516895766Z"
        } 
DEBUG  [2022-05-08T06:46:57+01:00] Connected to the websocket: wss://10.203.0.1:8443/1.0/events 
DEBUG  [2022-05-08T06:46:57+01:00] Sending request to LXD                        etag= method=POST url="https://10.203.0.1:8443/1.0/images"
DEBUG  [2022-05-08T06:46:57+01:00] 
        {
                "auto_update": false,
                "properties": {
                        "description": "IAC image: ubuntu-minio-22.04"
                },
                "public": false,
                "expires_at": "2022-05-26T08:51:17+01:00",
                "profiles": null,
                "filename": "",
                "source": {
                        "alias": "",
                        "certificate": "",
                        "protocol": "",
                        "server": "",
                        "image_type": "",
                        "mode": "push",
                        "type": "",
                        "url": "",
                        "name": "",
                        "fingerprint": "e53eb2fa269ee79775b38d6207796dd427826cd03babc3fd667b28cc6c3e14c0",
                        "secret": "",
                        "project": ""
                },
                "compression_algorithm": "",
                "aliases": null
        } 
DEBUG  [2022-05-08T06:46:57+01:00] Got operation from LXD                       
DEBUG  [2022-05-08T06:46:57+01:00] 
        {
                "id": "452b81cf-ec74-4d9c-883e-29e299ac4fac",
                "class": "token",
                "description": "Image download token",
                "created_at": "2022-05-08T06:46:57.896860241+01:00",
                "updated_at": "2022-05-08T06:46:57.896860241+01:00",
                "status": "Running",
                "status_code": 103,
                "resources": {
                        "images": [
                                "/1.0/images/e53eb2fa269ee79775b38d6207796dd427826cd03babc3fd667b28cc6c3e14c0"
                        ]
                },
                "metadata": {
                        "aliases": null,
                        "expires_at": "2022-05-26T08:51:17+01:00",
                        "properties": {
                                "description": "IAC image: ubuntu-minio-22.04"
                        },
                        "public": false,
                        "secret": "fbc575da2c7bc2c17eb6fa742796997998146a9500d7db1c3699775bc681cbe8"
                },
                "may_cancel": true,
                "err": "",
                "location": "none"
        } 
DEBUG  [2022-05-08T06:46:57+01:00] Connected to the websocket: ws://unix.socket/1.0/events 
DEBUG  [2022-05-08T06:46:57+01:00] Sending request to LXD                        etag= method=POST url="http://unix.socket/1.0/images/e53eb2fa269ee79775b38d6207796dd427826cd03babc3fd667b28cc6c3e14c0/export"
DEBUG  [2022-05-08T06:46:57+01:00] 
        {
                "target": "https://10.203.0.1:8443",
                "secret": "fbc575da2c7bc2c17eb6fa742796997998146a9500d7db1c3699775bc681cbe8",
                "certificate": "-----BEGIN CERTIFICATE-----\nMIICAjCCAYmgAwIBAgIQN4N01EkRTWt3KmQdy46muDAKBggqhkjOPQQDAzA0MRww\nGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QGJhY2t1\ncDAeFw0yMjA1MDcyMjI2MzNaFw0zMjA1MDQyMjI2MzNaMDQxHDAaBgNVBAoTE2xp\nbnV4Y29udGFpbmVycy5vcmcxFDASBgNVBAMMC3Jvb3RAYmFja3VwMHYwEAYHKoZI\nzj0CAQYFK4EEACIDYgAERizSl/gpUEExmNc27xXKpHIlqQOOWNh0oNhj+4Bgf0Sf\nHV6/ggxBIKGxzZvwGp/shDU2PONlwnATOt2OUUGVVVlKcZLleP5DV9Ku82ty1Y8k\ntbRYE4nGq55NRLBBXMvno2AwXjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYI\nKwYBBQUHAwEwDAYDVR0TAQH/BAIwADApBgNVHREEIjAgggZiYWNrdXCHBH8AAAGH\nEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDZwAwZAIwUZ1NZCLyKl6mfWjg\nZdRl6Ga7UwXVxzaw+YVmHFvZdDDslJWDODpRfHktYOcwmarWAjAmHKRMdv6Z2Ro3\nQgYeHZ1DkVsK7XX9HKHmHuUNmmafeeVmyyuKiztkKSl1h/8ty5U=\n-----END CERTIFICATE-----\n",
                "aliases": [
                        {
                                "name": "ubuntu-minio-22.04",
                                "description": ""
                        }
                ],
                "project": "default",
                "profiles": null
        } 
DEBUG  [2022-05-08T06:46:58+01:00] Got operation from LXD                       
DEBUG  [2022-05-08T06:46:58+01:00] 
        {
                "id": "f3472697-cdb2-4489-9761-3f9ae2ca8d1f",
                "class": "task",
                "description": "Downloading image",
                "created_at": "2022-05-08T06:46:58.185805989+01:00",
                "updated_at": "2022-05-08T06:46:58.185805989+01:00",
                "status": "Running",
                "status_code": 103,
                "resources": null,
                "metadata": null,
                "may_cancel": false,
                "err": "",
                "location": "none"
        } 
DEBUG  [2022-05-08T06:46:58+01:00] Sending request to LXD                        etag= method=GET url="http://unix.socket/1.0/operations/f3472697-cdb2-4489-9761-3f9ae2ca8d1f"
DEBUG  [2022-05-08T06:46:58+01:00] Got response struct from LXD                 
DEBUG  [2022-05-08T06:46:58+01:00] 
        {
                "id": "f3472697-cdb2-4489-9761-3f9ae2ca8d1f",
                "class": "task",
                "description": "Downloading image",
                "created_at": "2022-05-08T06:46:58.185805989+01:00",
                "updated_at": "2022-05-08T06:46:58.185805989+01:00",
                "status": "Running",
                "status_code": 103,
                "resources": null,
                "metadata": null,
                "may_cancel": false,
                "err": "",
                "location": "none"
        } 
DEBUG  [2022-05-08T06:48:19+01:00] Sending request to LXD                        etag= method=DELETE url="https://10.203.0.1:8443/1.0/operations/452b81cf-ec74-4d9c-883e-29e299ac4fac"
Error: Post "https://10.203.0.1:8443/1.0/images?project=default": net/http: timeout awaiting response headers
tomponline commented 2 years ago

Does using relay mode help by the way? Ive tried reproducing using tc to simulate a slower network so the transfer takes over 30s and have been able to reproduce.

itoffshore commented 2 years ago

yes relay mode completes without errors

tomponline commented 2 years ago

Is it possible to get the output of 'lxc monitor --type=logging --pretty' on source and target servers when you run the command. I'd like to see where the delay is occurring.

itoffshore commented 2 years ago

SOURCE MONITOR:

DEBUG  [2022-05-09T13:39:21+01:00] Event listener server handler started         listener=6facecee-8fc1-42d7-9f39-0f7e4d3c15c9 local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-09T13:39:31+01:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=stuart
DEBUG  [2022-05-09T13:39:31+01:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/images/aliases/archlinux-base-current username=stuart
DEBUG  [2022-05-09T13:39:31+01:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/images/4d142148795f0eb41fd6f638c613e12ca5914c189127f918b763ab381947960b username=stuart
DEBUG  [2022-05-09T13:39:32+01:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/events username=stuart
DEBUG  [2022-05-09T13:39:32+01:00] Event listener server handler started         listener=9bfd2080-d2a0-4f0b-981c-e4490723c241 local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-09T13:39:32+01:00] Handling API request                          ip=@ method=POST protocol=unix url=/1.0/images/4d142148795f0eb41fd6f638c613e12ca5914c189127f918b763ab381947960b/export username=stuart
DEBUG  [2022-05-09T13:39:32+01:00] Connecting to a remote LXD over HTTPS        
DEBUG  [2022-05-09T13:39:32+01:00] Sending request to LXD                        etag= method=GET url="https://10.203.0.1:8443/1.0"
DEBUG  [2022-05-09T13:39:32+01:00] Got response struct from LXD                 
DEBUG  [2022-05-09T13:39:32+01:00] 
        {
                "config": null,
                "api_extensions": [
                        "storage_zfs_remove_snapshots",
                        "container_host_shutdown_timeout",
                        "container_stop_priority",
                        "container_syscall_filtering",
                        "auth_pki",
                        "container_last_used_at",
                        "etag",
                        "patch",
                        "usb_devices",
                        "https_allowed_credentials",
                        "image_compression_algorithm",
                        "directory_manipulation",
                        "container_cpu_time",
                        "storage_zfs_use_refquota",
                        "storage_lvm_mount_options",
                        "network",
                        "profile_usedby",
                        "container_push",
                        "container_exec_recording",
                        "certificate_update",
                        "container_exec_signal_handling",
                        "gpu_devices",
                        "container_image_properties",
                        "migration_progress",
                        "id_map",
                        "network_firewall_filtering",
                        "network_routes",
                        "storage",
                        "file_delete",
                        "file_append",
                        "network_dhcp_expiry",
                        "storage_lvm_vg_rename",
                        "storage_lvm_thinpool_rename",
                        "network_vlan",
                        "image_create_aliases",
                        "container_stateless_copy",
                        "container_only_migration",
                        "storage_zfs_clone_copy",
                        "unix_device_rename",
                        "storage_lvm_use_thinpool",
                        "storage_rsync_bwlimit",
                        "network_vxlan_interface",
                        "storage_btrfs_mount_options",
                        "entity_description",
                        "image_force_refresh",
                        "storage_lvm_lv_resizing",
                        "id_map_base",
                        "file_symlinks",
                        "container_push_target",
                        "network_vlan_physical",
                        "storage_images_delete",
                        "container_edit_metadata",
                        "container_snapshot_stateful_migration",
                        "storage_driver_ceph",
                        "storage_ceph_user_name",
                        "resource_limits",
                        "storage_volatile_initial_source",
                        "storage_ceph_force_osd_reuse",
                        "storage_block_filesystem_btrfs",
                        "resources",
                        "kernel_limits",
                        "storage_api_volume_rename",
                        "macaroon_authentication",
                        "network_sriov",
                        "console",
                        "restrict_devlxd",
                        "migration_pre_copy",
                        "infiniband",
                        "maas_network",
                        "devlxd_events",
                        "proxy",
                        "network_dhcp_gateway",
                        "file_get_symlink",
                        "network_leases",
                        "unix_device_hotplug",
                        "storage_api_local_volume_handling",
                        "operation_description",
                        "clustering",
                        "event_lifecycle",
                        "storage_api_remote_volume_handling",
                        "nvidia_runtime",
                        "container_mount_propagation",
                        "container_backup",
                        "devlxd_images",
                        "container_local_cross_pool_handling",
                        "proxy_unix",
                        "proxy_udp",
                        "clustering_join",
                        "proxy_tcp_udp_multi_port_handling",
                        "network_state",
                        "proxy_unix_dac_properties",
                        "container_protection_delete",
                        "unix_priv_drop",
                        "pprof_http",
                        "proxy_haproxy_protocol",
                        "network_hwaddr",
                        "proxy_nat",
                        "network_nat_order",
                        "container_full",
                        "candid_authentication",
                        "backup_compression",
                        "candid_config",
                        "nvidia_runtime_config",
                        "storage_api_volume_snapshots",
                        "storage_unmapped",
                        "projects",
                        "candid_config_key",
                        "network_vxlan_ttl",
                        "container_incremental_copy",
                        "usb_optional_vendorid",
                        "snapshot_scheduling",
                        "snapshot_schedule_aliases",
                        "container_copy_project",
                        "clustering_server_address",
                        "clustering_image_replication",
                        "container_protection_shift",
                        "snapshot_expiry",
                        "container_backup_override_pool",
                        "snapshot_expiry_creation",
                        "network_leases_location",
                        "resources_cpu_socket",
                        "resources_gpu",
                        "resources_numa",
                        "kernel_features",
                        "id_map_current",
                        "event_location",
                        "storage_api_remote_volume_snapshots",
                        "network_nat_address",
                        "container_nic_routes",
                        "rbac",
                        "cluster_internal_copy",
                        "seccomp_notify",
                        "lxc_features",
                        "container_nic_ipvlan",
                        "network_vlan_sriov",
                        "storage_cephfs",
                        "container_nic_ipfilter",
                        "resources_v2",
                        "container_exec_user_group_cwd",
                        "container_syscall_intercept",
                        "container_disk_shift",
                        "storage_shifted",
                        "resources_infiniband",
                        "daemon_storage",
                        "instances",
                        "image_types",
                        "resources_disk_sata",
                        "clustering_roles",
                        "images_expiry",
                        "resources_network_firmware",
                        "backup_compression_algorithm",
                        "ceph_data_pool_name",
                        "container_syscall_intercept_mount",
                        "compression_squashfs",
                        "container_raw_mount",
                        "container_nic_routed",
                        "container_syscall_intercept_mount_fuse",
                        "container_disk_ceph",
                        "virtual-machines",
                        "image_profiles",
                        "clustering_architecture",
                        "resources_disk_id",
                        "storage_lvm_stripes",
                        "vm_boot_priority",
                        "unix_hotplug_devices",
                        "api_filtering",
                        "instance_nic_network",
                        "clustering_sizing",
                        "firewall_driver",
                        "projects_limits",
                        "container_syscall_intercept_hugetlbfs",
                        "limits_hugepages",
                        "container_nic_routed_gateway",
                        "projects_restrictions",
                        "custom_volume_snapshot_expiry",
                        "volume_snapshot_scheduling",
                        "trust_ca_certificates",
                        "snapshot_disk_usage",
                        "clustering_edit_roles",
                        "container_nic_routed_host_address",
                        "container_nic_ipvlan_gateway",
                        "resources_usb_pci",
                        "resources_cpu_threads_numa",
                        "resources_cpu_core_die",
                        "api_os",
                        "container_nic_routed_host_table",
                        "container_nic_ipvlan_host_table",
                        "container_nic_ipvlan_mode",
                        "resources_system",
                        "images_push_relay",
                        "network_dns_search",
                        "container_nic_routed_limits",
                        "instance_nic_bridged_vlan",
                        "network_state_bond_bridge",
                        "usedby_consistency",
                        "custom_block_volumes",
                        "clustering_failure_domains",
                        "resources_gpu_mdev",
                        "console_vga_type",
                        "projects_limits_disk",
                        "network_type_macvlan",
                        "network_type_sriov",
                        "container_syscall_intercept_bpf_devices",
                        "network_type_ovn",
                        "projects_networks",
                        "projects_networks_restricted_uplinks",
                        "custom_volume_backup",
                        "backup_override_name",
                        "storage_rsync_compression",
                        "network_type_physical",
                        "network_ovn_external_subnets",
                        "network_ovn_nat",
                        "network_ovn_external_routes_remove",
                        "tpm_device_type",
                        "storage_zfs_clone_copy_rebase",
                        "gpu_mdev",
                        "resources_pci_iommu",
                        "resources_network_usb",
                        "resources_disk_address",
                        "network_physical_ovn_ingress_mode",
                        "network_ovn_dhcp",
                        "network_physical_routes_anycast",
                        "projects_limits_instances",
                        "network_state_vlan",
                        "instance_nic_bridged_port_isolation",
                        "instance_bulk_state_change",
                        "network_gvrp",
                        "instance_pool_move",
                        "gpu_sriov",
                        "pci_device_type",
                        "storage_volume_state",
                        "network_acl",
                        "migration_stateful",
                        "disk_state_quota",
                        "storage_ceph_features",
                        "projects_compression",
                        "projects_images_remote_cache_expiry",
                        "certificate_project",
                        "network_ovn_acl",
                        "projects_images_auto_update",
                        "projects_restricted_cluster_target",
                        "images_default_architecture",
                        "network_ovn_acl_defaults",
                        "gpu_mig",
                        "project_usage",
                        "network_bridge_acl",
                        "warnings",
                        "projects_restricted_backups_and_snapshots",
                        "clustering_join_token",
                        "clustering_description",
                        "server_trusted_proxy",
                        "clustering_update_cert",
                        "storage_api_project",
                        "server_instance_driver_operational",
                        "server_supported_storage_drivers",
                        "event_lifecycle_requestor_address",
                        "resources_gpu_usb",
                        "clustering_evacuation",
                        "network_ovn_nat_address",
                        "network_bgp",
                        "network_forward",
                        "custom_volume_refresh",
                        "network_counters_errors_dropped",
                        "metrics",
                        "image_source_project",
                        "clustering_config",
                        "network_peer",
                        "linux_sysctl",
                        "network_dns",
                        "ovn_nic_acceleration",
                        "certificate_self_renewal",
                        "instance_project_move",
                        "storage_volume_project_move",
                        "cloud_init",
                        "network_dns_nat",
                        "database_leader",
                        "instance_all_projects",
                        "clustering_groups",
                        "ceph_rbd_du",
                        "instance_get_full",
                        "qemu_metrics",
                        "gpu_mig_uuid",
                        "event_project",
                        "clustering_evacuation_live",
                        "instance_allow_inconsistent_copy",
                        "network_state_ovn",
                        "storage_volume_api_filtering",
                        "image_restrictions",
                        "storage_zfs_export",
                        "network_dns_records",
                        "storage_zfs_reserve_space",
                        "network_acl_log",
                        "storage_zfs_blocksize",
                        "metrics_cpu_seconds",
                        "instance_snapshot_never",
                        "certificate_token",
                        "instance_nic_routed_neighbor_probe",
                        "event_hub",
                        "agent_nic_config",
                        "projects_restricted_intercept",
                        "metrics_authentication",
                        "images_target_project",
                        "cluster_migration_inconsistent_copy",
                        "cluster_ovn_chassis",
                        "container_syscall_intercept_sched_setscheduler",
                        "storage_lvm_thinpool_metadata_size",
                        "storage_volume_state_total",
                        "instance_file_head",
                        "instances_nic_host_name",
                        "image_copy_profile",
                        "container_syscall_intercept_sysinfo",
                        "clustering_evacuation_mode"
                ],
                "api_status": "stable",
                "api_version": "1.0",
                "auth": "untrusted",
                "public": false,
                "auth_methods": [
                        "tls"
                ],
                "environment": {
                        "addresses": null,
                        "architectures": null,
                        "certificate": "",
                        "certificate_fingerprint": "",
                        "driver": "",
                        "driver_version": "",
                        "firewall": "",
                        "kernel": "",
                        "kernel_architecture": "",
                        "kernel_features": null,
                        "kernel_version": "",
                        "lxc_features": null,
                        "os_name": "",
                        "os_version": "",
                        "project": "",
                        "server": "",
                        "server_clustered": false,
                        "server_event_mode": "",
                        "server_name": "",
                        "server_pid": 0,
                        "server_version": "",
                        "storage": "",
                        "storage_version": "",
                        "storage_supported_drivers": null
                }
        } 
DEBUG  [2022-05-09T13:39:32+01:00] New task Operation: 2313a3e6-77a1-4270-95c2-4483f85caa4e 
DEBUG  [2022-05-09T13:39:32+01:00] Started task operation: 2313a3e6-77a1-4270-95c2-4483f85caa4e 
DEBUG  [2022-05-09T13:39:32+01:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/operations/2313a3e6-77a1-4270-95c2-4483f85caa4e username=stuart
DEBUG  [2022-05-09T13:43:12+01:00] Failure for task operation: 2313a3e6-77a1-4270-95c2-4483f85caa4e: Post "https://10.203.0.1:8443/1.0/images?project=default": net/http: timeout awaiting response headers 
DEBUG  [2022-05-09T13:43:13+01:00] Event listener server handler stopped         listener=9bfd2080-d2a0-4f0b-981c-e4490723c241 local=/var/snap/lxd/common/lxd/unix.socket remote=@

DESTINATION MONITOR:

DEBUG  [2022-05-09T13:39:24+01:00] Event listener server handler started         listener=06f44bbc-1d78-46f0-af18-20369e26bf10 local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-09T13:39:31+01:00] Matched trusted cert                          fingerprint=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40 subject="CN=stuart@endeavour,O=linuxcontainers.org"
DEBUG  [2022-05-09T13:39:31+01:00] Matched trusted cert                          fingerprint=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40 subject="CN=stuart@endeavour,O=linuxcontainers.org"
DEBUG  [2022-05-09T13:39:31+01:00] Handling API request                          ip="10.0.0.1:37056" method=GET protocol=tls url=/1.0 username=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40
DEBUG  [2022-05-09T13:39:31+01:00] Matched trusted cert                          fingerprint=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40 subject="CN=stuart@endeavour,O=linuxcontainers.org"
DEBUG  [2022-05-09T13:39:31+01:00] Handling API request                          ip="10.0.0.1:37058" method=GET protocol=tls url=/1.0/events username=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40
DEBUG  [2022-05-09T13:39:31+01:00] Event listener server handler started         listener=4b20a751-b8fa-4b9e-b721-295a62d577d8 local="10.203.0.1:8443" remote="10.0.0.1:37058"
DEBUG  [2022-05-09T13:39:32+01:00] Matched trusted cert                          fingerprint=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40 subject="CN=stuart@endeavour,O=linuxcontainers.org"
DEBUG  [2022-05-09T13:39:32+01:00] Handling API request                          ip="10.0.0.1:37060" method=POST protocol=tls url=/1.0/images username=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40
DEBUG  [2022-05-09T13:39:32+01:00] Matched trusted cert                          fingerprint=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40 subject="CN=stuart@endeavour,O=linuxcontainers.org"
DEBUG  [2022-05-09T13:39:32+01:00] New token Operation: 61a3897d-c899-476c-85db-3ceee2c14e15 
DEBUG  [2022-05-09T13:39:32+01:00] Started token operation: 61a3897d-c899-476c-85db-3ceee2c14e15 
DEBUG  [2022-05-09T13:39:32+01:00] Allowing untrusted GET                        ip="10.0.0.1:37062" url=/1.0
DEBUG  [2022-05-09T13:39:32+01:00] Allowing untrusted POST                       ip="10.0.0.1:37064" url="/1.0/images?project=default"
DEBUG  [2022-05-09T13:39:32+01:00] Cancelling operation                          class=token operation=61a3897d-c899-476c-85db-3ceee2c14e15
DEBUG  [2022-05-09T13:39:32+01:00] Cancelled operation                           class=token operation=61a3897d-c899-476c-85db-3ceee2c14e15
DEBUG  [2022-05-09T13:43:13+01:00] Matched trusted cert                          fingerprint=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40 subject="CN=stuart@endeavour,O=linuxcontainers.org"
DEBUG  [2022-05-09T13:43:13+01:00] Handling API request                          ip="10.0.0.1:37066" method=DELETE protocol=tls url=/1.0/operations/61a3897d-c899-476c-85db-3ceee2c14e15 username=8d386358c1310194d00b6e53d1572e643fb400415fe8f5bc7a3fe3c0da9ece40
DEBUG  [2022-05-09T13:43:13+01:00] Database error                                err="Operation not found"
DEBUG  [2022-05-09T13:43:13+01:00] Event listener server handler stopped         listener=4b20a751-b8fa-4b9e-b721-295a62d577d8 local="10.203.0.1:8443" remote="10.0.0.1:37058"

10.0.0.1 is the source wireguard interface (with routing rules in /etc/systemd/network/99-wg0.network) 10.203.0.0/24 is the destination LXD subnet 10.201.0.0/24 is the source LXD subnet

Only trusted ip's are allowed to connect to the remote wireguard (allowed at the ingress hook level in nftables so there shouldn't be any mangling by the firewall)

gstanden commented 2 years ago

Hi, I'm seeing some issues with similar errors that I've never had before when creating lxd container on Fedora 34. Started in the last few days: "timeout awaiting response headers"

Creating oel84d10 Error: Failed instance creation: Post "http://unix.socket/1.0/instances": net/http: timeout awaiting response headers Error: Instance not found Error: Failed to validate "local:oel84d10": Instance not found Creating oel84d10 Error: Failed instance creation: Post "http://unix.socket/1.0/instances": net/http: timeout awaiting response headers Error: Instance not found Error: Failed to validate "local:oel84d10": Instance not found Creating oel84d10 Error: Failed instance creation: Post "http://unix.socket/1.0/instances": net/http: timeout awaiting response headers Error: Instance not found Error: Failed to validate "local:oel84d10": Instance not found Creating oel84d10 Error: Failed instance creation: Post "http://unix.socket/1.0/instances": net/http: timeout awaiting response headers Error: Instance not found Error: Failed to validate "local:oel84d10": Instance not found Creating oel84d10

tomponline commented 2 years ago

@gstanden can you rerun command with --debug and paste output please.

gstanden commented 2 years ago 
[orabuntu@f34sv1 ~]$ lxc launch -p olxc_sx1a images:oracle/$MajorRelease/amd64 oel$OracleRelease$SeedPostfix --debug
DEBUG  [2022-05-09T20:03:51-05:00] Connecting to a local LXD over a Unix socket 
DEBUG  [2022-05-09T20:03:51-05:00] Sending request to LXD                        etag= method=GET url="http://unix.socket/1.0"
DEBUG  [2022-05-09T20:03:51-05:00] Got response struct from LXD                 
DEBUG  [2022-05-09T20:03:51-05:00] 
    {
        "config": {
            "cluster.https_address": "10.209.53.1:8443",
            "core.https_address": "10.209.53.1:8443",
            "core.trust_password": true
        },
        "api_extensions": [
            "storage_zfs_remove_snapshots",
            "container_host_shutdown_timeout",
            "container_stop_priority",
            "container_syscall_filtering",
            "auth_pki",
            "container_last_used_at",
            "etag",
            "patch",
            "usb_devices",
            "https_allowed_credentials",
            "image_compression_algorithm",
            "directory_manipulation",
            "container_cpu_time",
            "storage_zfs_use_refquota",
            "storage_lvm_mount_options",
            "network",
            "profile_usedby",
            "container_push",
            "container_exec_recording",
            "certificate_update",
            "container_exec_signal_handling",
            "gpu_devices",
            "container_image_properties",
            "migration_progress",
            "id_map",
            "network_firewall_filtering",
            "network_routes",
            "storage",
            "file_delete",
            "file_append",
            "network_dhcp_expiry",
            "storage_lvm_vg_rename",
            "storage_lvm_thinpool_rename",
            "network_vlan",
            "image_create_aliases",
            "container_stateless_copy",
            "container_only_migration",
            "storage_zfs_clone_copy",
            "unix_device_rename",
            "storage_lvm_use_thinpool",
            "storage_rsync_bwlimit",
            "network_vxlan_interface",
            "storage_btrfs_mount_options",
            "entity_description",
            "image_force_refresh",
            "storage_lvm_lv_resizing",
            "id_map_base",
            "file_symlinks",
            "container_push_target",
            "network_vlan_physical",
            "storage_images_delete",
            "container_edit_metadata",
            "container_snapshot_stateful_migration",
            "storage_driver_ceph",
            "storage_ceph_user_name",
            "resource_limits",
            "storage_volatile_initial_source",
            "storage_ceph_force_osd_reuse",
            "storage_block_filesystem_btrfs",
            "resources",
            "kernel_limits",
            "storage_api_volume_rename",
            "macaroon_authentication",
            "network_sriov",
            "console",
            "restrict_devlxd",
            "migration_pre_copy",
            "infiniband",
            "maas_network",
            "devlxd_events",
            "proxy",
            "network_dhcp_gateway",
            "file_get_symlink",
            "network_leases",
            "unix_device_hotplug",
            "storage_api_local_volume_handling",
            "operation_description",
            "clustering",
            "event_lifecycle",
            "storage_api_remote_volume_handling",
            "nvidia_runtime",
            "container_mount_propagation",
            "container_backup",
            "devlxd_images",
            "container_local_cross_pool_handling",
            "proxy_unix",
            "proxy_udp",
            "clustering_join",
            "proxy_tcp_udp_multi_port_handling",
            "network_state",
            "proxy_unix_dac_properties",
            "container_protection_delete",
            "unix_priv_drop",
            "pprof_http",
            "proxy_haproxy_protocol",
            "network_hwaddr",
            "proxy_nat",
            "network_nat_order",
            "container_full",
            "candid_authentication",
            "backup_compression",
            "candid_config",
            "nvidia_runtime_config",
            "storage_api_volume_snapshots",
            "storage_unmapped",
            "projects",
            "candid_config_key",
            "network_vxlan_ttl",
            "container_incremental_copy",
            "usb_optional_vendorid",
            "snapshot_scheduling",
            "snapshot_schedule_aliases",
            "container_copy_project",
            "clustering_server_address",
            "clustering_image_replication",
            "container_protection_shift",
            "snapshot_expiry",
            "container_backup_override_pool",
            "snapshot_expiry_creation",
            "network_leases_location",
            "resources_cpu_socket",
            "resources_gpu",
            "resources_numa",
            "kernel_features",
            "id_map_current",
            "event_location",
            "storage_api_remote_volume_snapshots",
            "network_nat_address",
            "container_nic_routes",
            "rbac",
            "cluster_internal_copy",
            "seccomp_notify",
            "lxc_features",
            "container_nic_ipvlan",
            "network_vlan_sriov",
            "storage_cephfs",
            "container_nic_ipfilter",
            "resources_v2",
            "container_exec_user_group_cwd",
            "container_syscall_intercept",
            "container_disk_shift",
            "storage_shifted",
            "resources_infiniband",
            "daemon_storage",
            "instances",
            "image_types",
            "resources_disk_sata",
            "clustering_roles",
            "images_expiry",
            "resources_network_firmware",
            "backup_compression_algorithm",
            "ceph_data_pool_name",
            "container_syscall_intercept_mount",
            "compression_squashfs",
            "container_raw_mount",
            "container_nic_routed",
            "container_syscall_intercept_mount_fuse",
            "container_disk_ceph",
            "virtual-machines",
            "image_profiles",
            "clustering_architecture",
            "resources_disk_id",
            "storage_lvm_stripes",
            "vm_boot_priority",
            "unix_hotplug_devices",
            "api_filtering",
            "instance_nic_network",
            "clustering_sizing",
            "firewall_driver",
            "projects_limits",
            "container_syscall_intercept_hugetlbfs",
            "limits_hugepages",
            "container_nic_routed_gateway",
            "projects_restrictions",
            "custom_volume_snapshot_expiry",
            "volume_snapshot_scheduling",
            "trust_ca_certificates",
            "snapshot_disk_usage",
            "clustering_edit_roles",
            "container_nic_routed_host_address",
            "container_nic_ipvlan_gateway",
            "resources_usb_pci",
            "resources_cpu_threads_numa",
            "resources_cpu_core_die",
            "api_os",
            "container_nic_routed_host_table",
            "container_nic_ipvlan_host_table",
            "container_nic_ipvlan_mode",
            "resources_system",
            "images_push_relay",
            "network_dns_search",
            "container_nic_routed_limits",
            "instance_nic_bridged_vlan",
            "network_state_bond_bridge",
            "usedby_consistency",
            "custom_block_volumes",
            "clustering_failure_domains",
            "resources_gpu_mdev",
            "console_vga_type",
            "projects_limits_disk",
            "network_type_macvlan",
            "network_type_sriov",
            "container_syscall_intercept_bpf_devices",
            "network_type_ovn",
            "projects_networks",
            "projects_networks_restricted_uplinks",
            "custom_volume_backup",
            "backup_override_name",
            "storage_rsync_compression",
            "network_type_physical",
            "network_ovn_external_subnets",
            "network_ovn_nat",
            "network_ovn_external_routes_remove",
            "tpm_device_type",
            "storage_zfs_clone_copy_rebase",
            "gpu_mdev",
            "resources_pci_iommu",
            "resources_network_usb",
            "resources_disk_address",
            "network_physical_ovn_ingress_mode",
            "network_ovn_dhcp",
            "network_physical_routes_anycast",
            "projects_limits_instances",
            "network_state_vlan",
            "instance_nic_bridged_port_isolation",
            "instance_bulk_state_change",
            "network_gvrp",
            "instance_pool_move",
            "gpu_sriov",
            "pci_device_type",
            "storage_volume_state",
            "network_acl",
            "migration_stateful",
            "disk_state_quota",
            "storage_ceph_features",
            "projects_compression",
            "projects_images_remote_cache_expiry",
            "certificate_project",
            "network_ovn_acl",
            "projects_images_auto_update",
            "projects_restricted_cluster_target",
            "images_default_architecture",
            "network_ovn_acl_defaults",
            "gpu_mig",
            "project_usage",
            "network_bridge_acl",
            "warnings",
            "projects_restricted_backups_and_snapshots",
            "clustering_join_token",
            "clustering_description",
            "server_trusted_proxy",
            "clustering_update_cert",
            "storage_api_project",
            "server_instance_driver_operational",
            "server_supported_storage_drivers",
            "event_lifecycle_requestor_address",
            "resources_gpu_usb",
            "clustering_evacuation",
            "network_ovn_nat_address",
            "network_bgp",
            "network_forward",
            "custom_volume_refresh",
            "network_counters_errors_dropped",
            "metrics",
            "image_source_project",
            "clustering_config",
            "network_peer",
            "linux_sysctl",
            "network_dns",
            "ovn_nic_acceleration",
            "certificate_self_renewal",
            "instance_project_move",
            "storage_volume_project_move",
            "cloud_init",
            "network_dns_nat",
            "database_leader",
            "instance_all_projects",
            "clustering_groups",
            "ceph_rbd_du",
            "instance_get_full",
            "qemu_metrics",
            "gpu_mig_uuid",
            "event_project",
            "clustering_evacuation_live",
            "instance_allow_inconsistent_copy",
            "network_state_ovn",
            "storage_volume_api_filtering",
            "image_restrictions",
            "storage_zfs_export",
            "network_dns_records",
            "storage_zfs_reserve_space",
            "network_acl_log",
            "storage_zfs_blocksize",
            "metrics_cpu_seconds",
            "instance_snapshot_never",
            "certificate_token",
            "instance_nic_routed_neighbor_probe",
            "event_hub",
            "agent_nic_config",
            "projects_restricted_intercept",
            "metrics_authentication",
            "images_target_project",
            "cluster_migration_inconsistent_copy",
            "cluster_ovn_chassis",
            "container_syscall_intercept_sched_setscheduler",
            "storage_lvm_thinpool_metadata_size",
            "storage_volume_state_total",
            "instance_file_head",
            "instances_nic_host_name",
            "image_copy_profile",
            "container_syscall_intercept_sysinfo",
            "clustering_evacuation_mode"
        ],
        "api_status": "stable",
        "api_version": "1.0",
        "auth": "trusted",
        "public": false,
        "auth_methods": [
            "tls"
        ],
        "environment": {
            "addresses": [
                "10.209.53.1:8443"
            ],
            "architectures": [
                "x86_64",
                "i686"
            ],
            "certificate": "-----BEGIN CERTIFICATE-----\nMIICAzCCAYmgAwIBAgIQW1WMY5wWjGGh84OVrZUNxTAKBggqhkjOPQQDAzA0MRww\nGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QGYzNHN2\nMTAeFw0yMjA1MDkyMjI0NDhaFw0zMjA1MDYyMjI0NDhaMDQxHDAaBgNVBAoTE2xp\nbnV4Y29udGFpbmVycy5vcmcxFDASBgNVBAMMC3Jvb3RAZjM0c3YxMHYwEAYHKoZI\nzj0CAQYFK4EEACIDYgAE4p63DAbe7w7iED5wf+Ra54b9WEStQbvB/bwnSq9tXOiz\nqIVoqWxxxAho8HHASCVAVMdkY0vVGtffqiFAe7de+zH+FFGKZysqHNXoigDOtTVO\nHY3E5IMAMCfG4YNesffao2AwXjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYI\nKwYBBQUHAwEwDAYDVR0TAQH/BAIwADApBgNVHREEIjAgggZmMzRzdjGHBH8AAAGH\nEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDaAAwZQIwErX8iCV4YL6/gLyA\nSCDaVqTkge0PPsvlHVCgrzes569GZlPDndA9W9X3yDItHcAiAjEAkbMZOaB61PfR\nwAbUZ7YbNbVw8FSFGXMfVNdowhwyTod+oz+CstDbGd5QHnDHgHW/\n-----END CERTIFICATE-----\n",
            "certificate_fingerprint": "4797e3981caf2829d8c7be9fc1768f5ebc6d7bfc49e6bacd3fe20fcc678c07f3",
            "driver": "lxc",
            "driver_version": "4.0.12",
            "firewall": "nftables",
            "kernel": "Linux",
            "kernel_architecture": "x86_64",
            "kernel_features": {
                "idmapped_mounts": "true",
                "netnsid_getifaddrs": "true",
                "seccomp_listener": "true",
                "seccomp_listener_continue": "true",
                "shiftfs": "false",
                "uevent_injection": "true",
                "unpriv_fscaps": "true"
            },
            "kernel_version": "5.17.5-100.fc34.x86_64",
            "lxc_features": {
                "cgroup2": "true",
                "core_scheduling": "true",
                "devpts_fd": "true",
                "idmapped_mounts_v2": "true",
                "mount_injection_file": "true",
                "network_gateway_device_route": "true",
                "network_ipvlan": "true",
                "network_l2proxy": "true",
                "network_phys_macvlan_mtu": "true",
                "network_veth_router": "true",
                "pidfd": "true",
                "seccomp_allow_deny_syntax": "true",
                "seccomp_notify": "true",
                "seccomp_proxy_send_notify_fd": "true"
            },
            "os_name": "Fedora",
            "os_version": "34",
            "project": "default",
            "server": "lxd",
            "server_clustered": true,
            "server_event_mode": "full-mesh",
            "server_name": "f34sv1",
            "server_pid": 160714,
            "server_version": "5.1",
            "storage": "zfs | dir",
            "storage_version": "2.1.4-1 | 1",
            "storage_supported_drivers": [
                {
                    "Name": "zfs",
                    "Version": "2.1.4-1",
                    "Remote": false
                },
                {
                    "Name": "ceph",
                    "Version": "15.2.14",
                    "Remote": true
                },
                {
                    "Name": "btrfs",
                    "Version": "5.4.1",
                    "Remote": false
                },
                {
                    "Name": "cephfs",
                    "Version": "15.2.14",
                    "Remote": true
                },
                {
                    "Name": "dir",
                    "Version": "1",
                    "Remote": false
                },
                {
                    "Name": "lvm",
                    "Version": "2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0",
                    "Remote": false
                }
            ]
        }
    } 
Creating oel
DEBUG  [2022-05-09T20:03:51-05:00] Connecting to a remote simplestreams server   URL="https://images.linuxcontainers.org"
DEBUG  [2022-05-09T20:03:51-05:00] Connected to the websocket: ws://unix.socket/1.0/events 
DEBUG  [2022-05-09T20:03:51-05:00] Sending request to LXD                        etag= method=POST url="http://unix.socket/1.0/instances"
DEBUG  [2022-05-09T20:03:51-05:00] 
    {
        "architecture": "",
        "config": {},
        "devices": {},
        "ephemeral": false,
        "profiles": [
            "olxc_sx1a"
        ],
        "stateful": false,
        "description": "",
        "name": "oel",
        "source": {
            "type": "image",
            "certificate": "",
            "alias": "oracle//amd64",
            "server": "https://images.linuxcontainers.org",
            "protocol": "simplestreams",
            "mode": "pull",
            "allow_inconsistent": false
        },
        "instance_type": "",
        "type": "container"
    } 
Error: Failed instance creation: Post "http://unix.socket/1.0/instances": net/http: timeout awaiting response headers
[orabuntu@f34sv1 ~]$
tomponline commented 2 years ago

Thanks @gstanden can you also provide output of lxc monitor --type=logging --pretty on the server when you're running that command. I'm trying to see where the hold up in response headers are. Thanks

tomponline commented 2 years ago

Trying to reproduce now.

tomponline commented 2 years ago

@gstanden is it possible to get access to this server?

tomponline commented 2 years ago

@itoffshore is it possible to get access to this server for testing?

gstanden commented 2 years ago

@tomponline here you go ... also, it worked fine this time! of course - it always does when it "knows" we are watching :) ... did you make any changes ... if so they seem to be working. As before, this is the same Fedora 34 server (VM).

[orabuntu@f34sv1 ~]$ lxc monitor --type=logging --pretty
DEBUG  [2022-05-10T08:41:23-05:00] Event listener server handler started         listener=28f116df-c67a-48a6-bd00-b940aeb74256 local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-10T08:41:30-05:00] Starting heartbeat round                      local="10.209.53.1:8443" mode=normal
DEBUG  [2022-05-10T08:41:30-05:00] Heartbeat updating local raft members         members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
DEBUG  [2022-05-10T08:41:30-05:00] Completed heartbeat round                     duration=28.118492ms local="10.209.53.1:8443"
DEBUG  [2022-05-10T08:41:33-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
DEBUG  [2022-05-10T08:41:33-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/events username=orabuntu
DEBUG  [2022-05-10T08:41:33-05:00] Event listener server handler started         listener=ed32291c-37e0-45de-bac2-83eebdccb9da local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-10T08:41:33-05:00] Handling API request                          ip=@ method=POST protocol=unix url=/1.0/instances username=orabuntu
DEBUG  [2022-05-10T08:41:33-05:00] Responding to instance create                
DEBUG  [2022-05-10T08:41:33-05:00] Connecting to a remote simplestreams server   URL="https://images.linuxcontainers.org"
DEBUG  [2022-05-10T08:41:35-05:00] New task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:35-05:00] Started task operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:35-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/operations/baa997f7-c06a-40e6-a27f-01396220ba6b username=orabuntu
DEBUG  [2022-05-10T08:41:35-05:00] Connecting to a remote simplestreams server   URL="https://images.linuxcontainers.org"
DEBUG  [2022-05-10T08:41:35-05:00] Lock acquired for image "1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066" 
DEBUG  [2022-05-10T08:41:35-05:00] Acquiring lock for image "1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066" 
DEBUG  [2022-05-10T08:41:35-05:00] Database error                                err="Image not found"
DEBUG  [2022-05-10T08:41:35-05:00] Database error                                err="Image not found"
INFO   [2022-05-10T08:41:35-05:00] Downloading image                             alias=oracle/8/amd64 fingerprint=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 operation=baa997f7-c06a-40e6-a27f-01396220ba6b server="https://images.linuxcontainers.org" trigger=/1.0/operations/baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:41:35-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:36-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:36-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:37-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:37-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:38-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:38-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:38-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:39-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:39-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:40-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:40-05:00] Heartbeat updating local raft members         members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
DEBUG  [2022-05-10T08:41:40-05:00] Starting heartbeat round                      local="10.209.53.1:8443" mode=normal
DEBUG  [2022-05-10T08:41:40-05:00] Completed heartbeat round                     duration=5.473403ms local="10.209.53.1:8443"
DEBUG  [2022-05-10T08:41:40-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:41-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:41-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:41-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:42-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:42-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:42-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:43-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:43-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:44-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:44-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:45-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:45-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:46-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:46-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:47-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:47-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:48-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:48-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:49-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:49-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:50-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:50-05:00] Starting heartbeat round                      local="10.209.53.1:8443" mode=normal
DEBUG  [2022-05-10T08:41:50-05:00] Heartbeat updating local raft members         members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
DEBUG  [2022-05-10T08:41:50-05:00] Completed heartbeat round                     duration=21.990183ms local="10.209.53.1:8443"
DEBUG  [2022-05-10T08:41:50-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:51-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:51-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:51-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:52-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:52-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:52-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:53-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:53-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:54-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:54-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:55-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:55-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:55-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:56-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:56-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:57-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:57-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:57-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:58-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:59-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:41:59-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:00-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:00-05:00] Heartbeat updating local raft members         members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
DEBUG  [2022-05-10T08:42:00-05:00] Starting heartbeat round                      local="10.209.53.1:8443" mode=normal
DEBUG  [2022-05-10T08:42:00-05:00] Completed heartbeat round                     duration=22.543699ms local="10.209.53.1:8443"
DEBUG  [2022-05-10T08:42:00-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:00-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:01-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:01-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:02-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:02-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:02-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:03-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:03-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:04-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:04-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:05-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:05-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:05-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:06-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:06-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:07-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:07-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:08-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:08-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:09-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:09-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:10-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:10-05:00] Starting heartbeat round                      local="10.209.53.1:8443" mode=normal
DEBUG  [2022-05-10T08:42:10-05:00] Heartbeat updating local raft members         members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
DEBUG  [2022-05-10T08:42:10-05:00] Completed heartbeat round                     duration=23.069247ms local="10.209.53.1:8443"
DEBUG  [2022-05-10T08:42:10-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:10-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:11-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:11-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:12-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:12-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:13-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:13-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:13-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:14-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:14-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:14-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:15-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:16-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:16-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:17-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:17-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:17-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:18-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:18-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:19-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:19-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:19-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:20-05:00] Heartbeat updating local raft members         members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
DEBUG  [2022-05-10T08:42:20-05:00] Starting heartbeat round                      local="10.209.53.1:8443" mode=normal
DEBUG  [2022-05-10T08:42:20-05:00] Completed heartbeat round                     duration=10.655645ms local="10.209.53.1:8443"
DEBUG  [2022-05-10T08:42:20-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:20-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:21-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:21-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
INFO   [2022-05-10T08:42:22-05:00] Image downloaded                              alias=oracle/8/amd64 fingerprint=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 operation=baa997f7-c06a-40e6-a27f-01396220ba6b server="https://images.linuxcontainers.org" trigger=/1.0/operations/baa997f7-c06a-40e6-a27f-01396220ba6b
DEBUG  [2022-05-10T08:42:22-05:00] Acquiring lock for image "1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066" 
DEBUG  [2022-05-10T08:42:22-05:00] Lock acquired for image "1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066" 
DEBUG  [2022-05-10T08:42:22-05:00] Instance operation lock created               action=create instance=oel84d10 project=default reusable=false
INFO   [2022-05-10T08:42:22-05:00] Creating container                            ephemeral=false instance=oel84d10 instanceType=container project=default
DEBUG  [2022-05-10T08:42:22-05:00] Database error                                err="Network not found"
DEBUG  [2022-05-10T08:42:22-05:00] Database error                                err="Network not found"
DEBUG  [2022-05-10T08:42:22-05:00] Adding device                                 device=eth0 instance=oel84d10 instanceType=container project=default type=nic
DEBUG  [2022-05-10T08:42:22-05:00] Adding device                                 device=root instance=oel84d10 instanceType=container project=default type=disk
INFO   [2022-05-10T08:42:22-05:00] Created container                             ephemeral=false instance=oel84d10 instanceType=container project=default
DEBUG  [2022-05-10T08:42:22-05:00] CreateInstanceFromImage started               instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:22-05:00] EnsureImage started                           fingerprint=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066
DEBUG  [2022-05-10T08:42:22-05:00] Database error                                err="Storage pool volume not found"
INFO   [2022-05-10T08:42:22-05:00] Image unpack started                          imageFile=/var/snap/lxd/common/lxd/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 volName=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066
DEBUG  [2022-05-10T08:42:22-05:00] Mounted ZFS dataset                           dev=olxd-0933/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 pool=local
DEBUG  [2022-05-10T08:42:22-05:00] Running filler function                       dev= driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 pool=local
DEBUG  [2022-05-10T08:42:22-05:00] Updated metadata for task Operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
INFO   [2022-05-10T08:42:25-05:00] Image unpack stopped                          imageFile=/var/snap/lxd/common/lxd/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 volName=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066
DEBUG  [2022-05-10T08:42:30-05:00] Heartbeat updating local raft members         members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
DEBUG  [2022-05-10T08:42:30-05:00] Starting heartbeat round                      local="10.209.53.1:8443" mode=normal
DEBUG  [2022-05-10T08:42:30-05:00] Completed heartbeat round                     duration=16.878987ms local="10.209.53.1:8443"
DEBUG  [2022-05-10T08:42:31-05:00] Unmounted ZFS dataset                         dev=olxd-0933/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 pool=local volName=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066
DEBUG  [2022-05-10T08:42:31-05:00] EnsureImage finished                          fingerprint=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066
DEBUG  [2022-05-10T08:42:31-05:00] Checking volume size                          instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:31-05:00] Set new volume size                           instance=oel84d10 project=default size=
DEBUG  [2022-05-10T08:42:31-05:00] Mounted ZFS dataset                           dev=olxd-0933/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local
DEBUG  [2022-05-10T08:42:31-05:00] Unmounted ZFS dataset                         dev=olxd-0933/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local volName=oel84d10
DEBUG  [2022-05-10T08:42:31-05:00] UpdateInstanceBackupFile started              instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:31-05:00] CreateInstanceFromImage finished              instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:31-05:00] Mounted ZFS dataset                           dev=olxd-0933/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local
DEBUG  [2022-05-10T08:42:32-05:00] Instance operation lock finished              action=create err="<nil>" instance=oel84d10 project=default reusable=false
DEBUG  [2022-05-10T08:42:32-05:00] Unmounted ZFS dataset                         dev=olxd-0933/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local volName=oel84d10
DEBUG  [2022-05-10T08:42:32-05:00] UpdateInstanceBackupFile finished             instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:32-05:00] Success for task operation: baa997f7-c06a-40e6-a27f-01396220ba6b 
DEBUG  [2022-05-10T08:42:32-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/instances/oel84d10 username=orabuntu
DEBUG  [2022-05-10T08:42:32-05:00] Handling API request                          ip=@ method=PUT protocol=unix url=/1.0/instances/oel84d10/state username=orabuntu
DEBUG  [2022-05-10T08:42:32-05:00] New task Operation: 3b33fee1-63ac-4fa8-893f-d2ce36852ae6 
DEBUG  [2022-05-10T08:42:32-05:00] Instance operation lock created               action=start instance=oel84d10 project=default reusable=false
DEBUG  [2022-05-10T08:42:32-05:00] Start started                                 instance=oel84d10 instanceType=container project=default stateful=false
DEBUG  [2022-05-10T08:42:32-05:00] Started task operation: 3b33fee1-63ac-4fa8-893f-d2ce36852ae6 
DEBUG  [2022-05-10T08:42:32-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/operations/3b33fee1-63ac-4fa8-893f-d2ce36852ae6 username=orabuntu
DEBUG  [2022-05-10T08:42:32-05:00] MountInstance started                         instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:32-05:00] MountInstance finished                        instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:32-05:00] Mounted ZFS dataset                           dev=olxd-0933/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local
DEBUG  [2022-05-10T08:42:32-05:00] Updated metadata for task Operation: 3b33fee1-63ac-4fa8-893f-d2ce36852ae6 
DEBUG  [2022-05-10T08:42:32-05:00] Container idmap changed, remapping            instance=oel84d10 instanceType=container project=default
DEBUG  [2022-05-10T08:42:33-05:00] Updated metadata for task Operation: 3b33fee1-63ac-4fa8-893f-d2ce36852ae6 
DEBUG  [2022-05-10T08:42:33-05:00] Database error                                err="Network not found"
DEBUG  [2022-05-10T08:42:33-05:00] Starting device                               device=eth0 instance=oel84d10 instanceType=container project=default type=nic
DEBUG  [2022-05-10T08:42:33-05:00] Starting device                               device=root instance=oel84d10 instanceType=container project=default type=disk
DEBUG  [2022-05-10T08:42:33-05:00] UpdateInstanceBackupFile started              instance=oel84d10 project=default
INFO   [2022-05-10T08:42:33-05:00] Starting container                            action=start created="2022-05-10 13:42:22.057872521 +0000 UTC" ephemeral=false instance=oel84d10 instanceType=container project=default stateful=false used="1970-01-01 00:00:00 +0000 UTC"
DEBUG  [2022-05-10T08:42:33-05:00] UpdateInstanceBackupFile finished             instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:33-05:00] Skipping unmount as in use                    driver=zfs pool=local refCount=1 volName=oel84d10
DEBUG  [2022-05-10T08:42:33-05:00] Handling API request                          ip=@ method=GET protocol=unix url="/internal/containers/oel84d10/onstart?project=default" username=root
DEBUG  [2022-05-10T08:42:33-05:00] Scheduler: container oel84d10 started: re-balancing 
DEBUG  [2022-05-10T08:42:33-05:00] Success for task operation: 3b33fee1-63ac-4fa8-893f-d2ce36852ae6 
INFO   [2022-05-10T08:42:33-05:00] Started container                             action=start created="2022-05-10 13:42:22.057872521 +0000 UTC" ephemeral=false instance=oel84d10 instanceType=container project=default stateful=false used="1970-01-01 00:00:00 +0000 UTC"
DEBUG  [2022-05-10T08:42:33-05:00] Start finished                                instance=oel84d10 instanceType=container project=default stateful=false
DEBUG  [2022-05-10T08:42:33-05:00] Instance operation lock finished              action=start err="<nil>" instance=oel84d10 project=default reusable=false
DEBUG  [2022-05-10T08:42:33-05:00] Event listener server handler stopped         listener=ed32291c-37e0-45de-bac2-83eebdccb9da local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-10T08:42:40-05:00] Starting heartbeat round                      local="10.209.53.1:8443" mode=normal
DEBUG  [2022-05-10T08:42:40-05:00] Heartbeat updating local raft members         members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
DEBUG  [2022-05-10T08:42:40-05:00] Completed heartbeat round                     duration=46.256365ms local="10.209.53.1:8443"
DEBUG  [2022-05-10T08:42:48-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
DEBUG  [2022-05-10T08:42:48-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/events username=orabuntu
DEBUG  [2022-05-10T08:42:48-05:00] Event listener server handler started         listener=f94d1b77-620e-4b9e-bd7a-309f7cafbfcc local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-10T08:42:48-05:00] Handling API request                          ip=@ method=POST protocol=unix url=/1.0/instances/oel84d10/exec username=orabuntu
DEBUG  [2022-05-10T08:42:48-05:00] New websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587 
DEBUG  [2022-05-10T08:42:48-05:00] Waiting for exec websockets to connect       
DEBUG  [2022-05-10T08:42:48-05:00] Started websocket operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587 
DEBUG  [2022-05-10T08:42:48-05:00] Handled websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587 
DEBUG  [2022-05-10T08:42:48-05:00] Connected websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587 
DEBUG  [2022-05-10T08:42:48-05:00] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/operations/6ccbae38-bc50-4bcb-b7f0-1bc3b7840587/websocket?secret=46adcf39146218420742abbaeec8c24cd50f3ec5aa1884dc4094620474ea653b" username=orabuntu
DEBUG  [2022-05-10T08:42:48-05:00] Handled websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587 
DEBUG  [2022-05-10T08:42:48-05:00] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/operations/6ccbae38-bc50-4bcb-b7f0-1bc3b7840587/websocket?secret=ca9356d7c0ea567b1a811df478974372876d21629215f0491112e824e84419f0" username=orabuntu
DEBUG  [2022-05-10T08:42:48-05:00] Connected websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587 
DEBUG  [2022-05-10T08:42:48-05:00] Handled websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587 
DEBUG  [2022-05-10T08:42:48-05:00] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/operations/6ccbae38-bc50-4bcb-b7f0-1bc3b7840587/websocket?secret=89b3ab2d10db416808cf3d7f2f952be88d061b1a546ff278e86a6eaf7c2e167e" username=orabuntu
DEBUG  [2022-05-10T08:42:48-05:00] Connected websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587 
DEBUG  [2022-05-10T08:42:48-05:00] Handled websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587 
DEBUG  [2022-05-10T08:42:48-05:00] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/operations/6ccbae38-bc50-4bcb-b7f0-1bc3b7840587/websocket?secret=3925d5e121c6dbd82d84cc4f6d2ef318bc7f16936b84a39d17374ab86942db07" username=orabuntu
DEBUG  [2022-05-10T08:42:48-05:00] Connected websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587 
DEBUG  [2022-05-10T08:42:48-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/operations/6ccbae38-bc50-4bcb-b7f0-1bc3b7840587 username=orabuntu
DEBUG  [2022-05-10T08:42:49-05:00] Retrieved PID of executing child process      attachedPid=110357 instance=oel84d10 instanceType=container project=default
DEBUG  [2022-05-10T08:42:49-05:00] Exec mirror websocket started                 PID=110357 instance=oel84d10 interactive=false number=1 project=default
DEBUG  [2022-05-10T08:42:49-05:00] Exec control handler started                  PID=110357 instance=oel84d10 interactive=false project=default
DEBUG  [2022-05-10T08:42:49-05:00] Exec mirror websocket started                 PID=110357 instance=oel84d10 interactive=false number=0 project=default
DEBUG  [2022-05-10T08:42:49-05:00] WebsocketRecvStream got message barrier      
DEBUG  [2022-05-10T08:42:49-05:00] Exec mirror websocket finished                PID=110357 instance=oel84d10 interactive=false number=0 project=default
DEBUG  [2022-05-10T08:42:49-05:00] Instance process started                      PID=110357 instance=oel84d10 interactive=false project=default
DEBUG  [2022-05-10T08:42:49-05:00] Exec mirror websocket started                 PID=110357 instance=oel84d10 interactive=false number=2 project=default
DEBUG  [2022-05-10T08:42:49-05:00] Exec control handler finished                 PID=110357 instance=oel84d10 interactive=false project=default
DEBUG  [2022-05-10T08:42:49-05:00] Success for websocket operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587 
DEBUG  [2022-05-10T08:42:49-05:00] Instance process stopped                      PID=110357 exitStatus=0 instance=oel84d10 interactive=false project=default
DEBUG  [2022-05-10T08:42:49-05:00] Exec mirror websocket finished                PID=110357 instance=oel84d10 interactive=false number=1 project=default
DEBUG  [2022-05-10T08:42:49-05:00] Exec mirror websocket finished                PID=110357 instance=oel84d10 interactive=false number=2 project=default
DEBUG  [2022-05-10T08:42:49-05:00] Updated metadata for websocket Operation: 6ccbae38-bc50-4bcb-b7f0-1bc3b7840587 
DEBUG  [2022-05-10T08:42:49-05:00] Event listener server handler stopped         listener=f94d1b77-620e-4b9e-bd7a-309f7cafbfcc local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-10T08:42:49-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
DEBUG  [2022-05-10T08:42:49-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/instances/oel84d10 username=orabuntu
DEBUG  [2022-05-10T08:42:49-05:00] Handling API request                          ip=@ method=PUT protocol=unix url=/1.0/instances/oel84d10 username=orabuntu
DEBUG  [2022-05-10T08:42:49-05:00] Event listener server handler started         listener=40b2f8fd-916b-41f7-985a-31f7e94f2cec local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-10T08:42:49-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/events username=orabuntu
DEBUG  [2022-05-10T08:42:49-05:00] New task Operation: f30078f7-3d05-4f79-9312-5e6fc9889839 
DEBUG  [2022-05-10T08:42:49-05:00] Instance operation lock created               action=update instance=oel84d10 project=default reusable=false
DEBUG  [2022-05-10T08:42:49-05:00] Started task operation: f30078f7-3d05-4f79-9312-5e6fc9889839 
DEBUG  [2022-05-10T08:42:49-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/operations/f30078f7-3d05-4f79-9312-5e6fc9889839 username=orabuntu
DEBUG  [2022-05-10T08:42:49-05:00] Database error                                err="Network not found"
DEBUG  [2022-05-10T08:42:49-05:00] UpdateInstanceBackupFile started              instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:49-05:00] UpdateInstanceBackupFile finished             instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:49-05:00] Skipping unmount as in use                    driver=zfs pool=local refCount=1 volName=oel84d10
DEBUG  [2022-05-10T08:42:49-05:00] Success for task operation: f30078f7-3d05-4f79-9312-5e6fc9889839 
DEBUG  [2022-05-10T08:42:49-05:00] Instance operation lock finished              action=update err="<nil>" instance=oel84d10 project=default reusable=false
DEBUG  [2022-05-10T08:42:49-05:00] Event listener server handler stopped         listener=40b2f8fd-916b-41f7-985a-31f7e94f2cec local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-10T08:42:49-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
DEBUG  [2022-05-10T08:42:49-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/events username=orabuntu
DEBUG  [2022-05-10T08:42:49-05:00] Event listener server handler started         listener=d170a28f-c83a-4d6d-a7b7-adbf52695cdc local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-10T08:42:49-05:00] Handling API request                          ip=@ method=PUT protocol=unix url=/1.0/instances/oel84d10/state username=orabuntu
DEBUG  [2022-05-10T08:42:49-05:00] Started task operation: c7e0508c-3d04-4351-8b79-78b744f9fad7 
DEBUG  [2022-05-10T08:42:49-05:00] Instance operation lock created               action=stop instance=oel84d10 project=default reusable=true
DEBUG  [2022-05-10T08:42:49-05:00] Shutdown started                              instance=oel84d10 instanceType=container project=default timeout=-1s
DEBUG  [2022-05-10T08:42:49-05:00] New task Operation: c7e0508c-3d04-4351-8b79-78b744f9fad7 
DEBUG  [2022-05-10T08:42:49-05:00] Shutdown request sent to instance             instance=oel84d10 instanceType=container project=default
INFO   [2022-05-10T08:42:49-05:00] Shutting down container                       action=shutdown created="2022-05-10 13:42:22.057872521 +0000 UTC" ephemeral=false instance=oel84d10 instanceType=container project=default timeout=-1s used="2022-05-10 13:42:33.633858363 +0000 UTC"
DEBUG  [2022-05-10T08:42:49-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/operations/c7e0508c-3d04-4351-8b79-78b744f9fad7 username=orabuntu
DEBUG  [2022-05-10T08:42:49-05:00] Handling API request                          ip=@ method=GET protocol=unix url="/internal/containers/oel84d10/onstopns?netns=%2Fproc%2F109702%2Ffd%2F4&project=default&target=stop" username=root
DEBUG  [2022-05-10T08:42:49-05:00] Database error                                err="Network not found"
DEBUG  [2022-05-10T08:42:49-05:00] Stopping device                               device=eth0 instance=oel84d10 instanceType=container project=default type=nic
DEBUG  [2022-05-10T08:42:50-05:00] Handling API request                          ip=@ method=GET protocol=unix url="/internal/containers/oel84d10/onstop?project=default&target=stop" username=root
DEBUG  [2022-05-10T08:42:50-05:00] Container stopped, cleaning up                instance=oel84d10 instanceType=container project=default
DEBUG  [2022-05-10T08:42:50-05:00] Stopping device                               device=root instance=oel84d10 instanceType=container project=default type=disk
DEBUG  [2022-05-10T08:42:50-05:00] UnmountInstance started                       instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:50-05:00] UnmountInstance finished                      instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:50-05:00] Unmounted ZFS dataset                         dev=olxd-0933/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local volName=oel84d10
DEBUG  [2022-05-10T08:42:50-05:00] Success for task operation: c7e0508c-3d04-4351-8b79-78b744f9fad7 
DEBUG  [2022-05-10T08:42:50-05:00] Scheduler: container oel84d10 stopped: re-balancing 
DEBUG  [2022-05-10T08:42:50-05:00] Instance operation lock finished              action=stop err="<nil>" instance=oel84d10 project=default reusable=true
DEBUG  [2022-05-10T08:42:50-05:00] Shutdown finished                             instance=oel84d10 instanceType=container project=default timeout=-1s
DEBUG  [2022-05-10T08:42:50-05:00] Event listener server handler stopped         listener=d170a28f-c83a-4d6d-a7b7-adbf52695cdc local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-10T08:42:50-05:00] Heartbeat updating local raft members         members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
DEBUG  [2022-05-10T08:42:50-05:00] Starting heartbeat round                      local="10.209.53.1:8443" mode=normal
DEBUG  [2022-05-10T08:42:50-05:00] Completed heartbeat round                     duration=8.130839ms local="10.209.53.1:8443"
DEBUG  [2022-05-10T08:42:55-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
DEBUG  [2022-05-10T08:42:55-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/instances/oel84d10 username=orabuntu
DEBUG  [2022-05-10T08:42:55-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/events username=orabuntu
DEBUG  [2022-05-10T08:42:55-05:00] Event listener server handler started         listener=f090c5fd-f5fa-4922-b45b-7c7317a5823d local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-05-10T08:42:55-05:00] Handling API request                          ip=@ method=PUT protocol=unix url=/1.0/instances/oel84d10/state username=orabuntu
DEBUG  [2022-05-10T08:42:55-05:00] New task Operation: 454defe6-7771-4d4b-8941-2bbb260ea16c 
DEBUG  [2022-05-10T08:42:55-05:00] Started task operation: 454defe6-7771-4d4b-8941-2bbb260ea16c 
DEBUG  [2022-05-10T08:42:55-05:00] Start started                                 instance=oel84d10 instanceType=container project=default stateful=false
DEBUG  [2022-05-10T08:42:55-05:00] Instance operation lock created               action=start instance=oel84d10 project=default reusable=false
DEBUG  [2022-05-10T08:42:55-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/operations/454defe6-7771-4d4b-8941-2bbb260ea16c username=orabuntu
DEBUG  [2022-05-10T08:42:55-05:00] MountInstance started                         instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:55-05:00] MountInstance finished                        instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:55-05:00] Mounted ZFS dataset                           dev=olxd-0933/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local
DEBUG  [2022-05-10T08:42:55-05:00] Updated metadata for task Operation: 454defe6-7771-4d4b-8941-2bbb260ea16c 
DEBUG  [2022-05-10T08:42:55-05:00] Container idmap changed, remapping            instance=oel84d10 instanceType=container project=default
DEBUG  [2022-05-10T08:42:56-05:00] Updated metadata for task Operation: 454defe6-7771-4d4b-8941-2bbb260ea16c 
DEBUG  [2022-05-10T08:42:56-05:00] Database error                                err="Network not found"
DEBUG  [2022-05-10T08:42:56-05:00] Starting device                               device=eth0 instance=oel84d10 instanceType=container project=default type=nic
DEBUG  [2022-05-10T08:42:56-05:00] Starting device                               device=root instance=oel84d10 instanceType=container project=default type=disk
DEBUG  [2022-05-10T08:42:56-05:00] UpdateInstanceBackupFile started              instance=oel84d10 project=default
INFO   [2022-05-10T08:42:56-05:00] Starting container                            action=start created="2022-05-10 13:42:22.057872521 +0000 UTC" ephemeral=false instance=oel84d10 instanceType=container project=default stateful=false used="2022-05-10 13:42:33.633858363 +0000 UTC"
DEBUG  [2022-05-10T08:42:56-05:00] UpdateInstanceBackupFile finished             instance=oel84d10 project=default
DEBUG  [2022-05-10T08:42:56-05:00] Skipping unmount as in use                    driver=zfs pool=local refCount=1 volName=oel84d10
DEBUG  [2022-05-10T08:42:56-05:00] Handling API request                          ip=@ method=GET protocol=unix url="/internal/containers/oel84d10/onstart?project=default" username=root
DEBUG  [2022-05-10T08:42:56-05:00] Scheduler: container oel84d10 started: re-balancing 
DEBUG  [2022-05-10T08:42:56-05:00] Success for task operation: 454defe6-7771-4d4b-8941-2bbb260ea16c 
DEBUG  [2022-05-10T08:42:56-05:00] Start finished                                instance=oel84d10 instanceType=container project=default stateful=false
DEBUG  [2022-05-10T08:42:56-05:00] Instance operation lock finished              action=start err="<nil>" instance=oel84d10 project=default reusable=false
INFO   [2022-05-10T08:42:56-05:00] Started container                             action=start created="2022-05-10 13:42:22.057872521 +0000 UTC" ephemeral=false instance=oel84d10 instanceType=container project=default stateful=false used="2022-05-10 13:42:33.633858363 +0000 UTC"
DEBUG  [2022-05-10T08:42:56-05:00] Event listener server handler stopped         listener=f090c5fd-f5fa-4922-b45b-7c7317a5823d local=/var/snap/lxd/common/lxd/unix.socket remote=@
^C
[orabuntu@f34sv1 ~]$
tomponline commented 2 years ago

No changes made :)

Perhaps worth deleting the downloaded image and retrying?

gstanden commented 2 years ago

I have to set aside my non-paying opensource work on Orabuntu-LXC atm, and head off to my paying job shortly, but I will definitely do a retry later tonight or tomorrow morning and try to capture a timed-out session and post here. EDIT: In fact what I will do is add some code to Orabuntu-LXC that automatically starts up a logging service that runs "lxc monitor --type=logging --pretty" when lxc launch runs so that this gets captured on every run of the software. It will be a nice logging addition to the software anyway.

gstanden commented 2 years ago

ok finally captured the below two log files using this service:

[Unit]
Description=lxc-monitor Service
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=orabuntu
RemainAfterExit=yes
ExecStart=/var/lib/snapd/snap/bin/lxc monitor --type=logging --pretty
ExecStop=ps -aux | grep pretty | grep -v grep | grep monitor | awk '{print $2}' | xargs kill -9
StandardOutput=append:/home/orabuntu/lxc-monitor.log
StandardError=append:/home/orabuntu/lxc-monitor.err

[Install]
WantedBy=multi-user.target

The Orabuntu-LXC software starts this service just before it tries to create the first Oracle Linux LXD container. The logs of two runs that gave the error "Error: Failed instance creation: Post "http://unix.socket/1.0/instances": net/http: timeout awaiting response headers" are below:

lxc-monitor.seed.log lxc-monitor.run1.log

Now here is another "feature" of this issue; if I then immediately try to create an LXD container of Oracle Linux 8 by just incrementing my "$SeedPostfix" variable (increment from "d10" to "d11") the LXD container creates perfectly fine:

lxc-monitor.run3.log

So something is happening only with that first container attempt to create just after LXD has been setup.

The cluster looks ok:

[orabuntu@f34sv1 system]$ lxc cluster list
+--------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
|  NAME  |           URL            |      ROLES      | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      |
+--------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| f34sv1 | https://10.209.53.1:8443 | database-leader | x86_64       | default        |             | ONLINE | Fully operational |
|        |                          | database        |              |                |             |        |                   |
+--------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
[orabuntu@f34sv1 system]$ lxc storage list
+------------+--------+------------------+---------+---------+
|    NAME    | DRIVER |   DESCRIPTION    | USED BY |  STATE  |
+------------+--------+------------------+---------+---------+
| containerd | dir    |                  | 3       | CREATED |
+------------+--------+------------------+---------+---------+
| docker     | dir    |                  | 3       | CREATED |
+------------+--------+------------------+---------+---------+
| kubelet    | dir    |                  | 3       | CREATED |
+------------+--------+------------------+---------+---------+
| local      | zfs    | f34sv1-olxd-f2fc | 5       | CREATED |
+------------+--------+------------------+---------+---------+
[orabuntu@f34sv1 system]$ sudo zpool list
[sudo] password for orabuntu: 
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
olxd-f2fc   960M   578M   382M        -         -    13%    60%  1.00x    ONLINE  -
[orabuntu@f34sv1 system]$
tomponline commented 2 years ago

Are you sure "seed" was the same error the logs show something different:

time="2022-05-11T01:02:02-05:00" level=debug msg="Database error" err="FOREIGN KEY constraint failed"
time="2022-05-11T01:02:02-05:00" level=debug msg="Database error" err="Storage pool volume not found"
time="2022-05-11T01:02:02-05:00" level=debug msg="Instance operation lock finished" action=create err="<nil>" instance=oel84d10 project=default reusable=false
time="2022-05-11T01:02:02-05:00" level=debug msg="CreateInstanceFromImage finished" instance=oel84d10 project=default
time="2022-05-11T01:02:02-05:00" level=debug msg="Removing device" device=eth0 instance=oel84d10 instanceType=container project=default type=nic
time="2022-05-11T01:02:02-05:00" level=debug msg="Removing device" device=root instance=oel84d10 instanceType=container project=default type=disk
time="2022-05-11T01:02:02-05:00" level=debug msg="Failure for task operation: 9221fa4c-af35-4801-91b6-3cc32f2097c3: Failed creating instance from image: Failed to set apply_template volatile key: Failed to set volatile config: FOREIGN KEY constraint failed"
time="2022-05-11T01:02:02-05:00" level=debug msg="Database error" err="Query deleted 0 rows instead of 1"
gstanden commented 2 years ago

When my variables are set to create "oel84d10" (the first LXD container created after LXD has been installed and configured) I get the error.

If I increment to create "oel84d11" everything creates fine. It's always just that first LXD container creation attempt after LXD has been installed and configured.

Here's the log for oel84d11

lxc-monitor.run3.log

tomponline commented 2 years ago

Looks like you're trying to create the same instance concurrently:

time="2022-05-11T01:01:48-05:00" level=info msg="Image downloaded" alias=oracle/8/amd64 fingerprint=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066 operation=9221fa4c-af35-4801-91b6-3cc32f2097c3 server="https://images.linuxcontainers.org" trigger=/1.0/operations/9221fa4c-af35-4801-91b6-3cc32f2097c3
time="2022-05-11T01:01:48-05:00" level=debug msg="Lock acquired for image \"1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066\""
time="2022-05-11T01:01:48-05:00" level=debug msg="Acquiring lock for image \"1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066\""
time="2022-05-11T01:01:48-05:00" level=debug msg="Lock acquired for image \"1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066\""
time="2022-05-11T01:01:48-05:00" level=debug msg="Image already exists in the DB" fingerprint=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066
time="2022-05-11T01:01:48-05:00" level=debug msg="Acquiring lock for image \"1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066\""
time="2022-05-11T01:01:48-05:00" level=debug msg="Image already exists in the DB" fingerprint=1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066
time="2022-05-11T01:01:48-05:00" level=debug msg="Lock acquired for image \"1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066\""
time="2022-05-11T01:01:48-05:00" level=debug msg="Acquiring lock for image \"1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066\""
time="2022-05-11T01:01:48-05:00" level=debug msg="Lock acquired for image \"1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066\""
time="2022-05-11T01:01:48-05:00" level=debug msg="Lock acquired for image \"1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066\""
time="2022-05-11T01:01:48-05:00" level=debug msg="Lock acquired for image \"1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066\""
time="2022-05-11T01:01:48-05:00" level=debug msg="Instance operation lock created" action=create instance=oel84d10 project=default reusable=false
time="2022-05-11T01:01:48-05:00" level=debug msg="Database error" err="Add instance info to the database: This \"instances\" entry already exists"
time="2022-05-11T01:01:48-05:00" level=debug msg="Failure for task operation: 5ded6b69-dcd6-4fa4-9b5b-46eacff7b6fd: Failed creating instance record: Add instance info to the database: This \"instances\" entry already exists"
time="2022-05-11T01:01:48-05:00" level=debug msg="Failure for task operation: f00877e3-dcf3-42b3-b1c7-6a45f5d287b9: Failed creating instance record: Add instance info to the database: This \"instances\" entry already exists"
time="2022-05-11T01:01:48-05:00" level=debug msg="Database error" err="Add instance info to the database: This \"instances\" entry already exists"
time="2022-05-11T01:01:48-05:00" level=info msg="Creating container" ephemeral=false instance=oel84d10 instanceType=container project=default
gstanden commented 2 years ago

I never got these problems before in the lxd 4.x series. This really seems to be a new issue that was introduced in lxd 5.x I ran the following just now with no issue:


[orabuntu@f34sv1 ~]$ lxc launch -p olxc_sx1a images:oracle/$MajorRelease/amd64 oel$OracleRelease$SeedPostfix 
Creating oel84d10
Error: Failed instance creation: Failed creating instance from image: Failed to run: zfs clone olxd-f2fc/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066@readonly olxd-f2fc/containers/oel84d10: cannot create 'olxd-f2fc/containers/oel84d10': dataset already exists
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
[sudo] password for orabuntu: 
NAME                                                                                         USED  AVAIL     REFER  MOUNTPOINT
olxd-f2fc                                                                                    582M   250M       96K  legacy
olxd-f2fc/containers                                                                        4.09M   250M       96K  legacy
olxd-f2fc/containers/oel84d10                                                                 88K   250M      575M  legacy
olxd-f2fc/containers/oel84d11                                                               3.91M   250M      573M  legacy
olxd-f2fc/custom                                                                              96K   250M       96K  legacy
olxd-f2fc/deleted                                                                            480K   250M       96K  legacy
olxd-f2fc/deleted/containers                                                                  96K   250M       96K  legacy
olxd-f2fc/deleted/custom                                                                      96K   250M       96K  legacy
olxd-f2fc/deleted/images                                                                      96K   250M       96K  legacy
olxd-f2fc/deleted/virtual-machines                                                            96K   250M       96K  legacy
olxd-f2fc/images                                                                             575M   250M       96K  legacy
olxd-f2fc/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066            575M   250M      575M  legacy
olxd-f2fc/images/1534ec6dff3956ef7f6b609f8d7dd95e7766c7d86640563c982a12a79ac7a066@readonly     0B      -      575M  -
olxd-f2fc/virtual-machines                                                                    96K   250M       96K  legacy
[orabuntu@f34sv1 ~]$ sudo zfs destroy olxd-f2fc/containers/oel84d10 
[orabuntu@f34sv1 ~]$ lxc launch -p olxc_sx1a images:oracle/$MajorRelease/amd64 oel$OracleRelease$SeedPostfix 
Creating oel84d10
Starting oel84d10
[orabuntu@f34sv1 ~]$

I guess I will just put some code into Orabuntu-LXC to just do the "zfs destroy" step if this "timeout awaiting response headers" error is detected.

tomponline commented 2 years ago

I never got these problems before in the lxd 4.x series. This really seems to be a new issue that was introduced in lxd 5.x I ran the following just now with no issue:

The timeouts are in LXD 5.1, not LXD 5.0. My point is that we are trying to figure out why its taking over 30s to get the headers, and in this case it seems like you have multiple ongoing requests for creating the same instance, which is likely to incur delays due to locking.

I'm trying to understand your workflow, as I'm only getting part of the picture each time (either the server side logs or the client side logs), but never together with the command requests initiated. I need to build up a full picture of what you are doing to reproduce this issue.

tomponline commented 2 years ago

My suggestion would be to clear up any partially created containers and then re-run your script (showing the specific commands you are running) along with the --debug log of those commands and the server-side debug log, so we can see what is going on specifically.

gstanden commented 2 years ago

This is the code that was added when this problem started apparently with 5.1:

        LaunchCode=1
        n=0
        while [ $LaunchCode -ne 0 ] && [ $n -le 5 ]
        do
                eval echo "'/var/lib/snapd/snap/bin/lxc launch -p olxc_sx1a images:oracle/$MajorRelease/amd64 oel$OracleRelease$SeedPostfix' | sg lxd $CGROUP_SUFFIX"
                LaunchCode=`echo $?`
                if [ $LaunchCode -ne 0 ]
                then
                        eval echo "'/var/lib/snapd/snap/bin/lxc stop -f oel$OracleRelease$SeedPostfix' | sg lxd $CGROUP_SUFFIX"
                        eval echo "'/var/lib/snapd/snap/bin/lxc delete  oel$OracleRelease$SeedPostfix' | sg lxd $CGROUP_SUFFIX"
                        sudo zfs destroy $LXDStoragePoolName/containers/oel$OracleRelease$SeedPostfix
                        sleep 5
                fi
                n=$((n+1))
        done

Prior to that everything "just worked" and there was no retry code. I actually think this is really a good thing (especially once it's resolved) because there should be some retry code in Orabuntu-LXC to address the case that the container creation fails for whatever reason - so this has at least resulted it making Orabuntu-LXC more robust which is a good thing. :)

But prior to 5.1 this lxc launch step "just worked" always everytime - which is not proof that 5.1 is the culprit, but there is a possiblity that it is. The "zfs destroy" step was added just now.

gstanden commented 2 years ago

I also added "--debug" to the initial "launch" command for the next test of the entire Orabuntu-LXC deploy run:

        LaunchCode=1
        n=0
        while [ $LaunchCode -ne 0 ] && [ $n -le 5 ]
        do
                eval echo "'/var/lib/snapd/snap/bin/lxc launch -p olxc_sx1a images:oracle/$MajorRelease/amd64 oel$OracleRelease$SeedPostfix --debug' | sg lxd $CGROUP_SUFFIX"
                LaunchCode=`echo $?`
                if [ $LaunchCode -ne 0 ]
                then
                        eval echo "'/var/lib/snapd/snap/bin/lxc stop -f oel$OracleRelease$SeedPostfix' | sg lxd $CGROUP_SUFFIX"
                        eval echo "'/var/lib/snapd/snap/bin/lxc delete  oel$OracleRelease$SeedPostfix' | sg lxd $CGROUP_SUFFIX"
                        sudo zfs destroy $LXDStoragePoolName/containers/oel$OracleRelease$SeedPostfix
                        sleep 5
                fi
                n=$((n+1))
        done
tomponline commented 2 years ago

Thanks will try that.

tomponline commented 2 years ago

Does this still occur if you run lxc list first before running your script (to allow LXD to start first).

tomponline commented 2 years ago

Prior to that everything "just worked" and there was no retry code. I actually think this is really a good thing (especially once it's resolved) because there should be some retry code in Orabuntu-LXC to address the case that the container creation fails for whatever reason - so this has at least resulted it making Orabuntu-LXC more robust which is a good thing. :)

Can I also get the logs without the retry code, as I think that might be why the confusing logs are showing concurrent instance creation.

gstanden commented 2 years ago

First thank you so much for your valuable time on this. Really appreciated!

Doing a full run of Orabuntu-LXC now and will capture all the things you have requested. The code prior to the to the "lxc launch" is now as follows:

                echo "=============================================="
                echo "List LXD Cluster ...                          "
                echo "=============================================="
                echo ''
                eval echo "'/var/lib/snapd/snap/bin/lxc cluster list' | sg lxd $CGROUP_SUFFIX"
                sleep 5
                clear
                echo ''
                echo "=============================================="
                echo "Done: List LXD Cluster.                       "
                echo "=============================================="
                echo ''
                sleep 5
                clear
                echo ''
                echo "=============================================="
                echo "List LXD Containers...                        "
                echo "=============================================="
                echo ''
                eval echo "'/var/lib/snapd/snap/bin/lxc list' | sg lxd $CGROUP_SUFFIX"
                sleep 5
                clear
                echo ''
                echo "=============================================="
                echo "Done: List LXD Containers.                    "
                echo "=============================================="
                echo ''
                sleep 5
                clear

The "--debug" on the inital "lxc launch" should capture logging prior to the retry code.

The lxc-monitor service will log retries.

Does it all meet your requirements ?

gstanden commented 2 years ago

Also just FYI, regarding a side issue, in case you might be wondering: The $CGROUP_SUFFIX after the "sg lxd" is currently an empty null enviro variable. It was used previously for earlier issues on Fedora described below and was used to suppress errors during the run of Orabuntu-LXC on Fedora related to this issue (but this went away apparently due to snapd improvements so that variable is now empty). The issues on Fedora with snapd were related to the use of cgroupv2 and nftables 


        echo ''
            echo "=============================================="
            echo "On $LinuxFlavor $RedHatVersion the WARNING:   "
            echo "                                              "
            echo "WARNING: cgroup v2 is not fully supported yet "
            echo "proceeding with partial confinement.          "
            echo "                                              "
            echo "can be safely IGNORED.                        "
            echo "This is a snapd issue not an LXD issue.       "
            echo "                                              "
            echo "This specific warning has been suppressed     "
            echo "during this install of Orabuntu-LXC.          "
            echo "                                              "
            echo " More info here:                              "
            echo "                                              "
            echo "https://discuss.linuxcontainers.org/t/lxd-cgroup-v2-support/10455"
            echo "https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1850667"
            echo "                                              "
            echo "=============================================="
gstanden commented 2 years ago

Here is the --debug from the first attempt:

installed: 5.1-1f6f485

==============================================
Launch Oracle LXD Seed Container...           
==============================================

==============================================
(takes a minute or two ... patience) ...      
==============================================

DEBUG  [2022-05-11T07:59:45-05:00] Connecting to a local LXD over a Unix socket 
DEBUG  [2022-05-11T07:59:45-05:00] Sending request to LXD                        etag= method=GET url="http://unix.socket/1.0"
DEBUG  [2022-05-11T07:59:45-05:00] Got response struct from LXD                 
DEBUG  [2022-05-11T07:59:45-05:00] 
    {
        "config": {
            "cluster.https_address": "10.209.53.1:8443",
            "core.https_address": "10.209.53.1:8443",
            "core.trust_password": true
        },
        "api_extensions": [
            "storage_zfs_remove_snapshots",
            "container_host_shutdown_timeout",
            "container_stop_priority",
            "container_syscall_filtering",
            "auth_pki",
            "container_last_used_at",
            "etag",
            "patch",
            "usb_devices",
            "https_allowed_credentials",
            "image_compression_algorithm",
            "directory_manipulation",
            "container_cpu_time",
            "storage_zfs_use_refquota",
            "storage_lvm_mount_options",
            "network",
            "profile_usedby",
            "container_push",
            "container_exec_recording",
            "certificate_update",
            "container_exec_signal_handling",
            "gpu_devices",
            "container_image_properties",
            "migration_progress",
            "id_map",
            "network_firewall_filtering",
            "network_routes",
            "storage",
            "file_delete",
            "file_append",
            "network_dhcp_expiry",
            "storage_lvm_vg_rename",
            "storage_lvm_thinpool_rename",
            "network_vlan",
            "image_create_aliases",
            "container_stateless_copy",
            "container_only_migration",
            "storage_zfs_clone_copy",
            "unix_device_rename",
            "storage_lvm_use_thinpool",
            "storage_rsync_bwlimit",
            "network_vxlan_interface",
            "storage_btrfs_mount_options",
            "entity_description",
            "image_force_refresh",
            "storage_lvm_lv_resizing",
            "id_map_base",
            "file_symlinks",
            "container_push_target",
            "network_vlan_physical",
            "storage_images_delete",
            "container_edit_metadata",
            "container_snapshot_stateful_migration",
            "storage_driver_ceph",
            "storage_ceph_user_name",
            "resource_limits",
            "storage_volatile_initial_source",
            "storage_ceph_force_osd_reuse",
            "storage_block_filesystem_btrfs",
            "resources",
            "kernel_limits",
            "storage_api_volume_rename",
            "macaroon_authentication",
            "network_sriov",
            "console",
            "restrict_devlxd",
            "migration_pre_copy",
            "infiniband",
            "maas_network",
            "devlxd_events",
            "proxy",
            "network_dhcp_gateway",
            "file_get_symlink",
            "network_leases",
            "unix_device_hotplug",
            "storage_api_local_volume_handling",
            "operation_description",
            "clustering",
            "event_lifecycle",
            "storage_api_remote_volume_handling",
            "nvidia_runtime",
            "container_mount_propagation",
            "container_backup",
            "devlxd_images",
            "container_local_cross_pool_handling",
            "proxy_unix",
            "proxy_udp",
            "clustering_join",
            "proxy_tcp_udp_multi_port_handling",
            "network_state",
            "proxy_unix_dac_properties",
            "container_protection_delete",
            "unix_priv_drop",
            "pprof_http",
            "proxy_haproxy_protocol",
            "network_hwaddr",
            "proxy_nat",
            "network_nat_order",
            "container_full",
            "candid_authentication",
            "backup_compression",
            "candid_config",
            "nvidia_runtime_config",
            "storage_api_volume_snapshots",
            "storage_unmapped",
            "projects",
            "candid_config_key",
            "network_vxlan_ttl",
            "container_incremental_copy",
            "usb_optional_vendorid",
            "snapshot_scheduling",
            "snapshot_schedule_aliases",
            "container_copy_project",
            "clustering_server_address",
            "clustering_image_replication",
            "container_protection_shift",
            "snapshot_expiry",
            "container_backup_override_pool",
            "snapshot_expiry_creation",
            "network_leases_location",
            "resources_cpu_socket",
            "resources_gpu",
            "resources_numa",
            "kernel_features",
            "id_map_current",
            "event_location",
            "storage_api_remote_volume_snapshots",
            "network_nat_address",
            "container_nic_routes",
            "rbac",
            "cluster_internal_copy",
            "seccomp_notify",
            "lxc_features",
            "container_nic_ipvlan",
            "network_vlan_sriov",
            "storage_cephfs",
            "container_nic_ipfilter",
            "resources_v2",
            "container_exec_user_group_cwd",
            "container_syscall_intercept",
            "container_disk_shift",
            "storage_shifted",
            "resources_infiniband",
            "daemon_storage",
            "instances",
            "image_types",
            "resources_disk_sata",
            "clustering_roles",
            "images_expiry",
            "resources_network_firmware",
            "backup_compression_algorithm",
            "ceph_data_pool_name",
            "container_syscall_intercept_mount",
            "compression_squashfs",
            "container_raw_mount",
            "container_nic_routed",
            "container_syscall_intercept_mount_fuse",
            "container_disk_ceph",
            "virtual-machines",
            "image_profiles",
            "clustering_architecture",
            "resources_disk_id",
            "storage_lvm_stripes",
            "vm_boot_priority",
            "unix_hotplug_devices",
            "api_filtering",
            "instance_nic_network",
            "clustering_sizing",
            "firewall_driver",
            "projects_limits",
            "container_syscall_intercept_hugetlbfs",
            "limits_hugepages",
            "container_nic_routed_gateway",
            "projects_restrictions",
            "custom_volume_snapshot_expiry",
            "volume_snapshot_scheduling",
            "trust_ca_certificates",
            "snapshot_disk_usage",
            "clustering_edit_roles",
            "container_nic_routed_host_address",
            "container_nic_ipvlan_gateway",
            "resources_usb_pci",
            "resources_cpu_threads_numa",
            "resources_cpu_core_die",
            "api_os",
            "container_nic_routed_host_table",
            "container_nic_ipvlan_host_table",
            "container_nic_ipvlan_mode",
            "resources_system",
            "images_push_relay",
            "network_dns_search",
            "container_nic_routed_limits",
            "instance_nic_bridged_vlan",
            "network_state_bond_bridge",
            "usedby_consistency",
            "custom_block_volumes",
            "clustering_failure_domains",
            "resources_gpu_mdev",
            "console_vga_type",
            "projects_limits_disk",
            "network_type_macvlan",
            "network_type_sriov",
            "container_syscall_intercept_bpf_devices",
            "network_type_ovn",
            "projects_networks",
            "projects_networks_restricted_uplinks",
            "custom_volume_backup",
            "backup_override_name",
            "storage_rsync_compression",
            "network_type_physical",
            "network_ovn_external_subnets",
            "network_ovn_nat",
            "network_ovn_external_routes_remove",
            "tpm_device_type",
            "storage_zfs_clone_copy_rebase",
            "gpu_mdev",
            "resources_pci_iommu",
            "resources_network_usb",
            "resources_disk_address",
            "network_physical_ovn_ingress_mode",
            "network_ovn_dhcp",
            "network_physical_routes_anycast",
            "projects_limits_instances",
            "network_state_vlan",
            "instance_nic_bridged_port_isolation",
            "instance_bulk_state_change",
            "network_gvrp",
            "instance_pool_move",
            "gpu_sriov",
            "pci_device_type",
            "storage_volume_state",
            "network_acl",
            "migration_stateful",
            "disk_state_quota",
            "storage_ceph_features",
            "projects_compression",
            "projects_images_remote_cache_expiry",
            "certificate_project",
            "network_ovn_acl",
            "projects_images_auto_update",
            "projects_restricted_cluster_target",
            "images_default_architecture",
            "network_ovn_acl_defaults",
            "gpu_mig",
            "project_usage",
            "network_bridge_acl",
            "warnings",
            "projects_restricted_backups_and_snapshots",
            "clustering_join_token",
            "clustering_description",
            "server_trusted_proxy",
            "clustering_update_cert",
            "storage_api_project",
            "server_instance_driver_operational",
            "server_supported_storage_drivers",
            "event_lifecycle_requestor_address",
            "resources_gpu_usb",
            "clustering_evacuation",
            "network_ovn_nat_address",
            "network_bgp",
            "network_forward",
            "custom_volume_refresh",
            "network_counters_errors_dropped",
            "metrics",
            "image_source_project",
            "clustering_config",
            "network_peer",
            "linux_sysctl",
            "network_dns",
            "ovn_nic_acceleration",
            "certificate_self_renewal",
            "instance_project_move",
            "storage_volume_project_move",
            "cloud_init",
            "network_dns_nat",
            "database_leader",
            "instance_all_projects",
            "clustering_groups",
            "ceph_rbd_du",
            "instance_get_full",
            "qemu_metrics",
            "gpu_mig_uuid",
            "event_project",
            "clustering_evacuation_live",
            "instance_allow_inconsistent_copy",
            "network_state_ovn",
            "storage_volume_api_filtering",
            "image_restrictions",
            "storage_zfs_export",
            "network_dns_records",
            "storage_zfs_reserve_space",
            "network_acl_log",
            "storage_zfs_blocksize",
            "metrics_cpu_seconds",
            "instance_snapshot_never",
            "certificate_token",
            "instance_nic_routed_neighbor_probe",
            "event_hub",
            "agent_nic_config",
            "projects_restricted_intercept",
            "metrics_authentication",
            "images_target_project",
            "cluster_migration_inconsistent_copy",
            "cluster_ovn_chassis",
            "container_syscall_intercept_sched_setscheduler",
            "storage_lvm_thinpool_metadata_size",
            "storage_volume_state_total",
            "instance_file_head",
            "instances_nic_host_name",
            "image_copy_profile",
            "container_syscall_intercept_sysinfo",
            "clustering_evacuation_mode"
        ],
        "api_status": "stable",
        "api_version": "1.0",
        "auth": "trusted",
        "public": false,
        "auth_methods": [
            "tls"
        ],
        "environment": {
            "addresses": [
                "10.209.53.1:8443"
            ],
            "architectures": [
                "x86_64",
                "i686"
            ],
            "certificate": "-----BEGIN CERTIFICATE-----\nMIICBDCCAYqgAwIBAgIRAK7JizW5VTRBuz/K4S0mS+YwCgYIKoZIzj0EAwMwNDEc\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEUMBIGA1UEAwwLcm9vdEBmMzRz\ndjEwHhcNMjIwNTExMTI1ODIwWhcNMzIwNTA4MTI1ODIwWjA0MRwwGgYDVQQKExNs\naW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QGYzNHN2MTB2MBAGByqG\nSM49AgEGBSuBBAAiA2IABHwpvgG9f4rExLy1cvCCzHEHpJ7WV2uMHSxHRPJX7XP/\nVntjCOXvS1PvF78r+kUkd1bBbkVOdTGQuNQozTc8hTqm8uYazzfEDNgOoHTyOKs3\n2AOj82BrVDkKna5afEv/AqNgMF4wDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoG\nCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwKQYDVR0RBCIwIIIGZjM0c3YxhwR/AAAB\nhxAAAAAAAAAAAAAAAAAAAAABMAoGCCqGSM49BAMDA2gAMGUCMQDZUSLGDShkMb46\nqqb3sKmjq4bs9TdlEOFbjZ1t1zTAlwDVbsFvjmVtQ9Y2lBH0TYcCMCzv5gbzBmTO\nvWM8QmuCtpnI2fYlOi2CVdJdXVqj3NS2mDSPHqRCs+gJcUpoVkEzGA==\n-----END CERTIFICATE-----\n",
            "certificate_fingerprint": "e1975887f497482483ff68da74d24020a25b3b34451928c591348e549c794614",
            "driver": "lxc",
            "driver_version": "4.0.12",
            "firewall": "nftables",
            "kernel": "Linux",
            "kernel_architecture": "x86_64",
            "kernel_features": {
                "idmapped_mounts": "true",
                "netnsid_getifaddrs": "true",
                "seccomp_listener": "true",
                "seccomp_listener_continue": "true",
                "shiftfs": "false",
                "uevent_injection": "true",
                "unpriv_fscaps": "true"
            },
            "kernel_version": "5.17.5-100.fc34.x86_64",
            "lxc_features": {
                "cgroup2": "true",
                "core_scheduling": "true",
                "devpts_fd": "true",
                "idmapped_mounts_v2": "true",
                "mount_injection_file": "true",
                "network_gateway_device_route": "true",
                "network_ipvlan": "true",
                "network_l2proxy": "true",
                "network_phys_macvlan_mtu": "true",
                "network_veth_router": "true",
                "pidfd": "true",
                "seccomp_allow_deny_syntax": "true",
                "seccomp_notify": "true",
                "seccomp_proxy_send_notify_fd": "true"
            },
            "os_name": "Fedora",
            "os_version": "34",
            "project": "default",
            "server": "lxd",
            "server_clustered": true,
            "server_event_mode": "full-mesh",
            "server_name": "f34sv1",
            "server_pid": 107570,
            "server_version": "5.1",
            "storage": "zfs",
            "storage_version": "2.1.4-1",
            "storage_supported_drivers": [
                {
                    "Name": "lvm",
                    "Version": "2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0",
                    "Remote": false
                },
                {
                    "Name": "zfs",
                    "Version": "2.1.4-1",
                    "Remote": false
                },
                {
                    "Name": "ceph",
                    "Version": "15.2.16",
                    "Remote": true
                },
                {
                    "Name": "btrfs",
                    "Version": "5.4.1",
                    "Remote": false
                },
                {
                    "Name": "cephfs",
                    "Version": "15.2.16",
                    "Remote": true
                },
                {
                    "Name": "dir",
                    "Version": "1",
                    "Remote": false
                }
            ]
        }
    } 
Creating oel84d10
DEBUG  [2022-05-11T07:59:45-05:00] Connecting to a remote simplestreams server   URL="https://images.linuxcontainers.org"
DEBUG  [2022-05-11T07:59:45-05:00] Connected to the websocket: ws://unix.socket/1.0/events 
DEBUG  [2022-05-11T07:59:45-05:00] Sending request to LXD                        etag= method=POST url="http://unix.socket/1.0/instances"
DEBUG  [2022-05-11T07:59:45-05:00] 
    {
        "architecture": "",
        "config": {},
        "devices": {},
        "ephemeral": false,
        "profiles": [
            "olxc_sx1a"
        ],
        "stateful": false,
        "description": "",
        "name": "oel84d10",
        "source": {
            "type": "image",
            "certificate": "",
            "alias": "oracle/8/amd64",
            "server": "https://images.linuxcontainers.org",
            "protocol": "simplestreams",
            "mode": "pull",
            "allow_inconsistent": false
        },
        "instance_type": "",
        "type": "container"
    } 
Error: Failed instance creation: Post "http://unix.socket/1.0/instances": net/http: timeout awaiting response headers
Error: Instance not found
Error: Failed checking instance exists "local:oel84d10": Instance not found
cannot open 'olxd-693c/containers/oel84d10': dataset does not exist
gstanden commented 2 years ago

Also the retry code failed. There must be some additional cleanup steps needed:

lxc-monitor.seed.log

gstanden commented 2 years ago

Ok so I'm going to try a run with this setting (and all the previous debug and retry code as well) and see how it goes:

sudo snap install lxd --channel=4.0/stable

tomponline commented 2 years ago

I think we need to try without the retry code as it is confusing matters (by starting additional operations after the first one has timed out but still running on the server), making it impossible to see what is happening on the original operation. We aren't moving forward at the moment.

tomponline commented 2 years ago

I notice you're running a cluster, are the cluster members in the same LAN?

gstanden commented 2 years ago

Ok. I'm going to do a complete run of the entire program with the retry code (which should be moot anyway if 5.1 is the culprit): sudo snap install lxd --channel=4.0/stable

Then a complete run of the entire program without the retry code with: sudo snap install lxd

This node is the "first" node in the cluster, so it is created as a "single-node cluster". When additional cluster members are added, yes, they are all in the same LAN.

gstanden commented 2 years ago

Ok change of plan I decided to try the 5.0 again so that you'd have something hopefully to "crunch away" on while I'm at my daytime job ...

So first I made this change:

        LaunchCode=1
        n=0
        while [ $LaunchCode -ne 0 ] && [ $n -le 5 ]
        do
                eval echo "'/var/lib/snapd/snap/bin/lxc launch -p olxc_sx1a images:oracle/$MajorRelease/amd64 oel$OracleRelease$SeedPostfix --debug' | sg lxd $CGROUP_SUFFIX"
                LaunchCode=`echo $?`
                if [ $LaunchCode -ne 0 ]
                then
                        sleep 300
                        eval echo "'/var/lib/snapd/snap/bin/lxc stop -f oel$OracleRelease$SeedPostfix' | sg lxd $CGROUP_SUFFIX"
                        eval echo "'/var/lib/snapd/snap/bin/lxc delete  oel$OracleRelease$SeedPostfix' | sg lxd $CGROUP_SUFFIX"
                        sleep 300
                fi
                n=$((n+1))
        done

with the "sleep 300" in there bracketed so that this would not "step on" the logging as you had indicated.

Now here is what I'm getting from debug (and note the operations that went on in the next chunk of output after this that I was doing in another terminal session to "clean up" zfs).

==============================================
Launch Oracle LXD Seed Container...           
==============================================

==============================================
(takes a minute or two ... patience) ...      
==============================================

DEBUG  [2022-05-11T09:56:34-05:00] Connecting to a local LXD over a Unix socket 
DEBUG  [2022-05-11T09:56:34-05:00] Sending request to LXD                        etag= method=GET url="http://unix.socket/1.0"
DEBUG  [2022-05-11T09:56:34-05:00] Got response struct from LXD                 
DEBUG  [2022-05-11T09:56:34-05:00] 
    {
        "config": {
            "cluster.https_address": "10.209.53.1:8443",
            "core.https_address": "10.209.53.1:8443",
            "core.trust_password": true
        },
        "api_extensions": [
            "storage_zfs_remove_snapshots",
            "container_host_shutdown_timeout",
            "container_stop_priority",
            "container_syscall_filtering",
            "auth_pki",
            "container_last_used_at",
            "etag",
            "patch",
            "usb_devices",
            "https_allowed_credentials",
            "image_compression_algorithm",
            "directory_manipulation",
            "container_cpu_time",
            "storage_zfs_use_refquota",
            "storage_lvm_mount_options",
            "network",
            "profile_usedby",
            "container_push",
            "container_exec_recording",
            "certificate_update",
            "container_exec_signal_handling",
            "gpu_devices",
            "container_image_properties",
            "migration_progress",
            "id_map",
            "network_firewall_filtering",
            "network_routes",
            "storage",
            "file_delete",
            "file_append",
            "network_dhcp_expiry",
            "storage_lvm_vg_rename",
            "storage_lvm_thinpool_rename",
            "network_vlan",
            "image_create_aliases",
            "container_stateless_copy",
            "container_only_migration",
            "storage_zfs_clone_copy",
            "unix_device_rename",
            "storage_lvm_use_thinpool",
            "storage_rsync_bwlimit",
            "network_vxlan_interface",
            "storage_btrfs_mount_options",
            "entity_description",
            "image_force_refresh",
            "storage_lvm_lv_resizing",
            "id_map_base",
            "file_symlinks",
            "container_push_target",
            "network_vlan_physical",
            "storage_images_delete",
            "container_edit_metadata",
            "container_snapshot_stateful_migration",
            "storage_driver_ceph",
            "storage_ceph_user_name",
            "resource_limits",
            "storage_volatile_initial_source",
            "storage_ceph_force_osd_reuse",
            "storage_block_filesystem_btrfs",
            "resources",
            "kernel_limits",
            "storage_api_volume_rename",
            "macaroon_authentication",
            "network_sriov",
            "console",
            "restrict_devlxd",
            "migration_pre_copy",
            "infiniband",
            "maas_network",
            "devlxd_events",
            "proxy",
            "network_dhcp_gateway",
            "file_get_symlink",
            "network_leases",
            "unix_device_hotplug",
            "storage_api_local_volume_handling",
            "operation_description",
            "clustering",
            "event_lifecycle",
            "storage_api_remote_volume_handling",
            "nvidia_runtime",
            "container_mount_propagation",
            "container_backup",
            "devlxd_images",
            "container_local_cross_pool_handling",
            "proxy_unix",
            "proxy_udp",
            "clustering_join",
            "proxy_tcp_udp_multi_port_handling",
            "network_state",
            "proxy_unix_dac_properties",
            "container_protection_delete",
            "unix_priv_drop",
            "pprof_http",
            "proxy_haproxy_protocol",
            "network_hwaddr",
            "proxy_nat",
            "network_nat_order",
            "container_full",
            "candid_authentication",
            "backup_compression",
            "candid_config",
            "nvidia_runtime_config",
            "storage_api_volume_snapshots",
            "storage_unmapped",
            "projects",
            "candid_config_key",
            "network_vxlan_ttl",
            "container_incremental_copy",
            "usb_optional_vendorid",
            "snapshot_scheduling",
            "snapshot_schedule_aliases",
            "container_copy_project",
            "clustering_server_address",
            "clustering_image_replication",
            "container_protection_shift",
            "snapshot_expiry",
            "container_backup_override_pool",
            "snapshot_expiry_creation",
            "network_leases_location",
            "resources_cpu_socket",
            "resources_gpu",
            "resources_numa",
            "kernel_features",
            "id_map_current",
            "event_location",
            "storage_api_remote_volume_snapshots",
            "network_nat_address",
            "container_nic_routes",
            "rbac",
            "cluster_internal_copy",
            "seccomp_notify",
            "lxc_features",
            "container_nic_ipvlan",
            "network_vlan_sriov",
            "storage_cephfs",
            "container_nic_ipfilter",
            "resources_v2",
            "container_exec_user_group_cwd",
            "container_syscall_intercept",
            "container_disk_shift",
            "storage_shifted",
            "resources_infiniband",
            "daemon_storage",
            "instances",
            "image_types",
            "resources_disk_sata",
            "clustering_roles",
            "images_expiry",
            "resources_network_firmware",
            "backup_compression_algorithm",
            "ceph_data_pool_name",
            "container_syscall_intercept_mount",
            "compression_squashfs",
            "container_raw_mount",
            "container_nic_routed",
            "container_syscall_intercept_mount_fuse",
            "container_disk_ceph",
            "virtual-machines",
            "image_profiles",
            "clustering_architecture",
            "resources_disk_id",
            "storage_lvm_stripes",
            "vm_boot_priority",
            "unix_hotplug_devices",
            "api_filtering",
            "instance_nic_network",
            "clustering_sizing",
            "firewall_driver",
            "projects_limits",
            "container_syscall_intercept_hugetlbfs",
            "limits_hugepages",
            "container_nic_routed_gateway",
            "projects_restrictions",
            "custom_volume_snapshot_expiry",
            "volume_snapshot_scheduling",
            "trust_ca_certificates",
            "snapshot_disk_usage",
            "clustering_edit_roles",
            "container_nic_routed_host_address",
            "container_nic_ipvlan_gateway",
            "resources_usb_pci",
            "resources_cpu_threads_numa",
            "resources_cpu_core_die",
            "api_os",
            "container_nic_routed_host_table",
            "container_nic_ipvlan_host_table",
            "container_nic_ipvlan_mode",
            "resources_system",
            "images_push_relay",
            "network_dns_search",
            "container_nic_routed_limits",
            "instance_nic_bridged_vlan",
            "network_state_bond_bridge",
            "usedby_consistency",
            "custom_block_volumes",
            "clustering_failure_domains",
            "resources_gpu_mdev",
            "console_vga_type",
            "projects_limits_disk",
            "network_type_macvlan",
            "network_type_sriov",
            "container_syscall_intercept_bpf_devices",
            "network_type_ovn",
            "projects_networks",
            "projects_networks_restricted_uplinks",
            "custom_volume_backup",
            "backup_override_name",
            "storage_rsync_compression",
            "network_type_physical",
            "network_ovn_external_subnets",
            "network_ovn_nat",
            "network_ovn_external_routes_remove",
            "tpm_device_type",
            "storage_zfs_clone_copy_rebase",
            "gpu_mdev",
            "resources_pci_iommu",
            "resources_network_usb",
            "resources_disk_address",
            "network_physical_ovn_ingress_mode",
            "network_ovn_dhcp",
            "network_physical_routes_anycast",
            "projects_limits_instances",
            "network_state_vlan",
            "instance_nic_bridged_port_isolation",
            "instance_bulk_state_change",
            "network_gvrp",
            "instance_pool_move",
            "gpu_sriov",
            "pci_device_type",
            "storage_volume_state",
            "network_acl",
            "migration_stateful",
            "disk_state_quota",
            "storage_ceph_features",
            "projects_compression",
            "projects_images_remote_cache_expiry",
            "certificate_project",
            "network_ovn_acl",
            "projects_images_auto_update",
            "projects_restricted_cluster_target",
            "images_default_architecture",
            "network_ovn_acl_defaults",
            "gpu_mig",
            "project_usage",
            "network_bridge_acl",
            "warnings",
            "projects_restricted_backups_and_snapshots",
            "clustering_join_token",
            "clustering_description",
            "server_trusted_proxy",
            "clustering_update_cert",
            "storage_api_project",
            "server_instance_driver_operational",
            "server_supported_storage_drivers",
            "event_lifecycle_requestor_address",
            "resources_gpu_usb",
            "clustering_evacuation",
            "network_ovn_nat_address",
            "network_bgp",
            "network_forward",
            "custom_volume_refresh",
            "network_counters_errors_dropped",
            "metrics",
            "image_source_project",
            "clustering_config",
            "network_peer",
            "linux_sysctl",
            "network_dns",
            "ovn_nic_acceleration",
            "certificate_self_renewal",
            "instance_project_move",
            "storage_volume_project_move",
            "cloud_init",
            "network_dns_nat",
            "database_leader",
            "instance_all_projects",
            "clustering_groups",
            "ceph_rbd_du",
            "instance_get_full",
            "qemu_metrics",
            "gpu_mig_uuid",
            "event_project",
            "clustering_evacuation_live",
            "instance_allow_inconsistent_copy",
            "network_state_ovn",
            "storage_volume_api_filtering",
            "image_restrictions",
            "storage_zfs_export",
            "network_dns_records",
            "storage_zfs_reserve_space",
            "network_acl_log",
            "storage_zfs_blocksize",
            "metrics_cpu_seconds",
            "instance_snapshot_never",
            "certificate_token",
            "instance_nic_routed_neighbor_probe",
            "event_hub",
            "agent_nic_config",
            "projects_restricted_intercept",
            "metrics_authentication",
            "images_target_project",
            "cluster_migration_inconsistent_copy",
            "cluster_ovn_chassis",
            "container_syscall_intercept_sched_setscheduler",
            "storage_lvm_thinpool_metadata_size",
            "storage_volume_state_total",
            "instance_file_head",
            "instances_nic_host_name",
            "image_copy_profile",
            "container_syscall_intercept_sysinfo",
            "clustering_evacuation_mode"
        ],
        "api_status": "stable",
        "api_version": "1.0",
        "auth": "trusted",
        "public": false,
        "auth_methods": [
            "tls"
        ],
        "environment": {
            "addresses": [
                "10.209.53.1:8443"
            ],
            "architectures": [
                "x86_64",
                "i686"
            ],
            "certificate": "-----BEGIN CERTIFICATE-----\nMIICBDCCAYmgAwIBAgIQAwl2IyAmGY2+3ouePQ88zjAKBggqhkjOPQQDAzA0MRww\nGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QGYzNHN2\nMTAeFw0yMjA1MTExNDU1MTVaFw0zMjA1MDgxNDU1MTVaMDQxHDAaBgNVBAoTE2xp\nbnV4Y29udGFpbmVycy5vcmcxFDASBgNVBAMMC3Jvb3RAZjM0c3YxMHYwEAYHKoZI\nzj0CAQYFK4EEACIDYgAEts7SvdM2DhS4l/UYWeX/HsK7w07rUnAFwnAJQNDTCWac\nIXhK3VFCSQ0CjBjFozjPIaOQOklAsrz7qbfk13cthBsyygvbuKi1lfTPR7KBzJFY\ngpugSZwdApBkfPChIkOyo2AwXjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYI\nKwYBBQUHAwEwDAYDVR0TAQH/BAIwADApBgNVHREEIjAgggZmMzRzdjGHBH8AAAGH\nEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDaQAwZgIxALSKfieSrggNLT5/\nYPQrfS/xz1KDfjV5Dt2Fsaqi4RkKPNZ5SnCY61mhC5J46lTvCgIxAIfA+SiOoCdq\nzQv8qxTEyv9gonPNdXPrZmYhUwHokYW1NVWJH7bdn3HD5T8jqaqWJA==\n-----END CERTIFICATE-----\n",
            "certificate_fingerprint": "65cbbaa39a3fb9be2da92e754ecf3db4341d1fec2184b9e2b80802bd62147c0b",
            "driver": "lxc",
            "driver_version": "4.0.12",
            "firewall": "nftables",
            "kernel": "Linux",
            "kernel_architecture": "x86_64",
            "kernel_features": {
                "idmapped_mounts": "true",
                "netnsid_getifaddrs": "true",
                "seccomp_listener": "true",
                "seccomp_listener_continue": "true",
                "shiftfs": "false",
                "uevent_injection": "true",
                "unpriv_fscaps": "true"
            },
            "kernel_version": "5.17.6-100.fc34.x86_64",
            "lxc_features": {
                "cgroup2": "true",
                "core_scheduling": "true",
                "devpts_fd": "true",
                "idmapped_mounts_v2": "true",
                "mount_injection_file": "true",
                "network_gateway_device_route": "true",
                "network_ipvlan": "true",
                "network_l2proxy": "true",
                "network_phys_macvlan_mtu": "true",
                "network_veth_router": "true",
                "pidfd": "true",
                "seccomp_allow_deny_syntax": "true",
                "seccomp_notify": "true",
                "seccomp_proxy_send_notify_fd": "true"
            },
            "os_name": "Fedora",
            "os_version": "34",
            "project": "default",
            "server": "lxd",
            "server_clustered": true,
            "server_event_mode": "full-mesh",
            "server_name": "f34sv1",
            "server_pid": 49758,
            "server_version": "5.1",
            "storage": "zfs",
            "storage_version": "2.1.4-1",
            "storage_supported_drivers": [
                {
                    "Name": "btrfs",
                    "Version": "5.4.1",
                    "Remote": false
                },
                {
                    "Name": "cephfs",
                    "Version": "15.2.14",
                    "Remote": true
                },
                {
                    "Name": "dir",
                    "Version": "1",
                    "Remote": false
                },
                {
                    "Name": "lvm",
                    "Version": "2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0",
                    "Remote": false
                },
                {
                    "Name": "zfs",
                    "Version": "2.1.4-1",
                    "Remote": false
                },
                {
                    "Name": "ceph",
                    "Version": "15.2.14",
                    "Remote": true
                }
            ]
        }
    } 
Creating oel84d10
DEBUG  [2022-05-11T09:56:34-05:00] Connecting to a remote simplestreams server   URL="https://images.linuxcontainers.org"
DEBUG  [2022-05-11T09:56:34-05:00] Connected to the websocket: ws://unix.socket/1.0/events 
DEBUG  [2022-05-11T09:56:34-05:00] Sending request to LXD                        etag= method=POST url="http://unix.socket/1.0/instances"
DEBUG  [2022-05-11T09:56:34-05:00] 
    {
        "architecture": "",
        "config": {},
        "devices": {},
        "ephemeral": false,
        "profiles": [
            "olxc_sx1a"
        ],
        "stateful": false,
        "description": "",
        "name": "oel84d10",
        "source": {
            "type": "image",
            "certificate": "",
            "alias": "oracle/8/amd64",
            "server": "https://images.linuxcontainers.org",
            "protocol": "simplestreams",
            "mode": "pull",
            "allow_inconsistent": false
        },
        "instance_type": "",
        "type": "container"
    } 
Error: Failed instance creation: Post "http://unix.socket/1.0/instances": net/http: timeout awaiting response headers
Error: The instance is already stopped
DEBUG  [2022-05-11T10:07:05-05:00] Connecting to a local LXD over a Unix socket 
DEBUG  [2022-05-11T10:07:05-05:00] Sending request to LXD                        etag= method=GET url="http://unix.socket/1.0"
DEBUG  [2022-05-11T10:07:05-05:00] Got response struct from LXD                 
DEBUG  [2022-05-11T10:07:05-05:00] 
    {
        "config": {
            "cluster.https_address": "10.209.53.1:8443",
            "core.https_address": "10.209.53.1:8443",
            "core.trust_password": true
        },
        "api_extensions": [
            "storage_zfs_remove_snapshots",
            "container_host_shutdown_timeout",
            "container_stop_priority",
            "container_syscall_filtering",
            "auth_pki",
            "container_last_used_at",
            "etag",
            "patch",
            "usb_devices",
            "https_allowed_credentials",
            "image_compression_algorithm",
            "directory_manipulation",
            "container_cpu_time",
            "storage_zfs_use_refquota",
            "storage_lvm_mount_options",
            "network",
            "profile_usedby",
            "container_push",
            "container_exec_recording",
            "certificate_update",
            "container_exec_signal_handling",
            "gpu_devices",
            "container_image_properties",
            "migration_progress",
            "id_map",
            "network_firewall_filtering",
            "network_routes",
            "storage",
            "file_delete",
            "file_append",
            "network_dhcp_expiry",
            "storage_lvm_vg_rename",
            "storage_lvm_thinpool_rename",
            "network_vlan",
            "image_create_aliases",
            "container_stateless_copy",
            "container_only_migration",
            "storage_zfs_clone_copy",
            "unix_device_rename",
            "storage_lvm_use_thinpool",
            "storage_rsync_bwlimit",
            "network_vxlan_interface",
            "storage_btrfs_mount_options",
            "entity_description",
            "image_force_refresh",
            "storage_lvm_lv_resizing",
            "id_map_base",
            "file_symlinks",
            "container_push_target",
            "network_vlan_physical",
            "storage_images_delete",
            "container_edit_metadata",
            "container_snapshot_stateful_migration",
            "storage_driver_ceph",
            "storage_ceph_user_name",
            "resource_limits",
            "storage_volatile_initial_source",
            "storage_ceph_force_osd_reuse",
            "storage_block_filesystem_btrfs",
            "resources",
            "kernel_limits",
            "storage_api_volume_rename",
            "macaroon_authentication",
            "network_sriov",
            "console",
            "restrict_devlxd",
            "migration_pre_copy",
            "infiniband",
            "maas_network",
            "devlxd_events",
            "proxy",
            "network_dhcp_gateway",
            "file_get_symlink",
            "network_leases",
            "unix_device_hotplug",
            "storage_api_local_volume_handling",
            "operation_description",
            "clustering",
            "event_lifecycle",
            "storage_api_remote_volume_handling",
            "nvidia_runtime",
            "container_mount_propagation",
            "container_backup",
            "devlxd_images",
            "container_local_cross_pool_handling",
            "proxy_unix",
            "proxy_udp",
            "clustering_join",
            "proxy_tcp_udp_multi_port_handling",
            "network_state",
            "proxy_unix_dac_properties",
            "container_protection_delete",
            "unix_priv_drop",
            "pprof_http",
            "proxy_haproxy_protocol",
            "network_hwaddr",
            "proxy_nat",
            "network_nat_order",
            "container_full",
            "candid_authentication",
            "backup_compression",
            "candid_config",
            "nvidia_runtime_config",
            "storage_api_volume_snapshots",
            "storage_unmapped",
            "projects",
            "candid_config_key",
            "network_vxlan_ttl",
            "container_incremental_copy",
            "usb_optional_vendorid",
            "snapshot_scheduling",
            "snapshot_schedule_aliases",
            "container_copy_project",
            "clustering_server_address",
            "clustering_image_replication",
            "container_protection_shift",
            "snapshot_expiry",
            "container_backup_override_pool",
            "snapshot_expiry_creation",
            "network_leases_location",
            "resources_cpu_socket",
            "resources_gpu",
            "resources_numa",
            "kernel_features",
            "id_map_current",
            "event_location",
            "storage_api_remote_volume_snapshots",
            "network_nat_address",
            "container_nic_routes",
            "rbac",
            "cluster_internal_copy",
            "seccomp_notify",
            "lxc_features",
            "container_nic_ipvlan",
            "network_vlan_sriov",
            "storage_cephfs",
            "container_nic_ipfilter",
            "resources_v2",
            "container_exec_user_group_cwd",
            "container_syscall_intercept",
            "container_disk_shift",
            "storage_shifted",
            "resources_infiniband",
            "daemon_storage",
            "instances",
            "image_types",
            "resources_disk_sata",
            "clustering_roles",
            "images_expiry",
            "resources_network_firmware",
            "backup_compression_algorithm",
            "ceph_data_pool_name",
            "container_syscall_intercept_mount",
            "compression_squashfs",
            "container_raw_mount",
            "container_nic_routed",
            "container_syscall_intercept_mount_fuse",
            "container_disk_ceph",
            "virtual-machines",
            "image_profiles",
            "clustering_architecture",
            "resources_disk_id",
            "storage_lvm_stripes",
            "vm_boot_priority",
            "unix_hotplug_devices",
            "api_filtering",
            "instance_nic_network",
            "clustering_sizing",
            "firewall_driver",
            "projects_limits",
            "container_syscall_intercept_hugetlbfs",
            "limits_hugepages",
            "container_nic_routed_gateway",
            "projects_restrictions",
            "custom_volume_snapshot_expiry",
            "volume_snapshot_scheduling",
            "trust_ca_certificates",
            "snapshot_disk_usage",
            "clustering_edit_roles",
            "container_nic_routed_host_address",
            "container_nic_ipvlan_gateway",
            "resources_usb_pci",
            "resources_cpu_threads_numa",
            "resources_cpu_core_die",
            "api_os",
            "container_nic_routed_host_table",
            "container_nic_ipvlan_host_table",
            "container_nic_ipvlan_mode",
            "resources_system",
            "images_push_relay",
            "network_dns_search",
            "container_nic_routed_limits",
            "instance_nic_bridged_vlan",
            "network_state_bond_bridge",
            "usedby_consistency",
            "custom_block_volumes",
            "clustering_failure_domains",
            "resources_gpu_mdev",
            "console_vga_type",
            "projects_limits_disk",
            "network_type_macvlan",
            "network_type_sriov",
            "container_syscall_intercept_bpf_devices",
            "network_type_ovn",
            "projects_networks",
            "projects_networks_restricted_uplinks",
            "custom_volume_backup",
            "backup_override_name",
            "storage_rsync_compression",
            "network_type_physical",
            "network_ovn_external_subnets",
            "network_ovn_nat",
            "network_ovn_external_routes_remove",
            "tpm_device_type",
            "storage_zfs_clone_copy_rebase",
            "gpu_mdev",
            "resources_pci_iommu",
            "resources_network_usb",
            "resources_disk_address",
            "network_physical_ovn_ingress_mode",
            "network_ovn_dhcp",
            "network_physical_routes_anycast",
            "projects_limits_instances",
            "network_state_vlan",
            "instance_nic_bridged_port_isolation",
            "instance_bulk_state_change",
            "network_gvrp",
            "instance_pool_move",
            "gpu_sriov",
            "pci_device_type",
            "storage_volume_state",
            "network_acl",
            "migration_stateful",
            "disk_state_quota",
            "storage_ceph_features",
            "projects_compression",
            "projects_images_remote_cache_expiry",
            "certificate_project",
            "network_ovn_acl",
            "projects_images_auto_update",
            "projects_restricted_cluster_target",
            "images_default_architecture",
            "network_ovn_acl_defaults",
            "gpu_mig",
            "project_usage",
            "network_bridge_acl",
            "warnings",
            "projects_restricted_backups_and_snapshots",
            "clustering_join_token",
            "clustering_description",
            "server_trusted_proxy",
            "clustering_update_cert",
            "storage_api_project",
            "server_instance_driver_operational",
            "server_supported_storage_drivers",
            "event_lifecycle_requestor_address",
            "resources_gpu_usb",
            "clustering_evacuation",
            "network_ovn_nat_address",
            "network_bgp",
            "network_forward",
            "custom_volume_refresh",
            "network_counters_errors_dropped",
            "metrics",
            "image_source_project",
            "clustering_config",
            "network_peer",
            "linux_sysctl",
            "network_dns",
            "ovn_nic_acceleration",
            "certificate_self_renewal",
            "instance_project_move",
            "storage_volume_project_move",
            "cloud_init",
            "network_dns_nat",
            "database_leader",
            "instance_all_projects",
            "clustering_groups",
            "ceph_rbd_du",
            "instance_get_full",
            "qemu_metrics",
            "gpu_mig_uuid",
            "event_project",
            "clustering_evacuation_live",
            "instance_allow_inconsistent_copy",
            "network_state_ovn",
            "storage_volume_api_filtering",
            "image_restrictions",
            "storage_zfs_export",
            "network_dns_records",
            "storage_zfs_reserve_space",
            "network_acl_log",
            "storage_zfs_blocksize",
            "metrics_cpu_seconds",
            "instance_snapshot_never",
            "certificate_token",
            "instance_nic_routed_neighbor_probe",
            "event_hub",
            "agent_nic_config",
            "projects_restricted_intercept",
            "metrics_authentication",
            "images_target_project",
            "cluster_migration_inconsistent_copy",
            "cluster_ovn_chassis",
            "container_syscall_intercept_sched_setscheduler",
            "storage_lvm_thinpool_metadata_size",
            "storage_volume_state_total",
            "instance_file_head",
            "instances_nic_host_name",
            "image_copy_profile",
            "container_syscall_intercept_sysinfo",
            "clustering_evacuation_mode"
        ],
        "api_status": "stable",
        "api_version": "1.0",
        "auth": "trusted",
        "public": false,
        "auth_methods": [
            "tls"
        ],
        "environment": {
            "addresses": [
                "10.209.53.1:8443"
            ],
            "architectures": [
                "x86_64",
                "i686"
            ],
            "certificate": "-----BEGIN CERTIFICATE-----\nMIICBDCCAYmgAwIBAgIQAwl2IyAmGY2+3ouePQ88zjAKBggqhkjOPQQDAzA0MRww\nGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QGYzNHN2\nMTAeFw0yMjA1MTExNDU1MTVaFw0zMjA1MDgxNDU1MTVaMDQxHDAaBgNVBAoTE2xp\nbnV4Y29udGFpbmVycy5vcmcxFDASBgNVBAMMC3Jvb3RAZjM0c3YxMHYwEAYHKoZI\nzj0CAQYFK4EEACIDYgAEts7SvdM2DhS4l/UYWeX/HsK7w07rUnAFwnAJQNDTCWac\nIXhK3VFCSQ0CjBjFozjPIaOQOklAsrz7qbfk13cthBsyygvbuKi1lfTPR7KBzJFY\ngpugSZwdApBkfPChIkOyo2AwXjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYI\nKwYBBQUHAwEwDAYDVR0TAQH/BAIwADApBgNVHREEIjAgggZmMzRzdjGHBH8AAAGH\nEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDaQAwZgIxALSKfieSrggNLT5/\nYPQrfS/xz1KDfjV5Dt2Fsaqi4RkKPNZ5SnCY61mhC5J46lTvCgIxAIfA+SiOoCdq\nzQv8qxTEyv9gonPNdXPrZmYhUwHokYW1NVWJH7bdn3HD5T8jqaqWJA==\n-----END CERTIFICATE-----\n",
            "certificate_fingerprint": "65cbbaa39a3fb9be2da92e754ecf3db4341d1fec2184b9e2b80802bd62147c0b",
            "driver": "lxc",
            "driver_version": "4.0.12",
            "firewall": "nftables",
            "kernel": "Linux",
            "kernel_architecture": "x86_64",
            "kernel_features": {
                "idmapped_mounts": "true",
                "netnsid_getifaddrs": "true",
                "seccomp_listener": "true",
                "seccomp_listener_continue": "true",
                "shiftfs": "false",
                "uevent_injection": "true",
                "unpriv_fscaps": "true"
            },
            "kernel_version": "5.17.6-100.fc34.x86_64",
            "lxc_features": {
                "cgroup2": "true",
                "core_scheduling": "true",
                "devpts_fd": "true",
                "idmapped_mounts_v2": "true",
                "mount_injection_file": "true",
                "network_gateway_device_route": "true",
                "network_ipvlan": "true",
                "network_l2proxy": "true",
                "network_phys_macvlan_mtu": "true",
                "network_veth_router": "true",
                "pidfd": "true",
                "seccomp_allow_deny_syntax": "true",
                "seccomp_notify": "true",
                "seccomp_proxy_send_notify_fd": "true"
            },
            "os_name": "Fedora",
            "os_version": "34",
            "project": "default",
            "server": "lxd",
            "server_clustered": true,
            "server_event_mode": "full-mesh",
            "server_name": "f34sv1",
            "server_pid": 49758,
            "server_version": "5.1",
            "storage": "zfs",
            "storage_version": "2.1.4-1",
            "storage_supported_drivers": [
                {
                    "Name": "btrfs",
                    "Version": "5.4.1",
                    "Remote": false
                },
                {
                    "Name": "cephfs",
                    "Version": "15.2.14",
                    "Remote": true
                },
                {
                    "Name": "dir",
                    "Version": "1",
                    "Remote": false
                },
                {
                    "Name": "lvm",
                    "Version": "2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0",
                    "Remote": false
                },
                {
                    "Name": "zfs",
                    "Version": "2.1.4-1",
                    "Remote": false
                },
                {
                    "Name": "ceph",
                    "Version": "15.2.14",
                    "Remote": true
                }
            ]
        }
    } 
Creating oel84d10
DEBUG  [2022-05-11T10:07:05-05:00] Connecting to a remote simplestreams server   URL="https://images.linuxcontainers.org"
DEBUG  [2022-05-11T10:07:05-05:00] Connected to the websocket: ws://unix.socket/1.0/events 
DEBUG  [2022-05-11T10:07:05-05:00] Sending request to LXD                        etag= method=POST url="http://unix.socket/1.0/instances"
DEBUG  [2022-05-11T10:07:05-05:00] 
    {
        "architecture": "",
        "config": {},
        "devices": {},
        "ephemeral": false,
        "profiles": [
            "olxc_sx1a"
        ],
        "stateful": false,
        "description": "",
        "name": "oel84d10",
        "source": {
            "type": "image",
            "certificate": "",
            "alias": "oracle/8/amd64",
            "server": "https://images.linuxcontainers.org",
            "protocol": "simplestreams",
            "mode": "pull",
            "allow_inconsistent": false
        },
        "instance_type": "",
        "type": "container"
    } 
DEBUG  [2022-05-11T10:07:05-05:00] Got operation from LXD                       
DEBUG  [2022-05-11T10:07:05-05:00] 
    {
        "id": "1f005db2-b0c6-4ffd-80d7-ac85d454ac54",
        "class": "task",
        "description": "Creating instance",
        "created_at": "2022-05-11T10:07:05.167961848-05:00",
        "updated_at": "2022-05-11T10:07:05.167961848-05:00",
        "status": "Running",
        "status_code": 103,
        "resources": {
            "containers": [
                "/1.0/containers/oel84d10"
            ],
            "instances": [
                "/1.0/instances/oel84d10"
            ]
        },
        "metadata": null,
        "may_cancel": false,
        "err": "",
        "location": "f34sv1"
    } 
DEBUG  [2022-05-11T10:07:05-05:00] Sending request to LXD                        etag= method=GET url="http://unix.socket/1.0/operations/1f005db2-b0c6-4ffd-80d7-ac85d454ac54"
DEBUG  [2022-05-11T10:07:05-05:00] Got response struct from LXD                 
DEBUG  [2022-05-11T10:07:05-05:00] 
    {
        "id": "1f005db2-b0c6-4ffd-80d7-ac85d454ac54",
        "class": "task",
        "description": "Creating instance",
        "created_at": "2022-05-11T10:07:05.167961848-05:00",
        "updated_at": "2022-05-11T10:07:05.167961848-05:00",
        "status": "Running",
        "status_code": 103,
        "resources": {
            "containers": [
                "/1.0/containers/oel84d10"
            ],
            "instances": [
                "/1.0/instances/oel84d10"
            ]
        },
        "metadata": null,
        "may_cancel": false,
        "err": "",
        "location": "f34sv1"
    } 
Error: Failed instance creation: Failed creating instance from image: Error inserting volume "b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6" for project "default" in pool "local" of type "images" into database "UNIQUE constraint failed: storage_volumes.storage_pool_id, storage_volumes.node_id, storage_volumes.project_id, storage_volumes.name, storage_volumes.type"

Then over here in another terminal I was doing some additional steps to see if I could get the cleanup zfs steps down.

orabuntu@olxd1:~/Desktop$ ssh 192.168.1.146
orabuntu@192.168.1.146's password: 
Activate the web console with: systemctl enable --now cockpit.socket

Last login: Wed May 11 09:53:15 2022 from 192.168.1.99
[orabuntu@f34sv1 ~]$ lxc list
If this is your first time running LXD on this machine, you should also run: lxd init
To start your first container, try: lxc launch ubuntu:22.04
Or for a virtual machine: lxc launch ubuntu:22.04 --vm

+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+
[orabuntu@f34sv1 ~]$ lxc cluster list
+--------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
|  NAME  |           URL            |      ROLES      | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      |
+--------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| f34sv1 | https://10.209.53.1:8443 | database-leader | x86_64       | default        |             | ONLINE | Fully operational |
|        |                          | database        |              |                |             |        |                   |
+--------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
[orabuntu@f34sv1 ~]$ lxc list
+------+-------+------+------+------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+------+-------+------+------+------+-----------+----------+
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
[sudo] password for orabuntu: 
NAME                                 USED  AVAIL     REFER  MOUNTPOINT
olxd-5f5e                           1.86M  28.6G       96K  legacy
olxd-5f5e/containers                  96K  28.6G       96K  legacy
olxd-5f5e/custom                      96K  28.6G       96K  legacy
olxd-5f5e/deleted                    480K  28.6G       96K  legacy
olxd-5f5e/deleted/containers          96K  28.6G       96K  legacy
olxd-5f5e/deleted/custom              96K  28.6G       96K  legacy
olxd-5f5e/deleted/images              96K  28.6G       96K  legacy
olxd-5f5e/deleted/virtual-machines    96K  28.6G       96K  legacy
olxd-5f5e/images                      96K  28.6G       96K  legacy
olxd-5f5e/virtual-machines            96K  28.6G       96K  legacy
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME                                 USED  AVAIL     REFER  MOUNTPOINT
olxd-5f5e                           1.86M  28.6G       96K  legacy
olxd-5f5e/containers                  96K  28.6G       96K  legacy
olxd-5f5e/custom                      96K  28.6G       96K  legacy
olxd-5f5e/deleted                    480K  28.6G       96K  legacy
olxd-5f5e/deleted/containers          96K  28.6G       96K  legacy
olxd-5f5e/deleted/custom              96K  28.6G       96K  legacy
olxd-5f5e/deleted/images              96K  28.6G       96K  legacy
olxd-5f5e/deleted/virtual-machines    96K  28.6G       96K  legacy
olxd-5f5e/images                      96K  28.6G       96K  legacy
olxd-5f5e/virtual-machines            96K  28.6G       96K  legacy
[orabuntu@f34sv1 ~]$ ls -lrt
total 8
drwxr-xr-x. 3 orabuntu orabuntu  51 May 11 08:54 Downloads
-rw-r--r--. 1 orabuntu orabuntu 486 May 11 09:47 nameserver.lst
drwx------. 3 orabuntu orabuntu  17 May 11 09:55 snap
-rw-r--r--. 1 root     root       0 May 11 09:56 lxc-monitor.err
-rw-r--r--. 1 root     root     187 May 11 09:56 lxc-monitor.log
[orabuntu@f34sv1 ~]$ ls -lrt
total 8
drwxr-xr-x. 3 orabuntu orabuntu   51 May 11 08:54 Downloads
-rw-r--r--. 1 orabuntu orabuntu  486 May 11 09:47 nameserver.lst
drwx------. 3 orabuntu orabuntu   17 May 11 09:55 snap
-rw-r--r--. 1 root     root        0 May 11 09:56 lxc-monitor.err
-rw-r--r--. 1 root     root     3042 May 11 09:57 lxc-monitor.log
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME                                 USED  AVAIL     REFER  MOUNTPOINT
olxd-5f5e                           1.86M  28.6G       96K  legacy
olxd-5f5e/containers                  96K  28.6G       96K  legacy
olxd-5f5e/custom                      96K  28.6G       96K  legacy
olxd-5f5e/deleted                    480K  28.6G       96K  legacy
olxd-5f5e/deleted/containers          96K  28.6G       96K  legacy
olxd-5f5e/deleted/custom              96K  28.6G       96K  legacy
olxd-5f5e/deleted/images              96K  28.6G       96K  legacy
olxd-5f5e/deleted/virtual-machines    96K  28.6G       96K  legacy
olxd-5f5e/images                      96K  28.6G       96K  legacy
olxd-5f5e/virtual-machines            96K  28.6G       96K  legacy
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME                                 USED  AVAIL     REFER  MOUNTPOINT
olxd-5f5e                           1.86M  28.6G       96K  legacy
olxd-5f5e/containers                  96K  28.6G       96K  legacy
olxd-5f5e/custom                      96K  28.6G       96K  legacy
olxd-5f5e/deleted                    480K  28.6G       96K  legacy
olxd-5f5e/deleted/containers          96K  28.6G       96K  legacy
olxd-5f5e/deleted/custom              96K  28.6G       96K  legacy
olxd-5f5e/deleted/images              96K  28.6G       96K  legacy
olxd-5f5e/deleted/virtual-machines    96K  28.6G       96K  legacy
olxd-5f5e/images                      96K  28.6G       96K  legacy
olxd-5f5e/virtual-machines            96K  28.6G       96K  legacy
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME                                 USED  AVAIL     REFER  MOUNTPOINT
olxd-5f5e                           1.86M  28.6G       96K  legacy
olxd-5f5e/containers                  96K  28.6G       96K  legacy
olxd-5f5e/custom                      96K  28.6G       96K  legacy
olxd-5f5e/deleted                    480K  28.6G       96K  legacy
olxd-5f5e/deleted/containers          96K  28.6G       96K  legacy
olxd-5f5e/deleted/custom              96K  28.6G       96K  legacy
olxd-5f5e/deleted/images              96K  28.6G       96K  legacy
olxd-5f5e/deleted/virtual-machines    96K  28.6G       96K  legacy
olxd-5f5e/images                      96K  28.6G       96K  legacy
olxd-5f5e/virtual-machines            96K  28.6G       96K  legacy
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME                                 USED  AVAIL     REFER  MOUNTPOINT
olxd-5f5e                           1.86M  28.6G       96K  legacy
olxd-5f5e/containers                  96K  28.6G       96K  legacy
olxd-5f5e/custom                      96K  28.6G       96K  legacy
olxd-5f5e/deleted                    480K  28.6G       96K  legacy
olxd-5f5e/deleted/containers          96K  28.6G       96K  legacy
olxd-5f5e/deleted/custom              96K  28.6G       96K  legacy
olxd-5f5e/deleted/images              96K  28.6G       96K  legacy
olxd-5f5e/deleted/virtual-machines    96K  28.6G       96K  legacy
olxd-5f5e/images                      96K  28.6G       96K  legacy
olxd-5f5e/virtual-machines            96K  28.6G       96K  legacy
[orabuntu@f34sv1 ~]$ ls -lrt
total 12
drwxr-xr-x. 3 orabuntu orabuntu   51 May 11 08:54 Downloads
-rw-r--r--. 1 orabuntu orabuntu  486 May 11 09:47 nameserver.lst
drwx------. 3 orabuntu orabuntu   17 May 11 09:55 snap
-rw-r--r--. 1 root     root        0 May 11 09:56 lxc-monitor.err
-rw-r--r--. 1 root     root     4161 May 11 09:57 lxc-monitor.log
[orabuntu@f34sv1 ~]$ ls -lrt
total 12
drwxr-xr-x. 3 orabuntu orabuntu   51 May 11 08:54 Downloads
-rw-r--r--. 1 orabuntu orabuntu  486 May 11 09:47 nameserver.lst
drwx------. 3 orabuntu orabuntu   17 May 11 09:55 snap
-rw-r--r--. 1 root     root        0 May 11 09:56 lxc-monitor.err
-rw-r--r--. 1 root     root     4161 May 11 09:57 lxc-monitor.log
[orabuntu@f34sv1 ~]$ ls -lrt
total 12
drwxr-xr-x. 3 orabuntu orabuntu   51 May 11 08:54 Downloads
-rw-r--r--. 1 orabuntu orabuntu  486 May 11 09:47 nameserver.lst
drwx------. 3 orabuntu orabuntu   17 May 11 09:55 snap
-rw-r--r--. 1 root     root        0 May 11 09:56 lxc-monitor.err
-rw-r--r--. 1 root     root     4533 May 11 09:57 lxc-monitor.log
[orabuntu@f34sv1 ~]$ ls -lrt
total 12
drwxr-xr-x. 3 orabuntu orabuntu   51 May 11 08:54 Downloads
-rw-r--r--. 1 orabuntu orabuntu  486 May 11 09:47 nameserver.lst
drwx------. 3 orabuntu orabuntu   17 May 11 09:55 snap
-rw-r--r--. 1 root     root        0 May 11 09:56 lxc-monitor.err
-rw-r--r--. 1 root     root     4533 May 11 09:57 lxc-monitor.log
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME                                 USED  AVAIL     REFER  MOUNTPOINT
olxd-5f5e                           1.86M  28.6G       96K  legacy
olxd-5f5e/containers                  96K  28.6G       96K  legacy
olxd-5f5e/custom                      96K  28.6G       96K  legacy
olxd-5f5e/deleted                    480K  28.6G       96K  legacy
olxd-5f5e/deleted/containers          96K  28.6G       96K  legacy
olxd-5f5e/deleted/custom              96K  28.6G       96K  legacy
olxd-5f5e/deleted/images              96K  28.6G       96K  legacy
olxd-5f5e/deleted/virtual-machines    96K  28.6G       96K  legacy
olxd-5f5e/images                      96K  28.6G       96K  legacy
olxd-5f5e/virtual-machines            96K  28.6G       96K  legacy
[orabuntu@f34sv1 ~]$ tail -f lxc-monitor.log
time="2022-05-11T09:57:56-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:57:56-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:57:56-05:00" level=debug msg="Completed heartbeat round" duration=5.292361ms local="10.209.53.1:8443"
time="2022-05-11T09:58:06-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:58:06-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:58:06-05:00" level=debug msg="Completed heartbeat round" duration=18.984031ms local="10.209.53.1:8443"
time="2022-05-11T09:58:16-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:58:16-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:58:16-05:00" level=debug msg="Completed heartbeat round" duration=5.961386ms local="10.209.53.1:8443"
time="2022-05-11T09:58:17-05:00" level=debug msg="Success for task operation: 3dbc8217-c1b5-4cfe-91f5-b1b080bbecc3"
time="2022-05-11T09:58:26-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:58:26-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:58:26-05:00" level=debug msg="Completed heartbeat round" duration=22.35438ms local="10.209.53.1:8443"
time="2022-05-11T09:58:36-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:58:36-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:58:36-05:00" level=debug msg="Completed heartbeat round" duration=17.02173ms local="10.209.53.1:8443"
time="2022-05-11T09:58:36-05:00" level=debug msg="New task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:58:36-05:00" level=debug msg="Started task operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:58:36-05:00" level=debug msg="Connecting to a remote simplestreams server" URL="https://images.linuxcontainers.org"
time="2022-05-11T09:58:36-05:00" level=debug msg="Lock acquired for image \"b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6\""
time="2022-05-11T09:58:36-05:00" level=debug msg="Acquiring lock for image \"b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6\""
time="2022-05-11T09:58:36-05:00" level=debug msg="Database error" err="Image not found"
time="2022-05-11T09:58:36-05:00" level=info msg="Downloading image" alias=oracle/8/amd64 fingerprint=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 operation=8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4 server="https://images.linuxcontainers.org" trigger=/1.0/operations/8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4
time="2022-05-11T09:58:36-05:00" level=debug msg="Database error" err="Image not found"
^C
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME                                 USED  AVAIL     REFER  MOUNTPOINT
olxd-5f5e                           1.86M  28.6G       96K  legacy
olxd-5f5e/containers                  96K  28.6G       96K  legacy
olxd-5f5e/custom                      96K  28.6G       96K  legacy
olxd-5f5e/deleted                    480K  28.6G       96K  legacy
olxd-5f5e/deleted/containers          96K  28.6G       96K  legacy
olxd-5f5e/deleted/custom              96K  28.6G       96K  legacy
olxd-5f5e/deleted/images              96K  28.6G       96K  legacy
olxd-5f5e/deleted/virtual-machines    96K  28.6G       96K  legacy
olxd-5f5e/images                      96K  28.6G       96K  legacy
olxd-5f5e/virtual-machines            96K  28.6G       96K  legacy
[orabuntu@f34sv1 ~]$ tail -f lxc-monitor.log
time="2022-05-11T09:58:46-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:58:46-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:58:46-05:00" level=debug msg="Completed heartbeat round" duration=24.448558ms local="10.209.53.1:8443"
time="2022-05-11T09:58:56-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:58:56-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:58:56-05:00" level=debug msg="Completed heartbeat round" duration=3.37388ms local="10.209.53.1:8443"
time="2022-05-11T09:59:06-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:59:06-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:59:06-05:00" level=debug msg="Completed heartbeat round" duration=11.050051ms local="10.209.53.1:8443"
time="2022-05-11T09:59:07-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:16-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:59:16-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:59:16-05:00" level=debug msg="Completed heartbeat round" duration=5.742284ms local="10.209.53.1:8443"
time="2022-05-11T09:59:26-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:59:26-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:59:26-05:00" level=debug msg="Completed heartbeat round" duration=5.348088ms local="10.209.53.1:8443"
time="2022-05-11T09:59:36-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:59:36-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:59:36-05:00" level=debug msg="Completed heartbeat round" duration=13.097256ms local="10.209.53.1:8443"
time="2022-05-11T09:59:37-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:38-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:38-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:39-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:39-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:40-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:40-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:40-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:41-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:41-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:42-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:42-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:43-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:43-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:44-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:44-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:45-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:45-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:45-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:46-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:46-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:59:46-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:59:46-05:00" level=debug msg="Completed heartbeat round" duration=4.944594ms local="10.209.53.1:8443"
time="2022-05-11T09:59:47-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:47-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:48-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:48-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:48-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:49-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:49-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:50-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:50-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:51-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:51-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:52-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:52-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:52-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:53-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:53-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:54-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:54-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:54-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:55-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:55-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:56-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:56-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T09:59:56-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T09:59:56-05:00" level=debug msg="Completed heartbeat round" duration=17.596825ms local="10.209.53.1:8443"
time="2022-05-11T09:59:56-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:56-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:57-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:57-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:58-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:58-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:59-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T09:59:59-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:00-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:00-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:01-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:01-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:01-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:02-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:02-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:03-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:03-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:04-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:04-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:05-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:05-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:06-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:06-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:00:06-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:00:06-05:00" level=debug msg="Completed heartbeat round" duration=7.920559ms local="10.209.53.1:8443"
time="2022-05-11T10:00:06-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:06-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:07-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:07-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:08-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:08-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:09-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:09-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:10-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:10-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:10-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:11-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:11-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:12-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:12-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:13-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:13-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:14-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:14-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:14-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:15-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:16-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:16-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:00:16-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:00:16-05:00" level=debug msg="Completed heartbeat round" duration=9.288516ms local="10.209.53.1:8443"
time="2022-05-11T10:00:16-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:16-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:17-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:17-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:18-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:18-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:19-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:19-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:19-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:20-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:20-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:21-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:21-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:21-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:22-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:23-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:23-05:00" level=info msg="Image downloaded" alias=oracle/8/amd64 fingerprint=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 operation=8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4 server="https://images.linuxcontainers.org" trigger=/1.0/operations/8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4
time="2022-05-11T10:00:23-05:00" level=debug msg="Lock acquired for image \"b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6\""
time="2022-05-11T10:00:23-05:00" level=debug msg="Acquiring lock for image \"b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6\""
time="2022-05-11T10:00:23-05:00" level=debug msg="Instance operation lock created" action=create instance=oel84d10 project=default reusable=false
time="2022-05-11T10:00:23-05:00" level=info msg="Creating container" ephemeral=false instance=oel84d10 instanceType=container project=default
time="2022-05-11T10:00:23-05:00" level=debug msg="Database error" err="Network not found"
time="2022-05-11T10:00:23-05:00" level=debug msg="Database error" err="Network not found"
time="2022-05-11T10:00:23-05:00" level=debug msg="Adding device" device=eth0 instance=oel84d10 instanceType=container project=default type=nic
time="2022-05-11T10:00:23-05:00" level=debug msg="Adding device" device=root instance=oel84d10 instanceType=container project=default type=disk
time="2022-05-11T10:00:23-05:00" level=info msg="Created container" ephemeral=false instance=oel84d10 instanceType=container project=default
time="2022-05-11T10:00:23-05:00" level=debug msg="CreateInstanceFromImage started" instance=oel84d10 project=default
time="2022-05-11T10:00:23-05:00" level=debug msg="EnsureImage started" fingerprint=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:00:23-05:00" level=debug msg="Database error" err="Storage pool volume not found"
time="2022-05-11T10:00:23-05:00" level=debug msg="Running filler function" dev= driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 pool=local
time="2022-05-11T10:00:23-05:00" level=debug msg="Mounted ZFS dataset" dev=olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 pool=local
time="2022-05-11T10:00:23-05:00" level=info msg="Image unpack started" imageFile=/var/snap/lxd/common/lxd/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 volName=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:00:23-05:00" level=debug msg="Updated metadata for task Operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:26-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:00:26-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:00:26-05:00" level=debug msg="Completed heartbeat round" duration=11.115758ms local="10.209.53.1:8443"
time="2022-05-11T10:00:26-05:00" level=info msg="Image unpack stopped" imageFile=/var/snap/lxd/common/lxd/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 volName=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:00:33-05:00" level=debug msg="Unmounted ZFS dataset" dev=olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 pool=local volName=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:00:33-05:00" level=debug msg="EnsureImage finished" fingerprint=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:00:33-05:00" level=debug msg="Set new volume size" instance=oel84d10 project=default size=
time="2022-05-11T10:00:33-05:00" level=debug msg="Checking volume size" instance=oel84d10 project=default
time="2022-05-11T10:00:33-05:00" level=debug msg="Mounted ZFS dataset" dev=olxd-5f5e/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local
time="2022-05-11T10:00:33-05:00" level=debug msg="Unmounted ZFS dataset" dev=olxd-5f5e/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local volName=oel84d10
time="2022-05-11T10:00:33-05:00" level=debug msg="UpdateInstanceBackupFile started" instance=oel84d10 project=default
time="2022-05-11T10:00:33-05:00" level=debug msg="CreateInstanceFromImage finished" instance=oel84d10 project=default
time="2022-05-11T10:00:33-05:00" level=debug msg="Mounted ZFS dataset" dev=olxd-5f5e/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local
time="2022-05-11T10:00:33-05:00" level=debug msg="Unmounted ZFS dataset" dev=olxd-5f5e/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local volName=oel84d10
time="2022-05-11T10:00:33-05:00" level=debug msg="Success for task operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:33-05:00" level=debug msg="UpdateInstanceBackupFile finished" instance=oel84d10 project=default
time="2022-05-11T10:00:33-05:00" level=debug msg="Instance operation lock finished" action=create err="<nil>" instance=oel84d10 project=default reusable=false
time="2022-05-11T10:00:36-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:00:36-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:00:36-05:00" level=debug msg="Completed heartbeat round" duration=15.842063ms local="10.209.53.1:8443"
^C
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME                                                                                         USED  AVAIL     REFER  MOUNTPOINT
olxd-5f5e                                                                                    577M  28.0G       96K  legacy
olxd-5f5e/containers                                                                         220K  28.0G       96K  legacy
olxd-5f5e/containers/oel84d10                                                                124K  28.0G      575M  legacy
olxd-5f5e/custom                                                                              96K  28.0G       96K  legacy
olxd-5f5e/deleted                                                                            480K  28.0G       96K  legacy
olxd-5f5e/deleted/containers                                                                  96K  28.0G       96K  legacy
olxd-5f5e/deleted/custom                                                                      96K  28.0G       96K  legacy
olxd-5f5e/deleted/images                                                                      96K  28.0G       96K  legacy
olxd-5f5e/deleted/virtual-machines                                                            96K  28.0G       96K  legacy
olxd-5f5e/images                                                                             575M  28.0G       96K  legacy
olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6            575M  28.0G      575M  legacy
olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6@readonly     0B      -      575M  -
olxd-5f5e/virtual-machines                                                                    96K  28.0G       96K  legacy
[orabuntu@f34sv1 ~]$ tail -f lxc-monitor.log
time="2022-05-11T10:00:33-05:00" level=debug msg="Unmounted ZFS dataset" dev=olxd-5f5e/containers/oel84d10 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/containers/oel84d10 pool=local volName=oel84d10
time="2022-05-11T10:00:33-05:00" level=debug msg="Success for task operation: 8d692e82-e713-4ef4-a5fe-ef9fc5c08fa4"
time="2022-05-11T10:00:33-05:00" level=debug msg="UpdateInstanceBackupFile finished" instance=oel84d10 project=default
time="2022-05-11T10:00:33-05:00" level=debug msg="Instance operation lock finished" action=create err="<nil>" instance=oel84d10 project=default reusable=false
time="2022-05-11T10:00:36-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:00:36-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:00:36-05:00" level=debug msg="Completed heartbeat round" duration=15.842063ms local="10.209.53.1:8443"
time="2022-05-11T10:00:46-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:00:46-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:00:46-05:00" level=debug msg="Completed heartbeat round" duration=12.707894ms local="10.209.53.1:8443"
time="2022-05-11T10:00:56-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:00:56-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:00:56-05:00" level=debug msg="Completed heartbeat round" duration=18.441977ms local="10.209.53.1:8443"
^C
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME                                                                                         USED  AVAIL     REFER  MOUNTPOINT
olxd-5f5e                                                                                    577M  28.0G       96K  legacy
olxd-5f5e/containers                                                                         220K  28.0G       96K  legacy
olxd-5f5e/containers/oel84d10                                                                124K  28.0G      575M  legacy
olxd-5f5e/custom                                                                              96K  28.0G       96K  legacy
olxd-5f5e/deleted                                                                            480K  28.0G       96K  legacy
olxd-5f5e/deleted/containers                                                                  96K  28.0G       96K  legacy
olxd-5f5e/deleted/custom                                                                      96K  28.0G       96K  legacy
olxd-5f5e/deleted/images                                                                      96K  28.0G       96K  legacy
olxd-5f5e/deleted/virtual-machines                                                            96K  28.0G       96K  legacy
olxd-5f5e/images                                                                             575M  28.0G       96K  legacy
olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6            575M  28.0G      575M  legacy
olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6@readonly     0B      -      575M  -
olxd-5f5e/virtual-machines                                                                    96K  28.0G       96K  legacy
[orabuntu@f34sv1 ~]$ sudo zfs destroy olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
cannot destroy 'olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6': filesystem has children
use '-r' to destroy the following datasets:
olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6@readonly
[orabuntu@f34sv1 ~]$ sudo zfs destroy olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 -r
cannot destroy 'olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6': filesystem has dependent clones
use '-R' to destroy the following datasets:
olxd-5f5e/containers/oel84d10
[orabuntu@f34sv1 ~]$ sudo zfs destroy olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 -R
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME                                 USED  AVAIL     REFER  MOUNTPOINT
olxd-5f5e                           1.98M  28.6G       96K  legacy
olxd-5f5e/containers                  96K  28.6G       96K  legacy
olxd-5f5e/custom                      96K  28.6G       96K  legacy
olxd-5f5e/deleted                    480K  28.6G       96K  legacy
olxd-5f5e/deleted/containers          96K  28.6G       96K  legacy
olxd-5f5e/deleted/custom              96K  28.6G       96K  legacy
olxd-5f5e/deleted/images              96K  28.6G       96K  legacy
olxd-5f5e/deleted/virtual-machines    96K  28.6G       96K  legacy
olxd-5f5e/images                      96K  28.6G       96K  legacy
olxd-5f5e/virtual-machines            96K  28.6G       96K  legacy
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME                                 USED  AVAIL     REFER  MOUNTPOINT
olxd-5f5e                           1.98M  28.6G       96K  legacy
olxd-5f5e/containers                  96K  28.6G       96K  legacy
olxd-5f5e/custom                      96K  28.6G       96K  legacy
olxd-5f5e/deleted                    480K  28.6G       96K  legacy
olxd-5f5e/deleted/containers          96K  28.6G       96K  legacy
olxd-5f5e/deleted/custom              96K  28.6G       96K  legacy
olxd-5f5e/deleted/images              96K  28.6G       96K  legacy
olxd-5f5e/deleted/virtual-machines    96K  28.6G       96K  legacy
olxd-5f5e/images                      96K  28.6G       96K  legacy
olxd-5f5e/virtual-machines            96K  28.6G       96K  legacy
[orabuntu@f34sv1 ~]$ tail -f lxc-monitor.log
time="2022-05-11T10:02:06-05:00" level=debug msg="Completed heartbeat round" duration=9.257074ms local="10.209.53.1:8443"
time="2022-05-11T10:02:16-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:02:16-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:02:16-05:00" level=debug msg="Completed heartbeat round" duration=20.868486ms local="10.209.53.1:8443"
time="2022-05-11T10:02:26-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:02:26-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:02:26-05:00" level=debug msg="Completed heartbeat round" duration=9.358115ms local="10.209.53.1:8443"
time="2022-05-11T10:02:36-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:02:36-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:02:36-05:00" level=debug msg="Completed heartbeat round" duration=5.913498ms local="10.209.53.1:8443"
time="2022-05-11T10:02:46-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:02:46-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:02:46-05:00" level=debug msg="Completed heartbeat round" duration=10.019943ms local="10.209.53.1:8443"
time="2022-05-11T10:02:56-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:02:56-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:02:56-05:00" level=debug msg="Completed heartbeat round" duration=8.401611ms local="10.209.53.1:8443"
time="2022-05-11T10:03:06-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:03:06-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:03:06-05:00" level=debug msg="Completed heartbeat round" duration=18.892468ms local="10.209.53.1:8443"
time="2022-05-11T10:03:16-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:03:16-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:03:16-05:00" level=debug msg="Completed heartbeat round" duration=13.389547ms local="10.209.53.1:8443"
time="2022-05-11T10:03:26-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:03:26-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:03:26-05:00" level=debug msg="Completed heartbeat round" duration=11.055607ms local="10.209.53.1:8443"
time="2022-05-11T10:03:36-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:03:36-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:03:36-05:00" level=debug msg="Completed heartbeat round" duration=16.803647ms local="10.209.53.1:8443"
time="2022-05-11T10:03:46-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:03:46-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:03:46-05:00" level=debug msg="Completed heartbeat round" duration=20.290968ms local="10.209.53.1:8443"
time="2022-05-11T10:03:56-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:03:56-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:03:56-05:00" level=debug msg="Completed heartbeat round" duration=5.529903ms local="10.209.53.1:8443"
time="2022-05-11T10:04:06-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:04:06-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:04:06-05:00" level=debug msg="Completed heartbeat round" duration=22.841212ms local="10.209.53.1:8443"
time="2022-05-11T10:04:16-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:04:16-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:04:16-05:00" level=debug msg="Completed heartbeat round" duration=14.473922ms local="10.209.53.1:8443"
time="2022-05-11T10:04:26-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:04:26-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:04:26-05:00" level=debug msg="Completed heartbeat round" duration=17.520211ms local="10.209.53.1:8443"
time="2022-05-11T10:04:36-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:04:36-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:04:36-05:00" level=debug msg="Completed heartbeat round" duration=19.716728ms local="10.209.53.1:8443"
time="2022-05-11T10:04:46-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:04:46-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:04:46-05:00" level=debug msg="Completed heartbeat round" duration=10.958502ms local="10.209.53.1:8443"
time="2022-05-11T10:04:56-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:04:56-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:04:56-05:00" level=debug msg="Completed heartbeat round" duration=12.302142ms local="10.209.53.1:8443"
time="2022-05-11T10:05:06-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:05:06-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:05:06-05:00" level=debug msg="Completed heartbeat round" duration=6.756267ms local="10.209.53.1:8443"
time="2022-05-11T10:05:16-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:05:16-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:05:17-05:00" level=debug msg="Completed heartbeat round" duration=21.964416ms local="10.209.53.1:8443"
^C
[orabuntu@f34sv1 ~]$ sudo getenforce
Permissive
[orabuntu@f34sv1 ~]$ tail -f lxc-monitor.log
time="2022-05-11T10:04:56-05:00" level=debug msg="Completed heartbeat round" duration=12.302142ms local="10.209.53.1:8443"
time="2022-05-11T10:05:06-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:05:06-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:05:06-05:00" level=debug msg="Completed heartbeat round" duration=6.756267ms local="10.209.53.1:8443"
time="2022-05-11T10:05:16-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:05:16-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:05:17-05:00" level=debug msg="Completed heartbeat round" duration=21.964416ms local="10.209.53.1:8443"
time="2022-05-11T10:05:26-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:05:26-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:05:26-05:00" level=debug msg="Completed heartbeat round" duration=17.583889ms local="10.209.53.1:8443"
time="2022-05-11T10:05:36-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:05:36-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:05:36-05:00" level=debug msg="Completed heartbeat round" duration=10.055679ms local="10.209.53.1:8443"
time="2022-05-11T10:05:46-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:05:46-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:05:46-05:00" level=debug msg="Completed heartbeat round" duration=14.412593ms local="10.209.53.1:8443"
^C
[orabuntu@f34sv1 ~]$ lxc list
+------+-------+------+------+------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+------+-------+------+------+------+-----------+----------+
[orabuntu@f34sv1 ~]$ lxc cluste rlist
Error: unknown command "cluste" for "lxc"

Did you mean this?
    cluster

[orabuntu@f34sv1 ~]$ lxc cluster list
+--------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
|  NAME  |           URL            |      ROLES      | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      |
+--------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| f34sv1 | https://10.209.53.1:8443 | database-leader | x86_64       | default        |             | ONLINE | Fully operational |
|        |                          | database        |              |                |             |        |                   |
+--------+--------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
[orabuntu@f34sv1 ~]$ tail -f lxc-monitor.log
time="2022-05-11T10:05:46-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:05:46-05:00" level=debug msg="Completed heartbeat round" duration=14.412593ms local="10.209.53.1:8443"
time="2022-05-11T10:05:52-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
time="2022-05-11T10:05:52-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url="/1.0/instances?recursion=2" username=orabuntu
time="2022-05-11T10:05:56-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:05:56-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:05:56-05:00" level=debug msg="Completed heartbeat round" duration=20.03678ms local="10.209.53.1:8443"
time="2022-05-11T10:05:59-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
time="2022-05-11T10:05:59-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0/cluster username=orabuntu
time="2022-05-11T10:05:59-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url="/1.0/cluster/members?recursion=1" username=orabuntu
^C
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME                                 USED  AVAIL     REFER  MOUNTPOINT
olxd-5f5e                           1.98M  28.6G       96K  legacy
olxd-5f5e/containers                  96K  28.6G       96K  legacy
olxd-5f5e/custom                      96K  28.6G       96K  legacy
olxd-5f5e/deleted                    480K  28.6G       96K  legacy
olxd-5f5e/deleted/containers          96K  28.6G       96K  legacy
olxd-5f5e/deleted/custom              96K  28.6G       96K  legacy
olxd-5f5e/deleted/images              96K  28.6G       96K  legacy
olxd-5f5e/deleted/virtual-machines    96K  28.6G       96K  legacy
olxd-5f5e/images                      96K  28.6G       96K  legacy
olxd-5f5e/virtual-machines            96K  28.6G       96K  legacy
[orabuntu@f34sv1 ~]$ tail -f lxc-monitor.log
time="2022-05-11T10:05:56-05:00" level=debug msg="Completed heartbeat round" duration=20.03678ms local="10.209.53.1:8443"
time="2022-05-11T10:05:59-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
time="2022-05-11T10:05:59-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0/cluster username=orabuntu
time="2022-05-11T10:05:59-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url="/1.0/cluster/members?recursion=1" username=orabuntu
time="2022-05-11T10:06:06-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:06:06-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:06:06-05:00" level=debug msg="Completed heartbeat round" duration=18.965641ms local="10.209.53.1:8443"
time="2022-05-11T10:06:16-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:06:16-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:06:16-05:00" level=debug msg="Completed heartbeat round" duration=19.170734ms local="10.209.53.1:8443"
time="2022-05-11T10:06:26-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:06:26-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:06:26-05:00" level=debug msg="Completed heartbeat round" duration=5.272961ms local="10.209.53.1:8443"
^C
[orabuntu@f34sv1 ~]$ cd /var/lib/lxd
-bash: cd: /var/lib/lxd: No such file or directory
[orabuntu@f34sv1 ~]$ cd /var/lib
[orabuntu@f34sv1 lib]$ l s-lrt
-bash: l: command not found
[orabuntu@f34sv1 lib]$ ls -lrt
total 8
drwxr-xr-x.  2 root           root              6 Jan 26  2021 games
drwxr-xr-x.  2 root           root              6 Jan 26  2021 fprint
drwxr-xr-x.  2 root           root              6 Jan 27  2021 os-prober
drwxr-xr-x.  2 root           root              6 Jan 28  2021 sheepdog
drwx------.  2 tss            tss               6 Jan 28  2021 tpm
drwxr-xr-x.  2 root           root             91 Mar 22  2021 rpm
drwxr-xr-x. 10 root           root            120 Aug 16  2021 sss
drwxr-xr-x.  2 root           root              6 Aug 22  2021 iwd
drwxr-xr-x.  2 root           root              6 Aug 22  2021 ead
drwxr-xr-x.  2 root           root              6 Sep 10  2021 PackageKit
drwxr-xr-x.  2 root           root              6 Sep 28  2021 kdump
drwx------.  2 root           root              6 Oct  4  2021 udisks2
drwxr-xr-x.  2 geoclue        geoclue           6 Oct  7  2021 geoclue
drwxr-xr-x.  2 root           root              6 Oct 28  2021 initramfs
drwxr-xr-x.  2 root           root              6 Nov  2  2021 smartmontools
drwx------.  2 root           root              6 Dec 11 19:38 portables
drwx------.  2 root           root              6 Dec 11 19:38 private
drwxr-x---.  3 root           polkitd          28 Dec 11 19:38 polkit-1
drwxr-xr-x.  5 root           root             46 Dec 11 19:38 selinux
drwxr-xr-x.  8 root           root             90 Dec 11 19:38 iscsi
drwxr-xr-x.  3 root           root             20 Dec 11 19:40 tpm2-tss
drwxr-xr-x.  2 root           root             27 Dec 11 19:41 plymouth
drwxr-xr-x.  2 unbound        unbound          22 Dec 11 19:55 unbound
drwxr-xr-x.  2 root           root             42 Dec 11 19:56 abrt
drwxr-xr-x.  8 root           root            110 Jan 13 05:12 systemd
drwxr-xr-x.  4 root           root             33 Jan 31 10:57 samba
drwxr-xr-x.  2 root           dnsmasq           6 Feb 23 20:13 dnsmasq
drwxr-x---.  2 tss            root              6 Mar  9 09:36 swtpm-localca
drwxr-xr-x.  3 root           root             18 Mar 21 16:52 bluetooth
drwxr-xr-x.  2 root           root              6 Apr 14 09:52 dhclient
drwxr-xr-x.  2 root           root              6 Apr 15 08:55 flatpak
drwxr-xr-x.  3 root           root             20 May 10 07:51 pcp
drwxr-xr-x.  2 root           root             30 May 11 08:53 logrotate
drwxr-x---.  2 root           slocate          24 May 11 08:53 mlocate
drwxr-xr-x.  4 root           root             29 May 11 09:12 dkms
drwxr-xr-x.  5 root           root             55 May 11 09:15 scst
drwxr-x---.  2 chrony         chrony           19 May 11 09:20 chrony
drwxr-xr-x.  4 root           root             55 May 11 09:21 gssproxy
drwx------.  2 setroubleshoot setroubleshoot   71 May 11 09:21 setroubleshoot
drwxr-xr-x. 11 root           root            130 May 11 09:29 libvirt
drwx------.  2 root           root              6 May 11 09:29 machines
drwxr-xr-x.  2 root           root             58 May 11 09:37 dpkg
drwxr-xr-x.  2 root           root             35 May 11 09:37 misc
drwxr-xr-x.  2 root           root           4096 May 11 09:38 alternatives
drwxr-xr-x.  3 root           root             17 May 11 09:38 color
drwxr-xr-x.  3 colord         colord           17 May 11 09:38 colord
drwxr-xr-x.  3 root           root            209 May 11 09:38 authselect
drwx------.  3 root           root             17 May 11 09:39 ipsec
drwxr-xr-x.  3 root           root             17 May 11 09:39 openvswitch
drwxr-xr-x.  5 root           root            117 May 11 09:40 nfs
drwxr-xr-x.  3 root           root             23 May 11 09:47 xfsdump
drwxr-xr-x.  5 root           root             48 May 11 09:50 lxc
drwxr-xr-x.  3 root           root             20 May 11 09:52 rpm-state
drwxr-xr-x.  3 root           root             93 May 11 09:52 dnf
drwxr-xr-x. 18 root           root           4096 May 11 09:59 snapd
drwx------.  2 root           root            172 May 11 10:06 NetworkManager
drwx------.  2 root           root             29 May 11 10:06 rsyslog
[orabuntu@f34sv1 lib]$ cd
[orabuntu@f34sv1 ~]$ tail -f lxc-monitor.log
time="2022-05-11T10:06:26-05:00" level=debug msg="Completed heartbeat round" duration=5.272961ms local="10.209.53.1:8443"
time="2022-05-11T10:06:36-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:06:36-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:06:36-05:00" level=debug msg="Completed heartbeat round" duration=12.108497ms local="10.209.53.1:8443"
time="2022-05-11T10:06:46-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:06:46-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:06:47-05:00" level=debug msg="Completed heartbeat round" duration=19.480469ms local="10.209.53.1:8443"
time="2022-05-11T10:06:56-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:06:56-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:06:57-05:00" level=debug msg="Completed heartbeat round" duration=12.578933ms local="10.209.53.1:8443"
time="2022-05-11T10:07:05-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0 username=orabuntu
time="2022-05-11T10:07:05-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0/events username=orabuntu
time="2022-05-11T10:07:05-05:00" level=debug msg="Event listener server handler started" listener=64d2a265-8cd8-42a5-aafd-f7c9cf1206cd local=/var/snap/lxd/common/lxd/unix.socket remote=@
time="2022-05-11T10:07:05-05:00" level=debug msg="Responding to instance create"
time="2022-05-11T10:07:05-05:00" level=debug msg="Handling API request" ip=@ method=POST protocol=unix url=/1.0/instances username=orabuntu
time="2022-05-11T10:07:05-05:00" level=debug msg="Connecting to a remote simplestreams server" URL="https://images.linuxcontainers.org"
time="2022-05-11T10:07:05-05:00" level=debug msg="New task Operation: 1f005db2-b0c6-4ffd-80d7-ac85d454ac54"
time="2022-05-11T10:07:05-05:00" level=debug msg="Started task operation: 1f005db2-b0c6-4ffd-80d7-ac85d454ac54"
time="2022-05-11T10:07:05-05:00" level=debug msg="Connecting to a remote simplestreams server" URL="https://images.linuxcontainers.org"
time="2022-05-11T10:07:05-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0/operations/1f005db2-b0c6-4ffd-80d7-ac85d454ac54 username=orabuntu
time="2022-05-11T10:07:05-05:00" level=debug msg="Lock acquired for image \"b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6\""
time="2022-05-11T10:07:05-05:00" level=debug msg="Acquiring lock for image \"b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6\""
time="2022-05-11T10:07:05-05:00" level=debug msg="Image already exists in the DB" fingerprint=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:07:05-05:00" level=debug msg="Lock acquired for image \"b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6\""
time="2022-05-11T10:07:05-05:00" level=debug msg="Acquiring lock for image \"b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6\""
time="2022-05-11T10:07:05-05:00" level=debug msg="Instance operation lock created" action=create instance=oel84d10 project=default reusable=false
time="2022-05-11T10:07:05-05:00" level=info msg="Creating container" ephemeral=false instance=oel84d10 instanceType=container project=default
time="2022-05-11T10:07:05-05:00" level=debug msg="Database error" err="Network not found"
time="2022-05-11T10:07:05-05:00" level=debug msg="Database error" err="Network not found"
time="2022-05-11T10:07:05-05:00" level=debug msg="Adding device" device=eth0 instance=oel84d10 instanceType=container project=default type=nic
time="2022-05-11T10:07:05-05:00" level=debug msg="Adding device" device=root instance=oel84d10 instanceType=container project=default type=disk
time="2022-05-11T10:07:05-05:00" level=info msg="Created container" ephemeral=false instance=oel84d10 instanceType=container project=default
time="2022-05-11T10:07:05-05:00" level=debug msg="CreateInstanceFromImage started" instance=oel84d10 project=default
time="2022-05-11T10:07:05-05:00" level=debug msg="EnsureImage started" fingerprint=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:07:05-05:00" level=info msg="Image unpack started" imageFile=/var/snap/lxd/common/lxd/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 volName=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:07:05-05:00" level=debug msg="Mounted ZFS dataset" dev=olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 pool=local
time="2022-05-11T10:07:05-05:00" level=debug msg="Running filler function" dev= driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 pool=local
time="2022-05-11T10:07:05-05:00" level=debug msg="Updated metadata for task Operation: 1f005db2-b0c6-4ffd-80d7-ac85d454ac54"
time="2022-05-11T10:07:06-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:07:06-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:07:07-05:00" level=debug msg="Completed heartbeat round" duration=209.667138ms local="10.209.53.1:8443"
time="2022-05-11T10:07:08-05:00" level=info msg="Image unpack stopped" imageFile=/var/snap/lxd/common/lxd/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 volName=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:07:15-05:00" level=debug msg="Unmounted ZFS dataset" dev=olxd-5f5e/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/local/images/b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6 pool=local volName=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:07:15-05:00" level=debug msg="Database error" err="UNIQUE constraint failed: storage_volumes.storage_pool_id, storage_volumes.node_id, storage_volumes.project_id, storage_volumes.name, storage_volumes.type"
time="2022-05-11T10:07:15-05:00" level=debug msg="EnsureImage finished" fingerprint=b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6
time="2022-05-11T10:07:15-05:00" level=debug msg="CreateInstanceFromImage finished" instance=oel84d10 project=default
time="2022-05-11T10:07:15-05:00" level=debug msg="Instance operation lock finished" action=create err="<nil>" instance=oel84d10 project=default reusable=false
time="2022-05-11T10:07:15-05:00" level=debug msg="Removing device" device=eth0 instance=oel84d10 instanceType=container project=default type=nic
time="2022-05-11T10:07:15-05:00" level=debug msg="Removing device" device=root instance=oel84d10 instanceType=container project=default type=disk
time="2022-05-11T10:07:15-05:00" level=debug msg="Failure for task operation: 1f005db2-b0c6-4ffd-80d7-ac85d454ac54: Failed creating instance from image: Error inserting volume \"b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6\" for project \"default\" in pool \"local\" of type \"images\" into database \"UNIQUE constraint failed: storage_volumes.storage_pool_id, storage_volumes.node_id, storage_volumes.project_id, storage_volumes.name, storage_volumes.type\""
time="2022-05-11T10:07:15-05:00" level=debug msg="Event listener server handler stopped" listener=64d2a265-8cd8-42a5-aafd-f7c9cf1206cd local=/var/snap/lxd/common/lxd/unix.socket remote=@
time="2022-05-11T10:07:17-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:07:17-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:07:17-05:00" level=debug msg="Completed heartbeat round" duration=4.967137ms local="10.209.53.1:8443"
time="2022-05-11T10:07:27-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:07:27-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:07:27-05:00" level=debug msg="Completed heartbeat round" duration=3.641348ms local="10.209.53.1:8443"
time="2022-05-11T10:07:37-05:00" level=debug msg="Heartbeat updating local raft members" members="[{{1 10.209.53.1:8443 voter} f34sv1}]"
time="2022-05-11T10:07:37-05:00" level=debug msg="Starting heartbeat round" local="10.209.53.1:8443" mode=normal
time="2022-05-11T10:07:37-05:00" level=debug msg="Completed heartbeat round" duration=3.260698ms local="10.209.53.1:8443"
^C
[orabuntu@f34sv1 ~]$ lxc list
+------+-------+------+------+------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+------+-------+------+------+------+-----------+----------+
[orabuntu@f34sv1 ~]$ sudo zfs list -t all
NAME                                 USED  AVAIL     REFER  MOUNTPOINT
olxd-5f5e                           1.98M  28.6G       96K  legacy
olxd-5f5e/containers                  96K  28.6G       96K  legacy
olxd-5f5e/custom                      96K  28.6G       96K  legacy
olxd-5f5e/deleted                    480K  28.6G       96K  legacy
olxd-5f5e/deleted/containers          96K  28.6G       96K  legacy
olxd-5f5e/deleted/custom              96K  28.6G       96K  legacy
olxd-5f5e/deleted/images              96K  28.6G       96K  legacy
olxd-5f5e/deleted/virtual-machines    96K  28.6G       96K  legacy
olxd-5f5e/images                      96K  28.6G       96K  legacy
olxd-5f5e/virtual-machines            96K  28.6G       96K  legacy
[orabuntu@f34sv1 ~]$ sudo zpool list
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
olxd-5f5e  29.5G  1.98M  29.5G        -         -     0%     0%  1.00x    ONLINE  -
[orabuntu@f34sv1 ~]$ cd Downloads/orabuntu-lxc-master/uekulele/
[orabuntu@f34sv1 uekulele]$ vi uekulele-services-2.sh
[orabuntu@f34sv1 uekulele]$

Later about 12 hours or so from now I will try with

sudo snap install lxd --channel=4.0/stable

Finally here's the logging:

lxc-monitor.log

HTH Gil just enough time to zip off to work ...

gstanden commented 2 years ago

And there's still something missing from my "zfs cleanup" steps; --debug keeps giving this:

DEBUG  [2022-05-11T10:17:16-05:00] Sending request to LXD                        etag= method=GET url="http://unix.socket/1.0/operations/c970c925-4d1d-4958-93d0-d79725e75d7c"
DEBUG  [2022-05-11T10:17:16-05:00] Got response struct from LXD                 
DEBUG  [2022-05-11T10:17:16-05:00] 
    {
        "id": "c970c925-4d1d-4958-93d0-d79725e75d7c",
        "class": "task",
        "description": "Creating instance",
        "created_at": "2022-05-11T10:17:16.318963974-05:00",
        "updated_at": "2022-05-11T10:17:16.318963974-05:00",
        "status": "Running",
        "status_code": 103,
        "resources": {
            "containers": [
                "/1.0/containers/oel84d10"
            ],
            "instances": [
                "/1.0/instances/oel84d10"
            ]
        },
        "metadata": null,
        "may_cancel": false,
        "err": "",
        "location": "f34sv1"
    } 
Error: Failed instance creation: Failed creating instance from image: Error inserting volume "b9e15a417034605669640b3c23f6656f7ff4d3f70d5920533f1f8ce8197449f6" for project "default" in pool "local" of type "images" into database "UNIQUE constraint failed: storage_volumes.storage_pool_id, storage_volumes.node_id, storage_volumes.project_id, storage_volumes.name, storage_volumes.type"

I'd like to also get the cleanup steps figured out so that I can code that into the project, just to make it more bulletproof, aside from solving this current issue.

will try the 4.0 tonight much later...

tomponline commented 2 years ago

Only LXD 5.1 has the client timeouts so the tests need to be run on that version please, anything before that isn't that helpful.

tomponline commented 2 years ago

Although if you're seeing issues on < LXD 5.1, then resolving them may also resolve the timeout issue you see on LXD 5.1

gstanden commented 2 years ago

Everything I just posted was 5.1 :)

I'm just gonna do a 4.0 test tonight to prove to myself no issues with 4.0

"os_name": "Fedora", "os_version": "34", "project": "default", "server": "lxd", "server_clustered": true, "server_event_mode": "full-mesh", "server_name": "f34sv1", "server_pid": 49758, "server_version": "5.1", "storage": "zfs", "storage_version": "2.1.4-1",

tomponline commented 2 years ago

Everything I just posted was 5.1 :)

I'm just gonna do a 4.0 test tonight to prove to myself no issues with 4.0

Ah it was "Ok change of plan I decided to try the 5.0 again" that confused me.

gstanden commented 2 years ago

Sorry! I just thought it made more sense to try to generate better output for you than to do the 4.0 test. Sorry for the confusing flip flop. Yeah and the 5.0 was a typo sorry :) all the latest output above is 5.1