canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.32k stars 926 forks source link

Containers will not start after reboot. zpool exists but is reported missing #11446

Closed TN-Wade closed 1 year ago

TN-Wade commented 1 year ago

Required information

lxc info
config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- instances_nic_host_name
- image_copy_profile
- container_syscall_intercept_sysinfo
- clustering_evacuation_mode
- resources_pci_vpd
- qemu_raw_conf
- storage_cephfs_fscache
- network_load_balancer
- vsock_api
- instance_ready_state
- network_bgp_holdtime
- storage_volumes_all_projects
- metrics_memory_oom_total
- storage_buckets
- storage_buckets_create_credentials
- metrics_cpu_effective_total
- projects_networks_restricted_access
- storage_buckets_local
- loki
- acme
- internal_metrics
- cluster_join_token_expiry
- remote_token_expiry
- init_preseed
- storage_volumes_created_at
- cpu_hotplug
- projects_networks_zones
- network_txqueuelen
- cluster_member_state
- instances_placement_scriptlet
- storage_pool_source_wipe
- zfs_block_mode
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
    --omitted--
  certificate_fingerprint: --omitted--
  driver: lxc
  driver_version: 5.0.0
  firewall: xtables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    idmapped_mounts: "false"
    netnsid_getifaddrs: "false"
    seccomp_listener: "false"
    seccomp_listener_continue: "false"
    shiftfs: "false"
    uevent_injection: "false"
    unpriv_fscaps: "true"
  kernel_version: 4.15.0-197-generic
  lxc_features:
    cgroup2: "true"
    core_scheduling: "true"
    devpts_fd: "true"
    idmapped_mounts_v2: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: Ubuntu
  os_version: "18.04"
  project: default
  server: lxd
  server_clustered: false
  server_event_mode: full-mesh
  server_name: uhost
  server_pid: 2834
  server_version: "5.11"
  storage: ""
  storage_version: ""
  storage_supported_drivers:
  - name: btrfs
    version: 5.16.2
    remote: false
  - name: ceph
    version: 17.2.0
    remote: true
  - name: cephfs
    version: 17.2.0
    remote: true
  - name: cephobject
    version: 17.2.0
    remote: true
  - name: dir
    version: "1"
    remote: false
  - name: lvm
    version: 2.03.11(2) (2021-01-08) / 1.02.175 (2021-01-08) / 4.37.0
    remote: false

Issue description

Containers will not start after shutdown. lxd start container returns:

$ lxc start Plex
Error: Required tool 'zpool' is missing
Try `lxc info --show-log Plex` for more info

But zpool exists. I saw there's been some issues patched with initialising snap zfs so I tried an apt-update & upgrade to no avail. Rebooted a few times since too!

I found advice to restart the snap daemon which succeeds.


$ sudo systemctl reload snap.lxd.daemon
$
$ sudo systemctl status snap.lxd.daemon
● snap.lxd.daemon.service - Service for snap application lxd.daemon
   Loaded: loaded (/etc/systemd/system/snap.lxd.daemon.service; static; vendor preset: enabled)
   Active: active (running) since Tue 2023-03-07 19:20:54 ACDT; 2min 16s ago
  Process: 24500 ExecReload=/usr/bin/snap run --command=reload lxd.daemon (code=exited, status=0/SUCCESS)
 Main PID: 24541 (daemon.start)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/snap.lxd.daemon.service
           ‣ 24541 /bin/sh /snap/lxd/24561/commands/daemon.start

Mar 07 19:20:55 uhost lxd.daemon[24541]: ==> Rotating logs
Mar 07 19:20:55 uhost lxd.daemon[24541]: ==> Escaping the systemd cgroups
Mar 07 19:20:55 uhost lxd.daemon[24541]: ====> Detected cgroup V1
Mar 07 19:20:55 uhost lxd.daemon[24541]: ==> Escaping the systemd process resource limits
Mar 07 19:20:55 uhost lxd.daemon[24541]: ==> Disabling shiftfs on this kernel (auto)
Mar 07 19:20:55 uhost lxd.daemon[24541]: => Re-using existing LXCFS
Mar 07 19:20:55 uhost lxd.daemon[24541]: ==> Reloading LXCFS
Mar 07 19:20:55 uhost lxd.daemon[24541]: => Starting LXD
Mar 07 19:20:55 uhost lxd.daemon[24541]: time="2023-03-07T19:20:55+10:30" level=warning msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"
Mar 07 19:20:55 uhost lxd.daemon[24541]: time="2023-03-07T19:20:55+10:30" level=warning msg="Instance type not operational" driver=qemu err="KVM support is missing (no /dev/kvm)" type=virtual-machine

I've tried gathering as much information as possible based on similar posts. Dumped below.

Steps to reproduce

  1. Shutdown Ubuntu host gracefully. There was a stop running on a snap job, Hyper-V host proceeded to shutdown so RDP session was lost. No idea if the graceful shutdown finished, Hyper-V showed the guest as shutdown upon booting again.
  2. Containers now do not start even though ZFS does exist.

Information to attach


$ which zpool
/sbin/zpool

~$ zfs list
NAME                                                                                      USED  AVAIL  REFER  MOUNTPOINT
default                                                                                  13.3G  7.04G    24K  legacy
default/buckets                                                                            24K  7.04G    24K  legacy
default/containers                                                                       10.5G  7.04G    24K  legacy
default/containers/Plex                                                                  3.35G  7.04G  3.40G  legacy
default/containers/ctf                                                                    316M  7.04G   491M  none
default/containers/git                                                                    590M  7.04G   714M  none
default/containers/grafana                                                               1.08G  7.04G  1.22G  legacy
default/containers/jackett                                                                475M  7.04G   616M  legacy
default/containers/nginx                                                                  431M  7.04G   537M  legacy
default/containers/ombi                                                                   616M  7.04G   704M  legacy
default/containers/radarr                                                                1.21G  7.04G  1.27G  legacy
default/containers/sabnzbd                                                                620M  7.04G   677M  legacy
default/containers/sonarr                                                                 789M  7.04G   848M  legacy
default/containers/transm                                                                 524M  7.04G   617M  legacy
default/containers/vpn                                                                    577M  7.04G   651M  legacy
default/custom                                                                             24K  7.04G    24K  legacy
default/deleted                                                                          2.38G  7.04G    24K  legacy
default/deleted/buckets                                                                    24K  7.04G    24K  legacy
default/deleted/containers                                                                 24K  7.04G    24K  legacy
default/deleted/custom                                                                     24K  7.04G    24K  legacy
default/deleted/images                                                                   2.38G  7.04G    24K  legacy
default/deleted/images/19ebc24da9bba694a4128df8112b25ef69fa8cb6940e3e73458bd666ef72c256   371M  7.04G   371M  /var/snap/lxd/common/lxd/storage-pools/default/images/19ebc24da9bba694a4128df8112b25ef69fa8cb6940e3e73458bd666ef72c256
default/deleted/images/38219778c2cf02521f34f950580ce3af0e4b61fbaf2b4411a7a6c4f0736071f9   333M  7.04G   333M  none
default/deleted/images/39a93d0b355279d430e8ce21c689aa88515212ee99874276e77f7f31ad7bf810   359M  7.04G   359M  /var/snap/lxd/common/lxd/storage-pools/default/images/39a93d0b355279d430e8ce21c689aa88515212ee99874276e77f7f31ad7bf810
default/deleted/images/412fb387e01d8130016d300bbc33fbaee84c1b17ddfb7cb9f85ae63e0c4fa618   342M  7.04G   342M  none
default/deleted/images/6ae1c6e92017402f1aee655fa8d785ee9d2337a3369d76115cecad5e7a303e07   339M  7.04G   339M  none
default/deleted/images/95c0e536d361eb5ac953ad343e0342c2f615e4aea714ca8a64126a228b809cae   363M  7.04G   363M  /var/snap/lxd/common/lxd/storage-pools/default/images/95c0e536d361eb5ac953ad343e0342c2f615e4aea714ca8a64126a228b809cae
default/deleted/images/d72ae2e5073f20450c5260e6f227484c23452a46c6bb553ffe6be55e48602bb4   334M  7.04G   334M  none
default/deleted/virtual-machines                                                           24K  7.04G    24K  legacy
default/images                                                                            408M  7.04G    24K  legacy
default/images/f41dafa0754b7702cea2b43d95fb7c3a3b4de8e36b66d32264e84d72ac9b0b66           408M  7.04G   408M  legacy
default/snapshots                                                                          24K  7.04G    24K  none
default/virtual-machines                                                                   24K  7.04G    24K  legacy

$ sudo cat /var/snap/lxd/common/lxd/logs/lxd.log
time="2023-03-07T18:35:24+10:30" level=warning msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"
time="2023-03-07T18:35:24+10:30" level=warning msg="Instance type not operational" driver=qemu err="KVM support is missing (no /dev/kvm)" type=virtual-machine
time="2023-03-07T18:35:28+10:30" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=default
time="2023-03-07T18:35:28+10:30" level=warning msg="Failed to initialize fanotify, falling back on inotify" err="Failed to initialize fanotify: invalid argument"
time="2023-03-07T18:35:28+10:30" level=warning msg="Failed auto start instance attempt" attempt=1 err="Required tool 'zpool' is missing" instance=Plex maxAttempts=3 project=default
time="2023-03-07T18:35:29+10:30" level=error msg="Failed to update the image" err="Failed to create image \"eac8ed7b2d652de0edf14b6d163ace1948d24123fd7517258545ab420597a7c9\" on storage pool \"default\": Required tool 'zpool' is missing" fingerprint=f41dafa0754b7702cea2b43d95fb7c3a3b4de8e36b66d32264e84d72ac9b0b66
time="2023-03-07T18:35:33+10:30" level=warning msg="Failed auto start instance attempt" attempt=2 err="Required tool 'zpool' is missing" instance=Plex maxAttempts=3 project=default
time="2023-03-07T18:35:35+10:30" level=error msg="Error loading storage pool" err="Failed getting instance storage pool name: Required tool 'zpool' is missing" instance=transm instanceType=container project=default
time="2023-03-07T18:35:35+10:30" level=error msg="Error loading storage pool" err="Failed getting instance storage pool name: Required tool 'zpool' is missing" instance=sonarr instanceType=container project=default
time="2023-03-07T18:35:35+10:30" level=error msg="Error loading storage pool" err="Failed getting instance storage pool name: Required tool 'zpool' is missing" instance=Plex instanceType=container project=default
time="2023-03-07T18:35:35+10:30" level=error msg="Error loading storage pool" err="Failed getting instance storage pool name: Required tool 'zpool' is missing" instance=sabnzbd instanceType=container project=default
time="2023-03-07T18:35:35+10:30" level=error msg="Error loading storage pool" err="Failed getting instance storage pool name: Required tool 'zpool' is missing" instance=ombi instanceType=container project=default
time="2023-03-07T18:35:35+10:30" level=error msg="Error loading storage pool" err="Failed getting instance storage pool name: Required tool 'zpool' is missing" instance=nginx instanceType=container project=default
time="2023-03-07T18:35:35+10:30" level=error msg="Error loading storage pool" err="Failed getting instance storage pool name: Required tool 'zpool' is missing" instance=vpn instanceType=container project=default
time="2023-03-07T18:35:35+10:30" level=error msg="Error loading storage pool" err="Failed getting instance storage pool name: Required tool 'zpool' is missing" instance=git instanceType=container project=default
time="2023-03-07T18:35:35+10:30" level=error msg="Error loading storage pool" err="Failed getting instance storage pool name: Required tool 'zpool' is missing" instance=jackett instanceType=container project=default
time="2023-03-07T18:35:35+10:30" level=error msg="Error loading storage pool" err="Failed getting instance storage pool name: Required tool 'zpool' is missing" instance=radarr instanceType=container project=default
time="2023-03-07T18:35:35+10:30" level=error msg="Error loading storage pool" err="Failed getting instance storage pool name: Required tool 'zpool' is missing" instance=grafana instanceType=container project=default
time="2023-03-07T18:35:35+10:30" level=error msg="Error loading storage pool" err="Failed getting instance storage pool name: Required tool 'zpool' is missing" instance=ctf instanceType=container project=default
time="2023-03-07T18:35:38+10:30" level=warning msg="Failed auto start instance attempt" attempt=3 err="Required tool 'zpool' is missing" instance=Plex maxAttempts=3 project=default
time="2023-03-07T18:35:38+10:30" level=error msg="Failed to auto start instance" err="Required tool 'zpool' is missing" instance=Plex project=default
time="2023-03-07T18:35:38+10:30" level=warning msg="Failed auto start instance attempt" attempt=1 err="Required tool 'zpool' is missing" instance=grafana maxAttempts=3 project=default
time="2023-03-07T18:35:43+10:30" level=warning msg="Failed auto start instance attempt" attempt=2 err="Required tool 'zpool' is missing" instance=grafana maxAttempts=3 project=default
time="2023-03-07T18:35:48+10:30" level=warning msg="Failed auto start instance attempt" attempt=3 err="Required tool 'zpool' is missing" instance=grafana maxAttempts=3 project=default
time="2023-03-07T18:35:48+10:30" level=error msg="Failed to auto start instance" err="Required tool 'zpool' is missing" instance=grafana project=default
time="2023-03-07T18:35:48+10:30" level=warning msg="Failed auto start instance attempt" attempt=1 err="Required tool 'zpool' is missing" instance=jackett maxAttempts=3 project=default
time="2023-03-07T18:35:54+10:30" level=warning msg="Failed auto start instance attempt" attempt=2 err="Required tool 'zpool' is missing" instance=jackett maxAttempts=3 project=default
time="2023-03-07T18:35:59+10:30" level=warning msg="Failed auto start instance attempt" attempt=3 err="Required tool 'zpool' is missing" instance=jackett maxAttempts=3 project=default
time="2023-03-07T18:35:59+10:30" level=error msg="Failed to auto start instance" err="Required tool 'zpool' is missing" instance=jackett project=default
time="2023-03-07T18:35:59+10:30" level=warning msg="Failed auto start instance attempt" attempt=1 err="Required tool 'zpool' is missing" instance=nginx maxAttempts=3 project=default
time="2023-03-07T18:36:04+10:30" level=warning msg="Failed auto start instance attempt" attempt=2 err="Required tool 'zpool' is missing" instance=nginx maxAttempts=3 project=default
time="2023-03-07T18:36:09+10:30" level=warning msg="Failed auto start instance attempt" attempt=3 err="Required tool 'zpool' is missing" instance=nginx maxAttempts=3 project=default
time="2023-03-07T18:36:09+10:30" level=error msg="Failed to auto start instance" err="Required tool 'zpool' is missing" instance=nginx project=default
time="2023-03-07T18:36:09+10:30" level=warning msg="Failed auto start instance attempt" attempt=1 err="Required tool 'zpool' is missing" instance=ombi maxAttempts=3 project=default
time="2023-03-07T18:36:14+10:30" level=warning msg="Failed auto start instance attempt" attempt=2 err="Required tool 'zpool' is missing" instance=ombi maxAttempts=3 project=default
time="2023-03-07T18:36:19+10:30" level=warning msg="Failed auto start instance attempt" attempt=3 err="Required tool 'zpool' is missing" instance=ombi maxAttempts=3 project=default
time="2023-03-07T18:36:19+10:30" level=error msg="Failed to auto start instance" err="Required tool 'zpool' is missing" instance=ombi project=default
time="2023-03-07T18:36:19+10:30" level=warning msg="Failed auto start instance attempt" attempt=1 err="Required tool 'zpool' is missing" instance=radarr maxAttempts=3 project=default
time="2023-03-07T18:36:24+10:30" level=warning msg="Failed auto start instance attempt" attempt=2 err="Required tool 'zpool' is missing" instance=radarr maxAttempts=3 project=default
time="2023-03-07T18:36:28+10:30" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=default
time="2023-03-07T18:36:29+10:30" level=warning msg="Failed auto start instance attempt" attempt=3 err="Required tool 'zpool' is missing" instance=radarr maxAttempts=3 project=default
time="2023-03-07T18:36:29+10:30" level=error msg="Failed to auto start instance" err="Required tool 'zpool' is missing" instance=radarr project=default
time="2023-03-07T18:36:29+10:30" level=warning msg="Failed auto start instance attempt" attempt=1 err="Required tool 'zpool' is missing" instance=sabnzbd maxAttempts=3 project=default
time="2023-03-07T18:36:34+10:30" level=warning msg="Failed auto start instance attempt" attempt=2 err="Required tool 'zpool' is missing" instance=sabnzbd maxAttempts=3 project=default
time="2023-03-07T18:36:39+10:30" level=warning msg="Failed auto start instance attempt" attempt=3 err="Required tool 'zpool' is missing" instance=sabnzbd maxAttempts=3 project=default
time="2023-03-07T18:36:39+10:30" level=error msg="Failed to auto start instance" err="Required tool 'zpool' is missing" instance=sabnzbd project=default
time="2023-03-07T18:36:39+10:30" level=warning msg="Failed auto start instance attempt" attempt=1 err="Required tool 'zpool' is missing" instance=sonarr maxAttempts=3 project=default
time="2023-03-07T18:36:44+10:30" level=warning msg="Failed auto start instance attempt" attempt=2 err="Required tool 'zpool' is missing" instance=sonarr maxAttempts=3 project=default
time="2023-03-07T18:36:49+10:30" level=warning msg="Failed auto start instance attempt" attempt=3 err="Required tool 'zpool' is missing" instance=sonarr maxAttempts=3 project=default
time="2023-03-07T18:36:49+10:30" level=error msg="Failed to auto start instance" err="Required tool 'zpool' is missing" instance=sonarr project=default
time="2023-03-07T18:36:49+10:30" level=warning msg="Failed auto start instance attempt" attempt=1 err="Required tool 'zpool' is missing" instance=transm maxAttempts=3 project=default
time="2023-03-07T18:36:54+10:30" level=warning msg="Failed auto start instance attempt" attempt=2 err="Required tool 'zpool' is missing" instance=transm maxAttempts=3 project=default
time="2023-03-07T18:36:59+10:30" level=warning msg="Failed auto start instance attempt" attempt=3 err="Required tool 'zpool' is missing" instance=transm maxAttempts=3 project=default
time="2023-03-07T18:36:59+10:30" level=error msg="Failed to auto start instance" err="Required tool 'zpool' is missing" instance=transm project=default
time="2023-03-07T18:36:59+10:30" level=warning msg="Failed auto start instance attempt" attempt=1 err="Required tool 'zpool' is missing" instance=vpn maxAttempts=3 project=default
time="2023-03-07T18:37:04+10:30" level=warning msg="Failed auto start instance attempt" attempt=2 err="Required tool 'zpool' is missing" instance=vpn maxAttempts=3 project=default
time="2023-03-07T18:37:09+10:30" level=warning msg="Failed auto start instance attempt" attempt=3 err="Required tool 'zpool' is missing" instance=vpn maxAttempts=3 project=default
time="2023-03-07T18:37:09+10:30" level=error msg="Failed to auto start instance" err="Required tool 'zpool' is missing" instance=vpn project=default
time="2023-03-07T18:37:28+10:30" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=default
time="2023-03-07T18:38:28+10:30" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=default
time="2023-03-07T18:39:28+10:30" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=default
time="2023-03-07T18:40:28+10:30" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=default
time="2023-03-07T18:41:28+10:30" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=default
time="2023-03-07T18:42:29+10:30" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=default
time="2023-03-07T18:43:29+10:30" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=default
time="2023-03-07T18:44:29+10:30" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=default
time="2023-03-07T18:45:29+10:30" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=default
time="2023-03-07T18:46:29+10:30" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=default
time="2023-03-07T18:47:29+10:30" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=default
time="2023-03-07T18:48:29+10:30" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=default
time="2023-03-07T18:49:29+10:30" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=default
time="2023-03-07T18:50:29+10:30" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=default
time="2023-03-07T18:51:29+10:30" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=default
time="2023-03-07T18:52:29+10:30" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=default
time="2023-03-07T18:53:29+10:30" level=error msg="Failed loading storage pool" err="Required tool 'zpool' is missing" pool=default

$ snap list
Name    Version      Rev    Tracking       Publisher   Notes
core    16-2.58.2    14784  latest/stable  canonical✓  core
core18  20230207     2697   latest/stable  canonical✓  base
core20  20230126     1822   latest/stable  canonical✓  base
core22  20230210     522    latest/stable  canonical✓  base
lxd     git-c5795a8  24561  latest/edge    canonical✓  -
$
$
$ apt list lxd -a
Listing... Done
lxd/bionic-updates 3.0.3-0ubuntu1~18.04.2 amd64
lxd/bionic 3.0.0-0ubuntu4 amd64

$ apt list zfsutils-linux -a
Listing... Done
zfsutils-linux/bionic-updates,bionic-security,now 0.7.5-1ubuntu16.12 amd64 [installed]
zfsutils-linux/bionic 0.7.5-1ubuntu15 amd64

$ modinfo zfs
filename:       /lib/modules/4.15.0-197-generic/kernel/zfs/zfs.ko
version:        0.7.5-1ubuntu16.12
license:        CDDL
author:         OpenZFS on Linux
description:    ZFS
srcversion:     7C105EF1C775F5F2F9DF168
depends:        spl,znvpair,zcommon,zunicode,zavl,icp
retpoline:      Y
name:           zfs
vermagic:       4.15.0-197-generic SMP mod_unload modversions
signat:         PKCS#7
signer:
sig_key:
sig_hashalgo:   md4
--omitted--

snap-daemon.txt

mihalicyn commented 1 year ago

@TN-Wade this error is not about zpool presence on the host, but inside the snap. For some reason you are sitting on the edge channel of lxd snap. Is it intentionally or accidentally? You can try to make snap refresh lxd --channel=latest/stable, but it may not work cause AFAIK you can't downgrade LXD version if database structure was changed between revisions.

tomponline commented 1 year ago

@stgraber could this be to do with the switch to core22?

TN-Wade commented 1 year ago

@TN-Wade this error is not about zpool presence on the host, but inside the snap. For some reason you are sitting on the edge channel of lxd snap. Is it intentionally or accidentally? You can try to make snap refresh lxd --channel=latest/stable, but it may not work cause AFAIK you can't downgrade LXD version if database structure was changed between revisions.

My skillset is in networking and I know just enough Unix to get myself into trouble, I know very little about snap, sorry!

I setup lxd using a guide years ago so there's no rationale to being on the edge channel, entirely accidental. I did think it's strange my lxd version is a very recent git commit.

I just made a checkpoint and tried the snap refresh. It worked thankyou.


$ sudo snap refresh lxd --channel=latest/stable

2023-03-07T22:50:56+10:30 INFO Waiting for "snap.lxd.daemon.service" to stop.
lxd 5.11-ad0b61e from Canonical✓ refreshed

$ snap list
Name    Version       Rev    Tracking       Publisher   Notes
core    16-2.58.2     14784  latest/stable  canonical✓  core
core18  20230207      2697   latest/stable  canonical✓  base
core20  20230126      1822   latest/stable  canonical✓  base
core22  20230210      522    latest/stable  canonical✓  base
lxd     5.11-ad0b61e  24483  latest/stable  canonical✓  -

$ lxc list
+---------+---------+-------------------+------+-----------+-----------+
|  NAME   |  STATE  |       IPV4        | IPV6 |   TYPE    | SNAPSHOTS |
+---------+---------+-------------------+------+-----------+-----------+
| Plex    | RUNNING |       |      | CONTAINER | 0         |
+---------+---------+-------------------+------+-----------+-----------+
stgraber commented 1 year ago

Yeah, the core22 snap drops support for ZFS 0.6 and 0.7 as those aren't buildable anymore.

This affects Ubuntu 16.04 users that aren't on the HWE kernel but as neither Ubuntu nor ZFS of those versions are still supported, there's little we can do.

Worth noting that the same logic will make it to stable soon. On such systems you ideally want to update to a supported version of Ubuntu or at least switch to the HWE kernel to get ZFS 0.8.

stgraber commented 1 year ago

Actually, that's 18.04, so still supported for a couple of months, so only switching to HWE would be needed in this case.

stgraber commented 1 year ago

We won't be pushing the logic to the 3.0.or 4.0 tracks though, so the stock LXD version on those old Ubuntu distros won't be affected.

tomponline commented 1 year ago

Thanks!