search

lxc / incus

Powerful system container and virtual machine manager
https://linuxcontainers.org/incus
Apache License 2.0
2.5k stars 203 forks source link

Unable to start instance with custom storage block #534

Closed adamcstephens closed 7 months ago

adamcstephens commented 7 months ago

Required information

Issue description

After upgrading to 0.6 I'm unable to start a VM that has a custom storage volume attached.

Steps to reproduce

  1. incus create --ephemeral images:alpine/3.19 --vm -c security.secureboot=false
  2. incus storage volume create nvme_pool test1 --type block
  3. incus storage volume attach nvme_pool test1 joint-chicken
─❯ incus start joint-chicken
Error: Failed start validation for device "test1": Cannot add un-shared custom storage block volume to more than one instance
Try `incus info --show-log joint-chicken` for more info
─❯ incus info --show-log joint-chicken
Name: joint-chicken
Status: STOPPED
Type: virtual-machine (ephemeral)
Architecture: x86_64
Created: 2024/02/24 12:37 EST
Error: open /var/log/incus/joint-chicken/qemu.log: no such file or directory

debug log

Feb 24 12:51:20 bonk incusd[7231]: time="2024-02-24T12:51:20-05:00" level=debug msg="Start started" instance=stink2 instanceType=virtual-machine project=default stateful=false
Feb 24 12:51:20 bonk incusd[7231]: time="2024-02-24T12:51:20-05:00" level=debug msg="Instance operation lock created" action=start instance=stink2 project=default reusable=false
Feb 24 12:51:20 bonk incusd[7231]: time="2024-02-24T12:51:20-05:00" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0/operations/1859d059-e0a5-4d34-a7e7-e0ebe902e5a9 username=adam
Feb 24 12:51:20 bonk incusd[7231]: time="2024-02-24T12:51:20-05:00" level=debug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"id\": \"1859d059-e0a5-4d34-a7e7-e0ebe902e5a9\",\n\t\t\t\"class\": \"task\",\n\t\t\t\"description\": \"Starting instance\",\n\t\t\t\"created_at\": \"2024-02-24T12:51:20.971761887-05:00\",\n\t\t\t\"updated_at\": \"2024-02-24T12:51:20.971761887-05:00\",\n\t\t\t\"status\": \"Running\",\n\t\t\t\"status_code\": 103,\n\t\t\t\"resources\": {\n\t\t\t\t\"instances\": [\n\t\t\t\t\t\"/1.0/instances/stink2\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t\"metadata\": null,\n\t\t\t\"may_cancel\": false,\n\t\t\t\"err\": \"\",\n\t\t\t\"location\": \"none\"\n\t\t}\n\t}" http_code=200
Feb 24 12:51:20 bonk incusd[7231]: time="2024-02-24T12:51:20-05:00" level=debug msg="MountInstance started" driver=zfs instance=stink2 pool=default_pool project=default
Feb 24 12:51:21 bonk incusd[7231]: time="2024-02-24T12:51:21-05:00" level=debug msg="Activated ZFS volume" dev=main/lxd/virtual-machines/stink2.block driver=zfs pool=default_pool volName=stink2
Feb 24 12:51:21 bonk incusd[7231]: time="2024-02-24T12:51:21-05:00" level=debug msg="Mounted ZFS dataset" dev=main/lxd/virtual-machines/stink2 driver=zfs path=/var/lib/incus/storage-pools/default_pool/virtual-machines/stink2 pool=default_pool volName=stink2
Feb 24 12:51:21 bonk incusd[7231]: time="2024-02-24T12:51:21-05:00" level=debug msg="MountInstance finished" driver=zfs instance=stink2 pool=default_pool project=default
Feb 24 12:51:21 bonk incusd[7231]: time="2024-02-24T12:51:21-05:00" level=debug msg="Skipping incus-agent install as unchanged" installPath=/var/lib/incus/virtual-machines/stink2/config/incus-agent instance=stink2 instanceType=virtual-machine project=default srcPath=/nix/store/nyyk2j2q18s2gfjg74xp473559zyp55s-incus-unwrapped-0.6.0/bin/incus-agent
Feb 24 12:51:21 bonk incusd[7231]: time="2024-02-24T12:51:21-05:00" level=debug msg="Instance operation lock finished" action=start err="Failed start validation for device \"stink2-garage\": Cannot add un-shared custom storage block volume to more than one instance" instance=stink2 project=default reusable=false
Feb 24 12:51:21 bonk incusd[7231]: time="2024-02-24T12:51:21-05:00" level=debug msg="UnmountInstance started" driver=zfs instance=stink2 pool=default_pool project=default
Feb 24 12:51:21 bonk incusd[7231]: time="2024-02-24T12:51:21-05:00" level=debug msg="Unmounted ZFS dataset" dev=main/lxd/virtual-machines/stink2 driver=zfs path=/var/lib/incus/storage-pools/default_pool/virtual-machines/stink2 pool=default_pool volName=stink2
Feb 24 12:51:21 bonk incusd[7231]: time="2024-02-24T12:51:21-05:00" level=debug msg="Deactivated ZFS volume" dev=main/lxd/virtual-machines/stink2.block driver=zfs pool=default_pool volName=stink2
Feb 24 12:51:21 bonk incusd[7231]: time="2024-02-24T12:51:21-05:00" level=debug msg="UnmountInstance finished" driver=zfs instance=stink2 pool=default_pool project=default
Feb 24 12:51:21 bonk incusd[7231]: time="2024-02-24T12:51:21-05:00" level=debug msg="Start finished" instance=stink2 instanceType=virtual-machine project=default stateful=false
Feb 24 12:51:21 bonk incusd[7231]: time="2024-02-24T12:51:21-05:00" level=debug msg="Failure for operation" class=task description="Starting instance" err="Failed start validation for device \"stink2-garage\": Cannot add un-shared custom storage block volume to more than one instance" operation=1859d059-e0a5-4d34-a7e7-e0ebe902e5a9 project=default
Feb 24 12:51:21 bonk incusd[7231]: time="2024-02-24T12:51:21-05:00" level=debug msg="Event listener server handler stopped" listener=53cc072e-6345-4ed2-a29d-b6e074640911 local=/var/lib/incus/unix.socket remote=@

Information to attach

incus info ``` config: core.https_address: 0.0.0.0 api_extensions: - storage_zfs_remove_snapshots - container_host_shutdown_timeout - container_stop_priority - container_syscall_filtering - auth_pki - container_last_used_at - etag - patch - usb_devices - https_allowed_credentials - image_compression_algorithm - directory_manipulation - container_cpu_time - storage_zfs_use_refquota - storage_lvm_mount_options - network - profile_usedby - container_push - container_exec_recording - certificate_update - container_exec_signal_handling - gpu_devices - container_image_properties - migration_progress - id_map - network_firewall_filtering - network_routes - storage - file_delete - file_append - network_dhcp_expiry - storage_lvm_vg_rename - storage_lvm_thinpool_rename - network_vlan - image_create_aliases - container_stateless_copy - container_only_migration - storage_zfs_clone_copy - unix_device_rename - storage_lvm_use_thinpool - storage_rsync_bwlimit - network_vxlan_interface - storage_btrfs_mount_options - entity_description - image_force_refresh - storage_lvm_lv_resizing - id_map_base - file_symlinks - container_push_target - network_vlan_physical - storage_images_delete - container_edit_metadata - container_snapshot_stateful_migration - storage_driver_ceph - storage_ceph_user_name - resource_limits - storage_volatile_initial_source - storage_ceph_force_osd_reuse - storage_block_filesystem_btrfs - resources - kernel_limits - storage_api_volume_rename - network_sriov - console - restrict_dev_incus - migration_pre_copy - infiniband - dev_incus_events - proxy - network_dhcp_gateway - file_get_symlink - network_leases - unix_device_hotplug - storage_api_local_volume_handling - operation_description - clustering - event_lifecycle - storage_api_remote_volume_handling - nvidia_runtime - container_mount_propagation - container_backup - dev_incus_images - container_local_cross_pool_handling - proxy_unix - proxy_udp - clustering_join - proxy_tcp_udp_multi_port_handling - network_state - proxy_unix_dac_properties - container_protection_delete - unix_priv_drop - pprof_http - proxy_haproxy_protocol - network_hwaddr - proxy_nat - network_nat_order - container_full - backup_compression - nvidia_runtime_config - storage_api_volume_snapshots - storage_unmapped - projects - network_vxlan_ttl - container_incremental_copy - usb_optional_vendorid - snapshot_scheduling - snapshot_schedule_aliases - container_copy_project - clustering_server_address - clustering_image_replication - container_protection_shift - snapshot_expiry - container_backup_override_pool - snapshot_expiry_creation - network_leases_location - resources_cpu_socket - resources_gpu - resources_numa - kernel_features - id_map_current - event_location - storage_api_remote_volume_snapshots - network_nat_address - container_nic_routes - cluster_internal_copy - seccomp_notify - lxc_features - container_nic_ipvlan - network_vlan_sriov - storage_cephfs - container_nic_ipfilter - resources_v2 - container_exec_user_group_cwd - container_syscall_intercept - container_disk_shift - storage_shifted - resources_infiniband - daemon_storage - instances - image_types - resources_disk_sata - clustering_roles - images_expiry - resources_network_firmware - backup_compression_algorithm - ceph_data_pool_name - container_syscall_intercept_mount - compression_squashfs - container_raw_mount - container_nic_routed - container_syscall_intercept_mount_fuse - container_disk_ceph - virtual-machines - image_profiles - clustering_architecture - resources_disk_id - storage_lvm_stripes - vm_boot_priority - unix_hotplug_devices - api_filtering - instance_nic_network - clustering_sizing - firewall_driver - projects_limits - container_syscall_intercept_hugetlbfs - limits_hugepages - container_nic_routed_gateway - projects_restrictions - custom_volume_snapshot_expiry - volume_snapshot_scheduling - trust_ca_certificates - snapshot_disk_usage - clustering_edit_roles - container_nic_routed_host_address - container_nic_ipvlan_gateway - resources_usb_pci - resources_cpu_threads_numa - resources_cpu_core_die - api_os - container_nic_routed_host_table - container_nic_ipvlan_host_table - container_nic_ipvlan_mode - resources_system - images_push_relay - network_dns_search - container_nic_routed_limits - instance_nic_bridged_vlan - network_state_bond_bridge - usedby_consistency - custom_block_volumes - clustering_failure_domains - resources_gpu_mdev - console_vga_type - projects_limits_disk - network_type_macvlan - network_type_sriov - container_syscall_intercept_bpf_devices - network_type_ovn - projects_networks - projects_networks_restricted_uplinks - custom_volume_backup - backup_override_name - storage_rsync_compression - network_type_physical - network_ovn_external_subnets - network_ovn_nat - network_ovn_external_routes_remove - tpm_device_type - storage_zfs_clone_copy_rebase - gpu_mdev - resources_pci_iommu - resources_network_usb - resources_disk_address - network_physical_ovn_ingress_mode - network_ovn_dhcp - network_physical_routes_anycast - projects_limits_instances - network_state_vlan - instance_nic_bridged_port_isolation - instance_bulk_state_change - network_gvrp - instance_pool_move - gpu_sriov - pci_device_type - storage_volume_state - network_acl - migration_stateful - disk_state_quota - storage_ceph_features - projects_compression - projects_images_remote_cache_expiry - certificate_project - network_ovn_acl - projects_images_auto_update - projects_restricted_cluster_target - images_default_architecture - network_ovn_acl_defaults - gpu_mig - project_usage - network_bridge_acl - warnings - projects_restricted_backups_and_snapshots - clustering_join_token - clustering_description - server_trusted_proxy - clustering_update_cert - storage_api_project - server_instance_driver_operational - server_supported_storage_drivers - event_lifecycle_requestor_address - resources_gpu_usb - clustering_evacuation - network_ovn_nat_address - network_bgp - network_forward - custom_volume_refresh - network_counters_errors_dropped - metrics - image_source_project - clustering_config - network_peer - linux_sysctl - network_dns - ovn_nic_acceleration - certificate_self_renewal - instance_project_move - storage_volume_project_move - cloud_init - network_dns_nat - database_leader - instance_all_projects - clustering_groups - ceph_rbd_du - instance_get_full - qemu_metrics - gpu_mig_uuid - event_project - clustering_evacuation_live - instance_allow_inconsistent_copy - network_state_ovn - storage_volume_api_filtering - image_restrictions - storage_zfs_export - network_dns_records - storage_zfs_reserve_space - network_acl_log - storage_zfs_blocksize - metrics_cpu_seconds - instance_snapshot_never - certificate_token - instance_nic_routed_neighbor_probe - event_hub - agent_nic_config - projects_restricted_intercept - metrics_authentication - images_target_project - images_all_projects - cluster_migration_inconsistent_copy - cluster_ovn_chassis - container_syscall_intercept_sched_setscheduler - storage_lvm_thinpool_metadata_size - storage_volume_state_total - instance_file_head - instances_nic_host_name - image_copy_profile - container_syscall_intercept_sysinfo - clustering_evacuation_mode - resources_pci_vpd - qemu_raw_conf - storage_cephfs_fscache - network_load_balancer - vsock_api - instance_ready_state - network_bgp_holdtime - storage_volumes_all_projects - metrics_memory_oom_total - storage_buckets - storage_buckets_create_credentials - metrics_cpu_effective_total - projects_networks_restricted_access - storage_buckets_local - loki - acme - internal_metrics - cluster_join_token_expiry - remote_token_expiry - init_preseed - storage_volumes_created_at - cpu_hotplug - projects_networks_zones - network_txqueuelen - cluster_member_state - instances_placement_scriptlet - storage_pool_source_wipe - zfs_block_mode - instance_generation_id - disk_io_cache - amd_sev - storage_pool_loop_resize - migration_vm_live - ovn_nic_nesting - oidc - network_ovn_l3only - ovn_nic_acceleration_vdpa - cluster_healing - instances_state_total - auth_user - security_csm - instances_rebuild - numa_cpu_placement - custom_volume_iso - network_allocations - zfs_delegate - storage_api_remote_volume_snapshot_copy - operations_get_query_all_projects - metadata_configuration - syslog_socket - event_lifecycle_name_and_project - instances_nic_limits_priority - disk_initial_volume_configuration - operation_wait - image_restriction_privileged - cluster_internal_custom_volume_copy - disk_io_bus - storage_cephfs_create_missing - instance_move_config - ovn_ssl_config - certificate_description - disk_io_bus_virtio_blk - loki_config_instance - instance_create_start - clustering_evacuation_stop_options - boot_host_shutdown_action - agent_config_drive - network_state_ovn_lr - image_template_permissions - storage_bucket_backup - storage_lvm_cluster - shared_custom_block_volumes api_status: stable api_version: "1.0" auth: trusted public: false auth_methods: - tls auth_user_name: adam auth_user_method: unix environment: addresses: - 10.20.10.11:8443 architectures: - x86_64 - i686 certificate: | -----BEGIN CERTIFICATE----- MIIB/jCCAYOgAwIBAgIQGeWWR5EshUU2Sp2Gvi46XTAKBggqhkjOPQQDAzAyMRww GgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRIwEAYDVQQDDAlyb290QGJvbmsw HhcNMjMwODEzMjE1MzEzWhcNMzMwODEwMjE1MzEzWjAyMRwwGgYDVQQKExNsaW51 eGNvbnRhaW5lcnMub3JnMRIwEAYDVQQDDAlyb290QGJvbmswdjAQBgcqhkjOPQIB BgUrgQQAIgNiAAQ5PHduji7vWUnyo3y37yCIqarkvkTgS/kCnEiyBYDsqqh7f3T1 2vljyi92DIKvgB0CDrJOMUDkJrQB1wCRYf0s1Pm5BHHdIQp2hcdhUBzPNCW6hKzx ZOTRoVg5nVp//L+jXjBcMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEF BQcDATAMBgNVHRMBAf8EAjAAMCcGA1UdEQQgMB6CBGJvbmuHBH8AAAGHEAAAAAAA AAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDaQAwZgIxAKFxvDIUECjuDAGDkPXVT8rp /h/knaxBQSbljHHvdlu8xGRK8o/KSMlthqONSUjPcAIxANrJfDPs2E3KRUgJW2Li iv2DxqN9jdDgEAIPzL1yPsCTKnTAT+oAuiYRxFXBGkfiNg== -----END CERTIFICATE----- certificate_fingerprint: 8ff2b80f158c290cde9e4f78d4c662bba1f66fe5366bc3272eb1fb2b23965312 driver: lxc | qemu driver_version: 5.0.3 | 8.2.1 firewall: nftables kernel: Linux kernel_architecture: x86_64 kernel_features: idmapped_mounts: "true" netnsid_getifaddrs: "true" seccomp_listener: "true" seccomp_listener_continue: "true" uevent_injection: "true" unpriv_binfmt: "false" unpriv_fscaps: "true" kernel_version: 6.1.79 lxc_features: cgroup2: "true" core_scheduling: "true" devpts_fd: "true" idmapped_mounts_v2: "true" mount_injection_file: "true" network_gateway_device_route: "true" network_ipvlan: "true" network_l2proxy: "true" network_phys_macvlan_mtu: "true" network_veth_router: "true" pidfd: "true" seccomp_allow_deny_syntax: "true" seccomp_notify: "true" seccomp_proxy_send_notify_fd: "true" os_name: NixOS os_version: "24.05" project: default server: incus server_clustered: false server_event_mode: full-mesh server_name: bonk server_pid: 1870 server_version: "0.6" storage: zfs storage_version: 2.2.2-1 storage_supported_drivers: - name: dir version: "1" remote: false - name: lvm version: 2.03.23(2) (2023-11-21) / 1.02.197 (2023-11-21) / 4.47.0 remote: false - name: lvmcluster version: 2.03.23(2) (2023-11-21) / 1.02.197 (2023-11-21) / 4.47.0 remote: true - name: zfs version: 2.2.2-1 remote: false - name: btrfs version: 6.7.1 remote: false ```
stgraber commented 7 months ago

Duplicate of #530 which was fixed in #529

adamcstephens commented 7 months ago

Sorry. I searched but now realize why I missed that.

stgraber commented 7 months ago

It was a very quick turnaround on fixing that one as our CI did actually catch it but didn't report the updated result until after 0.6 was tagged, so there wasn't much time with the issue open for Google or Github to index (they're not great at indexing closed issues somehow).

adamcstephens commented 7 months ago

Yeah I noticed the fix number is one lower than the linked issue :)