search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.38k stars 931 forks source link

snap 5.15-002fa0f breaks redhat/centOS 7.9 VMs #12013

Closed sirtoobii closed 1 year ago

sirtoobii commented 1 year ago

Issue description

After an update to the snap package 5.15-002fa0f our Redhat 7.9 VMs fail to boot because they cannot see any disks:

[    3.990456] virtio-pci: probe of 0000:03:00.1 failed with error -22
[    3.991904] virtio-pci 0000:04:00.0: virtio_pci: unable to map virtio 56@4096 on bar 2
[    3.995995] virtio-pci: probe of 0000:04:00.0 failed with error -22
[snip]
[  187.585367] dracut-initqueue[288]: Warning: Could not boot.
[  187.755654] dracut-initqueue[288]: Warning: /dev/mapper/rhel-root does not exist
[  187.756036] dracut-initqueue[288]: Warning: /dev/rhel/root does not exist
[  187.756657] dracut-initqueue[288]: Warning: /dev/rhel/swap does not exist
         Starting Dracut Emergency Shell...
Warning: /dev/mapper/rhel-root does not exist
Warning: /dev/rhel/root does not exist
Warning: /dev/rhel/swap does not exist

Downgrading to 5.14-7072c7b resolves the issue.

Steps to reproduce

The issue is easily reproducible by booting directly from an official redhat iso (or centOS 7.9):

  1. Create a fresh vm:

     # create vm
 lxc init test-rhl7 --config limits.cpu=4 --config limits.memory=2048MiB --vm --empty

 # attach iso
 lxc config device add test-rhl7 rhl7iso disk source=/VM/iso/rhel-server-7.9-x86_64-dvd.iso
  2. Boot from iso

Information to attach

Container configuration ```yaml architecture: x86_64 config: limits.cpu: "4" limits.memory: 16384MiB volatile.cloud-init.instance-id: 020a77f6-d05b-42fb-8a8d-3c19134b3204 volatile.eth0.hwaddr: 00:16:3e:53:83:ba volatile.last_state.power: STOPPED volatile.last_state.ready: "false" volatile.uuid: 85e7bb4e-0c6f-4c34-b0b9-ea371c746558 volatile.uuid.generation: 85e7bb4e-0c6f-4c34-b0b9-ea371c746558 volatile.vsock_id: "7" devices: eth0: name: eth0 nictype: bridged parent: br-lxd type: nic rhl7iso: source: /VM/iso/rhel-server-7.9-x86_64-dvd.iso type: disk root: path: / pool: vm-disks size: 250GB type: disk ephemeral: false profiles: - default stateful: false description: "" ```
tomponline commented 1 year ago

So the VM doesn't book off of an iso?

tomponline commented 1 year ago

What does "lxc info --show-log" show when its running?

sirtoobii commented 1 year ago

What does "lxc info --show-log" show when its running?

lxc info --show-log ``` root@cad-vs01:~# /snap/bin/lxc info --show-log config: core.https_address: :8443 core.trust_password: true api_extensions: - storage_zfs_remove_snapshots - container_host_shutdown_timeout - container_stop_priority - container_syscall_filtering - auth_pki - container_last_used_at - etag - patch - usb_devices - https_allowed_credentials - image_compression_algorithm - directory_manipulation - container_cpu_time - storage_zfs_use_refquota - storage_lvm_mount_options - network - profile_usedby - container_push - container_exec_recording - certificate_update - container_exec_signal_handling - gpu_devices - container_image_properties - migration_progress - id_map - network_firewall_filtering - network_routes - storage - file_delete - file_append - network_dhcp_expiry - storage_lvm_vg_rename - storage_lvm_thinpool_rename - network_vlan - image_create_aliases - container_stateless_copy - container_only_migration - storage_zfs_clone_copy - unix_device_rename - storage_lvm_use_thinpool - storage_rsync_bwlimit - network_vxlan_interface - storage_btrfs_mount_options - entity_description - image_force_refresh - storage_lvm_lv_resizing - id_map_base - file_symlinks - container_push_target - network_vlan_physical - storage_images_delete - container_edit_metadata - container_snapshot_stateful_migration - storage_driver_ceph - storage_ceph_user_name - resource_limits - storage_volatile_initial_source - storage_ceph_force_osd_reuse - storage_block_filesystem_btrfs - resources - kernel_limits - storage_api_volume_rename - macaroon_authentication - network_sriov - console - restrict_devlxd - migration_pre_copy - infiniband - maas_network - devlxd_events - proxy - network_dhcp_gateway - file_get_symlink - network_leases - unix_device_hotplug - storage_api_local_volume_handling - operation_description - clustering - event_lifecycle - storage_api_remote_volume_handling - nvidia_runtime - container_mount_propagation - container_backup - devlxd_images - container_local_cross_pool_handling - proxy_unix - proxy_udp - clustering_join - proxy_tcp_udp_multi_port_handling - network_state - proxy_unix_dac_properties - container_protection_delete - unix_priv_drop - pprof_http - proxy_haproxy_protocol - network_hwaddr - proxy_nat - network_nat_order - container_full - candid_authentication - backup_compression - candid_config - nvidia_runtime_config - storage_api_volume_snapshots - storage_unmapped - projects - candid_config_key - network_vxlan_ttl - container_incremental_copy - usb_optional_vendorid - snapshot_scheduling - snapshot_schedule_aliases - container_copy_project - clustering_server_address - clustering_image_replication - container_protection_shift - snapshot_expiry - container_backup_override_pool - snapshot_expiry_creation - network_leases_location - resources_cpu_socket - resources_gpu - resources_numa - kernel_features - id_map_current - event_location - storage_api_remote_volume_snapshots - network_nat_address - container_nic_routes - rbac - cluster_internal_copy - seccomp_notify - lxc_features - container_nic_ipvlan - network_vlan_sriov - storage_cephfs - container_nic_ipfilter - resources_v2 - container_exec_user_group_cwd - container_syscall_intercept - container_disk_shift - storage_shifted - resources_infiniband - daemon_storage - instances - image_types - resources_disk_sata - clustering_roles - images_expiry - resources_network_firmware - backup_compression_algorithm - ceph_data_pool_name - container_syscall_intercept_mount - compression_squashfs - container_raw_mount - container_nic_routed - container_syscall_intercept_mount_fuse - container_disk_ceph - virtual-machines - image_profiles - clustering_architecture - resources_disk_id - storage_lvm_stripes - vm_boot_priority - unix_hotplug_devices - api_filtering - instance_nic_network - clustering_sizing - firewall_driver - projects_limits - container_syscall_intercept_hugetlbfs - limits_hugepages - container_nic_routed_gateway - projects_restrictions - custom_volume_snapshot_expiry - volume_snapshot_scheduling - trust_ca_certificates - snapshot_disk_usage - clustering_edit_roles - container_nic_routed_host_address - container_nic_ipvlan_gateway - resources_usb_pci - resources_cpu_threads_numa - resources_cpu_core_die - api_os - container_nic_routed_host_table - container_nic_ipvlan_host_table - container_nic_ipvlan_mode - resources_system - images_push_relay - network_dns_search - container_nic_routed_limits - instance_nic_bridged_vlan - network_state_bond_bridge - usedby_consistency - custom_block_volumes - clustering_failure_domains - resources_gpu_mdev - console_vga_type - projects_limits_disk - network_type_macvlan - network_type_sriov - container_syscall_intercept_bpf_devices - network_type_ovn - projects_networks - projects_networks_restricted_uplinks - custom_volume_backup - backup_override_name - storage_rsync_compression - network_type_physical - network_ovn_external_subnets - network_ovn_nat - network_ovn_external_routes_remove - tpm_device_type - storage_zfs_clone_copy_rebase - gpu_mdev - resources_pci_iommu - resources_network_usb - resources_disk_address - network_physical_ovn_ingress_mode - network_ovn_dhcp - network_physical_routes_anycast - projects_limits_instances - network_state_vlan - instance_nic_bridged_port_isolation - instance_bulk_state_change - network_gvrp - instance_pool_move - gpu_sriov - pci_device_type - storage_volume_state - network_acl - migration_stateful - disk_state_quota - storage_ceph_features - projects_compression - projects_images_remote_cache_expiry - certificate_project - network_ovn_acl - projects_images_auto_update - projects_restricted_cluster_target - images_default_architecture - network_ovn_acl_defaults - gpu_mig - project_usage - network_bridge_acl - warnings - projects_restricted_backups_and_snapshots - clustering_join_token - clustering_description - server_trusted_proxy - clustering_update_cert - storage_api_project - server_instance_driver_operational - server_supported_storage_drivers - event_lifecycle_requestor_address - resources_gpu_usb - clustering_evacuation - network_ovn_nat_address - network_bgp - network_forward - custom_volume_refresh - network_counters_errors_dropped - metrics - image_source_project - clustering_config - network_peer - linux_sysctl - network_dns - ovn_nic_acceleration - certificate_self_renewal - instance_project_move - storage_volume_project_move - cloud_init - network_dns_nat - database_leader - instance_all_projects - clustering_groups - ceph_rbd_du - instance_get_full - qemu_metrics - gpu_mig_uuid - event_project - clustering_evacuation_live - instance_allow_inconsistent_copy - network_state_ovn - storage_volume_api_filtering - image_restrictions - storage_zfs_export - network_dns_records - storage_zfs_reserve_space - network_acl_log - storage_zfs_blocksize - metrics_cpu_seconds - instance_snapshot_never - certificate_token - instance_nic_routed_neighbor_probe - event_hub - agent_nic_config - projects_restricted_intercept - metrics_authentication - images_target_project - cluster_migration_inconsistent_copy - cluster_ovn_chassis - container_syscall_intercept_sched_setscheduler - storage_lvm_thinpool_metadata_size - storage_volume_state_total - instance_file_head - instances_nic_host_name - image_copy_profile - container_syscall_intercept_sysinfo - clustering_evacuation_mode - resources_pci_vpd - qemu_raw_conf - storage_cephfs_fscache - network_load_balancer - vsock_api - instance_ready_state - network_bgp_holdtime - storage_volumes_all_projects - metrics_memory_oom_total - storage_buckets - storage_buckets_create_credentials - metrics_cpu_effective_total - projects_networks_restricted_access - storage_buckets_local - loki - acme - internal_metrics - cluster_join_token_expiry - remote_token_expiry - init_preseed - storage_volumes_created_at - cpu_hotplug - projects_networks_zones - network_txqueuelen - cluster_member_state - instances_placement_scriptlet - storage_pool_source_wipe - zfs_block_mode - instance_generation_id - disk_io_cache - amd_sev - storage_pool_loop_resize - migration_vm_live - ovn_nic_nesting - oidc - network_ovn_l3only - ovn_nic_acceleration_vdpa - cluster_healing - instances_state_total - auth_user - security_csm - instances_rebuild - numa_cpu_placement api_status: stable api_version: "1.0" auth: trusted public: false auth_methods: - tls auth_user_name: root auth_user_method: unix environment: addresses: - 192.168.249.153:8443 architectures: - x86_64 - i686 certificate: | -----BEGIN CERTIFICATE----- MIICCDCCAY+gAwIBAgIQMPbZC86BSUHoNYRpywlxuzAKBggqhkjOPQQDAzA2MRww GgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRYwFAYDVQQDDA1yb290QGNhZC12 czAxMB4XDTIzMDUxNTEzNDY0OFoXDTMzMDUxMjEzNDY0OFowNjEcMBoGA1UEChMT bGludXhjb250YWluZXJzLm9yZzEWMBQGA1UEAwwNcm9vdEBjYWQtdnMwMTB2MBAG ByqGSM49AgEGBSuBBAAiA2IABKyTn6xd9EguT38i2q3Ig2ERC2qUjN9uYUF2f6Wt ppcmv1lkUXcbBL5Rh5SlUnLERYyR7+YijM1Up3axnHPLQRe765uuYgsYzpjdhEZe Cg7DV1PJgaNj3/DzJ483e9q586NiMGAwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQM MAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwKwYDVR0RBCQwIoIIY2FkLXZzMDGH BH8AAAGHEAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDZwAwZAIwXHZ6A9kZ mh4EfyFjhQBngpL/ff0SmlJOxPvfEevhkcPWW2ok0ureT4fUwelOXv0HAjARljaf ChOlL19j+mgjpGadvyBr/KeaFbMIl+YK032+oJHxZ2Klapmw/WHMmhL6eo0= -----END CERTIFICATE----- certificate_fingerprint: e91124c7a6b114abdc8db307e6e069c860012706aea2aecd40792a154a9b7971 driver: qemu | lxc driver_version: 8.0.0 | 5.0.2 firewall: nftables kernel: Linux kernel_architecture: x86_64 kernel_features: idmapped_mounts: "true" netnsid_getifaddrs: "true" seccomp_listener: "true" seccomp_listener_continue: "true" shiftfs: "false" uevent_injection: "true" unpriv_fscaps: "true" kernel_version: 5.15.0-76-generic lxc_features: cgroup2: "true" core_scheduling: "true" devpts_fd: "true" idmapped_mounts_v2: "true" mount_injection_file: "true" network_gateway_device_route: "true" network_ipvlan: "true" network_l2proxy: "true" network_phys_macvlan_mtu: "true" network_veth_router: "true" pidfd: "true" seccomp_allow_deny_syntax: "true" seccomp_notify: "true" seccomp_proxy_send_notify_fd: "true" os_name: Ubuntu os_version: "22.04" project: default server: lxd server_clustered: false server_event_mode: full-mesh server_name: cad-vs01 server_pid: 38942 server_version: "5.15" storage: dir storage_version: "1" storage_supported_drivers: - name: dir version: "1" remote: false - name: lvm version: 2.03.11(2) (2021-01-08) / 1.02.175 (2021-01-08) / 4.45.0 remote: false - name: zfs version: 2.1.5-1ubuntu6~22.04.1 remote: false - name: btrfs version: 5.16.2 remote: false - name: ceph version: 17.2.5 remote: true - name: cephfs version: 17.2.5 remote: true - name: cephobject version: 17.2.5 remote: true ```

So the VM doesn't book off of an iso?

I just know about this specific redhat version. But an ubuntu iso works fine

tomponline commented 1 year ago

Will it boot off other isos?

tomponline commented 1 year ago

Also sorry, I meant

lxc info (instance) --show-log

sirtoobii commented 1 year ago

lxc info (instance) --show-log

The log: sections stays empty

Will it boot off other isos?

Yes, the latest ubuntu-server iso boots perfectly fine

tomponline commented 1 year ago

Interesting. Something different about the redhat iso. Are you able to reproduce with an iso we can try (maybe a centos iso?)

sirtoobii commented 1 year ago

@tomponline Just tested centOS 7.9, netinst iso - same behaviour.

tomponline commented 1 year ago

Ah good so we should he able to reproduce

mihalicyn commented 1 year ago

Between 5.15-002fa0f and 5.14-7072c7b UEFI firmware was updated and CSM support was added. I think it worth trying to check if it works with security.csm set to true.

tomponline commented 1 year ago

Certainly worth a try. Although that will disable UEFI which shouldn't be needed in this case AFAIK and seems undesirable.

tomponline commented 1 year ago

Can you try using the latest/edge channel (keeping in mind you likely won't be able to downgrade, so using a fresh/throwaway system for this would be good) and then also I noticed you've not set your boot.priority, which we normally need to do to ensure it boots from the iso.

See https://documentation.ubuntu.com/lxd/en/latest/howto/instances_create/#launch-a-vm-that-boots-from-an-iso

But I've just successfully booted from and installed centos 7.9 from the iso using latest/edge:

lxc config show vtest --expanded
architecture: x86_64
config:
  volatile.cloud-init.instance-id: d2d513de-be55-4d46-8459-1f78d95036d8
  volatile.eth0.host_name: tapca911747
  volatile.eth0.hwaddr: 00:16:3e:cc:4c:d2
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: 0b43fd4f-aee8-4787-9a95-a1840f199b5e
  volatile.uuid.generation: 0b43fd4f-aee8-4787-9a95-a1840f199b5e
  volatile.vsock_id: "4075497721"
devices:
  eth0:
    name: eth0
    network: lxdbr1
    type: nic
  iso:
    boot.priority: "10"
    source: /home/user/Downloads/CentOS-7-x86_64-DVD-2009.iso
    type: disk
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

If this works, then LXD 5.16 is being released next week and will be cut from latest/edge.

tomponline commented 1 year ago

I tried latest/stable too for good measure and it worked fine.

sirtoobii commented 1 year ago

and then also I noticed you've not set your boot.priority, which we normally need to do to ensure it boots from the iso.

Yes i thought there must be such an option -> I always booted with the graphical boot menu.

I tried latest/stable too for good measure and it worked fine.

You managed to boot with 5.15? If yes, did you change anything?

And at this point also thank you for looking into this so quickly!

tomponline commented 1 year ago

Yes I did manage to boot.

sirtoobii commented 1 year ago

Yes I did manage to boot.

@tomponline Just for clarification: You booted the centOS 7.9 iso with the 5.15-002fa0f lxd snap and without any modifications?

tomponline commented 1 year ago

Yes indeed. I only needed to set boot.priority, which is a normal requirement.

tomponline commented 1 year ago

What storage pool type is vm-disks?

tomponline commented 1 year ago

I tried on a dir pool.

Did you try latest/edge yet?

sirtoobii commented 1 year ago

Yes indeed. I only needed to set boot.priority, which is a normal requirement.

Hmm, I suspect this has nothing to do with the success of the booting itself (I mean I should make no difference if manually chose the ISO in the boot menu?)

What storage pool type is vm-disks?

Also dir

Did you try latest/edge yet?

Not yet, I probably won´t have time until this evening

tomponline commented 1 year ago

Hmm, I suspect this has nothing to do with the success of the booting itself (I mean I should make no difference if manually chose the ISO in the boot menu?)

Worth trying though right? In case somehow you are selecting a different item (perhaps a non-uefi enabled one).

tomponline commented 1 year ago

Seems like we've seen a similar issue here https://discourse.ubuntu.com/t/ubuntu-18-04-vm-doesnt-start-on-ubuntu-20-04-on-lxd-5-15

tomponline commented 1 year ago

Does lxc config set NAME raw.qemu='-cpu qemu64' fix it?

tomponline commented 1 year ago

In addition to the workaround above, we intend to address this issue via a patch to our edk build in the snap package.

For more details on the issue see

https://edk2.groups.io/g/devel/topic/94113626#104943