search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.32k stars 928 forks source link

Container stop/restart hanging after refresh from LXD 5.0.3 to LXD 5.21.1 #13331

Open mszywala opened 4 months ago

mszywala commented 4 months ago

Required information

Output of "lxc info"

 config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- backup_compression
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- instances_nic_host_name
- image_copy_profile
- container_syscall_intercept_sysinfo
- clustering_evacuation_mode
- resources_pci_vpd
- qemu_raw_conf
- storage_cephfs_fscache
- network_load_balancer
- vsock_api
- instance_ready_state
- network_bgp_holdtime
- storage_volumes_all_projects
- metrics_memory_oom_total
- storage_buckets
- storage_buckets_create_credentials
- metrics_cpu_effective_total
- projects_networks_restricted_access
- storage_buckets_local
- loki
- acme
- internal_metrics
- cluster_join_token_expiry
- remote_token_expiry
- init_preseed
- storage_volumes_created_at
- cpu_hotplug
- projects_networks_zones
- network_txqueuelen
- cluster_member_state
- instances_placement_scriptlet
- storage_pool_source_wipe
- zfs_block_mode
- instance_generation_id
- disk_io_cache
- amd_sev
- storage_pool_loop_resize
- migration_vm_live
- ovn_nic_nesting
- oidc
- network_ovn_l3only
- ovn_nic_acceleration_vdpa
- cluster_healing
- instances_state_total
- auth_user
- security_csm
- instances_rebuild
- numa_cpu_placement
- custom_volume_iso
- network_allocations
- storage_api_remote_volume_snapshot_copy
- zfs_delegate
- operations_get_query_all_projects
- metadata_configuration
- syslog_socket
- event_lifecycle_name_and_project
- instances_nic_limits_priority
- disk_initial_volume_configuration
- operation_wait
- cluster_internal_custom_volume_copy
- disk_io_bus
- storage_cephfs_create_missing
- instance_move_config
- ovn_ssl_config
- init_preseed_storage_volumes
- metrics_instances_count
- server_instance_type_info
- resources_disk_mounted
- server_version_lts
- oidc_groups_claim
- loki_config_instance
- storage_volatile_uuid
- import_instance_devices
- instances_uefi_vars
- instances_migration_stateful
- container_syscall_filtering_allow_deny_syntax
- access_management
- vm_disk_io_limits
- storage_volumes_all
- instances_files_modify_permissions
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
auth_user_name: root
auth_user_method: unix
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    CENSORED
    -----END CERTIFICATE-----
  certificate_fingerprint: CENSORED
  driver: lxc
  driver_version: 6.0.0
  instance_types:
  - container
  firewall: nftables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    idmapped_mounts: "true"
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    uevent_injection: "true"
    unpriv_fscaps: "true"
  kernel_version: 5.15.0-102-generic
  lxc_features:
    cgroup2: "true"
    core_scheduling: "true"
    devpts_fd: "true"
    idmapped_mounts_v2: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: Ubuntu
  os_version: "22.04"
  project: default
  server: lxd
  server_clustered: false
  server_event_mode: full-mesh
  server_name: ubuntu-jammy
  server_pid: 6383
  server_version: 5.21.1
  server_lts: true
  storage: dir
  storage_version: "1"
  storage_supported_drivers:
  - name: dir
    version: "1"
    remote: false
  - name: lvm
    version: 2.03.11(2) (2021-01-08) / 1.02.175 (2021-01-08) / 4.45.0
    remote: false
  - name: powerflex
    version: 1.16 (nvme-cli)
    remote: true
  - name: zfs
    version: 2.1.5-1ubuntu6~22.04.3
    remote: false
  - name: btrfs
    version: 5.16.2
    remote: false
  - name: ceph
    version: 17.2.7
    remote: true
  - name: cephfs
    version: 17.2.7
    remote: true
  - name: cephobject
    version: 17.2.7
    remote: true

Issue description

We're on the way to upgrading our servers from LXD 5.0.3 to LXD 5.21.1 LTS. I somehow experienced the issue that the stop and restart container operations were hanging after the refresh. I only encountered the issue for the first stop/restart after snap refresh lxd --channel=5.21/stable It seems not to make a difference if -f is used to stop or restart the instance.

After a while, the instance stops with the following error:

Error: Failed shutting down instance, status is "Running": context deadline exceeded
Try `lxc info --show-log u` for more info

Additionally, we are using a routed nic for our container instances.

Because the instance is not stopping properly, there are remaining routes and the veth Device of the instance is in the state "LOWERLAYERDOWN" and the instance is refusing to start back up. When we manage to shut down an instance, it is necessary to take down the veth interface of the instance with ip link set vethxxxx down, and we also have to clean up the remaining routes on the hostsystem

Expectation: lxc stop and lxc restart is working properly after a refresh.

Steps to reproduce

  1. Ubuntu 22.04 with LXD 5.0.3
  2. Create a container
  3. Perform a refresh snap refresh lxd --channel=5.21/stable
  4. Stop/Restart the container lxc stop <containerName>

Information to attach

Resources: Processes: 0 CPU usage: CPU usage (in seconds): 0

Log:

lxc u 20240415120709.358 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3621 - newuidmap binary is missing lxc u 20240415120709.358 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3627 - newgidmap binary is missing lxc u 20240415120709.359 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3621 - newuidmap binary is missing lxc u 20240415120709.359 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3627 - newgidmap binary is missing lxc u 20240415120831.600 ERROR conf - ../src/src/lxc/conf.c:run_buffer:322 - Script exited with status 1 lxc u 20240415120831.601 ERROR start - ../src/src/lxc/start.c:lxc_end:944 - Failed to run "lxc.hook.stop" hook lxc u 20240415120832.126 ERROR conf - ../src/src/lxc/conf.c:run_buffer:322 - Script exited with status 1 lxc u 20240415120832.126 ERROR start - ../src/src/lxc/start.c:lxc_end:985 - Failed to run lxc.hook.post-stop for container "u"

 - [X] Container configuration (`lxc config show NAME --expanded`)

architecture: x86_64 config: image.architecture: amd64 image.description: ubuntu 22.04 LTS amd64 (release) (20240319) image.label: release image.os: ubuntu image.release: jammy image.serial: "20240319" image.type: squashfs image.version: "22.04" volatile.base_image: ca08f2f359079cdb7fb2d4efa882df5974304d7ae9bb13cbac4904a1347626be volatile.cloud-init.instance-id: fd955459-ecfc-4d00-932e-896aa690f8f5 volatile.idmap.base: "0" volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]' volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]' volatile.last_state.idmap: '[]' volatile.last_state.power: RUNNING volatile.uuid: 72c6dd51-b44c-42ef-ac72-4064385a644e volatile.uuid.generation: 72c6dd51-b44c-42ef-ac72-4064385a644e devices: root: path: / pool: default type: disk ephemeral: false profiles:

location: none metadata: context: ip: '@' method: GET protocol: unix url: /1.0 username: root level: debug message: Handling API request timestamp: "2024-04-15T12:08:28.290919569Z" type: logging

location: none metadata: context: ip: '@' method: GET protocol: unix url: /1.0/events username: root level: debug message: Handling API request timestamp: "2024-04-15T12:08:28.292459798Z" type: logging

location: none metadata: context: id: 7206ca2e-6647-493d-acb0-9f2b739ad80f local: /var/snap/lxd/common/lxd/unix.socket remote: '@' level: debug message: Event listener server handler started timestamp: "2024-04-15T12:08:28.292742589Z" type: logging

location: none metadata: context: ip: '@' method: PUT protocol: unix url: /1.0/instances/u/state username: root level: debug message: Handling API request timestamp: "2024-04-15T12:08:28.293482232Z" type: logging

location: none metadata: class: task created_at: "2024-04-15T12:08:28.29445839Z" description: Stopping instance err: "" id: f74b35a7-28f7-4da7-a3a2-945f79b9d29a location: none may_cancel: false metadata: null resources: instances:

location: none metadata: context: class: task description: Stopping instance operation: f74b35a7-28f7-4da7-a3a2-945f79b9d29a project: default level: debug message: New operation timestamp: "2024-04-15T12:08:28.296017929Z" type: logging

location: none metadata: class: task created_at: "2024-04-15T12:08:28.29445839Z" description: Stopping instance err: "" id: f74b35a7-28f7-4da7-a3a2-945f79b9d29a location: none may_cancel: false metadata: null resources: instances:

location: none metadata: context: class: task description: Stopping instance operation: f74b35a7-28f7-4da7-a3a2-945f79b9d29a project: default level: debug message: Started operation timestamp: "2024-04-15T12:08:28.29604927Z" type: logging

location: none metadata: context: ip: '@' method: GET protocol: unix url: /1.0/operations/f74b35a7-28f7-4da7-a3a2-945f79b9d29a username: root level: debug message: Handling API request timestamp: "2024-04-15T12:08:28.297029355Z" type: logging

location: none metadata: context: instance: u instanceType: container project: default timeout: 10m0s level: debug message: Shutdown started timestamp: "2024-04-15T12:08:28.305543127Z" type: logging

location: none metadata: context: action: shutdown created: 2024-04-15 09:15:51.927293086 +0000 UTC ephemeral: "false" instance: u instanceType: container project: default timeout: 10m0s used: 2024-04-15 12:07:09.356193457 +0000 UTC level: info message: Shutting down instance timestamp: "2024-04-15T12:08:28.305832124Z" type: logging

location: none metadata: context: action: stop instance: u project: default reusable: "true" level: debug message: Instance operation lock created timestamp: "2024-04-15T12:08:28.305813353Z" type: logging

location: none metadata: context: instance: u instanceType: container project: default level: debug message: Shutdown request sent to instance timestamp: "2024-04-15T12:08:28.306351356Z" type: logging

location: none metadata: context: ip: '@' method: GET protocol: unix url: /1.0 username: root level: debug message: Handling API request timestamp: "2024-04-15T12:10:38.268495437Z" type: logging

location: none metadata: context: ip: '@' method: GET protocol: unix url: /1.0/instances/u?recursion=1 username: root level: debug message: Handling API request timestamp: "2024-04-15T12:10:38.269666709Z" type: logging

location: none metadata: context: driver: dir instance: u pool: default project: default level: debug message: GetInstanceUsage started timestamp: "2024-04-15T12:10:38.275039775Z" type: logging

location: none metadata: context: driver: dir instance: u pool: default project: default level: debug message: GetInstanceUsage finished timestamp: "2024-04-15T12:10:38.275631324Z" type: logging

location: none metadata: context: ip: '@' method: GET protocol: unix url: /1.0/instances/u/logs/lxc.log username: root level: debug message: Handling API request timestamp: "2024-04-15T12:10:38.294387288Z" type: logging

location: none metadata: action: instance-log-retrieved requestor: address: '@' protocol: unix username: root source: /1.0/instances/u/backups/lxc.log project: default timestamp: "2024-04-15T12:10:38.295641303Z" type: lifecycle

location: none metadata: class: task created_at: "2024-04-15T12:08:28.29445839Z" description: Stopping instance err: 'Failed shutting down instance, status is "Running": context deadline exceeded' id: f74b35a7-28f7-4da7-a3a2-945f79b9d29a location: none may_cancel: false metadata: null resources: instances:

location: none metadata: context: instance: u instanceType: container project: default timeout: 10m0s level: debug message: Shutdown finished timestamp: "2024-04-15T12:18:28.307150063Z" type: logging

location: none metadata: context: class: task description: Stopping instance err: 'Failed shutting down instance, status is "Running": context deadline exceeded' operation: f74b35a7-28f7-4da7-a3a2-945f79b9d29a project: default level: debug message: Failure for operation timestamp: "2024-04-15T12:18:28.307227229Z" type: logging

location: none metadata: context: listener: 7206ca2e-6647-493d-acb0-9f2b739ad80f local: /var/snap/lxd/common/lxd/unix.socket remote: '@' level: debug message: Event listener server handler stopped timestamp: "2024-04-15T12:18:28.312781582Z" type: logging

mszywala commented 2 months ago

Any updates on this topic?

hamistao commented 1 month ago

It seems that the Shutdown function is halting here, probably because LXD is not receiving the appropriate hook to finish the operation (neither the stop hook nor the stop namespace). Is is still to be determined what is the inconsistency on the hook.

tomponline commented 1 month ago

Took a look at this and found this in /var/snap/lxd/common/lxd/logs/c1/lxc.log after setting sudo snap set lxd daemon.debug=true; sudo systemctl reload snap.lxd.daemon:

lxc c1 20240719091404.652 DEBUG    conf - ../src/src/lxc/conf.c:run_buffer:311 - Script exec /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd "default" "c1" stop produced output: /snap/lxd/current/bin/lxd: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by /snap/lxd/current/bin/lxd)

lxc c1 20240719091404.652 DEBUG    conf - ../src/src/lxc/conf.c:run_buffer:311 - Script exec /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd "default" "c1" stop produced output: /snap/lxd/current/bin/lxd: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /snap/lxd/current/bin/lxd)

lxc c1 20240719091404.652 DEBUG    conf - ../src/src/lxc/conf.c:run_buffer:311 - Script exec /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd "default" "c1" stop produced output: /snap/lxd/current/bin/lxd: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /snap/lxd/current/bin/lxd)

lxc c1 20240719091404.652 DEBUG    conf - ../src/src/lxc/conf.c:run_buffer:311 - Script exec /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd "default" "c1" stop produced output: /snap/lxd/current/bin/lxd: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /snap/lxd/current/bin/lxd)

lxc c1 20240719091404.653 ERROR    conf - ../src/src/lxc/conf.c:run_buffer:322 - Script exited with status 1
lxc c1 20240719091404.653 ERROR    start - ../src/src/lxc/start.c:lxc_end:985 - Failed to run lxc.hook.post-stop for container "c1"

Looks like it could be to do with the core base change from 5.0 to 5.21 (core20 to core22).

tomponline commented 1 month ago

@simondeziel @mihalicyn btw ive confirmed this is also an issue going from 5.0/stable to 5.0/edge so it'll need to be sorted before we put out 5.0.4 with core22.