dn commented 6 months ago

Required information

Distribution: Ubuntu
Distribution version: 22.04.3 LTS

The output of "incus info" or if that fails:

config:
cluster.https_address: 172.17.53.1:8443
core.bgp_address: 172.17.53.1
core.bgp_asn: "4200000004"
core.bgp_routerid: 172.17.53.1
core.dns_address: 172.17.53.1
core.https_address: 172.17.53.1:8443
network.ovn.northbound_connection: tcp:172.17.53.1:6641,tcp:172.17.53.2:6641,tcp:172.17.53.3:6641
storage.images_volume: local/incus-images
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- network_sriov
- console
- restrict_dev_incus
- migration_pre_copy
- infiniband
- dev_incus_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- dev_incus_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- backup_compression
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- instances_nic_host_name
- image_copy_profile
- container_syscall_intercept_sysinfo
- clustering_evacuation_mode
- resources_pci_vpd
- qemu_raw_conf
- storage_cephfs_fscache
- network_load_balancer
- vsock_api
- instance_ready_state
- network_bgp_holdtime
- storage_volumes_all_projects
- metrics_memory_oom_total
- storage_buckets
- storage_buckets_create_credentials
- metrics_cpu_effective_total
- projects_networks_restricted_access
- storage_buckets_local
- loki
- acme
- internal_metrics
- cluster_join_token_expiry
- remote_token_expiry
- init_preseed
- storage_volumes_created_at
- cpu_hotplug
- projects_networks_zones
- network_txqueuelen
- cluster_member_state
- instances_placement_scriptlet
- storage_pool_source_wipe
- zfs_block_mode
- instance_generation_id
- disk_io_cache
- amd_sev
- storage_pool_loop_resize
- migration_vm_live
- ovn_nic_nesting
- oidc
- network_ovn_l3only
- ovn_nic_acceleration_vdpa
- cluster_healing
- instances_state_total
- auth_user
- security_csm
- instances_rebuild
- numa_cpu_placement
- custom_volume_iso
- network_allocations
- zfs_delegate
- storage_api_remote_volume_snapshot_copy
- operations_get_query_all_projects
- metadata_configuration
- syslog_socket
- event_lifecycle_name_and_project
- instances_nic_limits_priority
- disk_initial_volume_configuration
- operation_wait
- image_restriction_privileged
- cluster_internal_custom_volume_copy
- disk_io_bus
- storage_cephfs_create_missing
- instance_move_config
- ovn_ssl_config
- certificate_description
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
auth_user_name: root
auth_user_method: unix
environment:
addresses:
- 172.17.53.1:8443
architectures:
- x86_64
- i686
certificate: |
-----BEGIN CERTIFICATE-----
MIIB9DCCAXugAwIBAgIRAIh9whuiCtgrYAzZQs9d+eYwCgYIKoZIzj0EAwMwLjEZ
MBcGA1UEChMQTGludXggQ29udGFpbmVyczERMA8GA1UEAwwIcm9vdEBuMDEwHhcN
MjMxMjE0MTI1MDUzWhcNMzMxMjExMTI1MDUzWjAuMRkwFwYDVQQKExBMaW51eCBD
b250YWluZXJzMREwDwYDVQQDDAhyb290QG4wMTB2MBAGByqGSM49AgEGBSuBBAAi
A2IABPBIxGDQ5ROU1lJFViTVhN5NoqauuZKPm5q4Uz6+vwLjrhQwH2yKEdPXQGhq
bI1Ky2Apedf7bqdey4w0ONxJzysJJ19TLWLC9hN8rr5oaXTLBezbp7fLiZxtSfn+
iWMVl6NdMFswDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwG
A1UdEwEB/wQCMAAwJgYDVR0RBB8wHYIDbjAxhwR/AAABhxAAAAAAAAAAAAAAAAAA
AAABMAoGCCqGSM49BAMDA2cAMGQCMG+j1QDlLScbhtim0wZ2zl1dYiFPZ4G7pmRH
PjLgpt538pyVVNxM/jKsigfjSGwAbgIwVqwGqF1OaRUB430EcjhKiHC2hvtCKtZ7
aqKeQ+P6WsakBt0fM1yj/fbvvAu57aUH
-----END CERTIFICATE-----
certificate_fingerprint: b182ada30911c2cf96f407d5f0f997edb324e7c7f57e42f77f7937f6ff462d12
driver: lxc | qemu
driver_version: 5.0.3 | 8.1.3
firewall: nftables
kernel: Linux
kernel_architecture: x86_64
kernel_features:
idmapped_mounts: "true"
netnsid_getifaddrs: "true"
seccomp_listener: "true"
seccomp_listener_continue: "true"
uevent_injection: "true"
unpriv_fscaps: "true"
kernel_version: 5.15.0-91-generic
lxc_features:
cgroup2: "true"
core_scheduling: "true"
devpts_fd: "true"
idmapped_mounts_v2: "true"
mount_injection_file: "true"
network_gateway_device_route: "true"
network_ipvlan: "true"
network_l2proxy: "true"
network_phys_macvlan_mtu: "true"
network_veth_router: "true"
pidfd: "true"
seccomp_allow_deny_syntax: "true"
seccomp_notify: "true"
seccomp_proxy_send_notify_fd: "true"
os_name: Ubuntu
os_version: "22.04"
project: lbs.c1.dus2.wortbit.net
server: incus
server_clustered: true
server_event_mode: full-mesh
server_name: n01
server_pid: 1958311
server_version: "0.4"
storage: zfs
storage_version: 2.1.5-1ubuntu6~22.04.2
storage_supported_drivers:
- name: dir
version: "1"
remote: false
- name: lvm
version: 2.03.11(2) (2021-01-08) / 1.02.175 (2021-01-08) / 4.45.0
remote: false
- name: zfs
version: 2.1.5-1ubuntu6~22.04.2
remote: false
- name: btrfs
version: 5.16.2
remote: false

Issue description

I just updated (single node cluster) via apt-get from 0.3 to 0.4. Containers that were running before complain now that /proc is not mounted. Restarting the container will fix it.

Steps to reproduce

launch a container with incus 0.3
upgrade via apt-get to incus 0.4
incus exec container-from-03 bash

root@haproxy02:~# ps aux
Error: /proc must be mounted
  To mount /proc at boot you need an /etc/fstab line like:
      proc   /proc   proc    defaults
  In the meantime, run "mount proc /proc -t proc"

Information to attach

[ ] Any relevant kernel output (dmesg)
[x] Container log (incus info NAME --show-log)
[x] Container configuration (incus config show NAME --expanded)
[x] Main daemon log (at /var/log/incus/incusd.log)
[ ] Output of the client with --debug
[ ] Output of the daemon with --debug (alternatively output of incus monitor --pretty while reproducing the issue)

# incus info haproxy02  --project lbs.c1.dus2.wortbit.net --show-log
Name: haproxy02
Status: RUNNING
Type: container
Architecture: x86_64
Location: n01
PID: 1013403
Created: 2023/12/18 11:32 UTC
Last Used: 2023/12/18 11:32 UTC

Resources:
  Processes: 54
  Disk usage:
    root: 173.86MiB
  CPU usage:
    CPU usage (in seconds): 81
  Memory usage:
    Memory (current): 298.23MiB
  Network usage:
    eth0:
      Type: broadcast
      State: UP
      Host interface: veth6989dd45
      MAC address: 00:16:3e:a7:88:db
      MTU: 1442
      Bytes received: 17.14MB
      Bytes sent: 2.17MB
      Packets received: 54370
      Packets sent: 24044
      IP addresses:
        inet:  10.128.2.3/24 (global)
        inet6: fd42:6dc4:9380:428a:216:3eff:fea7:88db/64 (global)
        inet6: fe80::216:3eff:fea7:88db/64 (link)
    lo:
      Type: loopback
      State: UP
      MTU: 65536
      Bytes received: 28.00kB
      Bytes sent: 28.00kB
      Packets received: 234
      Packets sent: 234
      IP addresses:
        inet:  127.0.0.1/8 (local)
        inet6: ::1/128 (local)

Log:

lxc lbs.c1.dus2.wortbit.net_haproxy02 20231223151632.237 WARN     conf - ../src/lxc/conf.c:lxc_map_ids:3621 - newuidmap binary is missing
lxc lbs.c1.dus2.wortbit.net_haproxy02 20231223151632.237 WARN     conf - ../src/lxc/conf.c:lxc_map_ids:3627 - newgidmap binary is missing
lxc lbs.c1.dus2.wortbit.net_haproxy02 20231223152006.673 WARN     conf - ../src/lxc/conf.c:lxc_map_ids:3621 - newuidmap binary is missing
lxc lbs.c1.dus2.wortbit.net_haproxy02 20231223152006.673 WARN     conf - ../src/lxc/conf.c:lxc_map_ids:3627 - newgidmap binary is missing
lxc lbs.c1.dus2.wortbit.net_haproxy02 20231223152249.738 WARN     conf - ../src/lxc/conf.c:lxc_map_ids:3621 - newuidmap binary is missing
lxc lbs.c1.dus2.wortbit.net_haproxy02 20231223152249.738 WARN     conf - ../src/lxc/conf.c:lxc_map_ids:3627 - newgidmap binary is missing

# incus config show haproxy02  --project lbs.c1.dus2.wortbit.net --expanded

architecture: x86_64
config:
  image.architecture: amd64
  image.description: Ubuntu jammy amd64 (20231217_07:42)
  image.os: Ubuntu
  image.release: jammy
  image.serial: "20231217_07:42"
  image.type: squashfs
  image.variant: cloud
  volatile.base_image: 99233818fe61f26003a98763cdb753b824638410b794a4b972bea99e8b8f06df
  volatile.cloud-init.instance-id: 0fda9c4d-eb97-46c2-8741-d60475b57294
  volatile.eth0.host_name: veth6989dd45
  volatile.eth0.hwaddr: 00:16:3e:a7:88:db
  volatile.eth0.last_state.ip_addresses: 10.128.2.3,fd42:6dc4:9380:428a:216:3eff:fea7:88db
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 2f4cd7ad-dd9b-434e-a684-0b7200cc9712
  volatile.uuid.generation: 2f4cd7ad-dd9b-434e-a684-0b7200cc9712
devices:
  eth0:
    ipv4.address: 10.128.2.3
    name: eth0
    network: default
    type: nic
  root:
    path: /
    pool: local
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""


# cat /var/log/incus/incusd.log
time="2023-12-23T14:34:21Z" level=warning msg="Could not handover member's responsibilities" err="Failed to transfer leadership: No online voter found"
time="2023-12-23T14:34:21Z" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2023-12-23T14:34:21Z" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"
time="2023-12-23T14:34:34Z" level=warning msg="Could not handover member's responsibilities" err="Failed to transfer leadership: No online voter found"
time="2023-12-23T14:34:34Z" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
time="2023-12-23T14:34:34Z" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use per-device limits.priority instead"

stgraber commented 6 months ago

That wouldn't actually be related to Incus but to LXCFS. /proc also didn't get unmounted, it's just some files inside of it have become broken.

Can you show journalctl -u incus-lxcfs -n 300?

stgraber commented 6 months ago

On package updates, LXCFS is asked to reload itself by sending SIGUSR1 to the running LXCFS process. This then causes a library reload to get it on the new version.

In this case, it appears that LXCFS crashed while doing that, which results in LXCFS proc files becoming invalid until the container is restarted...

I didn't run into this on any of my systems during a 0.3 to 0.4 upgrade also similarly running on Ubuntu 22.04 so hopefully it's not a widespread issue.

@mihalicyn has been doing some kernel and userspace work to be able to automatically recover from such situations in the future as LXCFS crashing has been a bit of an issue for a number of years now...

dn commented 6 months ago

# journalctl -u incus-lxcfs -n 300
Dec 14 11:46:02 n01 lxcfs[10735]: - proc_loadavg
Dec 14 11:46:02 n01 lxcfs[10735]: - proc_meminfo
Dec 14 11:46:02 n01 lxcfs[10735]: - proc_stat
Dec 14 11:46:02 n01 lxcfs[10735]: - proc_swaps
Dec 14 11:46:02 n01 lxcfs[10735]: - proc_uptime
Dec 14 11:46:02 n01 lxcfs[10735]: - proc_slabinfo
Dec 14 11:46:02 n01 lxcfs[10735]: - shared_pidns
Dec 14 11:46:02 n01 lxcfs[10735]: - cpuview_daemon
Dec 14 11:46:02 n01 lxcfs[10735]: - loadavg_daemon
Dec 14 11:46:02 n01 lxcfs[10735]: - pidfds
Dec 14 11:46:04 n01 systemd[1]: Reloading Incus - LXCFS daemon...
Dec 14 11:46:04 n01 systemd[1]: Reloaded Incus - LXCFS daemon.
Dec 14 11:51:19 n01 systemd[1]: Stopping Incus - LXCFS daemon...
Dec 14 11:51:19 n01 lxcfs[10735]: Running destructor lxcfs_exit
Dec 14 11:51:19 n01 systemd[1]: incus-lxcfs.service: Main process exited, code=exited, status=1/FAILURE
Dec 14 11:51:19 n01 fusermount[13099]: /bin/fusermount: failed to unmount /var/lib/incus-lxcfs: Invalid argument
Dec 14 11:51:19 n01 systemd[1]: incus-lxcfs.service: Failed with result 'exit-code'.
Dec 14 11:51:19 n01 systemd[1]: Stopped Incus - LXCFS daemon.
-- Boot ce4d1f1d84bd4b0a9a61b6e4cae3eca1 --
Dec 14 11:55:15 n01 systemd[1]: Starting Incus - LXCFS daemon...
Dec 14 11:55:15 n01 systemd[1]: Started Incus - LXCFS daemon.
Dec 14 11:55:15 n01 lxcfs[5029]: Running constructor lxcfs_init to reload liblxcfs
Dec 14 11:55:15 n01 lxcfs[5029]: mount namespace: 5
Dec 14 11:55:15 n01 lxcfs[5029]: hierarchies:
Dec 14 11:55:15 n01 lxcfs[5029]:   0: fd:   6: cpuset,cpu,io,memory,hugetlb,pids,rdma,misc
Dec 14 11:55:15 n01 lxcfs[5029]: Kernel supports pidfds
Dec 14 11:55:15 n01 lxcfs[5029]: Kernel does not support swap accounting
Dec 14 11:55:15 n01 lxcfs[5029]: api_extensions:
Dec 14 11:55:15 n01 lxcfs[5029]: - cgroups
Dec 14 11:55:15 n01 lxcfs[5029]: - sys_cpu_online
Dec 14 11:55:15 n01 lxcfs[5029]: - proc_cpuinfo
Dec 14 11:55:15 n01 lxcfs[5029]: - proc_diskstats
Dec 14 11:55:15 n01 lxcfs[5029]: - proc_loadavg
Dec 14 11:55:15 n01 lxcfs[5029]: - proc_meminfo
Dec 14 11:55:15 n01 lxcfs[5029]: - proc_stat
Dec 14 11:55:15 n01 lxcfs[5029]: - proc_swaps
Dec 14 11:55:15 n01 lxcfs[5029]: - proc_uptime
Dec 14 11:55:15 n01 lxcfs[5029]: - proc_slabinfo
Dec 14 11:55:15 n01 lxcfs[5029]: - shared_pidns
Dec 14 11:55:15 n01 lxcfs[5029]: - cpuview_daemon
Dec 14 11:55:15 n01 lxcfs[5029]: - loadavg_daemon
Dec 14 11:55:15 n01 lxcfs[5029]: - pidfds
Dec 14 11:55:16 n01 systemd[1]: Reloading Incus - LXCFS daemon...
Dec 14 11:55:16 n01 systemd[1]: Reloaded Incus - LXCFS daemon.
Dec 14 11:55:41 n01 lxcfs[5029]: Closed liblxcfs.so
Dec 14 11:55:41 n01 lxcfs[5029]: Running destructor lxcfs_exit
Dec 14 11:55:41 n01 lxcfs[5029]: Running constructor lxcfs_init to reload liblxcfs
Dec 14 11:55:41 n01 lxcfs[5029]: mount namespace: 6
Dec 14 11:55:41 n01 lxcfs[5029]: hierarchies:
Dec 14 11:55:41 n01 lxcfs[5029]:   0: fd:   8: cpuset,cpu,io,memory,hugetlb,pids,rdma,misc
Dec 14 11:55:41 n01 lxcfs[5029]: Kernel supports pidfds
Dec 14 11:55:41 n01 lxcfs[5029]: Kernel does not support swap accounting
Dec 14 11:55:41 n01 lxcfs[5029]: api_extensions:
Dec 14 11:55:41 n01 lxcfs[5029]: - cgroups
Dec 14 11:55:41 n01 lxcfs[5029]: - sys_cpu_online
Dec 14 11:55:41 n01 lxcfs[5029]: - proc_cpuinfo
Dec 14 11:55:41 n01 lxcfs[5029]: - proc_diskstats
Dec 14 11:55:41 n01 lxcfs[5029]: - proc_loadavg
Dec 14 11:55:41 n01 lxcfs[5029]: - proc_meminfo
Dec 14 11:55:41 n01 lxcfs[5029]: - proc_stat
Dec 14 11:55:41 n01 lxcfs[5029]: - proc_swaps
Dec 14 11:55:41 n01 lxcfs[5029]: - proc_uptime
Dec 14 11:55:41 n01 lxcfs[5029]: - proc_slabinfo
Dec 14 11:55:41 n01 lxcfs[5029]: - shared_pidns
Dec 14 11:55:41 n01 lxcfs[5029]: - cpuview_daemon
Dec 14 11:55:41 n01 lxcfs[5029]: - loadavg_daemon
Dec 14 11:55:41 n01 lxcfs[5029]: - pidfds
Dec 14 11:55:41 n01 lxcfs[5029]: Reloaded LXCFS
Dec 14 12:14:02 n01 systemd[1]: Stopping Incus - LXCFS daemon...
Dec 14 12:14:02 n01 lxcfs[5029]: Running destructor lxcfs_exit
Dec 14 12:14:02 n01 systemd[1]: incus-lxcfs.service: Main process exited, code=exited, status=1/FAILURE
Dec 14 12:14:02 n01 fusermount[6577]: /bin/fusermount: failed to unmount /var/lib/incus-lxcfs: Invalid argument
Dec 14 12:14:02 n01 systemd[1]: incus-lxcfs.service: Failed with result 'exit-code'.
Dec 14 12:14:02 n01 systemd[1]: Stopped Incus - LXCFS daemon.
Dec 14 12:15:19 n01 systemd[1]: Starting Incus - LXCFS daemon...
Dec 14 12:15:19 n01 systemd[1]: Started Incus - LXCFS daemon.
Dec 14 12:15:19 n01 lxcfs[8264]: Running constructor lxcfs_init to reload liblxcfs
Dec 14 12:15:19 n01 lxcfs[8264]: mount namespace: 5
Dec 14 12:15:19 n01 lxcfs[8264]: hierarchies:
Dec 14 12:15:19 n01 lxcfs[8264]:   0: fd:   6: cpuset,cpu,io,memory,hugetlb,pids,rdma,mis

● incus-lxcfs.service - Incus - LXCFS daemon
     Loaded: loaded (/etc/systemd/system/incus-lxcfs.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2023-12-23 14:34:34 UTC; 1h 25min ago
    Process: 1958309 ExecStartPre=/usr/bin/mkdir -p /var/lib/incus-lxcfs (code=exited, status=0/SUCCESS)
   Main PID: 1958310 (lxcfs)
      Tasks: 4 (limit: 76890)
     Memory: 676.0K
        CPU: 24ms
     CGroup: /system.slice/incus-lxcfs.service
             └─1958310 /opt/incus/bin/lxcfs /var/lib/incus-lxcfs

Dec 23 14:34:34 n01 lxcfs[1958310]: - proc_loadavg
Dec 23 14:34:34 n01 lxcfs[1958310]: - proc_meminfo
Dec 23 14:34:34 n01 lxcfs[1958310]: - proc_stat
Dec 23 14:34:34 n01 lxcfs[1958310]: - proc_swaps
Dec 23 14:34:34 n01 lxcfs[1958310]: - proc_uptime
Dec 23 14:34:34 n01 lxcfs[1958310]: - proc_slabinfo
Dec 23 14:34:34 n01 lxcfs[1958310]: - shared_pidns
Dec 23 14:34:34 n01 lxcfs[1958310]: - cpuview_daemon
Dec 23 14:34:34 n01 lxcfs[1958310]: - loadavg_daemon
Dec 23 14:34:34 n01 lxcfs[1958310]: - pidfds

I just noticed the error here - but this is from 9 days ago and I didn't noticed anything odd before.

Dec 14 12:14:02 n01 systemd[1]: incus-lxcfs.service: Main process exited, code=exited, status=1/FAILURE
Dec 14 12:14:02 n01 fusermount[6577]: /bin/fusermount: failed to unmount /var/lib/incus-lxcfs: Invalid argument

stgraber commented 6 months ago

Looks like something happened at 12:14:02 that caused a full restart of LXCFS, so that should have been the beginning of the issue.

What's odd is that it looks like it was a whole 20mim after the upgrade to 0.4.

stgraber commented 6 months ago

Going to close this as the next upload will disable the LXCFS reload logic on the stable package.

janmuennich commented 2 months ago

I just experienced the same on all containers after upgrading Incus from 6.0-202404040304-ubuntu22.04 to 6.0-202404162314-ubuntu22.04.

Restarting the containers mounted lxcfs correctly again.

Is there any system info that I can supply for debugging? Or should I open a new issue?

stgraber commented 2 months ago

Hmm, that's pretty odd as the systemctl reload call on Incus package updates is disabled for stable and lts-6.0 repos, so either something weird is going on that's causing a service restart when it's not supposed to, or you hit a lxcfs crash for some reason.

Can you should journalctl -u incus-lxcfs -n 300?

janmuennich commented 2 months ago

Apr 16 15:38:58 srv7 systemd[1]: Starting Incus - LXCFS daemon...
Apr 16 15:38:58 srv7 systemd[1]: Started Incus - LXCFS daemon.
Apr 16 15:38:58 srv7 lxcfs[1202329]: Starting LXCFS at /opt/incus/bin/lxcfs
Apr 16 15:38:58 srv7 lxcfs[1202329]: Running constructor lxcfs_init to reload liblxcfs
Apr 16 15:38:58 srv7 lxcfs[1202329]: mount namespace: 5
Apr 16 15:38:58 srv7 lxcfs[1202329]: hierarchies:
Apr 16 15:38:58 srv7 lxcfs[1202329]:   0: fd:   6: cpuset,cpu,io,memory,hugetlb,pids,rdma,misc
Apr 16 15:38:58 srv7 lxcfs[1202329]: Kernel supports pidfds
Apr 16 15:38:58 srv7 lxcfs[1202329]: Kernel supports swap accounting
Apr 16 15:38:58 srv7 lxcfs[1202329]: api_extensions:
Apr 16 15:38:58 srv7 lxcfs[1202329]: - cgroups
Apr 16 15:38:58 srv7 lxcfs[1202329]: - sys_cpu_online
Apr 16 15:38:58 srv7 lxcfs[1202329]: - proc_cpuinfo
Apr 16 15:38:58 srv7 lxcfs[1202329]: - proc_diskstats
Apr 16 15:38:58 srv7 lxcfs[1202329]: - proc_loadavg
Apr 16 15:38:58 srv7 lxcfs[1202329]: - proc_meminfo
Apr 16 15:38:58 srv7 lxcfs[1202329]: - proc_stat
Apr 16 15:38:58 srv7 lxcfs[1202329]: - proc_swaps
Apr 16 15:38:58 srv7 lxcfs[1202329]: - proc_uptime
Apr 16 15:38:58 srv7 lxcfs[1202329]: - proc_slabinfo
Apr 16 15:38:58 srv7 lxcfs[1202329]: - shared_pidns
Apr 16 15:38:58 srv7 lxcfs[1202329]: - cpuview_daemon
Apr 16 15:38:58 srv7 lxcfs[1202329]: - loadavg_daemon
Apr 16 15:38:58 srv7 lxcfs[1202329]: - pidfds
Apr 18 14:49:03 srv7 systemd[1]: Stopping Incus - LXCFS daemon...
Apr 18 14:49:03 srv7 lxcfs[1202329]: Running destructor lxcfs_exit
Apr 18 14:49:03 srv7 systemd[1]: incus-lxcfs.service: Main process exited, code=exited, status=1/FAILURE
Apr 18 14:49:03 srv7 fusermount[4150742]: /bin/fusermount: failed to unmount /var/lib/incus-lxcfs: Invalid argument
Apr 18 14:49:03 srv7 systemd[1]: incus-lxcfs.service: Failed with result 'exit-code'.
Apr 18 14:49:03 srv7 systemd[1]: Stopped Incus - LXCFS daemon.
Apr 18 14:49:03 srv7 systemd[1]: incus-lxcfs.service: Consumed 15.057s CPU time.
Apr 18 14:49:03 srv7 systemd[1]: Starting Incus - LXCFS daemon...
Apr 18 14:49:03 srv7 systemd[1]: Started Incus - LXCFS daemon.
Apr 18 14:49:03 srv7 lxcfs[4150747]: Starting LXCFS at /opt/incus/bin/lxcfs
Apr 18 14:49:03 srv7 lxcfs[4150747]: Running constructor lxcfs_init to reload liblxcfs
Apr 18 14:49:03 srv7 lxcfs[4150747]: mount namespace: 5
Apr 18 14:49:03 srv7 lxcfs[4150747]: hierarchies:
Apr 18 14:49:03 srv7 lxcfs[4150747]:   0: fd:   6: cpuset,cpu,io,memory,hugetlb,pids,rdma,misc
Apr 18 14:49:03 srv7 lxcfs[4150747]: Kernel supports pidfds
Apr 18 14:49:03 srv7 lxcfs[4150747]: Kernel supports swap accounting
Apr 18 14:49:03 srv7 lxcfs[4150747]: api_extensions:
Apr 18 14:49:03 srv7 lxcfs[4150747]: - cgroups
Apr 18 14:49:03 srv7 lxcfs[4150747]: - sys_cpu_online
Apr 18 14:49:03 srv7 lxcfs[4150747]: - proc_cpuinfo
Apr 18 14:49:03 srv7 lxcfs[4150747]: - proc_diskstats
Apr 18 14:49:03 srv7 lxcfs[4150747]: - proc_loadavg
Apr 18 14:49:03 srv7 lxcfs[4150747]: - proc_meminfo
Apr 18 14:49:03 srv7 lxcfs[4150747]: - proc_stat
Apr 18 14:49:03 srv7 lxcfs[4150747]: - proc_swaps
Apr 18 14:49:03 srv7 lxcfs[4150747]: - proc_uptime
Apr 18 14:49:03 srv7 lxcfs[4150747]: - proc_slabinfo
Apr 18 14:49:03 srv7 lxcfs[4150747]: - shared_pidns
Apr 18 14:49:03 srv7 lxcfs[4150747]: - cpuview_daemon
Apr 18 14:49:03 srv7 lxcfs[4150747]: - loadavg_daemon
Apr 18 14:49:03 srv7 lxcfs[4150747]: - pidfds

At Apr 16 15:38:58 we migrated the server from LXD to Incus. At 2024-04-18 14:48:40 the Incus package was upgraded according to /var/log/apt/history.log

stgraber commented 2 months ago

That's very odd as it suggests something ran systemctl stop incus-lxcfs here followed by systemctl start incus-lxcfs (or maybe restart looks that way).

I just performed the same package update on one of my production servers and lxcfs wasn't touched at all:

root@shell01:~# cat /proc/uptime
332049.18 332049.18
root@shell01:~# exit
root@abydos:~# apt update
Hit:1 https://download.ceph.com/debian-reef jammy InRelease
Get:2 https://pkgs.zabbly.com/incus/stable jammy InRelease [7,358 B]                                                
Get:3 https://pkgs.zabbly.com/kernel/stable jammy InRelease [15.4 kB]                                               
Get:4 https://pkgs.zabbly.com/incus/stable jammy/main amd64 Packages [3,404 B]                                                   
Hit:5 http://us.archive.ubuntu.com/ubuntu jammy InRelease                                                                       
Get:6 http://us.archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]                    
Get:7 http://us.archive.ubuntu.com/ubuntu jammy-security InRelease [110 kB]                              
Hit:8 http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:9 https://ppa.launchpadcontent.net/stgraber/stgraber.net/ubuntu jammy InRelease
Get:10 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,562 kB]
Get:11 http://us.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,074 kB]
Get:12 http://us.archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,346 kB]
Get:13 http://us.archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB]
Fetched 5,090 kB in 1s (3,399 kB/s)                   
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
16 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@abydos:~# apt dist-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  incus incus-base incus-client incus-ui-canonical libc-bin libc-dev-bin libc-devtools libc6 libc6-dev locales openssh-client openssh-server
  openssh-sftp-server ubuntu-advantage-tools ubuntu-pro-client ubuntu-pro-client-l10n
16 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
6 standard LTS security updates
Need to get 103 MB of archives.
After this operation, 12.3 MB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libc-devtools amd64 2.35-0ubuntu3.7 [29.0 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libc6-dev amd64 2.35-0ubuntu3.7 [2,100 kB] 
Get:3 https://pkgs.zabbly.com/incus/stable jammy/main amd64 incus amd64 1:6.0-202404162314-ubuntu22.04 [25.3 MB]
Get:4 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libc-dev-bin amd64 2.35-0ubuntu3.7 [20.3 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libc6 amd64 2.35-0ubuntu3.7 [3,235 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libc-bin amd64 2.35-0ubuntu3.7 [706 kB]
Get:7 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 openssh-sftp-server amd64 1:8.9p1-3ubuntu0.7 [38.9 kB]
Get:8 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 openssh-server amd64 1:8.9p1-3ubuntu0.7 [435 kB]
Get:9 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 openssh-client amd64 1:8.9p1-3ubuntu0.7 [906 kB]
Get:10 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 locales all 2.35-0ubuntu3.7 [4,249 kB]
Get:11 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 ubuntu-pro-client-l10n amd64 31.2.2~22.04 [20.3 kB]
Get:12 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 ubuntu-pro-client amd64 31.2.2~22.04 [197 kB]
Get:13 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 ubuntu-advantage-tools all 31.2.2~22.04 [10.8 kB]
Get:14 https://pkgs.zabbly.com/incus/stable jammy/main amd64 incus-base amd64 1:6.0-202404162314-ubuntu22.04 [57.5 MB]
Get:15 https://pkgs.zabbly.com/incus/stable jammy/main amd64 incus-client amd64 1:6.0-202404162314-ubuntu22.04 [4,980 kB]
Get:16 https://pkgs.zabbly.com/incus/stable jammy/main amd64 incus-ui-canonical amd64 1:6.0-202404162314-ubuntu22.04 [3,067 kB]
Fetched 103 MB in 2s (60.5 MB/s)               
Preconfiguring packages ...
(Reading database ... 104768 files and directories currently installed.)
Preparing to unpack .../libc-devtools_2.35-0ubuntu3.7_amd64.deb ...
Unpacking libc-devtools (2.35-0ubuntu3.7) over (2.35-0ubuntu3.6) ...
Preparing to unpack .../libc6-dev_2.35-0ubuntu3.7_amd64.deb ...
Unpacking libc6-dev:amd64 (2.35-0ubuntu3.7) over (2.35-0ubuntu3.6) ...
Preparing to unpack .../libc-dev-bin_2.35-0ubuntu3.7_amd64.deb ...
Unpacking libc-dev-bin (2.35-0ubuntu3.7) over (2.35-0ubuntu3.6) ...
Preparing to unpack .../libc6_2.35-0ubuntu3.7_amd64.deb ...
Unpacking libc6:amd64 (2.35-0ubuntu3.7) over (2.35-0ubuntu3.6) ...
Setting up libc6:amd64 (2.35-0ubuntu3.7) ...
(Reading database ... 104768 files and directories currently installed.)
Preparing to unpack .../libc-bin_2.35-0ubuntu3.7_amd64.deb ...
Unpacking libc-bin (2.35-0ubuntu3.7) over (2.35-0ubuntu3.6) ...
Setting up libc-bin (2.35-0ubuntu3.7) ...
(Reading database ... 104768 files and directories currently installed.)
Preparing to unpack .../00-openssh-sftp-server_1%3a8.9p1-3ubuntu0.7_amd64.deb ...
Unpacking openssh-sftp-server (1:8.9p1-3ubuntu0.7) over (1:8.9p1-3ubuntu0.6) ...
Preparing to unpack .../01-openssh-server_1%3a8.9p1-3ubuntu0.7_amd64.deb ...
Unpacking openssh-server (1:8.9p1-3ubuntu0.7) over (1:8.9p1-3ubuntu0.6) ...
Preparing to unpack .../02-openssh-client_1%3a8.9p1-3ubuntu0.7_amd64.deb ...
Unpacking openssh-client (1:8.9p1-3ubuntu0.7) over (1:8.9p1-3ubuntu0.6) ...
Preparing to unpack .../03-locales_2.35-0ubuntu3.7_all.deb ...
Unpacking locales (2.35-0ubuntu3.7) over (2.35-0ubuntu3.6) ...
Preparing to unpack .../04-ubuntu-pro-client-l10n_31.2.2~22.04_amd64.deb ...
Unpacking ubuntu-pro-client-l10n (31.2.2~22.04) over (31.2~22.04) ...
Preparing to unpack .../05-ubuntu-pro-client_31.2.2~22.04_amd64.deb ...
Unpacking ubuntu-pro-client (31.2.2~22.04) over (31.2~22.04) ...
Preparing to unpack .../06-ubuntu-advantage-tools_31.2.2~22.04_all.deb ...
Unpacking ubuntu-advantage-tools (31.2.2~22.04) over (31.2~22.04) ...
Preparing to unpack .../07-incus_1%3a6.0-202404162314-ubuntu22.04_amd64.deb ...
Unpacking incus (1:6.0-202404162314-ubuntu22.04) over (6.0-202404040310-ubuntu22.04) ...
Preparing to unpack .../08-incus-base_1%3a6.0-202404162314-ubuntu22.04_amd64.deb ...
Unpacking incus-base (1:6.0-202404162314-ubuntu22.04) over (6.0-202404040310-ubuntu22.04) ...
Preparing to unpack .../09-incus-client_1%3a6.0-202404162314-ubuntu22.04_amd64.deb ...
Unpacking incus-client (1:6.0-202404162314-ubuntu22.04) over (6.0-202404040310-ubuntu22.04) ...
Preparing to unpack .../10-incus-ui-canonical_1%3a6.0-202404162314-ubuntu22.04_amd64.deb ...
Unpacking incus-ui-canonical (1:6.0-202404162314-ubuntu22.04) over (6.0-202404040310-ubuntu22.04) ...
Setting up openssh-client (1:8.9p1-3ubuntu0.7) ...
Setting up locales (2.35-0ubuntu3.7) ...
Generating locales (this might take a while)...
  en_AG.UTF-8... done
  en_AU.UTF-8... done
  en_BW.UTF-8... done
  en_CA.UTF-8... done
  en_DK.UTF-8... done
  en_GB.UTF-8... done
  en_HK.UTF-8... done
  en_IE.UTF-8... done
  en_IL.UTF-8... done
  en_IN.UTF-8... done
  en_NG.UTF-8... done
  en_NZ.UTF-8... done
  en_PH.UTF-8... done
  en_SG.UTF-8... done
  en_US.UTF-8... done
  en_ZA.UTF-8... done
  en_ZM.UTF-8... done
  en_ZW.UTF-8... done
Generation complete.
Setting up incus-client (1:6.0-202404162314-ubuntu22.04) ...
Setting up libc-dev-bin (2.35-0ubuntu3.7) ...
Setting up ubuntu-pro-client (31.2.2~22.04) ...
Setting up libc-devtools (2.35-0ubuntu3.7) ...
Setting up ubuntu-pro-client-l10n (31.2.2~22.04) ...
Setting up openssh-sftp-server (1:8.9p1-3ubuntu0.7) ...
Setting up openssh-server (1:8.9p1-3ubuntu0.7) ...
rescue-ssh.target is a disabled or a static unit not running, not starting it.
ssh.socket is a disabled or a static unit not running, not starting it.
Setting up incus-base (1:6.0-202404162314-ubuntu22.04) ...
Setting up incus (1:6.0-202404162314-ubuntu22.04) ...
Setting up incus-ui-canonical (1:6.0-202404162314-ubuntu22.04) ...
Setting up libc6-dev:amd64 (2.35-0ubuntu3.7) ...
Setting up ubuntu-advantage-tools (31.2.2~22.04) ...
Processing triggers for ufw (0.36.1-4ubuntu0.1) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.7) ...
root@abydos:~# incus exec shell01 bash
root@shell01:~# cat /proc/uptime
332138.34 332138.34
root@shell01:~# exit
root@abydos:~# ps aux | grep lxcfs
root        2177  0.4  0.0 1888848 3840 ?        Ssl  Apr14  24:14 /opt/incus/bin/lxcfs /var/lib/incus-lxcfs
root     2677509  0.0  0.0   9212  2304 pts/1    S+   18:32   0:00 grep --color=auto lxcfs
root@abydos:~# journalctl -u incus-lxcfs -n 30
Apr 08 20:44:08 abydos lxcfs[1983]: - proc_uptime
Apr 08 20:44:08 abydos lxcfs[1983]: - proc_slabinfo
Apr 08 20:44:08 abydos lxcfs[1983]: - shared_pidns
Apr 08 20:44:08 abydos lxcfs[1983]: - cpuview_daemon
Apr 08 20:44:08 abydos lxcfs[1983]: - loadavg_daemon
Apr 08 20:44:08 abydos lxcfs[1983]: - pidfds
-- Boot ee4c7dcce94b43adb7928253e1caad00 --
Apr 14 22:12:17 abydos systemd[1]: Starting Incus - LXCFS daemon...
Apr 14 22:12:17 abydos systemd[1]: Started Incus - LXCFS daemon.
Apr 14 22:12:17 abydos lxcfs[2177]: Starting LXCFS at /opt/incus/bin/lxcfs
Apr 14 22:12:17 abydos lxcfs[2177]: Running constructor lxcfs_init to reload liblxcfs
Apr 14 22:12:17 abydos lxcfs[2177]: mount namespace: 5
Apr 14 22:12:17 abydos lxcfs[2177]: hierarchies:
Apr 14 22:12:17 abydos lxcfs[2177]:   0: fd:   6: cpuset,cpu,io,memory,hugetlb,pids,rdma,misc
Apr 14 22:12:17 abydos lxcfs[2177]: Kernel supports pidfds
Apr 14 22:12:17 abydos lxcfs[2177]: Kernel supports swap accounting
Apr 14 22:12:17 abydos lxcfs[2177]: api_extensions:
Apr 14 22:12:17 abydos lxcfs[2177]: - cgroups
Apr 14 22:12:17 abydos lxcfs[2177]: - sys_cpu_online
Apr 14 22:12:17 abydos lxcfs[2177]: - proc_cpuinfo
Apr 14 22:12:17 abydos lxcfs[2177]: - proc_diskstats
Apr 14 22:12:17 abydos lxcfs[2177]: - proc_loadavg
Apr 14 22:12:17 abydos lxcfs[2177]: - proc_meminfo
Apr 14 22:12:17 abydos lxcfs[2177]: - proc_stat
Apr 14 22:12:17 abydos lxcfs[2177]: - proc_swaps
Apr 14 22:12:17 abydos lxcfs[2177]: - proc_uptime
Apr 14 22:12:17 abydos lxcfs[2177]: - proc_slabinfo
Apr 14 22:12:17 abydos lxcfs[2177]: - shared_pidns
Apr 14 22:12:17 abydos lxcfs[2177]: - cpuview_daemon
Apr 14 22:12:17 abydos lxcfs[2177]: - loadavg_daemon
Apr 14 22:12:17 abydos lxcfs[2177]: - pidfds
root@abydos:~#

stgraber commented 2 months ago

I also checked the packaging scripts here to confirm that there is no call to restart or stop+start for incus-lxcfs:

root@abydos:~# grep incus-lxcfs /var/lib/dpkg/info/*
/var/lib/dpkg/info/incus-base.list:/lib/systemd/system/incus-lxcfs.service
/var/lib/dpkg/info/incus-base.md5sums:ce18e4697abd2c24c0b6f41fc4e045f8  lib/systemd/system/incus-lxcfs.service
/var/lib/dpkg/info/incus-base.postinst:    for unit in incus.service incus.socket incus-user.service incus-user.socket incus-startup.service incus-lxcfs.service; do
/var/lib/dpkg/info/incus-base.postinst:        systemctl enable incus-lxcfs.service --now
/var/lib/dpkg/info/incus-base.postinst:        # if systemctl is-active incus-lxcfs.service -q; then
/var/lib/dpkg/info/incus-base.postinst:        #     systemctl reload incus-lxcfs.service
/var/lib/dpkg/info/incus-base.postinst:        systemctl enable incus-lxcfs.service
/var/lib/dpkg/info/incus-base.postrm:        rm -Rf /var/lib/incus-lxcfs || true
/var/lib/dpkg/info/incus-base.prerm:            systemctl disable --now incus-lxcfs.service
/var/lib/dpkg/info/incus-base.prerm:            systemctl disable incus-lxcfs.service

We can see that it does a enable --now on installation (postinst) and a disable --now on full package removal, but there is no mention of any similar action happening on upgrade (which would be postinst).

You can actually see the reload line being commented out in this case, though what you're showing isn't even a reload but a full service stop...

janmuennich commented 2 months ago

Okay, got it by reproducing the upgrades. This isn't Incus' fault, sorry!

At the same time libc-bin and libc6 were upgraded which caused a recommended restart of incus-lxcfs.service and incus.service by needrestart. Normally I don't run manual package upgrades so this time just hit enter without thinking about it.

I've added $nrconf{override_rc}->{q(^incus)} = 0; now to /etc/needrestart/conf.d/incus.conf

stgraber commented 2 months ago

Ah yeah, I have a profound dislike for needrestart so have it removed from all my servers :)

https://github.com/zabbly/incus/commit/6e72685bceedea3d84806eb72a01865ceb4ef65d

Restarting incus itself is fine, at worst it will disconnect some exec sessions after hitting a timeout, but restarting incus-lxcfs is a big no-no as that kills everything.

lxc / incus

upgrading incus from 0.3 to 0.4 unmounted `/proc` in the container #330

Required information

Issue description

Steps to reproduce

Information to attach