AceSlash commented 4 years ago

Required information

Distribution: Ubuntu
Distribution version: Ubuntu 18.04

The output of "lxc info" or if that fails:

config:
core.https_address: '[::]:8443'
core.trust_password: true
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
addresses:
- 10.24.98.217:8443
architectures:
- x86_64
- i686
certificate: |
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
certificate_fingerprint: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
driver: lxc
driver_version: 3.2.1
firewall: xtables
kernel: Linux
kernel_architecture: x86_64
kernel_features:
netnsid_getifaddrs: "false"
seccomp_listener: "false"
seccomp_listener_continue: "false"
shiftfs: "false"
uevent_injection: "false"
unpriv_fscaps: "true"
kernel_version: 4.15.0-70-generic
lxc_features:
cgroup2: "false"
mount_injection_file: "true"
network_gateway_device_route: "true"
network_ipvlan: "true"
network_l2proxy: "true"
network_phys_macvlan_mtu: "true"
network_veth_router: "true"
seccomp_notify: "true"
project: default
server: lxd
server_clustered: false
server_name: srv-217
server_pid: 11849
server_version: "3.22"
storage: zfs
storage_version: 0.7.5-1ubuntu16.6

Issue description

I already noticed during the last update of LXD with snap from 3.20 to 3.21 that /proc/stat wasn't available anymore for some containers and I rebooted them. But this time for the 3.21 to 3.22 update, the issue impacts a large majority of my containers (hard to evaluate with so many alerts but I'll say 70% of containers are affected this time):

# ll /proc |grep "?????"
ls: cannot access '/proc/stat': Transport endpoint is not connected
ls: cannot access '/proc/swaps': Transport endpoint is not connected
ls: cannot access '/proc/uptime': Transport endpoint is not connected
ls: cannot access '/proc/cpuinfo': Transport endpoint is not connected
ls: cannot access '/proc/loadavg': Transport endpoint is not connected
ls: cannot access '/proc/meminfo': Transport endpoint is not connected
ls: cannot access '/proc/diskstats': Transport endpoint is not connected
-?????????  ? ?               ?                  ?            ? cpuinfo
-?????????  ? ?               ?                  ?            ? diskstats
-?????????  ? ?               ?                  ?            ? loadavg
-?????????  ? ?               ?                  ?            ? meminfo
-?????????  ? ?               ?                  ?            ? stat
-?????????  ? ?               ?                  ?            ? swaps
-?????????  ? ?               ?                  ?            ? uptime

Restarting the containers does solve it but on some hosts, the containers then get the stats from the host. Meaning that they see the uptime of the host, the /proc/stat from the host (with all the cpu, memory, even if they have limites of their own).

This container was just stopped, then started:

# uptime 
 08:58:00 up 44 days, 22:23,  1 user,  load average: 2.47, 1.92, 1.70

When I see this last issue, dmesg shows (the container is not privileged):

[Fri Mar 13 08:37:26 2020] audit: type=1400 audit(1584085095.597:677): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-container-name_</var/snap/lxd/common/lxd>" pid=70044 
comm="apparmor_parser"

Not sure if that's relevant.

Steps to reproduce

Install LXD from snap
Wait for a LXD snap automatic update
Your containers will lose the access to /proc/stat, etc

Information to attach

I don't see anything unusual in the logs except the confirmation that this issue happened just after a lxd snap refresh.

On a side note, this time I will definitively stop the lxd automatic refresh for LXD. With an issue like this, I have lost all the monitoring of the containers since it uses /proc/stat inside of them.

I don't want to appear ungrateful for the LXD project which I like very much but this is a very serious issue. We can't use automatic update if they will randomly break parts of the system.

Stability is paramont for something as critical as LXD, I'm not sure if the "stable" release is stable enough to be used with automatic update. I'll gladly use an LTS snap channel with only critical updates.

Having to reboot hundreds of containers to fix this should not even be a thing.

stgraber commented 4 years ago

This actually isn't a LXD or even a snapd issue but a LXCFS reload bug which we've fixed just a few minutes ago and are actively rolling out a fix for now.

Note that to stay on specific version of the snap, we've had versioned tracks for a while now. It wouldn't have helped in this case but if your worry is around automatic jumps in LXD version, it does allow preventing that.

snap refresh lxd --channel=3.22

stgraber commented 4 years ago

https://discuss.linuxcontainers.org/t/mount-fails-when-starting-container-zfs-filesystem-already-mounted-lxd-3-22/7080/2 for some more details on this.

AceSlash commented 4 years ago

Ok thank you for the details.

canonical / lxd

/proc mountpoint issues after lxd snap automatic refresh #7016

Required information

Issue description

Steps to reproduce

Information to attach