rosbeef commented 2 years ago

Required information

Distribution: Description: Debian GNU/Linux 11 (bullseye)
More details Linux gre 5.4.199-odroidxu4 lxc/lxd#22.05.3 SMP PREEMPT Wed Jun 22 07:29:40 UTC 2022 armv7l

The output of "lxc info" or if that fails:

/snap/bin/lxc info
config:
core.https_address: 192.168.0.220:8443
core.trust_password: true
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_Release: 11
Codename:   bullseyehandling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- instances_nic_host_name
- image_copy_profile
- container_syscall_intercept_sysinfo
- clustering_evacuation_mode
- resources_pci_vpd
- qemu_raw_conf
- storage_cephfs_fscache
- network_load_balancer
- vsock_api
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
addresses:
- 192.168.0.220:8443
architectures:
- armv7l
certificate: |
-----BEGIN CERTIFICATE-----
MIIB+zCCAYGgAwIBAgIRAPEb677D8ktTOR+FKWyT6IkwCgYIKoZIzj0EAwMwMTEc
...
o+v2Ap/P/pIp3/EFLauErVYDFkLgssfapBscjow3ug==
-----END CERTIFICATE-----
certificate_fingerprint: 50f66adbac44369300aa0c06654011ed860fd3226e6f2326d9ae656ee4802c97
driver: lxc
driver_version: 5.0.1
firewall: nftables
kernel: Linux
kernel_architecture: armv7l
kernel_features:
idmapped_mounts: "false"
netnsid_getifaddrs: "true"
seccomp_listener: "true"
seccomp_listener_continue: "false"
shiftfs: "false"
uevent_injection: "true"
unpriv_fscaps: "true"
kernel_version: 5.4.199-odroidxu4
lxc_features:
cgroup2: "true"
core_scheduling: "true"
devpts_fd: "true"
idmapped_mounts_v2: "true"
mount_injection_file: "true"
network_gateway_device_route: "true"
network_ipvlan: "true"
network_l2proxy: "true"
network_phys_macvlan_mtu: "true"
network_veth_router: "true"
pidfd: "true"
seccomp_allow_deny_syntax: "true"
seccomp_notify: "true"
seccomp_proxy_send_notify_fd: "true"
os_name: Debian GNU/Linux
os_version: "11"
project: default
server: lxd
server_clustered: false
server_event_mode: full-mesh
server_name: gre
server_pid: 1800
server_version: "5.4"
storage: btrfs
storage_version: 5.4.1
storage_supported_drivers:
- name: btrfs
version: 5.4.1
remote: false
- name: cephfs
version: 15.2.16
remote: true
- name: dir
version: "1"
remote: false
- name: lvm
version: 2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.41.0
remote: false
- name: ceph
version: 15.2.16
remote: true

Issue description

no swap inside my container

/snap/bin/lxc exec yunh1 -- free -m
              total        used        free      shared  buff/cache   available
Mem:           1990          13        1969           7           7        1977
Swap:             0           0           0

Steps to reproduce

create container the free -hin the container show no swap available

Information to attach

[x] Any relevant kernel output (dmesg) Kernel command line: console=ttySAC2,115200n8 console=tty1 consoleblank=0 loglevel=1 root=UUID=50536e32-d033-4595-bfcd-113e6053042e rootfstype=btrfs rootwait rw smsc95xx.macaddr=00:xxx55 governor=performance hdmi_tx_amp_lvl=31 hdmi_tx_lvl_ch0=3 hdmi_tx_lvl_ch1=3 hdmi_tx_lvl_ch2=3 hdmi_tx_emp_lvl=6 hdmi_clk_amp_lvl=31 hdmi_tx_res=0 HPD=true vout=hdmi usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u swapaccount=1 cgroup_enable=memory cgroup_memory=1

[x] Container log (lxc info NAME --show-log)


/snap/bin/lxc info yunh1 --show-log
Name: yunh1
Status: RUNNING
Type: container
Architecture: armv7l
PID: 3597
Created: 2022/08/15 22:13 CEST
Last Used: 2022/08/15 23:20 CEST

Resources: Processes: 6 CPU usage: CPU usage (in seconds): 16 Memory usage: Memory (current): 19.95MiB Network usage: eth0: Type: broadcast State: UP Host interface: enx00xxx6b6 MAC address: 00:xxx:2e MTU: 1500 Bytes received: 3.45kB Bytes sent: 2.31kB Packets received: 24 Packets sent: 23 IP addresses: inet: 192.168.0.153/24 (global) inet6: 2a01:xxx:7b2e/64 (global) inet6: fe80:xxx:7b2e/64 (link) lo: Type: loopback State: UP MTU: 65536 Bytes received: 0B Bytes sent: 0B Packets received: 0 Packets sent: 0 IP addresses: inet: 127.0.0.1/8 (local) inet6: ::1/128 (local)

Log:

lxc yunh1 20220815212056.933 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing lxc yunh1 20220815212056.934 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing lxc yunh1 20220815212056.936 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing lxc yunh1 20220815212056.937 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing lxc yunh1 20220815212104.254 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing lxc yunh1 20220815212104.255 WARN conf - ../src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing


 - [x] Container configuration (`lxc config show NAME --expanded`)

/snap/bin/lxc config show yunh1 --expanded architecture: armv7l config: image.architecture: armhf image.description: Debian buster armhf (20220815_05:25) image.os: Debian image.release: buster image.serial: "20220815_05:25" image.type: squashfs image.variant: default limits.memory.swap: "true" security.syscalls.intercept.sysinfo: "true" volatile.base_image: 888106799c2e7b4a3ed66b5a0a5e32e53a65bdadabaf25e03adf80f71fabb1a2 volatile.cloud-init.instance-id: 89bafb63-157e-4f67-b6b5-b1c188f3048a volatile.eth0.host_name: mace6df26bd volatile.eth0.hwaddr: 00:1xxx:2e volatile.eth0.last_state.created: "false" volatile.eth0.name: eth0 volatile.idmap.base: "0" volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]' volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]' volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]' volatile.last_state.power: RUNNING volatile.uuid: d1606124-a0ca-4479-9e6f-2491e9a5d497 devices: eth0: nictype: macvlan parent: enx001e063686b6 type: nic root: path: / pool: default type: disk ephemeral: false profiles:

default stateful: false description: ""


- [ ] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
- [ ] Output of the client with --debug
- [ ] Output of the daemon with --debug (alternatively output of `lxc monitor` while reproducing the issue)

stgraber commented 2 years ago

Bouncing to lxcfs as it's not an actual LXD issue. I also seem to recall that it's a deliberate behavior of LXCFS as there's no better way to report this currently.

Basically, LXD sets up the swap under cgroup1 and cgroup2 to be 0 bytes. That doesn't actually mean no swap, it means no ADDITIONAL memory can be taken from swap. The total container allocation itself as set through memory limits can come from either swap or from main memory.

stgraber commented 2 years ago

We tried some approaches to report that in the past, but it's very tricky. Basically we know how much memory is used from swap and we know the total amount of memory the instance can use (ram+swap). But there is no fixed bucket of ram and fixed bucket of swap, which makes it impossible to report through the normal files.

In past LXCFS versions, we'd lie and always report the host swap as the swap and then report the container usage against that. But that had two issues:

The memory would no longer be counted as used in main memory, leading to software trying to allocate more RAM just to fail because of the ram+swap limit being hit.
Any software looking at the total amount of swap would wrongly assume that the container had a lot more spare memory available (in form of swap) despite the instance potentially only having a few MBs left, again causing crashes.

As a result, in such situations, LXCFS will simply not report the swap at all. You'll typically only find accurate swap reporting when no memory limit is applied to the container as in that case we can safely expose the host values and have everything line up nicely.

stgraber commented 2 years ago

https://github.com/lxc/lxcfs#swap-handling has some more details

rosbeef commented 2 years ago

nice to view your complete a clear answer. i can close this

lxc / lxcfs

No way to gain swap inside container #551

Required information

Issue description

Steps to reproduce

Information to attach