search

lxc / incus

Powerful system container and virtual machine manager
https://linuxcontainers.org/incus
Apache License 2.0
2.74k stars 223 forks source link

Memory limits ignored for instances. #1059

Closed alex14641 closed 3 months ago

alex14641 commented 3 months ago

Required information

Issue description

When creating an instance with memory limits, the limit is ignored.

Steps to reproduce

$ incus launch images:ubuntu/22.04 limited --config limits.cpu=1 --config limits.memory=192MiB $ incus exec limited -- free -h total used free shared buff/cache available Mem: 31Gi 2.1Gi 24Gi 9.0Mi 5.1Gi 28Gi Swap: 33Gi 0B 33Gi

What I expect to see: the limited memory of the container.

What I actually see: the memory of the host.

Information to attach

Resources: Processes: 13 Disk usage: root: 567.05MiB CPU usage: CPU usage (in seconds): 1 Memory usage: Memory (current): 85.67MiB Network usage: eth0: Type: broadcast State: UP Host interface: vetha7a061dd MAC address: 00:16:3e:6d:3c:f2 MTU: 1500 Bytes received: 5.88kB Bytes sent: 4.00kB Packets received: 50 Packets sent: 39 IP addresses: inet: 10.107.140.157/24 (global) inet6: fd42:b300:757d:5791:216:3eff:fe6d:3cf2/64 (global) inet6: fe80::216:3eff:fe6d:3cf2/64 (link) lo: Type: loopback State: UP MTU: 65536 Bytes received: 0B Bytes sent: 0B Packets received: 0 Packets sent: 0 IP addresses: inet: 127.0.0.1/8 (local) inet6: ::1/128 (local)

Log:

lxc limited 20240730233249.639 WARN attach - ../src/lxc/attach.c:get_attach_context:478 - No security context received lxc limited 20240730233438.607 WARN attach - ../src/lxc/attach.c:get_attach_context:478 - No security context received lxc limited 20240730235001.559 WARN attach - ../src/lxc/attach.c:get_attach_context:478 - No security context received

DEBUG [2024-07-30T20:10:51-04:00] Handling API request ip=@ method=GET protocol=unix url=/1.0 username=adavis DEBUG [2024-07-30T20:10:51-04:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/events username=adavis DEBUG [2024-07-30T20:10:51-04:00] Event listener server handler started id=be90c4cf-6bf6-4d89-ae93-9285b88ec3d5 local=/spare/incus-6.0/data/unix.socket remote=@ DEBUG [2024-07-30T20:10:51-04:00] Handling API request ip=@ method=POST protocol=unix url=/1.0/instances/limited/exec username=adavis DEBUG [2024-07-30T20:10:51-04:00] Started operation class=websocket description="Executing command" operation=9973b89c-eb37-47f2-a366-6107b1155dda project=default DEBUG [2024-07-30T20:10:51-04:00] New operation class=websocket description="Executing command" operation=9973b89c-eb37-47f2-a366-6107b1155dda project=default INFO [2024-07-30T20:10:51-04:00] ID: 9973b89c-eb37-47f2-a366-6107b1155dda, Class: websocket, Description: Executing command CreatedAt="2024-07-30 20:10:51.264352731 -0400 EDT" Err= Location=none MayCancel=false Metadata="map[command:[free -h] environment:map[HOME:/root LANG:C.UTF-8 PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM:xterm-256color USER:root] fds:map[0:a72a1819ae4b994178c9996cea67caa925c5bb3ea3515defc796825cda0cd329 control:13992b728b32f8d7601c1c9d8579811d3c337482e408c87592e9f1dfd0cc45f2] interactive:true]" Resources="map[instances:[/1.0/instances/limited]]" Status=Pending StatusCode=Pending UpdatedAt="2024-07-30 20:10:51.264352731 -0400 EDT" INFO [2024-07-30T20:10:51-04:00] ID: 9973b89c-eb37-47f2-a366-6107b1155dda, Class: websocket, Description: Executing command CreatedAt="2024-07-30 20:10:51.264352731 -0400 EDT" Err= Location=none MayCancel=false Metadata="map[command:[free -h] environment:map[HOME:/root LANG:C.UTF-8 PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM:xterm-256color USER:root] fds:map[0:a72a1819ae4b994178c9996cea67caa925c5bb3ea3515defc796825cda0cd329 control:13992b728b32f8d7601c1c9d8579811d3c337482e408c87592e9f1dfd0cc45f2] interactive:true]" Resources="map[instances:[/1.0/instances/limited]]" Status=Running StatusCode=Running UpdatedAt="2024-07-30 20:10:51.264352731 -0400 EDT" DEBUG [2024-07-30T20:10:51-04:00] Waiting for exec websockets to connect
DEBUG [2024-07-30T20:10:51-04:00] Handling API request ip=@ method=GET protocol=unix url="/1.0/operations/9973b89c-eb37-47f2-a366-6107b1155dda/websocket?secret=13992b728b32f8d7601c1c9d8579811d3c337482e408c87592e9f1dfd0cc45f2" username=adavis DEBUG [2024-07-30T20:10:51-04:00] Connected to operation class=websocket description="Executing command" operation=9973b89c-eb37-47f2-a366-6107b1155dda project=default DEBUG [2024-07-30T20:10:51-04:00] Connecting to operation class=websocket description="Executing command" operation=9973b89c-eb37-47f2-a366-6107b1155dda project=default DEBUG [2024-07-30T20:10:51-04:00] Connecting to operation class=websocket description="Executing command" operation=9973b89c-eb37-47f2-a366-6107b1155dda project=default DEBUG [2024-07-30T20:10:51-04:00] Handling API request ip=@ method=GET protocol=unix url="/1.0/operations/9973b89c-eb37-47f2-a366-6107b1155dda/websocket?secret=a72a1819ae4b994178c9996cea67caa925c5bb3ea3515defc796825cda0cd329" username=adavis DEBUG [2024-07-30T20:10:51-04:00] Connected to operation class=websocket description="Executing command" operation=9973b89c-eb37-47f2-a366-6107b1155dda project=default DEBUG [2024-07-30T20:10:51-04:00] Handling API request ip=@ method=GET protocol=unix url=/1.0/operations/9973b89c-eb37-47f2-a366-6107b1155dda username=adavis INFO [2024-07-30T20:10:51-04:00] Action: instance-exec, Source: /1.0/instances/limited command="[free -h]" DEBUG [2024-07-30T20:10:51-04:00] Retrieved PID of executing child process attachedPid=25058 instance=limited instanceType=container project=default DEBUG [2024-07-30T20:10:51-04:00] Instance process started PID=25058 instance=limited interactive=true project=default DEBUG [2024-07-30T20:10:51-04:00] Exec mirror websocket started PID=25058 instance=limited interactive=true number=0 project=default DEBUG [2024-07-30T20:10:51-04:00] Exec control handler started PID=25058 instance=limited interactive=true project=default DEBUG [2024-07-30T20:10:51-04:00] Websocket: Started write mirror address=@ DEBUG [2024-07-30T20:10:51-04:00] Websocket: Started read mirror address=@ DEBUG [2024-07-30T20:10:51-04:00] Websocket: Stopped read mirror address=@ err="" DEBUG [2024-07-30T20:10:51-04:00] Instance process stopped PID=25058 err="" exitStatus=0 instance=limited interactive=true project=default DEBUG [2024-07-30T20:10:51-04:00] Exec control handler finished PID=25058 instance=limited interactive=true project=default DEBUG [2024-07-30T20:10:51-04:00] Websocket: Sending barrier message address=@ DEBUG [2024-07-30T20:10:51-04:00] Websocket: Stopped write mirror address=@ err="websocket: close 1006 (abnormal closure): unexpected EOF" DEBUG [2024-07-30T20:10:51-04:00] Exec mirror websocket finished PID=25058 instance=limited interactive=true number=0 project=default readErr="" writeErr="websocket: close 1006 (abnormal closure): unexpected EOF" DEBUG [2024-07-30T20:10:51-04:00] Updated metadata for operation class=websocket description="Executing command" operation=9973b89c-eb37-47f2-a366-6107b1155dda project=default DEBUG [2024-07-30T20:10:51-04:00] Success for operation class=websocket description="Executing command" operation=9973b89c-eb37-47f2-a366-6107b1155dda project=default INFO [2024-07-30T20:10:51-04:00] ID: 9973b89c-eb37-47f2-a366-6107b1155dda, Class: websocket, Description: Executing command CreatedAt="2024-07-30 20:10:51.264352731 -0400 EDT" Err= Location=none MayCancel=false Metadata="map[command:[free -h] environment:map[HOME:/root LANG:C.UTF-8 PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM:xterm-256color USER:root] fds:map[0:a72a1819ae4b994178c9996cea67caa925c5bb3ea3515defc796825cda0cd329 control:13992b728b32f8d7601c1c9d8579811d3c337482e408c87592e9f1dfd0cc45f2] interactive:true return:0]" Resources="map[instances:[/1.0/instances/limited]]" Status=Running StatusCode=Running UpdatedAt="2024-07-30 20:10:51.538228811 -0400 EDT" INFO [2024-07-30T20:10:51-04:00] ID: 9973b89c-eb37-47f2-a366-6107b1155dda, Class: websocket, Description: Executing command CreatedAt="2024-07-30 20:10:51.264352731 -0400 EDT" Err= Location=none MayCancel=false Metadata="map[command:[free -h] environment:map[HOME:/root LANG:C.UTF-8 PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM:xterm-256color USER:root] fds:map[0:a72a1819ae4b994178c9996cea67caa925c5bb3ea3515defc796825cda0cd329 control:13992b728b32f8d7601c1c9d8579811d3c337482e408c87592e9f1dfd0cc45f2] interactive:true return:0]" Resources="map[instances:[/1.0/instances/limited]]" Status=Success StatusCode=Success UpdatedAt="2024-07-30 20:10:51.538228811 -0400 EDT" DEBUG [2024-07-30T20:10:51-04:00] Event listener server handler stopped listener=be90c4cf-6bf6-4d89-ae93-9285b88ec3d5 local=/spare/incus-6.0/data/unix.socket remote=@

stgraber commented 3 months ago

Some Linux distributions have been switching away from using /proc/meminfo to instead using the sysinfo system call to get the memory usage.

This doesn't mean the limit isn't effective, it just means some tools won't report it.

You can set security.syscalls.intercept.sysinfo=true on the container to have Incus set up system call interception for sysinfo.

alex14641 commented 3 months ago

That did not work: incus config set limited security.syscalls.intercept.sysinfo=true -p incus stop --all incus start limited incus exec limited -- free -h total used free shared buff/cache available Mem: 31Gi 3.1Gi 14Gi 14Mi 14Gi 27Gi Swap: 33Gi 0.0Ki 33Gi

incus config show limited architecture: x86_64 config: image.architecture: amd64 image.description: Ubuntu jammy amd64 (20240730_07:42) image.os: Ubuntu image.release: jammy image.serial: "20240730_07:42" image.type: squashfs image.variant: default limits.cpu: "1" limits.memory: 128MiB security.syscalls.intercept.sysinfo: "true" volatile.base_image: 8d2363d8bffb11d95f1c766879e283fe50f94d47a2834dbc725133d5c1817442 volatile.cloud-init.instance-id: ff74ee32-aa78-4ade-b7ad-95c17221931a volatile.eth0.host_name: vethad8735fe volatile.eth0.hwaddr: 00:16:3e:6a:db:a5 volatile.idmap.base: "0" volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]' volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]' volatile.last_state.idmap: '[]' volatile.last_state.power: RUNNING volatile.last_state.ready: "false" volatile.uuid: b739c273-3e9e-49ee-a8ca-66d9028313d7 volatile.uuid.generation: b739c273-3e9e-49ee-a8ca-66d9028313d7 devices: {} ephemeral: false profiles: