search

lxc / incus

Powerful system container and virtual machine manager
https://linuxcontainers.org/incus
Apache License 2.0
2.58k stars 214 forks source link

Cannot start Incus container in arm64 VM created on top of Raspberry Pi OS after adding security.syscalls.intercept.mount: "true" #1153

Closed x32767 closed 1 month ago

x32767 commented 1 month ago

Required information

Issue description

Container in above environment cannot start after setting security.syscalls.intercept.mount: "true" to config.

Firstly I was trying to install Waydroid headlessly into an Incus container inside a libvirt VM on Raspberry Pi OS Lite. Later I encounter this issue when trying to enable mount capability for the container. More info.

Later I found even for brand new VM, this issue still occurs.

Steps to reproduce

  1. Create a libvirt VM of arm64 with above environment on Raspberry Pi OS on Raspberry Pi 400 (or Raspberry Pi OS Lite on Raspberry Pi 5 Model B 8GB, both tested). Actually I think you may also reproduce it with any kind of arm64 machine or VM, but the above are the environments I tested.
  2. Install Incus stable (6.4) from https://github.com/zabbly/incus or 6.0.1 from bookworm-backports. 
  3. test@debian:~$ incus admin init
Would you like to use clustering? (yes/no) [default=no]: 
Do you want to configure a new storage pool? (yes/no) [default=yes]: 
Name of the new storage pool [default=default]: 
Would you like to create a new local network bridge? (yes/no) [default=yes]: 
What should the new bridge be called? [default=incusbr0]: 
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
Would you like the server to be available over the network? (yes/no) [default=no]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "init" preseed to be printed? (yes/no) [default=no]: 
test@debian:~$ incus list
+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+
test@debian:~$ incus launch images:ubuntu/22.04
Launching the instance
Instance name is: square-crayfish                    
test@debian:~$ incus list
+-----------------+---------+---------------------+-----------------------------------------------+-----------+-----------+
|      NAME       |  STATE  |        IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+-----------------+---------+---------------------+-----------------------------------------------+-----------+-----------+
| square-crayfish | RUNNING | 10.27.74.228 (eth0) | fd42:c039:54ad:a040:216:3eff:feae:da39 (eth0) | CONTAINER | 0         |
+-----------------+---------+---------------------+-----------------------------------------------+-----------+-----------+
test@debian:~$ incus stop square-crayfish
test@debian:~$ incus config set square-crayfish security.
security.guestapi           security.protection.delete  
test@debian:~$ incus config set square-crayfish security.syscalls.intercept.mount=true
test@debian:~$ incus list
+-----------------+---------+------+------+-----------+-----------+
|      NAME       |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |
+-----------------+---------+------+------+-----------+-----------+
| square-crayfish | STOPPED |      |      | CONTAINER | 0         |
+-----------------+---------+------+------+-----------+-----------+
test@debian:~$ incus start square-crayfish 
test@debian:~$ incus list
+-----------------+---------+------+------+-----------+-----------+
|      NAME       |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |
+-----------------+---------+------+------+-----------+-----------+
| square-crayfish | STOPPED |      |      | CONTAINER | 0         |
+-----------------+---------+------+------+-----------+-----------+
test@debian:~$ incus start square-crayfish --console 
To detach from the console, press: <ctrl>+a q
Error: Failed running forkconsole: "container is not running: \"square-crayfish\""
                                                                              Error: stat /proc/-1: no such file or directory
test@debian:~$

Information to attach

incus monitor --pretty log when starting the container. It seems the container started and then quickly stopped.

test@debian:~$ incus monitor --pretty 
DEBUG  [2024-08-24T21:36:57-05:00] Event listener server handler started         id=e8436c89-2b61-4c76-83b8-8d4156718e04 local=/var/lib/incus/unix.socket remote=@
DEBUG  [2024-08-24T21:37:01-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=test
DEBUG  [2024-08-24T21:37:01-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/instances/square-crayfish username=test
DEBUG  [2024-08-24T21:37:01-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/events username=test
DEBUG  [2024-08-24T21:37:01-05:00] Event listener server handler started         id=1076e0a6-5a43-4ac0-b5a0-d4d627e4497d local=/var/lib/incus/unix.socket remote=@
DEBUG  [2024-08-24T21:37:01-05:00] Handling API request                          ip=@ method=PUT protocol=unix url=/1.0/instances/square-crayfish/state username=test
DEBUG  [2024-08-24T21:37:01-05:00] New operation                                 class=task description="Starting instance" operation=dd34990a-865e-4fa8-bace-b90ccc987f08 project=default
INFO   [2024-08-24T21:37:01-05:00] ID: dd34990a-865e-4fa8-bace-b90ccc987f08, Class: task, Description: Starting instance  CreatedAt="2024-08-24 21:37:01.575465124 -0500 CDT" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/square-crayfish]]" Status=Pending StatusCode=Pending UpdatedAt="2024-08-24 21:37:01.575465124 -0500 CDT"
INFO   [2024-08-24T21:37:01-05:00] Starting instance                             action=start created="2024-08-25 01:53:39.678839496 +0000 UTC" ephemeral=false instance=square-crayfish instanceType=container project=default stateful=false used="2024-08-25 02:35:20.347393563 +0000 UTC"
DEBUG  [2024-08-24T21:37:01-05:00] Started operation                             class=task description="Starting instance" operation=dd34990a-865e-4fa8-bace-b90ccc987f08 project=default
INFO   [2024-08-24T21:37:01-05:00] ID: dd34990a-865e-4fa8-bace-b90ccc987f08, Class: task, Description: Starting instance  CreatedAt="2024-08-24 21:37:01.575465124 -0500 CDT" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/square-crayfish]]" Status=Running StatusCode=Running UpdatedAt="2024-08-24 21:37:01.575465124 -0500 CDT"
DEBUG  [2024-08-24T21:37:01-05:00] Start started                                 instance=square-crayfish instanceType=container project=default stateful=false
DEBUG  [2024-08-24T21:37:01-05:00] Instance operation lock created               action=start instance=square-crayfish project=default reusable=false
DEBUG  [2024-08-24T21:37:01-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/operations/dd34990a-865e-4fa8-bace-b90ccc987f08 username=test
DEBUG  [2024-08-24T21:37:01-05:00] MountInstance started                         driver=dir instance=square-crayfish pool=default project=default
DEBUG  [2024-08-24T21:37:01-05:00] MountInstance finished                        driver=dir instance=square-crayfish pool=default project=default
DEBUG  [2024-08-24T21:37:01-05:00] Starting device                               device=eth0 instance=square-crayfish instanceType=container project=default type=nic
DEBUG  [2024-08-24T21:37:01-05:00] Starting device                               device=root instance=square-crayfish instanceType=container project=default type=disk
DEBUG  [2024-08-24T21:37:01-05:00] UpdateInstanceBackupFile started              driver=dir instance=square-crayfish pool=default project=default
DEBUG  [2024-08-24T21:37:01-05:00] UpdateInstanceBackupFile finished             driver=dir instance=square-crayfish pool=default project=default
DEBUG  [2024-08-24T21:37:01-05:00] Skipping unmount as in use                    driver=dir pool=default refCount=1 volName=square-crayfish
DEBUG  [2024-08-24T21:37:01-05:00] Handling API request                          ip=@ method=GET protocol=unix url="/internal/containers/square-crayfish/onstart?project=default" username=root
DEBUG  [2024-08-24T21:37:02-05:00] Scheduler: container square-crayfish started: re-balancing 
DEBUG  [2024-08-24T21:37:02-05:00] Connected to seccomp socket: pid=2740        
DEBUG  [2024-08-24T21:37:02-05:00] Event listener server handler stopped         listener=1076e0a6-5a43-4ac0-b5a0-d4d627e4497d local=/var/lib/incus/unix.socket remote=@
INFO   [2024-08-24T21:37:02-05:00] Started instance                              action=start created="2024-08-25 01:53:39.678839496 +0000 UTC" ephemeral=false instance=square-crayfish instanceType=container project=default stateful=false used="2024-08-25 02:35:20.347393563 +0000 UTC"
INFO   [2024-08-24T21:37:02-05:00] Action: instance-started, Source: /1.0/instances/square-crayfish, Requestor: unix/test (@) 
DEBUG  [2024-08-24T21:37:02-05:00] Instance operation lock finished              action=start err="<nil>" instance=square-crayfish project=default reusable=false
DEBUG  [2024-08-24T21:37:02-05:00] Start finished                                instance=square-crayfish instanceType=container project=default stateful=false
DEBUG  [2024-08-24T21:37:02-05:00] Success for operation                         class=task description="Starting instance" operation=dd34990a-865e-4fa8-bace-b90ccc987f08 project=default
INFO   [2024-08-24T21:37:02-05:00] ID: dd34990a-865e-4fa8-bace-b90ccc987f08, Class: task, Description: Starting instance  CreatedAt="2024-08-24 21:37:01.575465124 -0500 CDT" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/square-crayfish]]" Status=Success StatusCode=Success UpdatedAt="2024-08-24 21:37:01.575465124 -0500 CDT"
DEBUG  [2024-08-24T21:37:02-05:00] Syscall handler received fds 37(/proc/<pid>), 38(/proc/<pid>/mem), and 40([seccomp notify]) 
DEBUG  [2024-08-24T21:37:02-05:00] Send seccomp notification for id(12030653473909927087) 
DEBUG  [2024-08-24T21:37:02-05:00] Syscall handler received fds 43(/proc/<pid>), 44(/proc/<pid>/mem), and 45([seccomp notify]) 
DEBUG  [2024-08-24T21:37:02-05:00] Send seccomp notification for id(12030653473909927088) 
DEBUG  [2024-08-24T21:37:02-05:00] Syscall handler received fds 37(/proc/<pid>), 38(/proc/<pid>/mem), and 40([seccomp notify]) 
DEBUG  [2024-08-24T21:37:02-05:00] Send seccomp notification for id(12030653473909927089) 
DEBUG  [2024-08-24T21:37:02-05:00] Handling API request                          ip=@ method=GET protocol=unix url="/internal/containers/square-crayfish/onstopns?netns=%2Fproc%2F2740%2Ffd%2F4&project=default&target=stop" username=root
DEBUG  [2024-08-24T21:37:02-05:00] Instance operation lock created               action=stop instance=square-crayfish project=default reusable=false
DEBUG  [2024-08-24T21:37:02-05:00] Instance initiated stop                       action=stop instance=square-crayfish instanceType=container project=default
DEBUG  [2024-08-24T21:37:02-05:00] Stopping device                               device=eth0 instance=square-crayfish instanceType=container project=default type=nic
DEBUG  [2024-08-24T21:37:03-05:00] Handling API request                          ip=@ method=GET protocol=unix url="/internal/containers/square-crayfish/onstop?project=default&target=stop" username=root
DEBUG  [2024-08-24T21:37:03-05:00] Instance operation lock inherited for stop    action=stop instance=square-crayfish instanceType=container project=default
DEBUG  [2024-08-24T21:37:03-05:00] Instance stopped, cleaning up                 instance=square-crayfish instanceType=container project=default
DEBUG  [2024-08-24T21:37:03-05:00] Stopping device                               device=root instance=square-crayfish instanceType=container project=default type=disk
DEBUG  [2024-08-24T21:37:03-05:00] Disconnected from seccomp socket after failed receive: pid=2740, err=EOF 
DEBUG  [2024-08-24T21:37:03-05:00] UnmountInstance started                       driver=dir instance=square-crayfish pool=default project=default
DEBUG  [2024-08-24T21:37:03-05:00] UnmountInstance finished                      driver=dir instance=square-crayfish pool=default project=default
DEBUG  [2024-08-24T21:37:03-05:00] Instance operation lock finished              action=stop err="<nil>" instance=square-crayfish project=default reusable=false
INFO   [2024-08-24T21:37:03-05:00] Shut down instance                            action=stop created="2024-08-25 01:53:39.678839496 +0000 UTC" ephemeral=false instance=square-crayfish instanceType=container project=default stateful=false used="2024-08-25 02:37:02.017136701 +0000 UTC"
INFO   [2024-08-24T21:37:03-05:00] Action: instance-shutdown, Source: /1.0/instances/square-crayfish 
DEBUG  [2024-08-24T21:37:03-05:00] Scheduler: container square-crayfish stopped: re-balancing

Here is the incus monitor --pretty log if I didn't add security.syscalls.intercept.mount: "true".

test@debian:~$ incus monitor --pretty 
DEBUG  [2024-08-24T21:35:17-05:00] Event listener server handler started         id=053cb52a-969e-43ed-b339-81929f1aa9d7 local=/var/lib/incus/unix.socket remote=@
DEBUG  [2024-08-24T21:35:19-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=test
DEBUG  [2024-08-24T21:35:19-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/instances/square-crayfish username=test
DEBUG  [2024-08-24T21:35:19-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/events username=test
DEBUG  [2024-08-24T21:35:19-05:00] Event listener server handler started         id=2ebbb767-cda2-4927-afc3-914402dbc7d4 local=/var/lib/incus/unix.socket remote=@
DEBUG  [2024-08-24T21:35:19-05:00] Handling API request                          ip=@ method=PUT protocol=unix url=/1.0/instances/square-crayfish/state username=test
DEBUG  [2024-08-24T21:35:19-05:00] New operation                                 class=task description="Starting instance" operation=a1873f7f-03d4-4331-bfbc-b04cee350e93 project=default
INFO   [2024-08-24T21:35:19-05:00] ID: a1873f7f-03d4-4331-bfbc-b04cee350e93, Class: task, Description: Starting instance  CreatedAt="2024-08-24 21:35:19.934425656 -0500 CDT" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/square-crayfish]]" Status=Pending StatusCode=Pending UpdatedAt="2024-08-24 21:35:19.934425656 -0500 CDT"
INFO   [2024-08-24T21:35:19-05:00] ID: a1873f7f-03d4-4331-bfbc-b04cee350e93, Class: task, Description: Starting instance  CreatedAt="2024-08-24 21:35:19.934425656 -0500 CDT" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/square-crayfish]]" Status=Running StatusCode=Running UpdatedAt="2024-08-24 21:35:19.934425656 -0500 CDT"
DEBUG  [2024-08-24T21:35:19-05:00] Started operation                             class=task description="Starting instance" operation=a1873f7f-03d4-4331-bfbc-b04cee350e93 project=default
DEBUG  [2024-08-24T21:35:19-05:00] Start started                                 instance=square-crayfish instanceType=container project=default stateful=false
INFO   [2024-08-24T21:35:19-05:00] Starting instance                             action=start created="2024-08-25 01:53:39.678839496 +0000 UTC" ephemeral=false instance=square-crayfish instanceType=container project=default stateful=false used="2024-08-25 02:34:11.233590415 +0000 UTC"
DEBUG  [2024-08-24T21:35:19-05:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/operations/a1873f7f-03d4-4331-bfbc-b04cee350e93 username=test
DEBUG  [2024-08-24T21:35:19-05:00] Instance operation lock created               action=start instance=square-crayfish project=default reusable=false
DEBUG  [2024-08-24T21:35:19-05:00] MountInstance started                         driver=dir instance=square-crayfish pool=default project=default
DEBUG  [2024-08-24T21:35:19-05:00] MountInstance finished                        driver=dir instance=square-crayfish pool=default project=default
DEBUG  [2024-08-24T21:35:19-05:00] Starting device                               device=eth0 instance=square-crayfish instanceType=container project=default type=nic
DEBUG  [2024-08-24T21:35:20-05:00] Starting device                               device=root instance=square-crayfish instanceType=container project=default type=disk
DEBUG  [2024-08-24T21:35:20-05:00] UpdateInstanceBackupFile started              driver=dir instance=square-crayfish pool=default project=default
DEBUG  [2024-08-24T21:35:20-05:00] UpdateInstanceBackupFile finished             driver=dir instance=square-crayfish pool=default project=default
DEBUG  [2024-08-24T21:35:20-05:00] Skipping unmount as in use                    driver=dir pool=default refCount=1 volName=square-crayfish
DEBUG  [2024-08-24T21:35:20-05:00] Handling API request                          ip=@ method=GET protocol=unix url="/internal/containers/square-crayfish/onstart?project=default" username=root
DEBUG  [2024-08-24T21:35:20-05:00] Scheduler: container square-crayfish started: re-balancing 
INFO   [2024-08-24T21:35:20-05:00] ID: a1873f7f-03d4-4331-bfbc-b04cee350e93, Class: task, Description: Starting instance  CreatedAt="2024-08-24 21:35:19.934425656 -0500 CDT" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/square-crayfish]]" Status=Success StatusCode=Success UpdatedAt="2024-08-24 21:35:19.934425656 -0500 CDT"
INFO   [2024-08-24T21:35:20-05:00] Started instance                              action=start created="2024-08-25 01:53:39.678839496 +0000 UTC" ephemeral=false instance=square-crayfish instanceType=container project=default stateful=false used="2024-08-25 02:34:11.233590415 +0000 UTC"
INFO   [2024-08-24T21:35:20-05:00] Action: instance-started, Source: /1.0/instances/square-crayfish, Requestor: unix/test (@) 
DEBUG  [2024-08-24T21:35:20-05:00] Instance operation lock finished              action=start err="<nil>" instance=square-crayfish project=default reusable=false
DEBUG  [2024-08-24T21:35:20-05:00] Start finished                                instance=square-crayfish instanceType=container project=default stateful=false
DEBUG  [2024-08-24T21:35:20-05:00] Success for operation                         class=task description="Starting instance" operation=a1873f7f-03d4-4331-bfbc-b04cee350e93 project=default
DEBUG  [2024-08-24T21:35:20-05:00] Event listener server handler stopped         listener=2ebbb767-cda2-4927-afc3-914402dbc7d4 local=/var/lib/incus/unix.socket remote=@
stgraber commented 1 month ago

Pretty odd.

Can you show incus console --show-log square-crayfish and incus info --show-log square-crayfish?

x32767 commented 1 month ago

Thank you for response.

test@debian:~$ incus console --show-log square-crayfish
Failed to mount tmpfs at /dev/shm: Invalid argument
Failed to mount tmpfs at /run: Invalid argument
Failed to mount tmpfs at /run/lock: Invalid argument
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...

test@debian:~$ incus info --show-log square-crayfish
Name: square-crayfish
Status: STOPPED
Type: container
Architecture: aarch64
Created: 2024/08/24 20:53 CDT
Last Used: 2024/09/01 13:46 CDT

Log:

lxc square-crayfish 20240901184641.129 WARN     idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:165 - newuidmap binary is missing
lxc square-crayfish 20240901184641.129 WARN     idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:171 - newgidmap binary is missing
lxc square-crayfish 20240901184641.131 WARN     idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:165 - newuidmap binary is missing
lxc square-crayfish 20240901184641.131 WARN     idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:171 - newgidmap binary is missing

I then tried to fix it with apt install rootlesskit, but after that, it still cannot start.

test@debian:~$ incus start square-crayfish
test@debian:~$ incus list 
+-----------------+---------+------+------+-----------+-----------+
|      NAME       |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |
+-----------------+---------+------+------+-----------+-----------+
| square-crayfish | STOPPED |      |      | CONTAINER | 0         |
+-----------------+---------+------+------+-----------+-----------+
test@debian:~$ incus info --show-log square-crayfish
Name: square-crayfish
Status: STOPPED
Type: container
Architecture: aarch64
Created: 2024/08/24 20:53 CDT
Last Used: 2024/09/01 13:50 CDT

Log:

test@debian:~$ incus console --show-log square-crayfish
Failed to mount tmpfs at /dev/shm: Invalid argument
Failed to mount tmpfs at /run: Invalid argument
Failed to mount tmpfs at /run/lock: Invalid argument
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...
stgraber commented 1 month ago

Interesting, I'll try it again on aarch64 here. I wonder if that's related to the new mount API somehow.

stgraber commented 1 month ago

Reproduced the issue here

stgraber commented 1 month ago

Sent a fix for this