MusicDin
commented
5 months ago Thanks. Quick link to previous discussion: https://discourse.ubuntu.com/t/ubuntu-24-04-broke-docker-inside-an-lxd-container/44499
Open WereCatf opened 5 months ago
MusicDin
commented
5 months ago Thanks. Quick link to previous discussion: https://discourse.ubuntu.com/t/ubuntu-24-04-broke-docker-inside-an-lxd-container/44499
tomponline
commented
5 months ago Most likely related to similar issue with Incus https://github.com/lxc/incus/issues/791#issuecomment-2083765381
tomponline
commented
4 months ago Required information
* Distribution: Ubuntu * Distribution version: 24.04 * The output of "snap list --all lxd core20 core22 core24 snapd":Name Version Rev Tracking Publisher Notes core22 20240408 1380 latest/stable canonical✓ base lxd 5.21.1-98dad8f 28323 5.21/stable/… canonical✓ - snapd 2.62 21465 latest/stable canonical✓ snapd* The output of "lxc info" or if that fails: * Kernel version: 6.8.0-31-generic * LXC version: 5.21.1 LTS * LXD version: 5.21.1 LTS * Storage backend in use: BtrfsIssue description
Attempting to run e.g.
docker run hello-worldinside a container fails even with security.nesting enabled. Docker spits out the following error:docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error jailing process inside rootfs: pivot_root .: permission denied: unknown. ERRO[0000] error waiting for container:Dmesg shows an AppArmor error after attempting to run Docker:
[ 108.876252] audit: type=1400 audit(1714085791.618:236): apparmor="DENIED" operation="pivotroot" class="mount" namespace="root//lxd-testi_<var-snap-lxd-common-lxd>" profile="runc" name="/var/lib/docker/overlay2/9664d056286b3d0a93188f5fd09aba3fd82a6326c6ab7c9e3197a99996078eff/merged/" pid=2580 comm="runc:[2:INIT]" srcname="/var/lib/docker/overlay2/9664d056286b3d0a93188f5fd09aba3fd82a6326c6ab7c9e3197a99996078eff/merged/"Steps to reproduce
1. Install Ubuntu 24.04 2. Set up LXD and launch a container with security.nesting=true 3. Attempt to run a Docker container inside the container
Which LXD container image were you using for this?
Yermo
commented
4 months ago @tomponline I have reproduced this on my end with Ubuntu 24.04, Ubuntu 22.04, and Ubuntu 18.04 system container images running on an Ubuntu 24.04 host.
On an Ubuntu 24.04 Desktop host, I installed Ubuntu 22.04 Server into a VirtualBox VM and repeated the experiment but got the same result.
As another test, I installed the latest non-snap version of lxc via apt-get onto an Ubuntu 24.04 host and was able to get docker running inside an Ubuntu 24.04 system container.
tomponline
commented
4 months ago Does this fix it inside the container
ln -s /etc/apparmor.d/runc /etc/apparmor.d/disable/
tomponline
commented
4 months ago Also what does
sudo cat /proc/sys/kernel/apparmor_restrict_unprivileged_userns
Show both on the host and inside the container?
Yermo
commented
4 months ago @tomponline
root@openMapTiles1:~# ln -s /etc/apparmor.d/runc /etc/apparmor.d/disable/
root@openMapTiles1:~# docker run hello-world
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "proc" to rootfs at "/proc": mount proc:/proc (via /proc/self/fd/6), flags: 0xe: permission denied: unknown.On the host:
root@mapmaker:~# sudo cat /proc/sys/kernel/apparmor_restrict_unprivileged_userns
0In the container:
root@openMapTiles1:~# sudo cat /proc/sys/kernel/apparmor_restrict_unprivileged_userns
0FWIW, I installed Debian Bookworm in a VirtualBox guest on my Ubuntu 24.04 Desktop host and installed the lxc packages through apt there and I am able to run docker hello world in a Debian Bookworm system container. lxc version 5.0.2.
tomponline
commented
4 months ago What do you see in dmesg re apparmor after that?
simondeziel
commented
4 months ago After creating the symlink to disable the runc profile, you also need to unload the profile for this to take effect:
# ln -s /etc/apparmor.d/runc /etc/apparmor.d/disable/
# apparmor_parser -R /etc/apparmor.d/runcRebooting the instance might also achieve the same.
Yermo
commented
4 months ago @tomponline In the system container I re-ran docker hello world while running sudo dmesg -w on the host. No messages were output.
I repeated the experiment on another machine running Ubuntu 22.04 Desktop with an Ubuntu 22.04 system container.
Running docker hello world before
ln -s /etc/apparmor.d/runc /etc/apparmor.d/disable/Docker reports the same error however dmesg -w yields:
[345587.192443] overlayfs: fs on '/var/lib/docker/overlay2/l/2DAZG7UGPSUNO7ZJXAN4IKKJKR' does not support file handles, falling back to xino=off.
[345592.320587] overlayfs: fs on '/var/lib/docker/overlay2/l/5UJ4HGRQGCQHWMRQHZZCSFLMEQ' does not support file handles, falling back to xino=off.
[345597.445972] overlayfs: fs on '/var/lib/docker/overlay2/l/5UJ4HGRQGCQHWMRQHZZCSFLMEQ' does not support file handles, falling back to xino=off.
[345597.457655] docker0: port 1(vethd264495) entered blocking state
[345597.457661] docker0: port 1(vethd264495) entered disabled state
[345597.457675] vethd264495: entered allmulticast mode
[345597.457736] vethd264495: entered promiscuous mode
[345597.737544] audit: type=1400 audit(1715198413.326:236): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="lxd-testContainer_</var/snap/lxd/common/lxd>" name="/var/lib/docker/overlay2/192f7cdf370b0c1e16733b8276c51a292c414f85832286038e07c7e938013005/merged/proc/" pid=873797 comm="runc:[2:INIT]" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
[345597.842745] docker0: port 1(vethd264495) entered disabled state
[345597.844227] vethd264495 (unregistering): left allmulticast mode
[345597.844234] vethd264495 (unregistering): left promiscuous mode
[345597.844237] docker0: port 1(vethd264495) entered disabled stateAfter setting the symlink and restarting the container no messages are output in dmesg -w however docker generates the same error as before.
WereCatf
commented
4 months ago [345597.737544] audit: type=1400 audit(1715198413.326:236): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="lxd-testContainer_</var/snap/lxd/common/lxd>" name="/var/lib/docker/overlay2/192f7cdf370b0c1e16733b8276c51a292c414f85832286038e07c7e938013005/merged/proc/" pid=873797 comm="runc:[2:INIT]" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"After setting the symlink and restarting the container no messages are output in dmesg -w however docker generates the same error as before.
You say no messages are output in dmesg, but there is actually a differently worded error in your log: Apparmor denied mount-operation from runc.
Yermo
commented
4 months ago @WereCatf Correct. The messages are generated before the symlink is added and the container restarted. After that no more messages are generated.
mihalicyn
commented
4 months ago Hi @WereCatf
Please can you show lxc config show <your_ct_name> -e?
WereCatf
commented
4 months ago Hi @WereCatf
Please can you show
lxc config show <your_ct_name> -e?
I'm not sure why you want that as I stated the steps for recreating the issue, but here are two different containers with Ubuntu 23.10 and 24.04 on a 24.04 host:
nita@ubuvm:~$ lxc config show test -e
architecture: x86_64
config:
image.architecture: amd64
image.description: ubuntu 23.10 amd64 (release) (20240514)
image.label: release
image.os: ubuntu
image.release: mantic
image.serial: "20240514"
image.type: squashfs
image.version: "23.10"
security.nesting: "true"
volatile.base_image: 7843f1211e7b93e7254197bff3d69d13950a46f3352be98aacbebc52b5665c4b
volatile.cloud-init.instance-id: 774764b2-2530-41e9-b54c-482d4e09d913
volatile.eth0.host_name: veth853c1bcb
volatile.eth0.hwaddr: 00:16:3e:7f:b7:f8
volatile.idmap.base: "0"
volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.last_state.idmap: '[]'
volatile.last_state.power: RUNNING
volatile.uuid: 5ed1ce54-03c2-4b91-a44a-43d34d6eba0a
volatile.uuid.generation: 5ed1ce54-03c2-4b91-a44a-43d34d6eba0a
devices:
eth0:
name: eth0
network: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""nita@ubuvm:~$ lxc config show test2404 -e
architecture: x86_64
config:
image.architecture: amd64
image.description: ubuntu 24.04 LTS amd64 (release) (20240423)
image.label: release
image.os: ubuntu
image.release: noble
image.serial: "20240423"
image.type: squashfs
image.version: "24.04"
security.nesting: "true"
volatile.base_image: c9fba5728bfe168aa73084b94deab3dd3a1e349b5f7e0b5e5a8e945899cb0378
volatile.cloud-init.instance-id: bb8fa7ea-20e1-4a7d-a9a0-46ac94f7b098
volatile.eth0.host_name: veth2d8f10fd
volatile.eth0.hwaddr: 00:16:3e:f8:02:d4
volatile.idmap.base: "0"
volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.last_state.idmap: '[]'
volatile.last_state.power: RUNNING
volatile.uuid: 462d38ed-cd29-4072-be68-d814ece56ad4
volatile.uuid.generation: 462d38ed-cd29-4072-be68-d814ece56ad4
devices:
eth0:
name: eth0
network: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""Hi @WereCatf !
Thanks a lot for providing extra info :)
I'm not sure why you want that as I stated the steps for recreating the issue
I'm asking because I wasn't able to reproduce this issue on my local environment.
Please, can you show docker version output from your container?
What I have:
root@ubuntu:/home/ubuntu# uname -a
Linux ubuntu 6.8.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 00:40:06 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
root@ubuntu:/home/ubuntu# cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
root@ubuntu:/home/ubuntu# snap list
Name Version Rev Tracking Publisher Notes
core22 20240408 1380 latest/stable canonical✓ base
lxd 5.21.1-98dad8f 28323 - canonical✓ -
snapd 2.62 21465 latest/stable canonical✓ snapd
root@ubuntu:/home/ubuntu# lxc launch ubuntu:n noblect
Creating noblect
Starting noblect
root@ubuntu:/home/ubuntu# lxc config set noblect security.nesting=true
root@ubuntu:/home/ubuntu# lxc restart noblect
root@ubuntu:/home/ubuntu# lxc config show noblect -e
architecture: x86_64
config:
image.architecture: amd64
image.description: ubuntu 24.04 LTS amd64 (release) (20240423)
image.label: release
image.os: ubuntu
image.release: noble
image.serial: "20240423"
image.type: squashfs
image.version: "24.04"
security.nesting: "true"
volatile.base_image: c9fba5728bfe168aa73084b94deab3dd3a1e349b5f7e0b5e5a8e945899cb0378
volatile.cloud-init.instance-id: 8bcc706f-f233-4759-83eb-83acbd03c789
volatile.eth0.host_name: vethf9b87fae
volatile.eth0.hwaddr: 00:16:3e:50:b0:00
volatile.idmap.base: "0"
volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.last_state.idmap: '[]'
volatile.last_state.power: RUNNING
volatile.last_state.ready: "false"
volatile.uuid: f948b0ec-91fb-4e2c-9a98-40642d185b9d
volatile.uuid.generation: f948b0ec-91fb-4e2c-9a98-40642d185b9d
devices:
eth0:
name: eth0
network: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""
root@ubuntu:/home/ubuntu# lxc shell noblect
root@noblect:~# curl -fsSL https://get.docker.com -o get-docker.sh
root@noblect:~# sh get-docker.sh
# Executing docker install script, commit: 6d9743e9656cc56f699a64800b098d5ea5a60020
+ sh -c apt-get update -qq >/dev/null
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
Scanning processes...
Scanning candidates...
+ sh -c install -m 0755 -d /etc/apt/keyrings
+ sh -c curl -fsSL "https://download.docker.com/linux/ubuntu/gpg" -o /etc/apt/keyrings/docker.asc
+ sh -c chmod a+r /etc/apt/keyrings/docker.asc
+ sh -c echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu noble stable" > /etc/apt/sources.list.d/docker.list
+ sh -c apt-get update -qq >/dev/null
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq docker-ce docker-ce-cli containerd.io docker-compose-plugin docker-ce-rootless-extras docker-buildx-plugin >/dev/null
Scanning processes...
+ sh -c docker version
Client: Docker Engine - Community
Version: 26.1.3
API version: 1.45
Go version: go1.21.10
Git commit: b72abbb
Built: Thu May 16 08:33:35 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 26.1.3
API version: 1.45 (minimum version 1.24)
Go version: go1.21.10
Git commit: 8e96db1
Built: Thu May 16 08:33:35 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.31
GitCommit: e377cd56a71523140ca6ae87e30244719194a521
runc:
Version: 1.1.12
GitCommit: v1.1.12-0-g51d5e94
docker-init:
Version: 0.19.0
GitCommit: de40ad0
================================================================================
To run Docker as a non-privileged user, consider setting up the
Docker daemon in rootless mode for your user:
dockerd-rootless-setuptool.sh install
Visit https://docs.docker.com/go/rootless/ to learn about rootless mode.
To run the Docker daemon as a fully privileged service, but granting non-root
users access, refer to https://docs.docker.com/go/daemon-access/
WARNING: Access to the remote API on a privileged Docker daemon is equivalent
to root access on the host. Refer to the 'Docker daemon attack surface'
documentation for details: https://docs.docker.com/go/attack-surface/
================================================================================
root@noblect:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c1ec31eb5944: Pull complete
Digest: sha256:266b191e926f65542fa8daaec01a192c4d292bff79426f47300a046e1bc576fd
Status: Downloaded newer image for hello-world:latest
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/@mihalicyn It never occurred to me to use Docker from their servers as I'm used to using Docker from Ubuntu's own repos -- never had a problem before this, so never had any good reason to seek upstream builds instead. Your reply made me realize that it's the builds from Ubuntu's own repos that break.
mihalicyn
commented
4 months ago @mihalicyn It never occurred to me to use Docker from their servers as I'm used to using Docker from Ubuntu's own repos -- never had a problem before this, so never had any good reason to seek upstream builds instead. Your reply made me realize that it's the builds from Ubuntu's own repos that break.
But it still makes sense to reproduce your problem. Because if any changes in the LXD/Linux kernel/AppArmor/Ubuntu on the host breaks anything inside the containers we must fix it. Because host must not affect what you have inside the container.
So, please tell me how precisely you are installing docker inside the container so I can reproduce it on my environment, then compare it with jammy and detect which component is broken.
As a temporary workaround I can propose you to do:
lxc config set <your ct> raw.lxc="lxc.apparmor.profile = unconfined"
this will disable AppArmor for one specific container which is not a big deal with unprivileged containers. But we still want to find the problem and fix it.
WereCatf
commented
4 months ago But it still makes sense to reproduce your problem.
I wasn't disagreeing with you, I was just saying I realized why you weren't able to reproduce the issue. apt update && apt -y install docker.io docker-compose-v2 is all I did inside a fresh container, followed by docker run hello-world.
As a temporary workaround I can propose you to do:
lxc config set <your ct> raw.lxc="lxc.apparmor.profile = unconfined"this will disable AppArmor for one specific container which is not a big deal with unprivileged containers. But we still want to find the problem and fix it.
That doesn't work, either, but results in a different error:
root@test2404:~# docker run hello-world
docker: Error response from daemon: Could not check if docker-default AppArmor profile was loaded: open /sys/kernel/security/apparmor/profiles: permission denied.
ERRO[0000] error waiting for container:I wasn't disagreeing with you, I was just saying I realized why you weren't able to reproduce the issue.
I understand that you are not disagreeing, just wanted to point that we want to fix this issue and it would be not great to force you to step to another docker version because of this bug. ;-)
apt update && apt -y install docker.io docker-compose-v2 is all I did inside a fresh container, followed by docker run hello-world.
Amazing! Now I have reproduced this.
That doesn't work, either, but results in a different error:
yep, I see. The same for me.
Ok, then I just need to investigate this more.
Thanks a lot for your patience and detailed information about the problem.
mihalicyn
commented
4 months ago Temporary workaround (please, do inside the container):
vim /etc/apparmor.d/runc
# add "pivot_root," line
systemctl reload apparmor.serviceOkay, it is clearly a Ubuntu kernel issue:
diff --git a/security/apparmor/mount.c b/security/apparmor/mount.c
index 74b7293ab971..b12e6bdfefb2 100644
--- a/security/apparmor/mount.c
+++ b/security/apparmor/mount.c
@@ -678,7 +678,7 @@ static struct aa_label *build_pivotroot(const struct cred *subj_cred,
AA_BUG(!new_path);
AA_BUG(!old_path);
- if (!RULE_MEDIATES(rules, AA_CLASS_MOUNT))
+ if (profile_unconfined(profile) || !RULE_MEDIATES(rules, AA_CLASS_MOUNT))
return aa_get_newest_label(&profile->label);
error = aa_path_name(old_path, path_flags(profile, old_path),(partial revert of https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/noble/commit/?h=master-next&id=dc757a645cfa82f6ac252365df20a36a9ff82760 )
fixed Ubuntu kernel.
Now it's a matter of time to get this fixed and new kernel released.
mihalicyn
commented
4 months ago 
softwarecreations
commented
2 months ago It seems this issue has nothing to do with docker or LXD. Rather LXC is broken in 24.04 I had 10 LXC containers running on Ubuntu 22.04. Upgraded to 23.10 had to do minor apparmour workarounds IIRC but it worked fine. Upgraded to 24.04. Now LXC containers won't start. Strangely a few people report having gotten LXC working but none of the "fixes"/"workarounds" worked for me. Incl "disabling" apparmour for a CT by attempting to use the unconfined apparmour profile that causes the pivot_root error.
lxc-start foo 20240708123456.853 ERROR conf - ../src/lxc/conf.c:lxc_pivot_root:1653 - Permission denied - Failed to turn new root mount tree into shared mount tree
lxc-start foo 20240708123456.854 ERROR conf - ../src/lxc/conf.c:lxc_setup:4486 - Failed to pivot root into rootfs
lxc-start foo 20240708123456.854 ERROR start - ../src/lxc/start.c:do_start:1272 - Failed to setup container "foo"
lxc-start foo 20240708123456.854 ERROR sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 4)
lxc-start foo 20240708123456.854 ERROR start - ../src/lxc/start.c:__lxc_start:2107 - Failed to spawn container "foo"
lxc-start foo 20240708123456.362 ERROR lxc_start - ../src/lxc/tools/lxc_start.c:main:306 - The container failed to start
lxc-start foo 20240708123456.362 ERROR lxc_start - ../src/lxc/tools/lxc_start.c:main:311 - Additional information can be obtained by setting the --logfile and --logpriority options@Yermo can you please post your working LXC 24.04 container on 24.04 host LXC CT config file and kernel version?
mihalicyn
commented
2 months ago Hi @softwarecreations
please, describe your setup. You have a bare-metal host with Ubuntu 24.04 and you run LXC (not LXD) containers on it? Or you are trying to run LXC containers inside LXD container?
softwarecreations
commented
2 months ago @mihalicyn
Last night it was not. I was running the exact same kernel on it last night 6.8.0-36-generic. I certainly rebooted it after upgrading the VM to 24.04, because I can see the 6.8.0-36-generic in last night's dmesg in the terminal scrollback. The only difference between last night and now, that I can think of, is an extra reboot.
My current status. Debian 12 bare-metal 6.1.0-18-amd64 REDACTED - for latest most accurate results scroll 2 posts down.
Only Ubuntu 24.04 CT's who's configs I add
Without lxc.apparmor.profile = unconfined will run.unconfined I still get the same error that I posted last night.
The linux-image-kvm apt package installs a KVM optimized kernel that lacks drivers for physical hardware and uses less RAM than the generic kernel.
LXC did not work on a 23.10 KVM VM with the KVM kernel, I had to change that one to a generic kernel.
I have not re-tested a KVM kernel on Ubuntu 24.04.
softwarecreations
commented
2 months ago I see the LXC containers on proxmox also can't run Ubuntu 24.04 without unconfined AppArmor profile.
https://forum.proxmox.com/threads/ubuntu-24-04-lxc-containers-fail-to-boot-after-upgrade.145848/page-2
or privileged + nesting enabled https://forum.proxmox.com/threads/ubuntu-24-04-unsupported-ubuntu-version-24-04.146454/page-2
softwarecreations
commented
2 months ago If I set the 24.04 containers to auto-start, they autostart successfully after I reboot the VM. I can manually stop the containers, but I cannot manually start them. ... so now my previous experience makes sense, that literally just rebooting made my Ubuntu 24.04 CT's run.
Ironically I chose Ubuntu for LXC because I thought it would be most reliable because Ubuntu works on LXD so I wrongly assumed they would care about LXC. I was tempted to switch everything to Debian now. But I'll hold off. For now I've reverted to 22.04 for VM2.
Ubuntu is making namespace related changes for good reasons, it's just a pity it was done so irresponsibly with respect to breaking LXC. https://ubuntu.com/blog/whats-new-in-security-for-ubuntu-24-04-lts
softwarecreations
commented
2 months ago Working: KVM VM1 new: Ubuntu 24.10 with 6.8.0-36-generic: Ubuntu 23.10
Working: KVM VM2 old: Ubuntu 22.04 with 5.15.0-1061-kvm : Ubuntu 22.04
Broken : KVM VM2 new: Ubuntu 24.04 with 6.8.0-36-generic: Ubuntu 24.04 (autostart reboot only)
Working: KVM VM3 : Ubuntu 24.10 with 6.8.0-36-generic: Debian 12 bookworm
Broken : KVM VM3 : Ubuntu 24.10 with 6.8.0-36-generic: Debian 13 trixie
Working: Baremetal : Debian 12 with 6.1.0-18-amd64: Debian 13 trixieUbuntu 24.04 LXC container only works for me with unconfined AppArmor profile when the LXC container auto-starts after host reboot, manual start is broken.
bpf-restrict-fs: BPF LSM hook not enabled in the kernel, BPF LSM not supported.
grep CONFIG_BPF_LSM /boot/config-6.8.0-36-generic shows CONFIG_BPF_LSM=y :face_with_diagonal_mouth: ... any ideas?
WereCatf
commented
1 month ago The issue is unfortunately still not fixed in Ubuntu 24.04.1 I had hoped they'd include the fix.
jay-tuckey
commented
1 month ago Are you kidding me now? This is still not fixed in the just-released Ubuntu 24.04.1, the issue is still happening!
Probably best to put a request via your support contract with Canonical if you need it fixed urgently.
tomponline
commented
1 month ago The issue is a combination of AppArmor rules and kernels in Ubuntu rather than LXD.
See https://bugs.launchpad.net/apparmor/+bug/2067900 for more details (@mihalicyn linked to this earlier in the thread).
There is also a workaround described in the first post there which may be of use.
mihalicyn
commented
1 month ago Hey @WereCatf
I can not see that AppArmor was updated in 24.04.1:
root@n:~# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.1 LTS"
root@n:~# apt update
Hit:1 http://archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:3 http://archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu noble-security InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
3 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@n:~# apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following upgrades have been deferred due to phasing:
cloud-init ubuntu-pro-client ubuntu-pro-client-l10n
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
root@n:~# apparmor_parser -V
AppArmor parser version 4.0.0~beta3 <==== OLD and buggy version
Copyright (C) 1999-2008 Novell Inc.
Copyright 2009-2018 Canonical Ltd.Sorry about that. But LXD team can not do anything with that. I did all the investigation initially and even helped AppArmor team by pointing out to the code which does not work correctly. As far as I know fix was landed to AppArmor (https://gitlab.com/apparmor/apparmor/-/commit/4bb134e4bb950a8c9a1f70a27eb2acd2a35df412) but I can't see that updated version in noble-updates track:
alex@amikhalitsyn:~$ rmadison apparmor -u ubuntu | grep noble
apparmor | 4.0.0-beta3-0ubuntu3 | noble | source, amd64, arm64, armhf, i386, ppc64el, riscv64, s390x
apparmor | 4.0.1really4.0.0-beta3-0ubuntu0.1 | noble-updates | source, amd64, arm64, armhf, i386, ppc64el, riscv64, s390x
apparmor | 4.0.1really4.0.1-0ubuntu0.24.04.3 | noble-proposed | source, amd64, arm64, armhf, i386, ppc64el, riscv64, s390xroot@n:~# dpkg -l apparmor
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=================================-============-======================================
ii apparmor 4.0.1really4.0.0-beta3-0ubuntu0.1 amd64 user-space parser utility for AppArmorAnd changelog says that update was reverted: https://changelogs.ubuntu.com/changelogs/pool/main/a/apparmor/apparmor_4.0.1really4.0.0-beta3-0ubuntu0.1/changelog
mihalicyn
commented
1 month ago @softwarecreations problem that you are describing is absolutely different. You need to create a separate issue for that.
And also, please, can you clarify what are you using (LXC or LXD)? Because if you are using LXC then you need to report your problem to LXC issues tracker https://github.com/lxc/lxc/issues
softwarecreations
commented
1 month ago Disclaimer: I have not re-tested since my last posts above, so my statements are in that context.
@mihalicyn I am not using LXD, only LXC.
The reason I added my post here is as above
Maybe I made a mistake but that was my logic.
mihalicyn
commented
1 month ago Hello @softwarecreations,
first of all, huge thanks for such a detailed report! It's definitely something for me to investigate.
Maybe I made a mistake but that was my logic.
No, your logic makes sense, but the problem is that actually LXD developers barely have more impact on Ubuntu in comparison to LXC developers. And that's not including that fact that I'm LXC developer too :-D Just to be clear, Ubuntu is an absolutely separate story and if something is going wrong with Ubuntu packages then I'll go and create an issue on Launchpad (the same as you or any external user would do). And don't think that my report will get more priority or attention than yours! Absolutely not.
Second point is that actually, LXC uses a different image server and Ubuntu images on that image server are different in comparison to what you'll get with LXD. Yes, it's still the same distro in general but with some extra changes/workarounds. So your point "It's not LXC's responsibility to fix Ubuntu." is right in general, but sometimes LXC developers actually fix Ubuntu to be run-able in LXC containers. We (LXD developers) do the same, but sometimes our fixes/workarounds can be different.
To conclude, it is better to report LXC issues to LXC tracker.
softwarecreations
commented
4 weeks ago Thanks for your very informative comment. If I re-test I'll be sure to make an issue on LXC :+1:
Required information
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error jailing process inside rootfs: pivot_root .: permission denied: unknown. ERRO[0000] error waiting for container: