search

MichaIng / DietPi

Lightweight justice for your single-board computer!
https://dietpi.com/
GNU General Public License v2.0
4.83k stars 495 forks source link

DietPi-Software | Docker: Service fails to start after upgrade (AppArmor) #6126

Closed samjw-nz closed 1 year ago

samjw-nz commented 1 year ago

ADMIN EDIT

Workaround

apt install apparmor
systemctl disable --now apparmor
systemctl restart docker

Creating a bug report/issue

Required Information

Additional Information (if applicable)

Steps to reproduce

  1. Install Docker from dietpi-software (optional)
  2. Install Portainer from dietpi-software (with or without first installing Docker, as it's installed if not already present)
  3. Allow Portainer image to pull and then fail by itself.

Expected behaviour

Actual behaviour

Extra details

Joulinar commented 1 year ago

uname -a (response: command not found)

That worries me a little. This command should display current kernel version. Can you reboot your system and check for kernel error messages afterwards.

dmesg -l err,crit,alert,emerg

As well check Docker logs

journalctl -u docker.service
samjw-nz commented 1 year ago

No kernel errors, or at least no response to the command anyway.

Docker log: 

-- Journal begins at Fri 2023-02-03 20:38:57 NZDT, ends at Fri 2023-02-03 20:59:21 NZDT. --
Feb 03 20:39:09 Sam-Pi systemd[1]: Started Docker Application Container Engine.
Feb 03 20:39:12 Sam-Pi dockerd[589]: time="2023-02-03T20:39:12.647336342+13:00" level=error msg="AppArmor enabled on system but the docker-default profile could not be loaded: running `apparmor_parser apparmor_parser --version` failed with output: \nerror: exec: \"apparmor_parser\": executable file not found in $PATH"

Edit: tried formatting it nicely and failed sorry.

samjw-nz commented 1 year ago

Ah, ran uname command again and it worked this time. Linux Sam-Pi 5.15.89-sunxi #22.11.4 SMP Mon Jan 23 21:58:30 UTC 2023 armv7l GNU/Linux

Joulinar commented 1 year ago

ok I found a known error for Docker v5.23 within Docker docs https://docs.docker.com/engine/release-notes/23.0/#known-issues

can you check the version you are running.

dpkg -l docker-ce
samjw-nz commented 1 year ago

5:23.0.0-1~debian.11~bullseye armhf

Looks like that includes me then. Want me to try their fix and report back?

Joulinar commented 1 year ago

according docker docs

apt install apparmor
systemctl stop apparmor
samjw-nz commented 1 year ago

All working again, no further errors. Cheers for the help, much appreciated.

Joulinar commented 1 year ago

@MichaIng tried to replicate issue on five different system. All ARM based systems working without issue. It failed on VM x86 only.

Not 100% sure if there is a real trigger or just a random issue. 🤔

MichaIng commented 1 year ago

Good to know that it is no a Bookworm-only issue. Faced it last night on our server. No good instructions in their release notes.

  1. Is this expected now or a bug that is being worked on? I guess/hope the second, since AppArmor is/was in fact not installed/enabled on affected systems.
  2. Instead of installing all AppArmor utils together with Python 3 and module, do apt install apparmor which is only a very small subset but suffiicient.
  3. Disable the service systemctl disable --now apparmor unless you are aware of the implications and know how to configure it to work with your applications.
MichaIng commented 1 year ago

Here is a good bug report, attached to 23.0.1 milestone 🤞: https://github.com/moby/moby/issues/44900

pedrom34 commented 1 year ago

@MichaIng tried to replicate issue on five different system. All ARM based systems working without issue. It failed on VM x86 only.

* RPi1 ARMv6 ✅

* RPi3B+ ARMv7 ✅

* RPi4B ARMv8 ✅

* NanoPi R5S ARMv8 ✅

* VM x86 AMD64 ❌

Not 100% sure if there is a real trigger or just a random issue. 🤔

Had the same issue yesterday on my Rock64. Had to apt install apparmor to solve it.

MichaIng commented 1 year ago

Interesting, so not only x86_64.

Joulinar commented 1 year ago

Nope, original issue was reported on a ZeroPi

cenizo commented 1 year ago

So for amd64 there is no solution?

Joulinar commented 1 year ago

Solution is described on Docker docs 

apt install apparmor-utils
cenizo commented 1 year ago

This solution doesn't work for me

 apt install apparmor-utils
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  apparmor python3-apparmor python3-libapparmor
Suggested packages:
  apparmor-profiles-extra vim-addon-manager
The following packages will be REMOVED:
  linux-image-5.10.0-21-amd64
The following NEW packages will be installed:
  apparmor apparmor-utils python3-apparmor python3-libapparmor
0 upgraded, 4 newly installed, 1 to remove and 0 not upgraded.
1 not fully installed or removed.
Need to get 0 B/1023 kB of archives.
After this operation, 314 MB disk space will be freed.
Do you want to continue? [Y/n] y
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 72244 files and directories currently installed.)
Removing linux-image-5.10.0-21-amd64 (5.10.162-1) ...
/etc/kernel/postrm.d/initramfs-tools:
update-initramfs: Deleting /boot/initrd.img-5.10.0-21-amd64
/etc/kernel/postrm.d/zz-update-grub:
/usr/sbin/grub-mkconfig: 9: /etc/default/grub: splash: not found
run-parts: /etc/kernel/postrm.d/zz-update-grub exited with return code 127
dpkg: error processing package linux-image-5.10.0-21-amd64 (--remove):
 installed linux-image-5.10.0-21-amd64 package post-removal script subprocess returned error exit status 1
dpkg: too many errors, stopping
Errors were encountered while processing:
 linux-image-5.10.0-21-amd64
Processing was halted because there were too many errors.
E: Sub-process /usr/bin/dpkg returned an error code (1)
Joulinar commented 1 year ago

As already stated on our forum, you have an issue with kernel update blocking your system now.

cenizo commented 1 year ago

I know, don't you know of a way to go back?

MichaIng commented 1 year ago

Please use the solution I added to the OP here:

apt install apparmor
systemctl stop apparmor

Otherwise you pull a lot of additional stuff you do not need here and run the service unnecessarily.

@cenizo Can you show the output of:

cat /etc/default/grub
cenizo commented 1 year ago 
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash acpi_osi=Linux”
GRUB_CMDLINE_LINUX="acpi_enforce_resources=lax"

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"
MichaIng commented 1 year ago

Funny that in your case the issue matches quite precisely the one I found coincidentally: https://askubuntu.com/questions/950331/grub-error-usr-sbin-grub-mkconfig-7-etc-default-grub-splash-not-found

Which guide did you follow to add these faulty ” quotes? I suggest to ask the author fixing this, to prevent all his readers from running into this problem.

Be careful when copy&pasting things from random guides if the code is not wrapped into proper verbatim tags, like here the code tags/fences on GitHub, where " and ” is well distinct. And when you do changes, test them immediately (in this case update-grub), so you see directly when something went wrong instead of later in a very unrelated step, like here 😉.

owenilc commented 1 year ago 
apt install apparmor
systemctl stop apparmor

The work round method works for my DietPi x86 AMD64 Docker service starts, all container restored. All functions works again.

But the same work round applied to all my DietPi SBC models: OrangePi Zero+, OrangePi Win+, NanoPi Neo2, NanoPi R2S

Only partial container were restored. The results of "docker ps -a" executed on the above SBC:

CONTAINER ID   IMAGE                           COMMAND                  CREATED        STATUS        PORTS      NAMES
5f23ee73a557   cloudflare/cloudflared:latest   "cloudflared --no-au…"   43 hours ago   Up 43 hours              cloudflared
55f7a11e0c41   containrrr/watchtower:latest    "/watchtower --clean…"   6 days ago     Up 45 hours   8080/tcp   watchtower

The result of "dpkg -l docker-ce" executed on the above SBC:

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version                       Architecture Description
+++-==============-=============================-============-====================================================
ii  docker-ce      5:23.0.0-1~debian.11~bullseye arm64        Docker: the open-source application container engine

The result of "ls -al /var/lib/docker/volumes/" executed on the above SBC:

total 44
drwx-----x  3 root root   4096 Feb  5 23:12 .
drwx--x--- 13 root root   4096 Feb  5 23:12 ..
brw-------  1 root root 179, 1 Feb  5 23:12 backingFsBlockDev
-rw-------  1 root root  65536 Feb  5 23:12 metadata.db
drwx-----x  3 root root   4096 Jul  1  2022 portainer_data

The result of "ls -al /var/lib/docker/image/" executed on the above SBC:

total 12
drwx------  3 root root 4096 Jun 30  2022 .
drwx--x--- 13 root root 4096 Feb  5 23:12 ..
drwx------  5 root root 4096 Feb  5 03:12 overlay2

The other SBC model that does not affected is Raspberry Rpi4B running Raspberry Pi OS.

Any suggestions for me?

Joulinar commented 1 year ago

The workaround is not needed for every system it just needed on systems having the error message with apparmor. I did a test on 5 systems and only 1 had this particular issue.

Any suggestions for me?

You did not share a single error message. Where should we know what issue you have starting containers?

owenilc commented 1 year ago

I was seeing the 'apparmor' error message with journalctl -u docker on my DietPi x86 AMD64 Feb 03 07:28:05 FutroS740 dockerd[695]: time="2023-02-03T07:28:05.327865626+08:00" level=error msg="AppArmor enabled on system but the docker-default profile could not be loaded: running apparmor_parser apparmor_parser --version failed with output: \nerror: exec: \"apparmor_parser\": executable file not found in $PATH"

After applied on the DietPi x86 AMD64, the docker service works agin.

Just forget to record error message on other SBCs. At that time, the docker service on SBCs were not function.

I thought the work round will fix them. Then I applied the work round and reboot SBCs. Then I found only partial container were started. many gone.

The results of journalctl -u docker on my SBCs were attached.

Any other instructions would you recommend to grab other log information?

many thanks.

R2S635003_journal(docker.service).txt Neo2-1_journal(docker.service).txt OrangePiWinPlus_journal(docker.service).txt OrangePiZeroPlus_journal(docker.service).txt R2S_634634_journal(docker.service).txt

Joulinar commented 1 year ago

This has nothing to do with the original issue and would need to be taken separately.

Did you tried to start the container manually? I found message hasBeenManuallyStopped=true.

owenilc commented 1 year ago

I used to login the Portainer UI web to manage the containers.

All container were set to start automatically.

The results of 'docker container ls --all' executed on the above SBC:

CONTAINER ID   IMAGE                           COMMAND                  CREATED        STATUS        PORTS      NAMES
5f23ee73a557   cloudflare/cloudflared:latest   "cloudflared --no-au…"   44 hours ago   Up 44 hours              cloudflared
55f7a11e0c41   containrrr/watchtower:latest    "/watchtower --clean…"   6 days ago     Up 46 hours   8080/tcp   watchtower

There are many other containers existed previously....

MichaIng commented 1 year ago

And... did you try to start the containers manually, probably even pulling them again via docker pull/docker run? Not sure but probably the new Docker version needs to re-register old containers somehow, maybe there were breaking changes. I'm sure the images and especially data is still there in /mnt/dietpi_userdata/docker-data.

Joulinar commented 1 year ago

just give it a try

docker container start portainer

And just to avoid a misunderstanding. We from DietPi, don't have any influence into Docker version / software. We install the official Docker version as it is provided. If there are issues, you might reach out to Docker support in parallel.

owenilc commented 1 year ago

And... did you try to start the containers manually, probably even pulling them again via docker pull/docker run? Not sure but probably the new Docker version needs to re-register old containers somehow, maybe there were breaking changes. I'm sure the images and especially data is still there in /mnt/dietpi_userdata/docker-data.

Thanks!

The results of 'ls -al /mnt/dietpi_userdata/' executed on my DietPi SBCs:

total 24
drwxrwxr-x 6 dietpi dietpi 4096 Apr  6  2022 .
drwxr-xr-x 7 root   root   4096 Apr 10  2022 ..
drwxrwxr-x 2 dietpi dietpi 4096 Apr  6  2022 Music
drwxrwxr-x 2 dietpi dietpi 4096 Apr  6  2022 Pictures
drwxrwxr-x 2 dietpi dietpi 4096 Apr  6  2022 Video
drwxrwxr-x 2 dietpi dietpi 4096 Apr  6  2022 downloads

no docker-data within /mnt/dietpi_userdata/, even the one of DietPi x86 AMD64

Joulinar commented 1 year ago

what does docker info | grep "Docker Root Dir" return?

owenilc commented 1 year ago

what does docker info | grep "Docker Root Dir" return?

Thank you for the quick reply.

The results of 'docker container start portainer' executed on my DietPi SBCs:

Error response from daemon: No such container: portainer
Error: failed to start containers: portainer

It is just a little strange, all my DietPi SBCs and DietPi x86 AMD64 being affected with apparmor error. Maybe a coincidence.

I will try to reach out to the Docker support. thanks.

demosthien commented 1 year ago

I encountered an issue, after updating the OS, where I could not access Docker containers via the browser. It looked to me like this issue and running the provided workaround seems to have fixed it.

FYI my system details:

DietPi: v8.13.2 Kernel: 5.10.0-21-amd64 Native PC (x86_64)

owenilc commented 1 year ago

docker info | grep "Docker Root Dir"

The result of docker info | grep "Docker Root Dir" from one of my SBC

Docker Root Dir: /var/lib/docker
Joulinar commented 1 year ago

Docker Root Dir: /var/lib/docker

Did you install Docker via dietpi-software? Because this is not our default value.

owenilc commented 1 year ago

Docker Root Dir: /var/lib/docker

Did you install Docker via dietpi-software? Because this is not our default value.

No, I install Docker from command line.

Joulinar commented 1 year ago

all my DietPi SBCs and DietPi x86 AMD64 being affected with apparmor error.

The apparmor is gone after applying the workaround as provided by Docker guys. You have different issues now.

Joulinar commented 1 year ago

No, I install Docker from command line.

Some essential information as we assumed you installed it from our software catalogue. This is changing things. Have a look to /var/lib/docker for your container / images.

owenilc commented 1 year ago

No, I install Docker from command line.

Some essential information as we assumed you installed it from our software catalogue. This is changing things. Have a look to /var/lib/docker for your container / images.

The result of "ls -al /var/lib/docker/volumes/" executed on my SBCs:

total 44
drwx-----x  3 root root   4096 Feb  5 23:12 .
drwx--x--- 13 root root   4096 Feb  5 23:12 ..
brw-------  1 root root 179, 1 Feb  5 23:12 backingFsBlockDev
-rw-------  1 root root  65536 Feb  5 23:12 metadata.db
drwx-----x  3 root root   4096 Jul  1  2022 portainer_data

The result of "ls -al /var/lib/docker/image/" executed on my SBC:

total 12
drwx------  3 root root 4096 Jun 30  2022 .
drwx--x--- 13 root root 4096 Feb  5 23:12 ..
drwx------  5 root root 4096 Feb  5 03:12 overlay2
Joulinar commented 1 year ago

At least your Portainer volume seems to be there. Can you check

docker volume ls
samjw-nz commented 1 year ago

Need to unsubscribe from this issue thread, getting too many notifications since my original issue was resolved. @Joulinar feel free to message me if there's anything further you need from me. Cheers

Joulinar commented 1 year ago

@samjwass Don't think we need anything on addition as issue is outside of our area. It's a challenge of Docker software and basically, they are aware as it's documented as know issue inside Docker release note.

irousom commented 1 year ago

One workaround that doesn't require installing additional software is explicitly disabling AppArmor in the grub configuration; for example:

sudo echo 'GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT apparmor=0"' | sudo tee /etc/default/grub.d/apparmor.cfg
sudo update-grub
sudo reboot

It might be desirable to include this explicit disabling of AppArmor in future DietPi releases and to only explicitly enable it if the user installs all the necessary AppArmor elements, but since this is clearly an issue with Docker itself that may be premature.

Joulinar commented 1 year ago

One workaround that doesn't require installing additional software is explicitly disabling AppArmor in the grub configuration; for example:

This applies for x86 systems only. But we have reports on ARM SBC as well. :wink:

MichaIng commented 1 year ago

Can be added via /boot/dietpiEnv.txt etc. But if users then want to actually use AppArmor, they need to be aware of it and remove it again. Not nice. Docker needs to fix the check. The fact that no cmdline argument is passed to disable AppArmor is not at all a prove that AppArmor is actually available, neither that CLI utils are present, nor that the kernel driver is actually loaded. Docker uses a specific utility apparmor_parser, so it needs to check whether this specific utility is present and at best do an additional modprobe apparmor which must be allowed to fail, to assure that the driver is loaded, or, if it fails to load, skip the whole thing.

But aside of that, of course it is nice that there are alternative workarounds, just not one that we could implement into dietpi-software 😉.

This might work as well:

echo -e '#!/bin/dash\n:' > /usr/local/bin/apparmor_parser
Joulinar commented 1 year ago

@MichaIng are we able to push a notification about Docker issues on latest Docker build via MOTD?

Joulinar commented 1 year ago

This might work as well:

echo -e '#!/bin/dash\n:' > /usr/local/bin/apparmor_parser

nope, it's not going to change anything.

MichaIng commented 1 year ago

nope, it's not going to change anything.

At least the error message should look different then:

running `apparmor_parser apparmor_parser --version` failed with output: \nerror: exec: \"apparmor_parser\": executable file not found in $PATH"
Roalkege commented 1 year ago

I also have the problem on my Odroid N2+ But I could fix it with:

apt install apparmor
systemctl stop apparmor
Joulinar commented 1 year ago

there ist some light at the end of the tunnel https://github.com/moby/moby/issues/44900#issuecomment-1420032331

Fix fill become available on Docker v23.0.1

pedrom34 commented 1 year ago

Since we had to install apparmor as a workaround, with the upcoming fix, will it be possible to remove apparmor? I must admit I do not know what this package does...