Closed samjw-nz closed 1 year ago
uname -a (response: command not found)
That worries me a little. This command should display current kernel version. Can you reboot your system and check for kernel error messages afterwards.
dmesg -l err,crit,alert,emerg
As well check Docker logs
journalctl -u docker.service
No kernel errors, or at least no response to the command anyway.
Docker log:
-- Journal begins at Fri 2023-02-03 20:38:57 NZDT, ends at Fri 2023-02-03 20:59:21 NZDT. --
Feb 03 20:39:09 Sam-Pi systemd[1]: Started Docker Application Container Engine.
Feb 03 20:39:12 Sam-Pi dockerd[589]: time="2023-02-03T20:39:12.647336342+13:00" level=error msg="AppArmor enabled on system but the docker-default profile could not be loaded: running `apparmor_parser apparmor_parser --version` failed with output: \nerror: exec: \"apparmor_parser\": executable file not found in $PATH"
Edit: tried formatting it nicely and failed sorry.
Ah, ran uname command again and it worked this time. Linux Sam-Pi 5.15.89-sunxi #22.11.4 SMP Mon Jan 23 21:58:30 UTC 2023 armv7l GNU/Linux
ok I found a known error for Docker v5.23 within Docker docs https://docs.docker.com/engine/release-notes/23.0/#known-issues
can you check the version you are running.
dpkg -l docker-ce
5:23.0.0-1~debian.11~bullseye armhf
Looks like that includes me then. Want me to try their fix and report back?
according docker docs
apt install apparmor
systemctl stop apparmor
All working again, no further errors. Cheers for the help, much appreciated.
@MichaIng tried to replicate issue on five different system. All ARM based systems working without issue. It failed on VM x86 only.
Not 100% sure if there is a real trigger or just a random issue. 🤔
Good to know that it is no a Bookworm-only issue. Faced it last night on our server. No good instructions in their release notes.
apt install apparmor
which is only a very small subset but suffiicient.systemctl disable --now apparmor
unless you are aware of the implications and know how to configure it to work with your applications.Here is a good bug report, attached to 23.0.1 milestone 🤞: https://github.com/moby/moby/issues/44900
@MichaIng tried to replicate issue on five different system. All ARM based systems working without issue. It failed on VM x86 only.
* RPi1 ARMv6 ✅ * RPi3B+ ARMv7 ✅ * RPi4B ARMv8 ✅ * NanoPi R5S ARMv8 ✅ * VM x86 AMD64 ❌
Not 100% sure if there is a real trigger or just a random issue. 🤔
Had the same issue yesterday on my Rock64. Had to apt install apparmor
to solve it.
Interesting, so not only x86_64.
Nope, original issue was reported on a ZeroPi
So for amd64 there is no solution?
Solution is described on Docker docs
apt install apparmor-utils
This solution doesn't work for me
apt install apparmor-utils
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
apparmor python3-apparmor python3-libapparmor
Suggested packages:
apparmor-profiles-extra vim-addon-manager
The following packages will be REMOVED:
linux-image-5.10.0-21-amd64
The following NEW packages will be installed:
apparmor apparmor-utils python3-apparmor python3-libapparmor
0 upgraded, 4 newly installed, 1 to remove and 0 not upgraded.
1 not fully installed or removed.
Need to get 0 B/1023 kB of archives.
After this operation, 314 MB disk space will be freed.
Do you want to continue? [Y/n] y
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 72244 files and directories currently installed.)
Removing linux-image-5.10.0-21-amd64 (5.10.162-1) ...
/etc/kernel/postrm.d/initramfs-tools:
update-initramfs: Deleting /boot/initrd.img-5.10.0-21-amd64
/etc/kernel/postrm.d/zz-update-grub:
/usr/sbin/grub-mkconfig: 9: /etc/default/grub: splash: not found
run-parts: /etc/kernel/postrm.d/zz-update-grub exited with return code 127
dpkg: error processing package linux-image-5.10.0-21-amd64 (--remove):
installed linux-image-5.10.0-21-amd64 package post-removal script subprocess returned error exit status 1
dpkg: too many errors, stopping
Errors were encountered while processing:
linux-image-5.10.0-21-amd64
Processing was halted because there were too many errors.
E: Sub-process /usr/bin/dpkg returned an error code (1)
As already stated on our forum, you have an issue with kernel update blocking your system now.
I know, don't you know of a way to go back?
Please use the solution I added to the OP here:
apt install apparmor
systemctl stop apparmor
Otherwise you pull a lot of additional stuff you do not need here and run the service unnecessarily.
@cenizo Can you show the output of:
cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'
GRUB_DEFAULT=0
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT=”quiet splash acpi_osi=Linux”
GRUB_CMDLINE_LINUX="acpi_enforce_resources=lax"
# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"
# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console
# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480
# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true
# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"
# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"
Funny that in your case the issue matches quite precisely the one I found coincidentally: https://askubuntu.com/questions/950331/grub-error-usr-sbin-grub-mkconfig-7-etc-default-grub-splash-not-found
Which guide did you follow to add these faulty ”
quotes? I suggest to ask the author fixing this, to prevent all his readers from running into this problem.
Be careful when copy&pasting things from random guides if the code is not wrapped into proper verbatim tags, like here the code tags/fences on GitHub, where "
and ”
is well distinct. And when you do changes, test them immediately (in this case update-grub
), so you see directly when something went wrong instead of later in a very unrelated step, like here 😉.
apt install apparmor
systemctl stop apparmor
The work round method works for my DietPi x86 AMD64 Docker service starts, all container restored. All functions works again.
But the same work round applied to all my DietPi SBC models: OrangePi Zero+, OrangePi Win+, NanoPi Neo2, NanoPi R2S
Only partial container were restored. The results of "docker ps -a" executed on the above SBC:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5f23ee73a557 cloudflare/cloudflared:latest "cloudflared --no-au…" 43 hours ago Up 43 hours cloudflared
55f7a11e0c41 containrrr/watchtower:latest "/watchtower --clean…" 6 days ago Up 45 hours 8080/tcp watchtower
The result of "dpkg -l docker-ce" executed on the above SBC:
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=============================-============-====================================================
ii docker-ce 5:23.0.0-1~debian.11~bullseye arm64 Docker: the open-source application container engine
The result of "ls -al /var/lib/docker/volumes/" executed on the above SBC:
total 44
drwx-----x 3 root root 4096 Feb 5 23:12 .
drwx--x--- 13 root root 4096 Feb 5 23:12 ..
brw------- 1 root root 179, 1 Feb 5 23:12 backingFsBlockDev
-rw------- 1 root root 65536 Feb 5 23:12 metadata.db
drwx-----x 3 root root 4096 Jul 1 2022 portainer_data
The result of "ls -al /var/lib/docker/image/" executed on the above SBC:
total 12
drwx------ 3 root root 4096 Jun 30 2022 .
drwx--x--- 13 root root 4096 Feb 5 23:12 ..
drwx------ 5 root root 4096 Feb 5 03:12 overlay2
The other SBC model that does not affected is Raspberry Rpi4B running Raspberry Pi OS.
Any suggestions for me?
The workaround is not needed for every system it just needed on systems having the error message with apparmor
. I did a test on 5 systems and only 1 had this particular issue.
Any suggestions for me?
You did not share a single error message. Where should we know what issue you have starting containers?
I was seeing the 'apparmor' error message with journalctl -u docker on my DietPi x86 AMD64
Feb 03 07:28:05 FutroS740 dockerd[695]: time="2023-02-03T07:28:05.327865626+08:00" level=error msg="AppArmor enabled on system but the docker-default profile could not be loaded: running apparmor_parser apparmor_parser --version failed with output: \nerror: exec: \"apparmor_parser\": executable file not found in $PATH"
After applied on the DietPi x86 AMD64, the docker service works agin.
Just forget to record error message on other SBCs. At that time, the docker service on SBCs were not function.
I thought the work round will fix them. Then I applied the work round and reboot SBCs. Then I found only partial container were started. many gone.
The results of journalctl -u docker on my SBCs were attached.
Any other instructions would you recommend to grab other log information?
many thanks.
R2S635003_journal(docker.service).txt Neo2-1_journal(docker.service).txt OrangePiWinPlus_journal(docker.service).txt OrangePiZeroPlus_journal(docker.service).txt R2S_634634_journal(docker.service).txt
This has nothing to do with the original issue and would need to be taken separately.
Did you tried to start the container manually? I found message hasBeenManuallyStopped=true
.
I used to login the Portainer UI web to manage the containers.
All container were set to start automatically.
The results of 'docker container ls --all' executed on the above SBC:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5f23ee73a557 cloudflare/cloudflared:latest "cloudflared --no-au…" 44 hours ago Up 44 hours cloudflared
55f7a11e0c41 containrrr/watchtower:latest "/watchtower --clean…" 6 days ago Up 46 hours 8080/tcp watchtower
There are many other containers existed previously....
And... did you try to start the containers manually, probably even pulling them again via docker pull
/docker run
? Not sure but probably the new Docker version needs to re-register old containers somehow, maybe there were breaking changes. I'm sure the images and especially data is still there in /mnt/dietpi_userdata/docker-data
.
just give it a try
docker container start portainer
And just to avoid a misunderstanding. We from DietPi, don't have any influence into Docker version / software. We install the official Docker version as it is provided. If there are issues, you might reach out to Docker support in parallel.
And... did you try to start the containers manually, probably even pulling them again via
docker pull
/docker run
? Not sure but probably the new Docker version needs to re-register old containers somehow, maybe there were breaking changes. I'm sure the images and especially data is still there in/mnt/dietpi_userdata/docker-data
.
Thanks!
The results of 'ls -al /mnt/dietpi_userdata/' executed on my DietPi SBCs:
total 24
drwxrwxr-x 6 dietpi dietpi 4096 Apr 6 2022 .
drwxr-xr-x 7 root root 4096 Apr 10 2022 ..
drwxrwxr-x 2 dietpi dietpi 4096 Apr 6 2022 Music
drwxrwxr-x 2 dietpi dietpi 4096 Apr 6 2022 Pictures
drwxrwxr-x 2 dietpi dietpi 4096 Apr 6 2022 Video
drwxrwxr-x 2 dietpi dietpi 4096 Apr 6 2022 downloads
no docker-data within /mnt/dietpi_userdata/, even the one of DietPi x86 AMD64
what does docker info | grep "Docker Root Dir"
return?
what does
docker info | grep "Docker Root Dir"
return?
Thank you for the quick reply.
The results of 'docker container start portainer' executed on my DietPi SBCs:
Error response from daemon: No such container: portainer
Error: failed to start containers: portainer
It is just a little strange, all my DietPi SBCs and DietPi x86 AMD64 being affected with apparmor error. Maybe a coincidence.
I will try to reach out to the Docker support. thanks.
I encountered an issue, after updating the OS, where I could not access Docker containers via the browser. It looked to me like this issue and running the provided workaround seems to have fixed it.
FYI my system details:
DietPi: v8.13.2 Kernel: 5.10.0-21-amd64 Native PC (x86_64)
docker info | grep "Docker Root Dir"
The result of docker info | grep "Docker Root Dir" from one of my SBC
Docker Root Dir: /var/lib/docker
Docker Root Dir: /var/lib/docker
Did you install Docker via dietpi-software
? Because this is not our default value.
Docker Root Dir: /var/lib/docker
Did you install Docker via
dietpi-software
? Because this is not our default value.
No, I install Docker from command line.
all my DietPi SBCs and DietPi x86 AMD64 being affected with apparmor error.
The apparmor
is gone after applying the workaround as provided by Docker guys. You have different issues now.
No, I install Docker from command line.
Some essential information as we assumed you installed it from our software catalogue. This is changing things. Have a look to /var/lib/docker
for your container / images.
No, I install Docker from command line.
Some essential information as we assumed you installed it from our software catalogue. This is changing things. Have a look to
/var/lib/docker
for your container / images.
The result of "ls -al /var/lib/docker/volumes/" executed on my SBCs:
total 44
drwx-----x 3 root root 4096 Feb 5 23:12 .
drwx--x--- 13 root root 4096 Feb 5 23:12 ..
brw------- 1 root root 179, 1 Feb 5 23:12 backingFsBlockDev
-rw------- 1 root root 65536 Feb 5 23:12 metadata.db
drwx-----x 3 root root 4096 Jul 1 2022 portainer_data
The result of "ls -al /var/lib/docker/image/" executed on my SBC:
total 12
drwx------ 3 root root 4096 Jun 30 2022 .
drwx--x--- 13 root root 4096 Feb 5 23:12 ..
drwx------ 5 root root 4096 Feb 5 03:12 overlay2
At least your Portainer volume seems to be there. Can you check
docker volume ls
Need to unsubscribe from this issue thread, getting too many notifications since my original issue was resolved. @Joulinar feel free to message me if there's anything further you need from me. Cheers
@samjwass Don't think we need anything on addition as issue is outside of our area. It's a challenge of Docker software and basically, they are aware as it's documented as know issue inside Docker release note.
One workaround that doesn't require installing additional software is explicitly disabling AppArmor in the grub configuration; for example:
sudo echo 'GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT apparmor=0"' | sudo tee /etc/default/grub.d/apparmor.cfg
sudo update-grub
sudo reboot
It might be desirable to include this explicit disabling of AppArmor in future DietPi releases and to only explicitly enable it if the user installs all the necessary AppArmor elements, but since this is clearly an issue with Docker itself that may be premature.
One workaround that doesn't require installing additional software is explicitly disabling AppArmor in the grub configuration; for example:
This applies for x86 systems only. But we have reports on ARM SBC as well. :wink:
Can be added via /boot/dietpiEnv.txt
etc. But if users then want to actually use AppArmor, they need to be aware of it and remove it again. Not nice. Docker needs to fix the check. The fact that no cmdline argument is passed to disable AppArmor is not at all a prove that AppArmor is actually available, neither that CLI utils are present, nor that the kernel driver is actually loaded. Docker uses a specific utility apparmor_parser
, so it needs to check whether this specific utility is present and at best do an additional modprobe apparmor
which must be allowed to fail, to assure that the driver is loaded, or, if it fails to load, skip the whole thing.
But aside of that, of course it is nice that there are alternative workarounds, just not one that we could implement into dietpi-software
😉.
This might work as well:
echo -e '#!/bin/dash\n:' > /usr/local/bin/apparmor_parser
@MichaIng are we able to push a notification about Docker issues on latest Docker build via MOTD?
This might work as well:
echo -e '#!/bin/dash\n:' > /usr/local/bin/apparmor_parser
nope, it's not going to change anything.
nope, it's not going to change anything.
At least the error message should look different then:
running `apparmor_parser apparmor_parser --version` failed with output: \nerror: exec: \"apparmor_parser\": executable file not found in $PATH"
I also have the problem on my Odroid N2+ But I could fix it with:
apt install apparmor
systemctl stop apparmor
there ist some light at the end of the tunnel https://github.com/moby/moby/issues/44900#issuecomment-1420032331
Fix fill become available on Docker v23.0.1
Since we had to install apparmor as a workaround, with the upcoming fix, will it be possible to remove apparmor? I must admit I do not know what this package does...
ADMIN EDIT
Workaround
Creating a bug report/issue
Required Information
Linux Sam-Pi 5.15.89-sunxi #22.11.4 SMP Mon Jan 23 21:58:30 UTC 2023 armv7l GNU/Linux
Additional Information (if applicable)
Steps to reproduce
Expected behaviour
Actual behaviour
Extra details