mailcow / mailcow-dockerized

mailcow: dockerized - 🐮 + 🐋 = 💕
https://mailcow.email
GNU General Public License v3.0
8.34k stars 1.13k forks source link

apparmor problems #5057

Closed Oratorian closed 11 months ago

Oratorian commented 1 year ago

Contribution guidelines

I've found a bug and checked that ...

Description

My system updated to docker-ce version 23.0.1 today and after the update mailcow broke.
Restarting mailcow or running updates errors out. 

Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: unable to apply apparmor profile: apparmor failed to apply profile: write /proc/self/attr/apparmor/exec: no such file or directory: unknown

I tested it by downgrading to 23.0.0 and everything works fine.

Temporary fix for me on 23.0.1 was to add privileged : true to all containers in the compose file.

Logs:

root@mx:# docker compose up -d
[+] Running 13/18
 ⠿ Container mailcowdockerized-redis-mailcow-1      Started                                                                                                                   8.7s
 ⠿ Container mailcowdockerized-sogo-mailcow-1       Started                                                                                                                   5.3s
 ⠿ Container mailcowdockerized-unbound-mailcow-1    Started                                                                                                                   3.5s
 ⠿ Container mailcowdockerized-watchdog-mailcow-1   Started                                                                                                                   4.1s
 ⠿ Container mailcowdockerized-php-fpm-mailcow-1    Started                                                                                                                  11.0s
 ⠿ Container mailcowdockerized-mysql-mailcow-1      Started                                                                                                                   8.8s
 ⠿ Container mailcowdockerized-clamd-mailcow-1      Started                                                                                                                   7.2s
 ⠿ Container mailcowdockerized-postfix-mailcow-1    Started                                                                                                                  12.4s
 ⠿ Container mailcowdockerized-dovecot-mailcow-1    Started                                                                                                                  11.0s
 ⠿ Container mailcowdockerized-nginx-mailcow-1      Started                                                                                                                  14.3s
 ⠿ Container mailcowdockerized-acme-mailcow-1       Started                                                                                                                  17.7s
 ⠿ Container mailcowdockerized-ofelia-mailcow-1     Starting                                                                                                                 18.1s
 ⠿ Container mailcowdockerized-rspamd-mailcow-1     Started                                                                                                                  15.9s
 ⠿ Container mailcowdockerized-netfilter-mailcow-1  Started                                                                                                                  14.1s
 ⠧ Container mailcowdockerized-olefy-mailcow-1      Starting                                                                                                                 17.6s
 ⠧ Container mailcowdockerized-memcached-mailcow-1  Starting                                                                                                                 17.6s
 ⠧ Container mailcowdockerized-solr-mailcow-1       Starting                                                                                                                 17.6s
 ⠧ Container mailcowdockerized-dockerapi-mailcow-1  Starting                                                                                                                 17.6s
Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: unable to apply apparmor profile: apparmor failed to apply profile: write /proc/self/attr/apparmor/exec: no such file or directory: unknown.

Steps to reproduce:

1. Update to docker-ce version 23.0.1
2. Stop and restart mailcow

Which branch are you using?

master

Operating System:

Ubuntu 22.04.1 LTS

Server/VM specifications:

32GB / 4 Cores / 8 Threads

Is Apparmor, SELinux or similar active?

yes

Virtualization technology:

none

Docker version:

23.0.1

docker-compose version or docker compose version:

2.16.0

mailcow version:

2023-02

Reverse proxy:

none

Logs of git diff:

none

Logs of iptables -L -vn:

Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
 6509 3722K DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 6509 3722K DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 5447 3618K ACCEPT     all  --  *      br0-mailcow  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
  371 23970 DOCKER     all  --  *      br0-mailcow  0.0.0.0/0            0.0.0.0/0
  691 80023 ACCEPT     all  --  br0-mailcow !br0-mailcow  0.0.0.0/0            0.0.0.0/0
  365 23593 ACCEPT     all  --  br0-mailcow br0-mailcow  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     tcp  --  !br0-mailcow br0-mailcow  0.0.0.0/0            172.22.1.5           tcp dpt:8983
    0     0 ACCEPT     tcp  --  !br0-mailcow br0-mailcow  0.0.0.0/0            172.22.1.249         tcp dpt:6379
    0     0 ACCEPT     tcp  --  !br0-mailcow br0-mailcow  0.0.0.0/0            172.22.1.7           tcp dpt:3306
    0     0 ACCEPT     tcp  --  !br0-mailcow br0-mailcow  0.0.0.0/0            172.22.1.253         tcp dpt:587
    1    60 ACCEPT     tcp  --  !br0-mailcow br0-mailcow  0.0.0.0/0            172.22.1.253         tcp dpt:465
    0     0 ACCEPT     tcp  --  !br0-mailcow br0-mailcow  0.0.0.0/0            172.22.1.253         tcp dpt:25
    0     0 ACCEPT     tcp  --  !br0-mailcow br0-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:12345
    0     0 ACCEPT     tcp  --  !br0-mailcow br0-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:4190
    0     0 ACCEPT     tcp  --  !br0-mailcow br0-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:995
    0     0 ACCEPT     tcp  --  !br0-mailcow br0-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:993
    0     0 ACCEPT     tcp  --  !br0-mailcow br0-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:143
    0     0 ACCEPT     tcp  --  !br0-mailcow br0-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:110
    4   224 ACCEPT     tcp  --  !br0-mailcow br0-mailcow  0.0.0.0/0            172.22.1.10          tcp dpt:443
    1    93 ACCEPT     tcp  --  !br0-mailcow br0-mailcow  0.0.0.0/0            172.22.1.10          tcp dpt:80

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination
  691 80023 DOCKER-ISOLATION-STAGE-2  all  --  br0-mailcow !br0-mailcow  0.0.0.0/0            0.0.0.0/0
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
 8993 4859K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  *      br0-mailcow  0.0.0.0/0            0.0.0.0/0
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
  948  106K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination
26944   16M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Logs of ip6tables -L -vn:

# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER-USER  all      *      *       ::/0                 ::/0
    0     0 DOCKER-ISOLATION-STAGE-1  all      *      *       ::/0                 ::/0
    0     0 ACCEPT     all      *      docker0  ::/0                 ::/0                 ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all      *      docker0  ::/0                 ::/0
    0     0 ACCEPT     all      docker0 !docker0  ::/0                 ::/0
    0     0 ACCEPT     all      docker0 docker0  ::/0                 ::/0

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER-ISOLATION-STAGE-2  all      br0-mailcow !br0-mailcow  ::/0                 ::/0
    0     0 DOCKER-ISOLATION-STAGE-2  all      docker0 !docker0  ::/0                 ::/0
    0     0 RETURN     all      *      *       ::/0                 ::/0

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all      *      br0-mailcow  ::/0                 ::/0
    0     0 DROP       all      *      docker0  ::/0                 ::/0
    0     0 RETURN     all      *      *       ::/0                 ::/0

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all      *      *       ::/0                 ::/0

Logs of iptables -L -vn -t nat:

# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
   97  6815 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  366 26344 MASQUERADE  all  --  *      !br0-mailcow  172.22.1.0/24        0.0.0.0/0
    0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.5           172.22.1.5           tcp dpt:8983
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.249         172.22.1.249         tcp dpt:6379
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.7           172.22.1.7           tcp dpt:3306
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.253         172.22.1.253         tcp dpt:587
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.253         172.22.1.253         tcp dpt:465
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.253         172.22.1.253         tcp dpt:25
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:12345
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:4190
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:995
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:993
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:143
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:110
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.10          172.22.1.10          tcp dpt:443
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.10          172.22.1.10          tcp dpt:80

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    7   420 RETURN     all  --  br0-mailcow *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0
    0     0 DNAT       tcp  --  !br0-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:18983 to:172.22.1.5:8983
    0     0 DNAT       tcp  --  !br0-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:7654 to:172.22.1.249:6379
    0     0 DNAT       tcp  --  !br0-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:13306 to:172.22.1.7:3306
    0     0 DNAT       tcp  --  !br0-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:587 to:172.22.1.253:587
    1    60 DNAT       tcp  --  !br0-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:465 to:172.22.1.253:465
    0     0 DNAT       tcp  --  !br0-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:25 to:172.22.1.253:25
    0     0 DNAT       tcp  --  !br0-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:19991 to:172.22.1.250:12345
    0     0 DNAT       tcp  --  !br0-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:4190 to:172.22.1.250:4190
    0     0 DNAT       tcp  --  !br0-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:995 to:172.22.1.250:995
    0     0 DNAT       tcp  --  !br0-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:993 to:172.22.1.250:993
    0     0 DNAT       tcp  --  !br0-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:143 to:172.22.1.250:143
    0     0 DNAT       tcp  --  !br0-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:110 to:172.22.1.250:110
    4   224 DNAT       tcp  --  !br0-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443 to:172.22.1.10:443
    1    93 DNAT       tcp  --  !br0-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 to:172.22.1.10:80

Logs of ip6tables -L -vn -t nat:

# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER     all      *      *       ::/0                 ::/0                 ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER     all      *      *       ::/0                !::1                  ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MASQUERADE  all      *      !docker0  fd00:dead:beef:c0::/80  ::/0

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all      docker0 *       ::/0                 ::/

DNS check:

151.101.1.69
151.101.193.69
151.101.65.69
151.101.129.69
cnbhl commented 1 year ago

Exactly the same here. Seem to be a problem with docker-ce itself. Should be solved in docker-ce=23.0.1. See here: https://github.com/moby/moby/issues/44900

This is not working for me. Temporary solution: revert to previous docker-ce version e.g.:

apt install docker-ce=5:20.10.23~3-0~debian-bullseye

cnbhl commented 1 year ago

Also a solution: Install apparmor prior to upgrading to docker 23.0.1 or restart docker after installation of apparmor. apt install apparmor and then restart docker, e.g. systemctl restart docker. Finally restart mailcow docker-compose up -d.

or simply set apparmor=0 on the kernel command line. Edit: not a good security advice if forgotten...

See: https://github.com/moby/moby/issues/44970#issuecomment-1427428554

stefan2904 commented 1 year ago

for the record, upgrading to docker-ce=23.0.1 did not solve the problem for me, but manually installing the apparmor package and then restarting docker did. Thanks cnbhl!

Margrie commented 1 year ago

i struggled with this problem the last 2 days. I try'd Debian 11 and Ubuntu 22.10. On Debian11 apt install apparmor solved the problem. On Ubuntu 22.10 didnt. Maybe can someone post where in the Docker Install script i can specify the version to 23.0 ? curl -sSL https://get.docker.com/ | CHANNEL=stable sh

cnbhl commented 1 year ago

i struggled with this problem the last 2 days. I try'd Debian 11 and Ubuntu 22.10. On Debian11 apt install apparmor solved the problem. On Ubuntu 22.10 didnt. Maybe can someone post where in the Docker Install script i can specify the version to 23.0 ? curl -sSL https://get.docker.com/ | CHANNEL=stable sh

The people from moby/moby are aware of the incomplete fix for the missing apparmor check.

I tried with Ubuntu 22.10 and it worked for me. Maybe you try apt install docker-ce=5:20.10.23~3-0~ubuntu-kinetic to downgrade to a previous docker version and wait until there is the final fix for the apparmor check.

Oratorian commented 1 year ago

for the record, upgrading to docker-ce=23.0.1 did not solve the problem for me, but manually installing the apparmor package and then restarting docker did. Thanks cnbhl!

@stefan2904 The solution was never to update to 23.0.1, docker-ce 23.0.1 is the problem, the solution is either to downgrade back to 23.0.0 or set all containers to priviliged mode or stop mailcow, docker install apparmor and restart everything.

celangoni commented 1 year ago

I have the same issue on vanilla OS, based on ubuntu 22.10. It seens there is a missing file on /etc/apparmor.d/tunebles, called home. sudo systemctl status apparmor should show inactive due to errors on startup. /etc/apparmor.d/tunebles/home on debian11 is:

#
#    Copyright (C) 2006-2009 Novell/SUSE
#    Copyright (C) 2010 Canonical Ltd.
#
#    This program is free software; you can redistribute it and/or
#    modify it under the terms of version 2 of the GNU General Public
#    License published by the Free Software Foundation.
#
# ------------------------------------------------------------------

# @{HOME} is a space-separated list of all user home directories. While
# it doesn't refer to a specific home directory (AppArmor doesn't
# enforce discretionary access controls) it can be used as if it did
# refer to a specific home directory
@{HOME}=@{HOMEDIRS}/*/ /root/

# @{HOMEDIRS} is a space-separated list of where user home directories
# are stored, for programs that must enumerate all home directories on a
# system.
@{HOMEDIRS}=/home/

# Also, include files in tunables/home.d for site-specific adjustments to
# @{HOMEDIRS}.
include <tunables/home.d>

Just create this file and restart the system, to make sure everything is restarted, solved my problem

zandercodes commented 1 year ago

Problem Solved by installing this packages for me: apt install apparmor apparmor-utils apparmor-notify apparmor-profiles apparmor-profiles-extra and reboot.

landonstewart commented 1 year ago

WTF

sthempura commented 1 year ago

If you are using a security module other than AppArmor and docker-ce = 23.0.1, you can work around the issue by disabling AppArmor on the kernel's command line. This can be achieved by adding 'apparmor=0' in the kernel command line, which can be done as follows:

  1. Locate the 'GRUB_CMDLINE_LINUX=' parameter in the /etc/default/grub file and add 'apparmor=0' to it along with any other security module you are using.

For example:

$ grep "GRUB_CMDLINE_LINUX=" /etc/default/grub GRUB_CMDLINE_LINUX="apparmor=0 security=xxx"

  1. Run the command 'update-grub' to update grub.
  2. Run the command 'update-initramfs -k all -u' to update initramfs.
  3. Reboot your machine.

Once your machine has restarted, your containers should start without any issues.

alimp5 commented 1 year ago

@zandercodes Thanks you. I had same problem on docker 23.0.1 on debian 11. your solution fixed my problem: apt install apparmor apparmor-utils apparmor-notify apparmor-profiles apparmor-profiles-extra

milkmaker commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

milkmaker commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.