mailcow fails to start after update of docker-ce and docker-ce-cli from amd64 5:20.10.23~3-0~debian-bullseye to amd64 5:5:23.0.0-1~debian.11~bullseye

purejava commented 1 year ago

Contribution guidelines

[X] I've read the contribution guidelines and wholeheartedly agree

I've found a bug and checked that ...

[X] ... I understand that not following the below instructions will result in immediate closure and/or deletion of my issue.
[X] ... I have understood that this bug report is dedicated for bugs, and not for support-related inquiries.
[X] ... I have understood that answers are voluntary and community-driven, and not commercial support.
[X] ... I have verified that my issue has not been already answered in the past. I also checked previous issues.

Description

mailcow is up to date via update.sh.

Today, an update for the two packages mentioned above was offered.

========================================
[UPGRADE] docker-ce:amd64 5:20.10.23~3-0~debian-bullseye -> 5:23.0.0-1~debian.11~bullseye
[UPGRADE] docker-ce-cli:amd64 5:20.10.23~3-0~debian-bullseye -> 5:23.0.0-1~debian.11~bullseye
========================================

After the update, mailcow fails to start with an error message: Error response from daemon: AppArmor enabled on system but the docker-default profile could not be loaded: running apparmor_parser apparmor_parser --version failed with output: error: exec: "apparmor_parser": executable file not found in $PATH

Downgrading the two packages makes mailcow start again.

The packages are installed from: https://download.docker.com/linux/debian bullseye stable

Logs:

root@mail /opt/mailcow-dockerized # docker-compose up -d
[+] Running 0/1
 ⠙ Network mailcowdockerized_mailcow-network  Creating                                                                       0.2s
WARN[0000] mount of type `volume` should not define `bind` option 
WARN[0000] mount of type `volume` should not define `bind` option 
WARN[0000] mount of type `volume` should not define `bind` option 
WARN[0000] mount of type `volume` should not define `bind` option 
[+] Running 11/20of type `volume` should not define `bind` option 
 ⠿ Network mailcowdockerized_mailcow-network        Created                                                                  0.2s
 ⠿ Container mailcowdockerized-watchdog-mailcow-1   Starting                                                                 1.8s
 ⠿ Container mailcowdockerized-borgmatic-mailcow-1  Starting                                                                 1.8s
 ⠿ Container mailcowdockerized-memcached-mailcow-1  Starting                                                                 1.8s
 ⠿ Container mailcowdockerized-olefy-mailcow-1      Starting                                                                 1.8s
 ⠿ Container mailcowdockerized-solr-mailcow-1       Starting                                                                 1.8s
 ⠿ Container mailcowdockerized-dockerapi-mailcow-1  Starting                                                                 1.8s
 ⠿ Container mailcowdockerized-redis-mailcow-1      Starting                                                                 1.8s
 ⠿ Container mailcowdockerized-unbound-mailcow-1    Starting                                                                 1.8s
 ⠿ Container mailcowdockerized-sogo-mailcow-1       Starting                                                                 1.8s
 ⠿ Container mailcowdockerized-php-fpm-mailcow-1    Created                                                                  0.1s
 ⠿ Container mailcowdockerized-mysql-mailcow-1      Created                                                                  0.1s
 ⠿ Container mailcowdockerized-clamd-mailcow-1      Created                                                                  0.0s
 ⠿ Container mailcowdockerized-dovecot-mailcow-1    Created                                                                  0.1s
 ⠿ Container mailcowdockerized-postfix-mailcow-1    Created                                                                  0.1s
 ⠿ Container mailcowdockerized-nginx-mailcow-1      Created                                                                  0.1s
 ⠿ Container mailcowdockerized-acme-mailcow-1       Created                                                                  0.1s
 ⠿ Container mailcowdockerized-netfilter-mailcow-1  Created                                                                  0.0s
 ⠿ Container mailcowdockerized-ofelia-mailcow-1     Created                                                                  0.1s
 ⠿ Container mailcowdockerized-rspamd-mailcow-1     Created                                                                  0.1s
Error response from daemon: AppArmor enabled on system but the docker-default profile could not be loaded: running `apparmor_parser apparmor_parser --version` failed with output: 
error: exec: "apparmor_parser": executable file not found in $PATH
root@mail /opt/mailcow-dockerized #

Steps to reproduce:

See above

Which branch are you using?

master

Operating System:

Debian GNU/Linux 11

Server/VM specifications:

16 GB, 4 vCores

Is Apparmor, SELinux or similar active?

no

Virtualization technology:

KVM

Docker version:

23.0.0

docker-compose version or docker compose version:

v2.9.0

mailcow version:

2023-02

Reverse proxy:

none

Logs of git diff:

diff --git a/data/conf/postfix/main.cf b/data/conf/postfix/main.cf
index a445b60c..ed6d4cde 100644
--- a/data/conf/postfix/main.cf
+++ b/data/conf/postfix/main.cf
@@ -198,3 +198,6 @@ parent_domain_matches_subdomains = debug_peer_list,fast_flush_domains,mynetworks

 # DO NOT EDIT ANYTHING BELOW #
 # User overrides #
+
+myhostname = mail.plawetzki.net
+
diff --git a/docker-compose.yml b/docker-compose.yml
index 05a2f9aa..2b339235 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -582,36 +582,6 @@ services:
           aliases:
             - ofelia

-    ipv6nat-mailcow:
-      depends_on:
-        - unbound-mailcow
-        - mysql-mailcow
-        - redis-mailcow
-        - clamd-mailcow
-        - rspamd-mailcow
-        - php-fpm-mailcow
-        - sogo-mailcow
-        - dovecot-mailcow
-        - postfix-mailcow
-        - memcached-mailcow
-        - nginx-mailcow
-        - acme-mailcow
-        - netfilter-mailcow
-        - watchdog-mailcow
-        - dockerapi-mailcow
-        - solr-mailcow
-      environment:
-        - TZ=${TZ}
-      image: robbertkl/ipv6nat
-      security_opt:
-        - label=disable
-      restart: always
-      privileged: true
-      network_mode: "host"
-      volumes:
-        - /var/run/docker.sock:/var/run/docker.sock:ro
-        - /lib/modules:/lib/modules:ro
-
 networks:
   mailcow-network:
     driver: bridge
(END)

Logs of iptables -L -vn:

n.a.

Logs of ip6tables -L -vn:

n.a.

Logs of iptables -L -vn -t nat:

n.a.

Logs of ip6tables -L -vn -t nat:

n.a.

DNS check:

n.a.

DerLinkman commented 1 year ago

That's not a mailcow problem.

Looks like a configuration issue from docker and app armor.

purejava commented 1 year ago

I never touched the AppArmor configuration nor configured docker other than the hints in your documentation - basically added borgmatic back up and that's it.

Downgrading the two packages fixes the problem, upgrading shows the issue again. As you state, you support Debian 11, mailcow should run on a current Debian 11 without issues. Even, if this "is not a mailcow problem".

purejava commented 1 year ago

According to this, AppAmor can be someway half installed.

Just adding the packages apparmor and apparmor-utils to the system fixed the issue for me.

accolon commented 1 year ago

This is a Debian issue: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989781

Either disable AppArmor or install the userland tools for AppArmor since according to the error message your system is missing apparmor_parser:

apt install apparmor-utils

(EDIT: I was too late posting my answer since I got distracted, will leave this here.)

purejava commented 1 year ago

This is a Debian issue:

This is debatable. Since mailcow installation guide points to Docker CE as a requirement in opposite to install docker.io from the Debian repo (which probably wouldn't cause configuration issues) and apparmor is needed in conjunction with Docker CE, the documentation could mention this instead of the devs arguing "it's a Debian issue".

Edit: it's apparmor that contains /sbin/apparmor_parser, not apparmor-utils

DerLinkman commented 1 year ago

I never touched the AppArmor configuration nor configured docker other than the hints in your documentation - basically added borgmatic back up and that's it.

Downgrading the two packages fixes the problem, upgrading shows the issue again. As you state, you support Debian 11, mailcow should run on a current Debian 11 without issues. Even, if this "is not a mailcow problem".

It does. No one else have reported the issue so far exept you. And yes we run 200+ mailcow dockerized Instances for our customers running on Debian 11 rock stable. So that's a bit unfair to say don't you think?

purejava commented 1 year ago

Let's see. The new version of the two Debian packages mentioned above were released just yesterday.

The release notes contain changes regarding AppAmor: https://github.com/moby/moby/pull/42276

purejava commented 1 year ago

So that's a bit unfair to say don't you think?

In my mind it's unfair to answer a bug report to which someone has invested the time to open it by writing: looks like a configuration issue from docker and app armor.

Did you try to reproduce the issue before suspecting what might be behind it?

Probably not, because changing the suspected Debian packages leads to the problem.

I explained in a previous comment, why I think this relates to mailcow.

Every software developer has bugs in his code. Even the Linux kernel maintainers have an average error quote of 1 bug ~every 500 lines of code.

This is the only project on GitHub where developers feel their honor is violated once a bug report is opened.

From my earlier experience with this project I am happy to not being yelled at yet. But I already regret that I was trying to help you.

DerLinkman commented 1 year ago

How /when did I yelled at you? I was only saying that this issue is not a mailcow issue.

And of course did I reproduced it to be exact a few hours earlier roundabout the time I wrote my previous statement about it.

I'm going to close this pr as it seems to be undiscussible in your eyes...

mailcow / mailcow-dockerized