Clamd Container is not Healthy

mritzmann commented 2 years ago

Contribution guidelines

[X] I've read the contribution guidelines and wholeheartedly agree

I've found a bug and checked that ...

[X] ... I understand that not following the below instructions will result in immediate closure and/or deletion of my issue.
[X] ... I have understood that this bug report is dedicated for bugs, and not for support-related inquiries.
[X] ... I have understood that answers are voluntary and community-driven, and not commercial support.
[X] ... I have verified that my issue has not been already answered in the past. I also checked previous issues.

Description

The simplest way to monitor Docker services is to check if all containers are running/healthy.
However, since the new Mailcow update, i think there seems to be a bug here.

Affected image: mailcow/clamd:1.52
The last working image: mailcow/clamd:1.51

After the last update, the Docker Compose service clamd-mailcow is unhealthy.

$ docker-compose ps | grep unhealthy
mailcowdockerized_c   /sbin/tini -g --      Up (unhealthy)   3310/tcp, 7357/tcp

$ docker compose ps | grep unhealthy
mailcowdockerized_clamd-mailcow_1       "/sbin/tini -g -- /c…"   clamd-mailcow       running (unhealthy)   7357/tcp

Otherwise, the image works as intended for me (I have clamd disabled).

Logs

$ docker-compose logs clamd-mailcow
Attaching to mailcow_clamd-mailcow_1
clamd-mailcow_1  | SKIP_CLAMD=y, skipping ClamAV...

Steps to reproduce

The easiest way to reproduce that is the following docker-compose file.

Before update / working state:

Create the following docker-compose.yml

version: '2.1'
services:
clamd-mailcow:
  image: mailcow/clamd:1.51
  environment:
    - SKIP_CLAMD=y

Run docker-compose up -d

Run docker-compose ps

$ docker-compose ps
     Name                      Command             State   Ports
--------------------------------------------------------------------
mailcow_clamd-mailcow_1   /sbin/tini -g -- /clamd.sh   Up

After update / non working state:

Create the following docker-compose.yml

version: '2.1'
services:
clamd-mailcow:
  image: mailcow/clamd:1.52
  environment:
    - SKIP_CLAMD=y

Run docker-compose up -d

Run docker-compose ps

$ docker-compose ps
     Name                      Command                     State                 Ports       
-------------------------------------------------------------------------------------------------
mailcow_clamd-mailcow_1   /sbin/tini -g -- /clamd.sh   Up (health: starting)   3310/tcp, 7357/tcp

System information

Question	Answer
My operating system	Debian GNU/Linux 11 (bullseye)
Is Apparmor, SELinux or similar active?	no
Virtualization technology (KVM, VMware, Xen, etc - LXC and OpenVZ are not supported	KVM
Server/VM specifications (Memory, CPU Cores)	8 GB Memory, 2 CPU
Docker version (`docker version`)	20.10.17
docker-compose version (`docker-compose version`)	1.25.0
mailcow version (git describe --tags `git rev-list --tags --max-count=1`)	2022-06
Reverse proxy (custom solution)	Yes, NGINX

Output of git diff origin/master, any other changes to the code? If so, please post them:

diff --git a/data/conf/postfix/main.cf b/data/conf/postfix/main.cf
index a445b60c..b11fb9df 100644
--- a/data/conf/postfix/main.cf
+++ b/data/conf/postfix/main.cf
@@ -198,3 +198,7 @@ parent_domain_matches_subdomains = debug_peer_list,fast_flush_domains,mynetworks

 # DO NOT EDIT ANYTHING BELOW #
 # User overrides #
+
+myhostname = mail.example.com
+submission_smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3
+smtps_smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3
diff --git a/data/conf/rspamd/local.d/composites.conf b/data/conf/rspamd/local.d/composites.conf
index 337a2eb1..66152a9f 100644
--- a/data/conf/rspamd/local.d/composites.conf
+++ b/data/conf/rspamd/local.d/composites.conf
@@ -29,7 +29,7 @@ SPOOFED_UNAUTH {
 # Only apply to inbound unauthed and not whitelisted
 OLEFY_MACRO {
   expression = "!MAILCOW_AUTH & !MAILCOW_WHITE & OLETOOLS";
-  score = 20.0;
+  score = 5.0;
   policy = "remove_weight";
 }
 # Applies to a content filter map
diff --git a/data/conf/sogo/sogo.conf b/data/conf/sogo/sogo.conf
index 1d6a9d3f..165b65f1 100644
--- a/data/conf/sogo/sogo.conf
+++ b/data/conf/sogo/sogo.conf
@@ -5,7 +5,7 @@
         PrivateDAndTViewer
     );

-    WOWorkersCount = "20";
+    WOWorkersCount = "4";
     SOGoACLsSendEMailNotifications = YES;
     SOGoAppointmentSendEMailNotifications = YES;
     SOGoDraftsFolderName = "Drafts";
diff --git a/docker-compose.yml b/docker-compose.yml
index e3b08637..f87c0933 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -447,7 +447,7 @@ services:
         - /lib/modules:/lib/modules:ro

     watchdog-mailcow:
-      image: mailcow/watchdog:1.96
+      image: mailcow/watchdog:1.97
       dns:
         - ${IPV4_NETWORK:-172.22.1}.254
       tmpfs:
diff --git a/generate_config.sh b/generate_config.sh
index 800f0b53..601d8ca2 100755
--- a/generate_config.sh
+++ b/generate_config.sh
@@ -32,9 +32,6 @@ done
 echo "checking docker compose version...";
 if docker compose >/dev/null 2>&1; then
   echo -e "\e[32mFound Compose v2!\e[0m"
-elif docker-compose version --short | grep -m1 "^2" > /dev/null 2>&1; then
-  echo -e "\e[32mFound Compose v2!\e[0m"
-  COMPOSE_COMMAND="docker-compose"  
 elif docker-compose version --short | grep -m1 "^1" > /dev/null 2>&1; then
   echo -e "\e[33mWARN: Your machine is using Docker-Compose v1!\e[0m"
   echo -e "\e[33mmailcow will drop the Docker-Compose v1 Support in December 2022\e[0m"
diff --git a/helper-scripts/_cold-standby.sh b/helper-scripts/_cold-standby.sh
index 14200954..ca205605 100755
--- a/helper-scripts/_cold-standby.sh
+++ b/helper-scripts/_cold-standby.sh
@@ -88,9 +88,6 @@ function preflight_local_checks() {
   echo "checking docker compose version...";
   if docker compose >/dev/null 2>&1; then
     echo -e "\e[32mFound Compose v2 on local machine!\e[0m"
-  elif docker-compose version --short | grep -m1 "^2" > /dev/null 2>&1; then
-  echo -e "\e[32mFound Compose v2!\e[0m"
-  COMPOSE_COMMAND="docker-compose"  
   elif docker-compose version --short | grep -m1 "^1" > /dev/null 2>&1; then
     echo -e "\e[33mWARN: Your machine is using Docker-Compose v1!\e[0m"
     echo -e "\e[33mmailcow will drop the Docker-Compose v1 Support in December 2022\e[0m"
@@ -150,13 +147,6 @@ function preflight_remote_checks() {
      -t 'docker compose' >/dev/null 2>&1; then
     echo -e "\e[32mFound Compose v2 on remote!\e[0m"
     COMPOSE_COMMAND="docker compose"
-  elif ssh -q -o StrictHostKeyChecking=no \
-      -i "${REMOTE_SSH_KEY}" \
-      ${REMOTE_SSH_HOST} \
-      -p ${REMOTE_SSH_PORT} \
-      -t 'docker-compose version --short' | grep -m1 "^2" > /dev/null 2>&1; then
-    echo -e "\e[32mFound Compose v2!\e[0m"
-    COMPOSE_COMMAND="docker-compose"
   elif ssh -q -o StrictHostKeyChecking=no \
       -i "${REMOTE_SSH_KEY}" \
       ${REMOTE_SSH_HOST} \
diff --git a/helper-scripts/backup_and_restore.sh b/helper-scripts/backup_and_restore.sh
index 8136def3..b89c19d4 100755
--- a/helper-scripts/backup_and_restore.sh
+++ b/helper-scripts/backup_and_restore.sh
@@ -80,9 +80,6 @@ echo "checking docker compose version...";
 if docker compose >/dev/null 2>&1; then
   echo -e "\e[32mFound Compose v2!\e[0m"
   COMPOSE_COMMAND="docker compose"
-elif docker-compose version --short | grep -m1 "^2" > /dev/null 2>&1; then
-  echo -e "\e[32mFound Compose v2!\e[0m"
-  COMPOSE_COMMAND="docker-compose"  
 elif docker-compose version --short | grep -m1 "^1" > /dev/null 2>&1; then
   echo -e "\e[33mWARN: Your machine is using Docker-Compose v1!\e[0m"
   echo -e "\e[33mmailcow will drop the Docker-Compose v1 Support in December 2022\e[0m"
diff --git a/update.sh b/update.sh
index 0b9e12d2..6d01edba 100755
--- a/update.sh
+++ b/update.sh
@@ -49,9 +49,6 @@ echo "checking docker compose version...";
 if docker compose >/dev/null 2>&1; then
   echo -e "\e[32mFound Compose v2!\e[0m"
   COMPOSE_COMMAND="docker compose"
-elif docker-compose version --short | grep -m1 "^2" > /dev/null 2>&1; then
-  echo -e "\e[32mFound Compose v2!\e[0m"
-  COMPOSE_COMMAND="docker-compose"
 elif docker-compose version --short | grep -m1 "^1" > /dev/null 2>&1; then
   echo -e "\e[33mWARN: Your machine is using Docker-Compose v1!\e[0m"
   echo -e "\e[33mmailcow will drop the Docker-Compose v1 Support in December 2022\e[0m"

All third-party firewalls and custom iptables rules are unsupported. Please check the Docker docs about how to use Docker with your own ruleset. Nevertheless, iptabels output can help us to help you: iptables -L -vn:

I can submit later if necessary

ip6tables -L -vn:

I can submit later if necessary

iptables -L -vn -t nat:

I can submit later if necessary

ip6tables -L -vn -t nat:

I can submit later if necessary

DNS problems? Please run docker exec -it $(docker ps -qf name=acme-mailcow) dig +short stackoverflow.com @172.22.1.254 (set the IP accordingly, if you changed the internal mailcow network) and post the output:

$ docker exec -it $(docker ps -qf name=acme-mailcow) dig +short stackoverflow.com @172.22.1.254
151.101.1.69
151.101.129.69
151.101.193.69
151.101.65.69

mritzmann commented 2 years ago

The image mailcow/clamd:1.52 seems to have a predefined healthcheck, but the old image does not.

$ docker inspect mailcow_clamd-mailcow_1 | grep Healthcheck -A7
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "\"clamdcheck.sh\""
                ],
                "StartPeriod": 360000000000
            },
            "Image": "mailcow/clamd:1.52",

But the predefined healthcheck makes no sense if SKIP_CLAMD is set to y.

mritzmann commented 2 years ago

Guess we would have to override the predefined HEALTHCHECK in the Dockerfile with a own script.

If SKIP_CLAMD=n: Execute clamdcheck.sh
If SKIP_CLAMD=y: exit 0

Should I create a pull request for this?

DerLinkman commented 2 years ago

I cannot reproduce that.

In all my testing machines :(

DerLinkman commented 2 years ago

Why are you changing to much of the code files? E.G update.sh? As i see you have docker-compose v 1.25.0 which is the oldest one i´ve ever saw. Maybe you should start with updating that to 1.29.0 first?

mritzmann commented 2 years ago

Why are you changing to much of the code files? E.G update.sh?

The issue template requires a diff with origin/master. The most changes you see are changes that are currently in the master and have not yet been released (no release tag). See: https://github.com/mailcow/mailcow-dockerized/compare/2022-06...5d14baa43a433638dc9500866039eea1546ed926 (diff between latest release and master)

Maybe you should start with updating that to 1.29.0 first?

Same with docker-compose 1.29.2 and docker compose.

$ docker-compose --version
docker-compose version 1.29.2, build 5becea4c

$ docker-compose ps | grep clamd
mailcowdockerized_clamd-mailcow_1       /sbin/tini -g -- /clamd.sh       Up (health: starting)   3310/tcp, 7357/tcp

And after 6m (predefined HEALTHCHECK timeout of the image used as base for mailcow/clamd:1.52):

$ docker-compose ps | grep clamd
mailcowdockerized_clamd-mailcow_1       /sbin/tini -g -- /clamd.sh       Up (unhealthy)   3310/tcp, 7357/tcp

I cannot reproduce that.

Have you tested it with SKIP_CLAMD=y? The problem only exists if mailcow/clamd:1.52 is used and clamd is disabled.

DerLinkman commented 2 years ago

Ah yes now i see that.

You can create a pull request on that yes, but please on the staging branch :)

Thanks

mritzmann commented 2 years ago

See #4634

mritzmann commented 2 years ago

Fixed in latest Release https://github.com/mailcow/mailcow-dockerized/releases/tag/2022-07

mailcow / mailcow-dockerized