ARM64 php-fpm upgrade to alpine 3.20 recreates known "empty" /debug startpage issue

ReqX commented 3 days ago

Contribution guidelines

[X] I've read the contribution guidelines and wholeheartedly agree

I've found a bug and checked that ...

[X] ... I understand that not following the below instructions will result in immediate closure and/or deletion of my issue.
[X] ... I have understood that this bug report is dedicated for bugs, and not for support-related inquiries.
[X] ... I have understood that answers are voluntary and community-driven, and not commercial support.
[X] ... I have verified that my issue has not been already answered in the past. I also checked previous issues.

Description

After upgrade to alpine 3.20 php: upgrade to alpine 3.20 (base os) #6106 the bug resolved in e.g. https://github.com/mailcow/mailcow-dockerized/issues/5927 is back.

Manually downgrading to image: mailcow/phpfpm:1.90 temporary solves the issue.

Logs:

php-fpm-mailcow-1  | Waiting for SQL...
php-fpm-mailcow-1  | Uptime: 1  Threads: 2  Questions: 2  Slow queries: 0  Opens: 17  Open tables: 10  Queries per second avg: 2.000
php-fpm-mailcow-1  | Could not get mysql-mailcow container id... trying again
php-fpm-mailcow-1  | Could not get mysql-mailcow container id... trying again
php-fpm-mailcow-1  | Could not get mysql-mailcow container id... trying again
php-fpm-mailcow-1  | Could not get mysql-mailcow container id... trying again
php-fpm-mailcow-1  | MySQL @ a17033cb9ea8021b0c0bbe8cdd081ddea6899972406e3db7bf6c7e2305427ce5
php-fpm-mailcow-1  | SQL upgrade iteration #1
php-fpm-mailcow-1  | MySQL is up-to-date - debug output:
php-fpm-mailcow-1  | { "type": "success", "msg": "mysql_upgrade: already upgraded", "text": "This installation of MariaDB is already upgraded to 10.5.23-MariaDB.\nThere is no need to run mysql_upgrade again for 10.5.26-MariaDB.\nYou can use --force if you still want to run mysql_upgrade\n" }
php-fpm-mailcow-1  | We are master, preparing...
php-fpm-mailcow-1  | Running DB init...
php-fpm-mailcow-1  | Cleaned up memcached
php-fpm-mailcow-1  | Rebuilding domain map in Redis...
php-fpm-mailcow-1  | [09-Nov-2024 15:25:40] NOTICE: fpm is running, pid 1
php-fpm-mailcow-1  | [09-Nov-2024 15:25:40] NOTICE: ready to handle connections
php-fpm-mailcow-1  | 172.22.1.13 -  09/Nov/2024:15:26:30 +0100 "POST /pipe.php" 505
php-fpm-mailcow-1  | [09-Nov-2024 15:26:30] WARNING: [pool system-worker] child 52 said into stderr: "NOTICE: PHP message: QUARANTINE: Message too large: 52 b exceeds 0 b"
php-fpm-mailcow-1  | fd4d:6169:6c63:6f77::12 -  09/Nov/2024:15:27:45 +0100 "POST /pipe.php" 505
php-fpm-mailcow-1  | [09-Nov-2024 15:27:45] WARNING: [pool system-worker] child 52 said into stderr: "NOTICE: PHP message: QUARANTINE: Message too large: 52 b exceeds 0 b"
php-fpm-mailcow-1  | 172.22.1.13 -  09/Nov/2024:15:28:43 +0100 "GET /forwardinghosts.php" 200
php-fpm-mailcow-1  | fd4d:6169:6c63:6f77::12 -  09/Nov/2024:15:28:58 +0100 "POST /pipe.php" 505
php-fpm-mailcow-1  | [09-Nov-2024 15:28:58] WARNING: [pool system-worker] child 53 said into stderr: "NOTICE: PHP message: QUARANTINE: Message too large: 52 b exceeds 0 b"
php-fpm-mailcow-1  | 172.22.1.13 -  09/Nov/2024:15:29:18 +0100 "POST /pipe.php" 505
php-fpm-mailcow-1  | [09-Nov-2024 15:29:18] WARNING: [pool system-worker] child 53 said into stderr: "NOTICE: PHP message: QUARANTINE: Message too large: 52 b exceeds 0 b"
php-fpm-mailcow-1  | fd4d:6169:6c63:6f77::12 -  09/Nov/2024:15:29:40 +0100 "POST /pipe.php" 505
php-fpm-mailcow-1  | [09-Nov-2024 15:29:40] WARNING: [pool system-worker] child 52 said into stderr: "NOTICE: PHP message: QUARANTINE: Message too large: 52 b exceeds 0 b"
php-fpm-mailcow-1  | 172.22.1.13 -  09/Nov/2024:15:30:33 +0100 "POST /pipe.php" 505
php-fpm-mailcow-1  | [09-Nov-2024 15:30:33] WARNING: [pool system-worker] child 52 said into stderr: "NOTICE: PHP message: QUARANTINE: Message too large: 52 b exceeds 0 b"
php-fpm-mailcow-1  | 172.22.1.13 -  09/Nov/2024:15:30:48 +0100 "GET /debug.php" 200

Steps to reproduce:

1. follow normal 2024-11 update on aarch64
2. login to mailcow UI
3. blank page is shown

Which branch are you using?

master

Which architecture are you using?

ARM64 (aarch64)

Operating System:

Ubuntu 24.04.1 LTS

Server/VM specifications:

tested on 2 different installs e.g. 20GB RAM / 3vCores

Is Apparmor, SELinux or similar active?

no

Virtualization technology:

KVM

Docker version:

27.3.1

docker-compose version or docker compose version:

2.29.7

mailcow version:

2024-11

Reverse proxy:

Cloudflare

Logs of git diff:

diff --git a/data/conf/postfix/main.cf b/data/conf/postfix/main.cf
index 6721204c..5443b6e9 100644
--- a/data/conf/postfix/main.cf
+++ b/data/conf/postfix/main.cf
@@ -175,3 +175,37 @@ lmtp_destination_recipient_limit=1

 # DO NOT EDIT ANYTHING BELOW #
 # Overrides #
+
+postscreen_dnsbl_sites = wl.mailspike.net=127.0.0.[18;19;20]*-2
+  hostkarma.junkemailfilter.com=127.0.0.1*-2
+  list.dnswl.org=127.0.[0..255].0*-2
+  list.dnswl.org=127.0.[0..255].1*-4
+  list.dnswl.org=127.0.[0..255].2*-6
+  list.dnswl.org=127.0.[0..255].3*-8
+  ix.dnsbl.manitu.net*2
+  bl.spamcop.net*2
+  bl.suomispam.net*2
+  hostkarma.junkemailfilter.com=127.0.0.2*3
+  hostkarma.junkemailfilter.com=127.0.0.4*2
+  hostkarma.junkemailfilter.com=127.0.1.2*1
+  backscatter.spameatingmonkey.net*2
+  bl.ipv6.spameatingmonkey.net*2
+  bl.spameatingmonkey.net*2
+  b.barracudacentral.org=127.0.0.2*7
+  bl.mailspike.net=127.0.0.2*5
+  bl.mailspike.net=127.0.0.[10;11;12]*4
+  dnsbl.sorbs.net=127.0.0.10*8
+  dnsbl.sorbs.net=127.0.0.5*6
+  dnsbl.sorbs.net=127.0.0.7*3
+  dnsbl.sorbs.net=127.0.0.8*2
+  dnsbl.sorbs.net=127.0.0.6*2
+  dnsbl.sorbs.net=127.0.0.9*2
+  zen.spamhaus.org=127.0.0.[10;11]*8
+  zen.spamhaus.org=127.0.0.[4..7]*6
+  zen.spamhaus.org=127.0.0.3*4
+  zen.spamhaus.org=127.0.0.2*3
+
+# User Overrides
+myhostname = *************.***
+#smtpd_forbid_unauth_pipelining = yes
+smtpd_data_restrictions = reject_unauth_pipelining
diff --git a/docker-compose.yml b/docker-compose.yml
index c462ba88..3c79f5fc 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -113,6 +113,7 @@ services:

     php-fpm-mailcow:
       image: mailcow/phpfpm:1.91
+        #image: mailcow/phpfpm:1.90
       command: "php-fpm -d date.timezone=${TZ} -d expose_php=0"
       depends_on:
         - redis-mailcow
@@ -614,36 +615,6 @@ services:
           aliases:
             - ofelia

-    ipv6nat-mailcow:
-      depends_on:
-        - unbound-mailcow
-        - mysql-mailcow
-        - redis-mailcow
-        - clamd-mailcow
-        - rspamd-mailcow
-        - php-fpm-mailcow
-        - sogo-mailcow
-        - dovecot-mailcow
-        - postfix-mailcow

Logs of iptables -L -vn:

Chain INPUT (policy ACCEPT 1703K packets, 206M bytes)
 pkts bytes target     prot opt in     out     source               destination
  234 18140 MAILCOW    0    --  *      *       0.0.0.0/0            0.0.0.0/0            /* mailcow */

Chain FORWARD (policy ACCEPT 1050 packets, 728K bytes)
 pkts bytes target     prot opt in     out     source               destination
   16  1344 MAILCOW    0    --  *      *       0.0.0.0/0            0.0.0.0/0            /* mailcow */
 2697  788K DOCKER-USER  0    --  *      *       0.0.0.0/0            0.0.0.0/0
 2697  788K DOCKER-ISOLATION-STAGE-1  0    --  *      *       0.0.0.0/0            0.0.0.0/0
 1163  663K ACCEPT     0    --  *      br-mailcow  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
   62  2500 DOCKER     0    --  *      br-mailcow  0.0.0.0/0            0.0.0.0/0
 1472  123K ACCEPT     0    --  br-mailcow !br-mailcow  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     0    --  br-mailcow br-mailcow  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     0    --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     0    --  *      docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     0    --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     0    --  docker0 docker0  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 1939K packets, 360M bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.249         tcp dpt:6379
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.5           tcp dpt:8983
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.6           tcp dpt:3306
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:110
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:143
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:993
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:995
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:4190
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:12345
    1    60 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.253         tcp dpt:25
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.253         tcp dpt:465
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.253         tcp dpt:587
   61  2440 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.13          tcp dpt:80
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.13          tcp dpt:443

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination
 1472  123K DOCKER-ISOLATION-STAGE-2  0    --  br-mailcow !br-mailcow  0.0.0.0/0            0.0.0.0/0
    0     0 DOCKER-ISOLATION-STAGE-2  0    --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
 653K  274M RETURN     0    --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       0    --  *      br-mailcow  0.0.0.0/0            0.0.0.0/0
    0     0 DROP       0    --  *      docker0  0.0.0.0/0            0.0.0.0/0
 367K   32M RETURN     0    --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination
 661K  277M RETURN     0    --  *      *       0.0.0.0/0            0.0.0.0/0

Chain InstanceServices (0 references)
 pkts bytes target     prot opt in     out     source               destination

Chain MAILCOW (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       0    --  *      *       194.169.175.10       0.0.0.0/0
    0     0 DROP       0    --  *      *       80.244.11.0/24       0.0.0.0/0
    0     0 DROP       0    --  *      *       194.169.175.0/24     0.0.0.0/0
    0     0 DROP       0    --  *      *       78.128.113.68        0.0.0.0/0
    0     0 DROP       0    --  *      *       123.20.160.210       0.0.0.0/0
    0     0 DROP       0    --  *      *       87.246.7.34          0.0.0.0/0
    0     0 DROP       0    --  *      *       46.148.40.0/24       0.0.0.0/0
    0     0 DROP       0    --  *      *       178.184.127.103      0.0.0.0/0
    0     0 DROP       0    --  *      *       91.215.85.0/24       0.0.0.0/0
    0     0 DROP       0    --  *      *       113.21.113.176       0.0.0.0/0
    0     0 DROP       0    --  *      *       78.128.113.73        0.0.0.0/0
    0     0 DROP       0    --  *      *       14.186.172.81        0.0.0.0/0
    0     0 DROP       6    --  !br-mailcow br-mailcow  0.0.0.0/0            0.0.0.0/0            /* mailcow isolation */

Logs of ip6tables -L -vn:

Chain INPUT (policy ACCEPT 39848 packets, 145M bytes)
 pkts bytes target     prot opt in     out     source               destination
40035  146M MAILCOW    0    --  *      *       ::/0                 ::/0                 /* mailcow */
40038  146M MAILCOW    0    --  *      *       ::/0                 ::/0                 /* mailcow */
39848  145M MAILCOW    0    --  *      *       ::/0                 ::/0                 /* mailcow */

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
 1193  249K DOCKER-USER  0    --  *      *       ::/0                 ::/0
 1193  249K DOCKER-ISOLATION-STAGE-1  0    --  *      *       ::/0                 ::/0
  592  177K ACCEPT     0    --  *      br-mailcow  ::/0                 ::/0                 ctstate RELATED,ESTABLISHED
    0     0 DOCKER     0    --  *      br-mailcow  ::/0                 ::/0
  601 71650 ACCEPT     0    --  br-mailcow !br-mailcow  ::/0                 ::/0
    0     0 ACCEPT     0    --  br-mailcow br-mailcow  ::/0                 ::/0
 1647  367K MAILCOW    0    --  *      *       ::/0                 ::/0                 /* mailcow */
 1647  367K MAILCOW    0    --  *      *       ::/0                 ::/0                 /* mailcow */
  987  193K MAILCOW    0    --  *      *       ::/0                 ::/0                 /* mailcow */
  987  193K DOCKER-USER  0    --  *      *       ::/0                 ::/0
  987  193K DOCKER-ISOLATION-STAGE-1  0    --  *      *       ::/0                 ::/0
    0     0 ACCEPT     0    --  *      docker0  ::/0                 ::/0                 ctstate RELATED,ESTABLISHED
    0     0 DOCKER     0    --  *      docker0  ::/0                 ::/0
    0     0 ACCEPT     0    --  docker0 !docker0  ::/0                 ::/0
    0     0 ACCEPT     0    --  docker0 docker0  ::/0                 ::/0

Chain OUTPUT (policy ACCEPT 28995 packets, 4105K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::c  tcp dpt:25
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::c  tcp dpt:465
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::c  tcp dpt:587
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::12  tcp dpt:80
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::12  tcp dpt:443

Chain DOCKER-ISOLATION-STAGE-1 (2 references)
 pkts bytes target     prot opt in     out     source               destination
  601 71650 DOCKER-ISOLATION-STAGE-2  0    --  br-mailcow !br-mailcow  ::/0                 ::/0
    0     0 DOCKER-ISOLATION-STAGE-2  0    --  docker0 !docker0  ::/0                 ::/0
53494   10M RETURN     0    --  *      *       ::/0                 ::/0

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       0    --  *      br-mailcow  ::/0                 ::/0
    0     0 DROP       0    --  *      docker0  ::/0                 ::/0
26833 2801K RETURN     0    --  *      *       ::/0                 ::/0

Chain DOCKER-USER (2 references)
 pkts bytes target     prot opt in     out     source               destination
56862   11M RETURN     0    --  *      *       ::/0                 ::/0

Chain MAILCOW (6 references)
 pkts bytes target     prot opt in     out     source               destination

Logs of iptables -L -vn -t nat:

Chain PREROUTING (policy ACCEPT 129K packets, 9894K bytes)
 pkts bytes target     prot opt in     out     source               destination
25410 1075K DOCKER     0    --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 1041 packets, 61489 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 72645 packets, 4924K bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER     0    --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 97463 packets, 5958K bytes)
 pkts bytes target     prot opt in     out     source               destination
  838 76713 MASQUERADE  0    --  *      !br-mailcow  172.22.1.0/24        0.0.0.0/0
    0     0 MASQUERADE  6    --  *      *       172.22.1.9           172.22.1.9           tcp dpt:80
    0     0 MASQUERADE  6    --  *      *       172.22.1.9           172.22.1.9           tcp dpt:443
    0     0 MASQUERADE  6    --  *      *       172.22.1.12          172.22.1.12          tcp dpt:3306
    0     0 MASQUERADE  6    --  *      *       172.22.1.13          172.22.1.13          tcp dpt:8983
    0     0 MASQUERADE  0    --  *      !docker0  172.17.0.0/16        0.0.0.0/0
    0     0 MASQUERADE  6    --  *      *       172.22.1.9           172.22.1.9           tcp dpt:80
    0     0 MASQUERADE  6    --  *      *       172.22.1.9           172.22.1.9           tcp dpt:443
    0     0 MASQUERADE  6    --  *      *       172.22.1.12          172.22.1.12          tcp dpt:3306
    0     0 MASQUERADE  6    --  *      *       172.22.1.13          172.22.1.13          tcp dpt:8983
    0     0 MASQUERADE  6    --  *      *       172.22.1.249         172.22.1.249         tcp dpt:6379
    0     0 MASQUERADE  6    --  *      *       172.22.1.5           172.22.1.5           tcp dpt:8983
    0     0 MASQUERADE  6    --  *      *       172.22.1.6           172.22.1.6           tcp dpt:3306
    0     0 MASQUERADE  6    --  *      *       172.22.1.250         172.22.1.250         tcp dpt:110
    0     0 MASQUERADE  6    --  *      *       172.22.1.250         172.22.1.250         tcp dpt:143
    0     0 MASQUERADE  6    --  *      *       172.22.1.250         172.22.1.250         tcp dpt:993
    0     0 MASQUERADE  6    --  *      *       172.22.1.250         172.22.1.250         tcp dpt:995
    0     0 MASQUERADE  6    --  *      *       172.22.1.250         172.22.1.250         tcp dpt:4190
    0     0 MASQUERADE  6    --  *      *       172.22.1.250         172.22.1.250         tcp dpt:12345
    0     0 MASQUERADE  6    --  *      *       172.22.1.253         172.22.1.253         tcp dpt:25
    0     0 MASQUERADE  6    --  *      *       172.22.1.253         172.22.1.253         tcp dpt:465
    0     0 MASQUERADE  6    --  *      *       172.22.1.253         172.22.1.253         tcp dpt:587
    0     0 MASQUERADE  6    --  *      *       172.22.1.13          172.22.1.13          tcp dpt:80
    0     0 MASQUERADE  6    --  *      *       172.22.1.13          172.22.1.13          tcp dpt:443

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     0    --  br-mailcow *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     0    --  docker0 *       0.0.0.0/0            0.0.0.0/0
    0     0 DNAT       6    --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:7654 to:172.22.1.249:6379
    0     0 DNAT       6    --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:18983 to:172.22.1.5:8983
    0     0 DNAT       6    --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:13306 to:172.22.1.6:3306
    0     0 DNAT       6    --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:110 to:172.22.1.250:110
    0     0 DNAT       6    --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:143 to:172.22.1.250:143
    0     0 DNAT       6    --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:993 to:172.22.1.250:993
    0     0 DNAT       6    --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:995 to:172.22.1.250:995
    0     0 DNAT       6    --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:4190 to:172.22.1.250:4190
    0     0 DNAT       6    --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:19991 to:172.22.1.250:12345
    1    60 DNAT       6    --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:25 to:172.22.1.253:25
    0     0 DNAT       6    --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:465 to:172.22.1.253:465
    0     0 DNAT       6    --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:587 to:172.22.1.253:587
   79  3160 DNAT       6    --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 to:172.22.1.13:80
    0     0 DNAT       6    --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443 to:172.22.1.13:443

Logs of ip6tables -L -vn -t nat:

Chain PREROUTING (policy ACCEPT 27785 packets, 2775K bytes)
 pkts bytes target     prot opt in     out     source               destination
  318 25448 DOCKER     0    --  *      *       ::/0                 ::/0                 ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 278 packets, 22416 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 383 packets, 30960 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER     0    --  *      *       ::/0                !::1                  ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 423 packets, 33992 bytes)
 pkts bytes target     prot opt in     out     source               destination
  588 68362 MASQUERADE  0    --  *      !br-mailcow  fd4d:6169:6c63:6f77::/64  ::/0
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::7  fd4d:6169:6c63:6f77::7  tcp dpt:25
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::a  fd4d:6169:6c63:6f77::a  tcp dpt:80
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::7  fd4d:6169:6c63:6f77::7  tcp dpt:465
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::a  fd4d:6169:6c63:6f77::a  tcp dpt:443
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::7  fd4d:6169:6c63:6f77::7  tcp dpt:587
    0     0 MASQUERADE  0    --  *      !docker0  fd00:dead:beef:c0::/80  ::/0
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::7  fd4d:6169:6c63:6f77::7  tcp dpt:25
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::a  fd4d:6169:6c63:6f77::a  tcp dpt:80
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::7  fd4d:6169:6c63:6f77::7  tcp dpt:465
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::a  fd4d:6169:6c63:6f77::a  tcp dpt:443
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::7  fd4d:6169:6c63:6f77::7  tcp dpt:587
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::c  fd4d:6169:6c63:6f77::c  tcp dpt:25
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::c  fd4d:6169:6c63:6f77::c  tcp dpt:465
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::c  fd4d:6169:6c63:6f77::c  tcp dpt:587
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::12  fd4d:6169:6c63:6f77::12  tcp dpt:80
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::12  fd4d:6169:6c63:6f77::12  tcp dpt:443

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    1    80 RETURN     0    --  br-mailcow *       ::/0                 ::/0
    0     0 RETURN     0    --  docker0 *       ::/0                 ::/0
    0     0 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:25 to:[fd4d:6169:6c63:6f77::c]:25
    0     0 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:465 to:[fd4d:6169:6c63:6f77::c]:465
    0     0 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:587 to:[fd4d:6169:6c63:6f77::c]:587
    0     0 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:80 to:[fd4d:6169:6c63:6f77::12]:80
    0     0 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:443 to:[fd4d:6169:6c63:6f77::12]:443

DNS check:

104.18.32.7
172.64.155.249

stijnos1991 commented 2 days ago

I am also experiencing this bug after the upgrade on an arm64 system. Mail seems to be working luckily.

mitchplze commented 2 days ago

Same issue for me on a Debian 12 host with 2024-11, had to roll back.

lexi-o99 commented 2 days ago

I'm also having this issue on Ubuntu 22.04 LTS host.

FreddleSpl0it commented 1 day ago

I could not replicate the issue on my ARM64 machine.

Can you also post the nginx logs? After navigating to /debug try to navigate to /admin and see if there is an Error Alert in the bottom right corner. If you don't see an Error Alert try setting DEV_MODE=y in mailcow.conf and do a docker compose up -d. Does navigating to /debug now shows a php error? (Disable the DEV_MODE after that)

mitchplze commented 1 day ago

After navigating to /debug and back to /admin, I get the "Array" popup pictured:

Enabling DEV_MODE=y and browsing to /debug gives me:

Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /web/debug.php:32 Stack trace: #0 {main} thrown in /web/debug.php on line 32

NGINX logs from the most recent run, with dev mode:

2024/11/10 23:39:09 [notice] 30#30: gracefully shutting down
2024/11/10 23:39:09 [notice] 34#34: gracefully shutting down
2024/11/10 23:39:09 [notice] 33#33: gracefully shutting down
2024/11/10 23:39:09 [notice] 35#35: gracefully shutting down
2024/11/10 23:39:09 [notice] 28#28: exiting
2024/11/10 23:39:09 [notice] 29#29: gracefully shutting down
2024/11/10 23:39:09 [notice] 31#31: exiting
2024/11/10 23:39:09 [notice] 32#32: gracefully shutting down
2024/11/10 23:39:09 [notice] 30#30: exiting
2024/11/10 23:39:09 [notice] 34#34: exiting
2024/11/10 23:39:09 [notice] 35#35: exiting
2024/11/10 23:39:09 [notice] 29#29: exiting
2024/11/10 23:39:09 [notice] 33#33: exiting
2024/11/10 23:39:09 [notice] 32#32: exiting
2024/11/10 23:39:09 [notice] 26#26: exiting
2024/11/10 23:39:09 [notice] 36#36: exiting
2024/11/10 23:39:09 [notice] 28#28: exit
2024/11/10 23:39:09 [notice] 27#27: exit
2024/11/10 23:39:09 [notice] 35#35: exit
2024/11/10 23:39:09 [notice] 21#21: exit
2024/11/10 23:39:09 [notice] 31#31: exit
2024/11/10 23:39:09 [notice] 25#25: exit
2024/11/10 23:39:09 [notice] 29#29: exit
2024/11/10 23:39:09 [notice] 30#30: exit
2024/11/10 23:39:09 [notice] 33#33: exit
2024/11/10 23:39:09 [notice] 34#34: exit
2024/11/10 23:39:09 [notice] 32#32: exit
2024/11/10 23:39:09 [notice] 20#20: exit
2024/11/10 23:39:09 [notice] 26#26: exit
2024/11/10 23:39:09 [notice] 1#1: signal 17 (SIGCHLD) received from 25
2024/11/10 23:39:09 [notice] 1#1: worker process 25 exited with code 0
2024/11/10 23:39:09 [notice] 1#1: worker process 27 exited with code 0
2024/11/10 23:39:09 [notice] 1#1: worker process 26 exited with code 0
2024/11/10 23:39:09 [notice] 1#1: worker process 30 exited with code 0
2024/11/10 23:39:09 [notice] 1#1: signal 29 (SIGIO) received
2024/11/10 23:39:09 [notice] 1#1: signal 17 (SIGCHLD) received from 27
2024/11/10 23:39:09 [notice] 1#1: signal 17 (SIGCHLD) received from 35
2024/11/10 23:39:09 [notice] 1#1: worker process 35 exited with code 0
2024/11/10 23:39:09 [notice] 1#1: worker process 32 exited with code 0
2024/11/10 23:39:09 [notice] 1#1: signal 29 (SIGIO) received
2024/11/10 23:39:09 [notice] 1#1: signal 17 (SIGCHLD) received from 32
2024/11/10 23:39:09 [notice] 1#1: signal 17 (SIGCHLD) received from 29
2024/11/10 23:39:09 [notice] 1#1: worker process 29 exited with code 0
2024/11/10 23:39:09 [notice] 1#1: signal 29 (SIGIO) received
2024/11/10 23:39:09 [notice] 1#1: signal 17 (SIGCHLD) received from 34
2024/11/10 23:39:09 [notice] 1#1: worker process 34 exited with code 0
2024/11/10 23:39:09 [notice] 1#1: cache manager process 36 exited with code 0
2024/11/10 23:39:09 [notice] 1#1: worker process 31 exited with code 0
2024/11/10 23:39:09 [notice] 1#1: worker process 20 exited with code 0
2024/11/10 23:39:09 [notice] 1#1: worker process 21 exited with code 0
2024/11/10 23:39:09 [notice] 1#1: worker process 33 exited with code 0
2024/11/10 23:39:09 [notice] 1#1: signal 29 (SIGIO) received
2024/11/10 23:39:09 [notice] 1#1: signal 17 (SIGCHLD) received from 36
2024/11/10 23:39:09 [notice] 1#1: signal 17 (SIGCHLD) received from 28
2024/11/10 23:39:09 [notice] 1#1: worker process 28 exited with code 0
2024/11/10 23:39:09 [notice] 1#1: signal 29 (SIGIO) received
2024/11/10 23:39:21 [notice] 1#1: using the "epoll" event method
2024/11/10 23:39:21 [notice] 1#1: nginx/1.27.2
2024/11/10 23:39:21 [notice] 1#1: built by gcc 13.2.1 20240309 (Alpine 13.2.1_git20240309) 
2024/11/10 23:39:21 [notice] 1#1: OS: Linux 6.1.0-25-arm64
2024/11/10 23:39:21 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2024/11/10 23:39:21 [notice] 1#1: start worker processes
2024/11/10 23:39:21 [notice] 1#1: start worker process 20
2024/11/10 23:39:21 [notice] 1#1: start worker process 21
2024/11/10 23:39:21 [notice] 1#1: start worker process 22
2024/11/10 23:39:21 [notice] 1#1: start worker process 23
2024/11/10 23:39:21 [notice] 1#1: start worker process 24
2024/11/10 23:39:21 [notice] 1#1: start worker process 25
2024/11/10 23:39:21 [notice] 1#1: start worker process 26
2024/11/10 23:39:21 [notice] 1#1: start worker process 27
2024/11/10 23:39:21 [notice] 1#1: start worker process 28
2024/11/10 23:39:21 [notice] 1#1: start worker process 29
2024/11/10 23:39:21 [notice] 1#1: start worker process 30
2024/11/10 23:39:21 [notice] 1#1: start worker process 31
2024/11/10 23:39:21 [notice] 1#1: start worker process 32
2024/11/10 23:39:21 [notice] 1#1: start worker process 33
2024/11/10 23:39:21 [notice] 1#1: start worker process 34
2024/11/10 23:39:21 [notice] 1#1: start worker process 35
2024/11/10 23:39:21 [notice] 1#1: start cache manager process 36
2024/11/10 23:39:21 [notice] 1#1: start cache loader process 37
2024/11/10 23:39:22 [error] 21#21: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.22.111.11, server: _, request: "HEAD /settings.php HTTP/1.1", upstream: "fastcgi://172.22.111.7:9001", host: "nginx"
2024/11/10 23:39:22 [error] 21#21: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.22.111.11, server: _, request: "HEAD /settings.php HTTP/1.1", upstream: "fastcgi://[fd4d:6169:6c63:6f77::a]:9001", host: "nginx"
172.22.111.11 - - [10/Nov/2024:23:39:22 -0800] "HEAD /settings.php HTTP/1.1" 502 0 "-" "rspamd-3.10.2"
172.22.111.7 - - [10/Nov/2024:23:39:35 -0800] "GET /settings.php HTTP/1.1" 200 2170 "-" "-"
135.181.111.11 - - [10/Nov/2024:23:39:35 -0800] "GET /admin HTTP/1.1" 499 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1 Safari/605.1.15"
135.181.111.11 - - [10/Nov/2024:23:39:35 -0800] "GET /debug HTTP/1.1" 200 176 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1 Safari/605.1.15"
172.22.111.11 - - [10/Nov/2024:23:39:45 -0800] "HEAD /forwardinghosts.php HTTP/1.1" 200 0 "-" "rspamd-3.10.2"
172.22.111.11 - - [10/Nov/2024:23:39:45 -0800] "GET /forwardinghosts.php HTTP/1.1" 200 27 "-" "rspamd-3.10.2"
fd4d:6169:6c63:6f77::f - - [10/Nov/2024:23:39:46 -0800] "GET /forwardinghosts.php?host=23.94.160.137 HTTP/1.1" 200 19 "-" "curl/7.88.1"
fd4d:6169:6c63:6f77::f - - [10/Nov/2024:23:39:48 -0800] "GET /forwardinghosts.php?host=80.94.95.239 HTTP/1.1" 200 19 "-" "curl/7.88.1"
172.22.111.12 - - [10/Nov/2024:23:40:17 -0800] "GET / HTTP/1.1" 200 15 "-" "check_http/v (nagios-plugins 2.4.5)"
127.0.0.1 - nadia@na0.ca [10/Nov/2024:23:40:17 -0800] "GET /sogo-auth HTTP/1.0" 200 0 "-" "Apple-iPhone14C4/2202.83"
2024/11/10 23:40:21 [notice] 37#37: http file cache: /tmp 0.000M, bsize: 4096
2024/11/10 23:40:21 [notice] 1#1: signal 17 (SIGCHLD) received from 37
2024/11/10 23:40:21 [notice] 1#1: cache loader process 37 exited with code 0
2024/11/10 23:40:21 [notice] 1#1: signal 29 (SIGIO) received
127.0.0.1 - redacted@redacted [10/Nov/2024:23:40:21 -0800] "GET /sogo-auth HTTP/1.0" 200 0 "-" "Apple-iPhone16C1/2202.83"
127.0.0.1 - redacted@redacted [10/Nov/2024:23:40:25 -0800] "GET /sogo-auth HTTP/1.0" 200 0 "-" "Apple-iPhone16C1/2202.83"
172.22.111.11 - - [10/Nov/2024:23:40:26 -0800] "HEAD /forwardinghosts.php HTTP/1.1" 200 0 "-" "rspamd-3.10.2"
172.22.111.11 - - [10/Nov/2024:23:40:26 -0800] "GET /forwardinghosts.php HTTP/1.1" 200 27 "-" "rspamd-3.10.2"

Hope this helps.

FreddleSpl0it commented 1 day ago

please try the following commands. it seems that the broken c-ares package is still in the alpine:3.20 docker image docker compose exec -it php-fpm-mailcow apk update docker compose exec -it php-fpm-mailcow apk upgrade

mitchplze commented 1 day ago

please try the following commands. it seems that the broken c-ares package is still in the alpine:3.20 docker image docker compose exec -it php-fpm-mailcow apk update docker compose exec -it php-fpm-mailcow apk upgrade

No change for me. Performed updates okay, restarted stack, same error in debug mode.

Output from the upgrade was:

root@hs1:/srv/mailcow-dockerized# docker compose exec -it php-fpm-mailcow apk upgrade
Upgrading critical system libraries and apk-tools:
(1/1) Upgrading apk-tools (2.14.4-r0 -> 2.14.4-r1)
Executing busybox-1.36.1-r29.trigger
Continuing the upgrade transaction with new apk-tools:
(1/21) Upgrading libcrypto3 (3.3.2-r0 -> 3.3.2-r1)
(2/21) Upgrading libssl3 (3.3.2-r0 -> 3.3.2-r1)
(3/21) Upgrading c-ares (1.28.1-r0 -> 1.33.1-r0)
(4/21) Upgrading libcurl (8.9.1-r2 -> 8.11.0-r1)
(5/21) Upgrading curl (8.9.1-r2 -> 8.11.0-r1)
(6/21) Upgrading libexpat (2.6.3-r0 -> 2.6.4-r0)
(7/21) Upgrading librsvg (2.58.0-r0 -> 2.58.5-r0)
Executing librsvg-2.58.5-r0.post-upgrade
(8/21) Upgrading mariadb-common (10.11.8-r0 -> 10.11.10-r0)
Executing mariadb-common-10.11.10-r0.post-upgrade
(9/21) Upgrading mariadb-client (10.11.8-r0 -> 10.11.10-r0)
(10/21) Upgrading mysql-client (10.11.8-r0 -> 10.11.10-r0)
(11/21) Upgrading openssl (3.3.2-r0 -> 3.3.2-r1)
(12/21) Upgrading samba-util-libs (4.19.6-r0 -> 4.19.9-r0)
(13/21) Upgrading libwbclient (4.19.6-r0 -> 4.19.9-r0)
(14/21) Upgrading ldb (2.8.0-r1 -> 2.8.2-r0)
(15/21) Upgrading samba-libs (4.19.6-r0 -> 4.19.9-r0)
(16/21) Upgrading samba-common (4.19.6-r0 -> 4.19.9-r0)
(17/21) Upgrading libarchive (3.7.6-r0 -> 3.7.7-r0)
(18/21) Upgrading libauth-samba (4.19.6-r0 -> 4.19.9-r0)
(19/21) Upgrading samba-client-libs (4.19.6-r0 -> 4.19.9-r0)
(20/21) Upgrading libsmbclient (4.19.6-r0 -> 4.19.9-r0)
(21/21) Upgrading samba-client (4.19.6-r0 -> 4.19.9-r0)
Executing busybox-1.36.1-r29.trigger
Executing ca-certificates-20240705-r0.trigger
Executing gdk-pixbuf-2.42.12-r0.trigger
OK: 253 MiB in 174 packages

FreddleSpl0it commented 1 day ago

don't restart the stack after the apk update. just update and try if the issue is resolved

mitchplze commented 1 day ago

Forgot to mention, I did try it before restarting, no joy. Re-running your apk upgrades says up-to-date, even after restart.

DerLinkman commented 1 day ago

@mitchplze Which timezone did you setup inside mailcow.conf?

mitchplze commented 1 day ago

@mitchplze Which timezone did you setup inside mailcow.conf?

America/Vancouver

FreddleSpl0it commented 1 day ago

@mitchplze since I can't reproduce the issue, could you try out this fix? Replace the fixed code here https://github.com/mailcow/mailcow-dockerized/blob/0a58aa293a69d15f4f0c5385cc57d2a55ca7295b/data/web/debug.php#L25-L54

// containers
$containers = (array) docker('info');
if ($clamd_status === false) unset($containers['clamd-mailcow']);
if ($solr_status === false) unset($containers['solr-mailcow']);
ksort($containers);
foreach ($containers as $container => $container_info) {
  date_default_timezone_set('UTC');
  if (isset($container_info['State']) && is_array($container_info['State']) && isset($container_info['State']['StartedAt'])){
    $StartedAt = date_parse($container_info['State']['StartedAt']);
  } else {
    $StartedAt = null;
  }
  if (isset($StartedAt) && $StartedAt['hour'] !== false) {
    $date = new \DateTime();
    $date->setTimestamp(mktime(
      $StartedAt['hour'],
      $StartedAt['minute'],
      $StartedAt['second'],
      $StartedAt['month'],
      $StartedAt['day'],
      $StartedAt['year']));
    try {
      $user_tz = new DateTimeZone(getenv('TZ'));
      $date->setTimezone($user_tz);
      $started = $date->format('r');
    } catch(Exception $e) {
      $started = '?';
    }
  }
  else {
    $started = '?';
  }
  $containers[$container]['State']['StartedAtHR'] = $started;
}

There is probably a container now showing ? as the starting time on the debug page

andzeil commented 1 day ago

@FreddleSpl0it: not @mitchplze, but I got the same problem on an x64 machine.

docker compose exec -it php-fpm-mailcow apk update
docker compose exec -it php-fpm-mailcow apk upgrade

didn't help, and the changes in debug.php unfortunately seem to not fix the problem, either.

EinPinsel commented 1 day ago

I'm in the same boat. If you wait (or forget) your tab for a while, I got greeted with a very strange page.

mitchplze commented 1 day ago

@mitchplze since I can't reproduce the issue, could you try out this fix? Replace the fixed code here

There is probably a container now showing ? as the starting time on the debug page

@FreddleSpl0it: I applied your suggested changes to debug.php, commenting out the existing block altogether, and adding yours below.

No change really. debug.php shows: Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /web/debug.php:88 Stack trace: #0 {main} thrown in /web/debug.php on line 88

Restarted stack completely, no change

Btw not sure if needed, but my host:

root@hs1:/srv/mailcow-dockerized# uname -a
Linux hs1 6.1.0-25-arm64 #1 SMP Debian 6.1.106-3 (2024-08-26) aarch64 GNU/Linux

ilya9876 commented 22 hours ago

I have the same issue so it seems not to be ARM realted. /debug after login is a blank page; typing /admin shows the admin page; but clicking around on the admin page results in: An unknown error occured: TypeError Object( [message:protected] => Cannot access offset of type string on string [string:Error:private] => [code:protected] => 0 [file:protected] => /web/debug.php [line:protected] => 32 [trace:Error:private] => Array ( ) [previous:Error:private] => ) My system is as following: ubuntu@mailcow:/opt/mailcow-dockerized$ uname -a Linux mailcow 6.8.0-1015-oracle #15~22.04.1-Ubuntu SMP Wed Oct 9 15:47:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Update:

every other URL except /debug seems to work correctly
/queue is working, /admin is working, /quarantine is working /sogo is working
the error message I get is not connected to some certain switches between config screens, it seems to show up random

I suggest to remove ARM64 from the issue subject to better address this issue.

FreddleSpl0it commented 21 hours ago

@mitchplze i need to see the $containers value to debug this. Could you add the following on line 61 before the foreach loop and send me the file?

file_put_contents("/web/templates/cache/containers_res.json", json_encode($containers, JSON_PRETTY_PRINT));

ilya9876 commented 21 hours ago

Please be aware that the solution seems to be also discussed here: https://github.com/mailcow/mailcow-dockerized/issues/5927 This hint helped me, all working fine now: Editing /opt/mailcow-dockerized/docker-compose.yml and changing line: php-fpm-mailcow: image: mailcow/phpfpm:1.91 to php-fpm-mailcow: image: mailcow/phpfpm:1.90

then saving the file and running ./update.sh again fixed the issue for me. Thanks @ajnadox for providing

DerLinkman commented 21 hours ago

Please be aware that the solution seems to be also discussed here: #5927 This hint helped me, all working fine now: Editing /opt/mailcow-dockerized/docker-compose.yml and changing line: php-fpm-mailcow: image: mailcow/phpfpm:1.91 to php-fpm-mailcow: image: mailcow/phpfpm:1.90

then saving the file and running ./update.sh again fixed the issue for me. Thanks @ajnadox for providing

That is not a solution, only a temporary workaround.

ilya9876 commented 21 hours ago

Please be aware that the solution seems to be also discussed here: #5927 This hint helped me, all working fine now: Editing /opt/mailcow-dockerized/docker-compose.yml and changing line: php-fpm-mailcow: image: mailcow/phpfpm:1.91 to php-fpm-mailcow: image: mailcow/phpfpm:1.90 then saving the file and running ./update.sh again fixed the issue for me. Thanks @ajnadox for providing

That is not a solution, only a temporary workaround.

Agree on this

FreddleSpl0it commented 14 hours ago

I'm still not sure what the problem is, but I've added a workaround and hope it helps to display the /debug page https://github.com/mailcow/mailcow-dockerized/pull/6160/files#diff-1ce35b9f8d77569adc5c31790cf2f8dd8bc9646a23a03174d23b2984c954325d.

mitchplze commented 11 hours ago

@mitchplze i need to see the $containers value to debug this. Could you add the following on line 61 before the foreach loop and send me the file?
file_put_contents("/web/templates/cache/containers_res.json", json_encode($containers, JSON_PRETTY_PRINT));

Thank you.

@FreddleSpl0it: Sorry but that doesn't appear to do what you need. I checked that folder and there is no output file as expected.

I also checked after a full stack restart and no change.

mitchplze commented 11 hours ago

I'm still not sure what the problem is, but I've added a workaround and hope it helps to display the /debug page https://github.com/mailcow/mailcow-dockerized/pull/6160/files#diff-1ce35b9f8d77569adc5c31790cf2f8dd8bc9646a23a03174d23b2984c954325d.

I just upgraded to 2024-11a, and my Mailcow is now working with a blank override file (not overriding to mailcow/phpfpm:1.90)!

Which I think means I'm now on 1.91.1 and works ok.

stijnos1991 commented 11 hours ago

Can confirm after the update the issue appears to be resolved.

andzeil commented 11 hours ago

Yup, same here. Thanks for the quick delivery of the fix, appreciated!

ReqX commented 10 hours ago

Dear all, Sorry for late reply from my side, was on a business trip. Thx everyone for stepping in and of course for the fast fix.

Can also confirm that upgrading and reverting manual downgrade fixed it.

Are we fine to close?

mitchplze commented 8 hours ago

Can also confirm that upgrading and reverting manual downgrade fixed it.

Are we fine to close?

I believe today's patch to 2024-11a only works-around the problem temporarily, to un-break the product. AFAIU, devs still need to figure out what's causing the issue.

mailcow / mailcow-dockerized