mailcow / mailcow-dockerized

mailcow: dockerized - 🐮 + 🐋 = 💕
https://mailcow.email
GNU General Public License v3.0
8.66k stars 1.17k forks source link

Web interface keeps crashing #2332

Closed DJIronic closed 5 years ago

DJIronic commented 5 years ago

My web interface of Mailcow just keeps showing this error after some time:

Fatal error: Uncaught RedisException: Connection refused in /web/inc/prerequisites.inc.php:41 Stack trace: #0 /web/inc/prerequisites.inc.php(41): Redis->connect('redis-mailcow', 6379) #1 /web/index.php(2): require_once('/web/inc/prereq...') #2 {main} thrown in /web/inc/prerequisites.inc.php on line 41

image

the container itself will not crash. My server is just "vanilla" Mailcow-dockerized installation without custom edits.

Some users even reported an invalid SSL certificate detected by their outlook.

Any Ideas? Thanks

andryyy commented 5 years ago

Please see the bug report template. iptables are important here...

DJIronic commented 5 years ago

I am sorry, here are my iptables.

root@MailServer:~/mailcow-dockerized# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
MAILCOW    all  --  anywhere             anywhere

Chain FORWARD (policy DROP)
target     prot opt source               destination
MAILCOW    all  --  anywhere             anywhere
DOCKER-ISOLATION  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (2 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             172.22.1.6           tcp dpt:12345
ACCEPT     tcp  --  anywhere             172.22.1.6           tcp dpt:sieve
ACCEPT     tcp  --  anywhere             172.22.1.6           tcp dpt:pop3s
ACCEPT     tcp  --  anywhere             172.22.1.6           tcp dpt:imaps
ACCEPT     tcp  --  anywhere             172.22.1.6           tcp dpt:imap2
ACCEPT     tcp  --  anywhere             172.22.1.6           tcp dpt:pop3
ACCEPT     tcp  --  anywhere             172.22.1.5           tcp dpt:submission
ACCEPT     tcp  --  anywhere             172.22.1.5           tcp dpt:urd
ACCEPT     tcp  --  anywhere             172.22.1.5           tcp dpt:smtp
ACCEPT     tcp  --  anywhere             172.22.1.9           tcp dpt:mysql

Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain MAILCOW (2 references)
target     prot opt source               destination

I did not edit them. I just used the manual to install mailcow.

By the way, I just started the update script to ensure that I have the latest version of the software and now I am just on "connection refused".

andryyy commented 5 years ago

This is still missing more information... these are not all iptables commands listed in the template. :-(

Any container logs, too? php-fpm-mailcow for example?

System info? Network info?

Okay, there are still 100 things your system can do wrong. It does not matter if it's a clean installation.

DJIronic commented 5 years ago

Your OS (is Apparmor or SELinux active?): Debian 8 64-bit, no, no

Chain FORWARD (policy DROP 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 1006K 166M DOCKER-ISOLATION all -- 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 0 0 DOCKER all -- docker0 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0 833K 141M ACCEPT all -- br-f3a07dd6e7a1 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 74137 4858K DOCKER all -- br-f3a07dd6e7a1 0.0.0.0/0 0.0.0.0/0 98678 20M ACCEPT all -- br-f3a07dd6e7a1 !br-f3a07dd6e7a1 0.0.0.0/0 0.0.0.0/0 68369 4513K ACCEPT all -- br-f3a07dd6e7a1 br-f3a07dd6e7a1 0.0.0.0/0 0.0.0.0/0

Chain OUTPUT (policy ACCEPT 46269 packets, 5338K bytes) pkts bytes target prot opt in out source destination

Chain DOCKER (2 references) pkts bytes target prot opt in out source destination 0 0 ACCEPT tcp -- !br-f3a07dd6e7a1 br-f3a07dd6e7a1 0.0.0.0/0 172.22.1.7 tcp dpt:3306 0 0 ACCEPT tcp -- !br-f3a07dd6e7a1 br-f3a07dd6e7a1 0.0.0.0/0 172.22.1.9 tcp dpt:12345 0 0 ACCEPT tcp -- !br-f3a07dd6e7a1 br-f3a07dd6e7a1 0.0.0.0/0 172.22.1.9 tcp dpt:4190 212 12288 ACCEPT tcp -- !br-f3a07dd6e7a1 br-f3a07dd6e7a1 0.0.0.0/0 172.22.1.9 tcp dpt:995 1895 114K ACCEPT tcp -- !br-f3a07dd6e7a1 br-f3a07dd6e7a1 0.0.0.0/0 172.22.1.9 tcp dpt:993 107 6795 ACCEPT tcp -- !br-f3a07dd6e7a1 br-f3a07dd6e7a1 0.0.0.0/0 172.22.1.9 tcp dpt:143 1 40 ACCEPT tcp -- !br-f3a07dd6e7a1 br-f3a07dd6e7a1 0.0.0.0/0 172.22.1.9 tcp dpt:110 44 2636 ACCEPT tcp -- !br-f3a07dd6e7a1 br-f3a07dd6e7a1 0.0.0.0/0 172.22.1.8 tcp dpt:587 1586 95180 ACCEPT tcp -- !br-f3a07dd6e7a1 br-f3a07dd6e7a1 0.0.0.0/0 172.22.1.8 tcp dpt:465 1923 114K ACCEPT tcp -- !br-f3a07dd6e7a1 br-f3a07dd6e7a1 0.0.0.0/0 172.22.1.8 tcp dpt:25

Chain DOCKER-ISOLATION (1 references) pkts bytes target prot opt in out source destination 0 0 DROP all -- br-f3a07dd6e7a1 docker0 0.0.0.0/0 0.0.0.0/0 0 0 DROP all -- docker0 br-f3a07dd6e7a1 0.0.0.0/0 0.0.0.0/0 1006K 166M RETURN all -- 0.0.0.0/0 0.0.0.0/0 ` root@MailServer:~/mailcow-dockerized# ip6tables -L -vn Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination

root@MailServer:~/mailcow-dockerized# ip6tables -L -vn
Chain INPUT (policy ACCEPT 4 packets, 272 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 26 packets, 2183 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 6 packets, 550 bytes)
 pkts bytes target     prot opt in     out     source               destination
root@MailServer:~/mailcow-dockerized# iptables -L -vn -t nat
Chain PREROUTING (policy ACCEPT 133K packets, 8999K bytes)
 pkts bytes target     prot opt in     out     source               destination
18565  935K DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 12823 packets, 604K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 2335 packets, 172K bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 76953 packets, 5062K bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0
51387 3852K MASQUERADE  all  --  *      !br-f3a07dd6e7a1  172.22.1.0/24        0.0.0.0/0
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.7           172.22.1.7           tcp dpt:3306
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.9           172.22.1.9           tcp dpt:12345
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.9           172.22.1.9           tcp dpt:4190
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.9           172.22.1.9           tcp dpt:995
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.9           172.22.1.9           tcp dpt:993
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.9           172.22.1.9           tcp dpt:143
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.9           172.22.1.9           tcp dpt:110
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.8           172.22.1.8           tcp dpt:587
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.8           172.22.1.8           tcp dpt:465
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.8           172.22.1.8           tcp dpt:25

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  br-f3a07dd6e7a1 *       0.0.0.0/0            0.0.0.0/0
    0     0 DNAT       tcp  --  !br-f3a07dd6e7a1 *       0.0.0.0/0            127.0.0.1            tcp dpt:13306 to:172.22.1.7:3306
    0     0 DNAT       tcp  --  !br-f3a07dd6e7a1 *       0.0.0.0/0            127.0.0.1            tcp dpt:19991 to:172.22.1.9:12345
    0     0 DNAT       tcp  --  !br-f3a07dd6e7a1 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:4190 to:172.22.1.9:4190
  214 12408 DNAT       tcp  --  !br-f3a07dd6e7a1 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:995 to:172.22.1.9:995
 1909  115K DNAT       tcp  --  !br-f3a07dd6e7a1 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:993 to:172.22.1.9:993
  107  6795 DNAT       tcp  --  !br-f3a07dd6e7a1 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:143 to:172.22.1.9:143
    1    40 DNAT       tcp  --  !br-f3a07dd6e7a1 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:110 to:172.22.1.9:110
   45  2696 DNAT       tcp  --  !br-f3a07dd6e7a1 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:587 to:172.22.1.8:587
 1600 96020 DNAT       tcp  --  !br-f3a07dd6e7a1 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:465 to:172.22.1.8:465
 1927  115K DNAT       tcp  --  !br-f3a07dd6e7a1 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:25 to:172.22.1.8:25
root@MailServer:~/mailcow-dockerized# ip6tables -L -vn -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER (2 references) pkts bytes target prot opt in out source destination 0 0 RETURN all -- docker0 0.0.0.0/0 0.0.0.0/0 0 0 RETURN all -- br-f3a07dd6e7a1 0.0.0.0/0 0.0.0.0/0 0 0 DNAT tcp -- !br-f3a07dd6e7a1 0.0.0.0/0 127.0.0.1 tcp dpt:13306 to:172.22.1.7:3306 0 0 DNAT tcp -- !br-f3a07dd6e7a1 0.0.0.0/0 127.0.0.1 tcp dpt:19991 to:172.22.1.9:12345 0 0 DNAT tcp -- !br-f3a07dd6e7a1 0.0.0.0/0 0.0.0.0/0 tcp dpt:4190 to:172.22.1.9:4190 214 12408 DNAT tcp -- !br-f3a07dd6e7a1 0.0.0.0/0 0.0.0.0/0 tcp dpt:995 to:172.22.1.9:995 1909 115K DNAT tcp -- !br-f3a07dd6e7a1 0.0.0.0/0 0.0.0.0/0 tcp dpt:993 to:172.22.1.9:993 107 6795 DNAT tcp -- !br-f3a07dd6e7a1 0.0.0.0/0 0.0.0.0/0 tcp dpt:143 to:172.22.1.9:143 1 40 DNAT tcp -- !br-f3a07dd6e7a1 0.0.0.0/0 0.0.0.0/0 tcp dpt:110 to:172.22.1.9:110 45 2696 DNAT tcp -- !br-f3a07dd6e7a1 0.0.0.0/0 0.0.0.0/0 tcp dpt:587 to:172.22.1.8:587 1600 96020 DNAT tcp -- !br-f3a07dd6e7a1 0.0.0.0/0 0.0.0.0/0 tcp dpt:465 to:172.22.1.8:465 1927 115K DNAT tcp -- !br-f3a07dd6e7a1 0.0.0.0/0 0.0.0.0/0 tcp dpt:25 to:172.22.1.8:25 root@MailServer:~/mailcow-dockerized# ip6tables -L -vn -t nat Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination

Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination root@MailServer:~/mailcow-dockerized# docker exec -it $(docker ps -qf name=acme-mailcow) dig +short stackoverflow.com @172.22.1.254 Error response from daemon: No such container: dig root@MailServer:~/mailcow-dockerized# docker exec -it $(docker ps -qf name=acme-mailcow) dig +short stackoverflow.com @172.22.1.254 Error response from daemon: No such container: dig

and docker exec -it $(docker ps -qf name=acme-mailcow) dig +short stackoverflow.com @1.1.1.1 - output? Timeout?

root@MailServer:~/mailcow-dockerized# docker exec -it $(docker ps -qf name=acme-mailcow) dig +short stackoverflow.com @1.1.1.1
Error response from daemon: No such container: dig

I hope this is all now. the server is still giving timeout on interface and both SMTP and IMAP does not work. Thank you very much for your help.

DJIronic commented 5 years ago

OK.

I updated the Linux OS and then Mailcow again. Webserver an everything is running, but now I got this email from watchdog:

Service was restarted on Fri Feb 15 22:24:21 CET 2019, please check your mailcow installation.

andryyy commented 5 years ago

Something is totally messed up there.

Error response from daemon: No such container: dig

It cannot resolve "docker ps -qf name=acme-mailcow". Or mailcow was not running (I guess that's the case).

andryyy commented 5 years ago

Service was restarted on Fri Feb 15 22:24:21 CET 2019, please check your mailcow installation.

Which service? :)

DJIronic commented 5 years ago

Oh sorry, I missed that it's not in the text but in the header :)

Watchdog: postfix-mailcow hit the error rate limit

andryyy commented 5 years ago

This happened once?

DJIronic commented 5 years ago

Well I know there was one like two months ago. But there were some issues with the network setup above the server at the time, so I blame it on it.

DJIronic commented 5 years ago

image OK, there is definitely some issue. Just got these two emails.

The file attached to SOGo emails has this line in it: CRITICAL - Socket timeout

DJIronic commented 5 years ago

OK. System crashed again.

Update & restart fixed that, but there was an error in the update log. image

andryyy commented 5 years ago

I don't know, definitely not default behavior. I suggest you use a clean, new Linux installation without ufw/firewalld/netfilter rules and enough resources.

DJIronic commented 5 years ago

Got it, but is there any tool/tutorial how to transfer emails, users, domains and basically everything? :) I will migrate the server, but there is a lot of inboxes.

DJIronic commented 5 years ago

Maybe even an option to run two servers on two places "synchronized"? I am pretty much newbie in docker, so some documentation from mailcow on multiserver setup will be cool!

andryyy commented 5 years ago

You could also check why there were active endpoints, first.

Can you post docker ps -q?

DJIronic commented 5 years ago

Here you go. Sorry for delayed answer.

root@MailServer:~# docker ps -q
a7321a45cfee
6f179953ba6a
7943b5ddfa06
b6a9cde97c85
dee0cb5d45d8
c31726271c96
a41285640a1d
58c7939d033b
ebb5c591d724
4e873436d7de
b547d4b22c98
f403004edea9
7e5ff16523a0
f6c095e39b0d
da7b4feca76a
226e04976f4d
cc8ead8fb722
andryyy commented 5 years ago

Oh, ps -a :)

DJIronic commented 5 years ago

image

root@MailServer:~# docker ps -a
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                                                                                                                        NAMES
a7321a45cfee        robbertkl/ipv6nat        "/docker-ipv6nat -..."   4 hours ago         Up 3 hours                                                                                                                                                       mailcowdockerized_ipv6nat_1
6f179953ba6a        mailcow/rspamd:1.34      "/docker-entrypoin..."   4 hours ago         Up 4 hours                                                                                                                                                       mailcowdockerized_rspamd-mailcow_1
7943b5ddfa06        mailcow/acme:1.48        "/sbin/tini -g -- ..."   4 hours ago         Up 4 hours                                                                                                                                                       mailcowdockerized_acme-mailcow_1
b6a9cde97c85        nginx:mainline-alpine    "/bin/sh -c 'envsu..."   4 hours ago         Up 4 hours          0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                                                                                                     mailcowdockerized_nginx-mailcow_1
dee0cb5d45d8        mailcow/netfilter:1.22   "python2 -u /serve..."   4 hours ago         Up 4 hours                                                                                                                                                       mailcowdockerized_netfilter-mailcow_1
c31726271c96        mailcow/phpfpm:1.34      "/docker-entrypoin..."   4 hours ago         Up 4 hours          9000/tcp                                                                                                                                     mailcowdockerized_php-fpm-mailcow_1
a41285640a1d        mailcow/solr:1.2         "/docker-entrypoin..."   4 hours ago         Up 4 hours          8983/tcp                                                                                                                                     mailcowdockerized_solr-mailcow_1
58c7939d033b        mailcow/dockerapi:1.26   "python2 -u /serve..."   4 hours ago         Up 4 hours                                                                                                                                                       mailcowdockerized_dockerapi-mailcow_1
ebb5c591d724        mailcow/unbound:1.6      "/docker-entrypoin..."   4 hours ago         Up 4 hours          53/tcp, 53/udp                                                                                                                               mailcowdockerized_unbound-mailcow_1
4e873436d7de        mailcow/postfix:1.29     "/bin/sh -c 'exec ..."   4 hours ago         Up 4 hours          0.0.0.0:25->25/tcp, 0.0.0.0:465->465/tcp, 0.0.0.0:587->587/tcp, 588/tcp                                                                      mailcowdockerized_postfix-mailcow_1
b547d4b22c98        mailcow/sogo:1.51        "/bin/sh -c 'exec ..."   4 hours ago         Up 4 hours                                                                                                                                                       mailcowdockerized_sogo-mailcow_1
f403004edea9        mailcow/dovecot:1.62     "/docker-entrypoin..."   4 hours ago         Up 4 hours          0.0.0.0:110->110/tcp, 0.0.0.0:143->143/tcp, 0.0.0.0:993->993/tcp, 0.0.0.0:995->995/tcp, 0.0.0.0:4190->4190/tcp, 127.0.0.1:19991->12345/tcp   mailcowdockerized_dovecot-mailcow_1
7e5ff16523a0        memcached:alpine         "docker-entrypoint..."   4 hours ago         Up 4 hours          11211/tcp                                                                                                                                    mailcowdockerized_memcached-mailcow_1
f6c095e39b0d        mailcow/clamd:1.21       "/sbin/tini -g -- ..."   4 hours ago         Up 4 hours                                                                                                                                                       mailcowdockerized_clamd-mailcow_1
da7b4feca76a        mailcow/watchdog:1.34    "/sbin/tini -g -- ..."   4 hours ago         Up 4 hours                                                                                                                                                       mailcowdockerized_watchdog-mailcow_1
226e04976f4d        mariadb:10.2             "docker-entrypoint..."   4 hours ago         Up 4 hours          127.0.0.1:13306->3306/tcp                                                                                                                    mailcowdockerized_mysql-mailcow_1
cc8ead8fb722        redis:5-alpine           "docker-entrypoint..."   2 days ago          Up 4 hours          6379/tcp                                                                                                                                     mailcowdockerized_redis-mailcow_1
DJIronic commented 5 years ago

btw just got another crash notification about SOGo and postfix by the watchdog.

Watchdog: postfix-mailcow hit the error rate limit

In he the file: SMTP OK - 0.283 sec. response time|time=0.282872s;;;0.000000

andryyy commented 5 years ago

I'd change the server. :)

DJIronic commented 5 years ago

Same here. Will create a clean VPS with more space and power. But I need to transfer at least inboxes, DKIM keys and etc (will migrate the IPv4). Is that possible?

DJIronic commented 5 years ago

OK. Is there any chance to migrate the whole system? I mean inboxes, domains and DKIM keys.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.