mailcow / mailcow-dockerized

mailcow: dockerized - 🐮 + 🐋 = 💕
https://mailcow.email
GNU General Public License v3.0
8.95k stars 1.17k forks source link

DNS, Redis and MySQL errors after fresh install #4416

Closed SoulofAkuma closed 2 years ago

SoulofAkuma commented 2 years ago

Prior to placing the issue, please check following: (fill out each checkbox with an X once done)

Summary

After the old "undockerized" version of mailcow, which was set up on the server by someone else, stopped working, I decided to set up the dockerized mailcow version three days ago and since then I've tried to fix the issues I've had with the dockerized one, but no luck. I've first tried to set it up with apache as a proxy but after apache gave me repeated proxy errors I've tried it without. Still, trying to curl the HTTP port still got me a curl: (56) Recv failure: Connection reset by peer. If I use my browser, I get an ERR_CONNECTION_RESET or ERR_CONNECTION_CLOSED. Telnet also doesn't work (telnet: Unable to connect to remote host: Connection refused). The logs repeatedly show MySQL, Redis, and DNS errors for various containers. I've attached the output of nestat -tulpn at the end of the issue, to show that docker is listening on these ports, but that's all that shows any sign of success on my machine network-wise. I'll happily provide any required information.

Logs

Logs from postfix and dovecot related to DNS (these occur periodically in the logs)

dovecot-mailcow_1    | Waiting for DNS...
postfix-mailcow_1    | Waiting for DNS...

Logs from sogo and ofelia related to Redis (watchdog and acme repeatedly print Waiting for... redis) (these also repeatedly occur periodically in the logs)

ofelia-mailcow_1     | 2022-01-07T19:34:14.024Z  common.go:125 NOTICE [Job "dovecot_trim_logs" (9f898a4414ca)] StdOut: Command redis-cli -h redis -p 6379 LTRIM ACME_LOG 0 9999 failed to execute, exit code was 1
ofelia-mailcow_1     | Command redis-cli -h redis -p 6379 LTRIM POSTFIX_MAILLOG 0 9999 failed to execute, exit code was 1
ofelia-mailcow_1     | Command redis-cli -h redis -p 6379 LTRIM DOVECOT_MAILLOG 0 9999 failed to execute, exit code was 1
ofelia-mailcow_1     | Command redis-cli -h redis -p 6379 LTRIM SOGO_LOG 0 9999 failed to execute, exit code was 1
ofelia-mailcow_1     | Command redis-cli -h redis -p 6379 LTRIM NETFILTER_LOG 0 9999 failed to execute, exit code was 1
ofelia-mailcow_1     | Command redis-cli -h redis -p 6379 LTRIM AUTODISCOVER_LOG 0 9999 failed to execute, exit code was 1
ofelia-mailcow_1     | Command redis-cli -h redis -p 6379 LTRIM API_LOG 0 9999 failed to execute, exit code was 1
ofelia-mailcow_1     | Command redis-cli -h redis -p 6379 LTRIM RL_LOG 0 9999 failed to execute, exit code was 1
ofelia-mailcow_1     | Command redis-cli -h redis -p 6379 LTRIM WATCHDOG_LOG 0 9999 failed to execute, exit code was 1
ofelia-mailcow_1     | 2022-01-07T19:34:14.024Z  common.go:125 NOTICE [Job "dovecot_trim_logs" (9f898a4414ca)] StdErr: /usr/local/bin/trim_logs.sh: line 10: /source_env.sh: No such file or directory
ofelia-mailcow_1     | Could not connect to Redis at redis:6379: Connection timed out
ofelia-mailcow_1     | Could not connect to Redis at redis:6379: Connection timed out
ofelia-mailcow_1     | Could not connect to Redis at redis:6379: Connection timed out
ofelia-mailcow_1     | Could not connect to Redis at redis:6379: Connection timed out
ofelia-mailcow_1     | Could not connect to Redis at redis:6379: Connection timed out
ofelia-mailcow_1     | Could not connect to Redis at redis:6379: Connection timed out
ofelia-mailcow_1     | Could not connect to Redis at redis:6379: Connection timed out
ofelia-mailcow_1     | Could not connect to Redis at redis:6379: Connection timed out
ofelia-mailcow_1     | Could not connect to Redis at redis:6379: Connection timed out
ofelia-mailcow_1     | 2022-01-07T19:34:14.024Z  common.go:125 NOTICE [Job "dovecot_trim_logs" (9f898a4414ca)] Finished in "19m38.014201362s", failed: false, skipped: false, error: none
sogo-mailcow_1       | Jan  7 20:33:54 8bc396398bd0 syslog-ng[8]: REDIS server error during connection; driver='d_redis_ui_log#0', error='Connection timed out', time_reopen='60'
sogo-mailcow_1       | Jan  7 20:33:54 8bc396398bd0 syslog-ng[8]: REDIS server error during connection; driver='d_redis_f2b_channel#0', error='Connection timed out', time_reopen='60'

Logs from sogo related to MySQL error (these also repeatedly occur periodically in the logs):

| Waiting for schema update...
sogo-mailcow_1       | ERROR 1146 (42S02) at line 1: Table 'mailcow.versions' doesn't exist

Reproduction

I was not able to reproduce the issue on another machine. I've tried it on another local development VM and it worked there. However, on the machine it isn't working on, I've been able to reproduce it every time. As to what I've done to fix it, here's a list:

System information

Question Answer
My operating system Debian 10
Is Apparmor, SELinux or similar active? No
Virtualization technlogy (KVM, VMware, Xen, etc - LXC and OpenVZ are not supported VMWare
Server/VM specifications (Memory, CPU Cores) 8GB RAM, 2 Cores
Docker Version (docker version) 20.10.12
Docker-Compose Version (docker-compose version) 1.29.2
Reverse proxy (custom solution) No

Output from iptables -L -vn

Chain INPUT (policy ACCEPT 12M packets, 5771M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      br-mailcow  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      br-mailcow  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  br-mailcow !br-mailcow  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  br-mailcow br-mailcow  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 11M packets, 2681M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  br-mailcow !br-mailcow  0.0.0.0/0            0.0.0.0/0           
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    2   448 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      br-mailcow  0.0.0.0/0            0.0.0.0/0           
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           
# Warning: iptables-legacy tables present, use iptables-legacy to see them
root@mail:/home/sysadmin# iptables -L -vn
Chain INPUT (policy ACCEPT 12M packets, 5772M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      br-mailcow  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      br-mailcow  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  br-mailcow !br-mailcow  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  br-mailcow br-mailcow  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 11M packets, 2682M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.5           tcp dpt:8983
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.249         tcp dpt:6379
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.8           tcp dpt:3306
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.253         tcp dpt:587
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:12345
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:4190
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.253         tcp dpt:465
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:995
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.253         tcp dpt:25
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.10          tcp dpt:443
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:993
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.10          tcp dpt:80
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:143
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:110

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  br-mailcow !br-mailcow  0.0.0.0/0            0.0.0.0/0           
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    2   448 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      br-mailcow  0.0.0.0/0            0.0.0.0/0           
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           
# Warning: iptables-legacy tables present, use iptables-legacy to see them

Output from iptables -L -vn -t nat

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  *      !br-mailcow  172.22.1.0/24        0.0.0.0/0           
    0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0           
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.5           172.22.1.5           tcp dpt:8983
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.249         172.22.1.249         tcp dpt:6379
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.8           172.22.1.8           tcp dpt:3306
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.253         172.22.1.253         tcp dpt:587
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:12345
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:4190
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.253         172.22.1.253         tcp dpt:465
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:995
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.253         172.22.1.253         tcp dpt:25
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.10          172.22.1.10          tcp dpt:443
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:993
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.10          172.22.1.10          tcp dpt:80
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:143
    0     0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:110

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  br-mailcow *       0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0           
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:18983 to:172.22.1.5:8983
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:7654 to:172.22.1.249:6379
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:13306 to:172.22.1.8:3306
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:587 to:172.22.1.253:587
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:19991 to:172.22.1.250:12345
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:4190 to:172.22.1.250:4190
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:465 to:172.22.1.253:465
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:995 to:172.22.1.250:995
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:25 to:172.22.1.253:25
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443 to:172.22.1.10:443
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:993 to:172.22.1.250:993
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 to:172.22.1.10:80
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:143 to:172.22.1.250:143
    0     0 DNAT       tcp  --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:110 to:172.22.1.250:110
# Warning: iptables-legacy tables present, use iptables-legacy to see them

Output from ip6tables -L -vn

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
76036 5141K RETURN     all      *      *       ::/0                 ::/0                

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::d  tcp dpt:25
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::d  tcp dpt:465
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::d  tcp dpt:587
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::e  tcp dpt:4190
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::e  tcp dpt:993
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::e  tcp dpt:995
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::e  tcp dpt:110
    0     0 ACCEPT     tcp      !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::e  tcp dpt:143

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-ISOLATION-STAGE-2  all      br-mailcow !br-mailcow  ::/0                 ::/0                
 1376 81348 RETURN     all      *      *       ::/0                 ::/0                

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all      *      br-mailcow  ::/0                 ::/0                
    0     0 RETURN     all      *      *       ::/0                 ::/0                

Chain FORWARD (policy ACCEPT 15 packets, 840 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 1360 80452 DOCKER-USER  all      *      *       ::/0                 ::/0                
77034 5216K DOCKER-ISOLATION-STAGE-1  all      *      *       ::/0                 ::/0                
77034 5216K DOCKER     all      *      br-mailcow  ::/0                 ::/0                
  383 25935 ACCEPT     all      *      br-mailcow  ::/0                 ::/0                 ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all      br-mailcow !br-mailcow  ::/0                 ::/0                
76636 5189K ACCEPT     all      br-mailcow br-mailcow  ::/0                 ::/0                

Chain INPUT (policy ACCEPT 4891 packets, 1253K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 6148 packets, 1447K bytes)
 pkts bytes target     prot opt in     out     source               destination         
# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them

Output from ip6tables -L -vn -t nat

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all      br-mailcow *       ::/0                 ::/0                
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:25 to:[fd4d:6169:6c63:6f77::d]:25
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:465 to:[fd4d:6169:6c63:6f77::d]:465
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:587 to:[fd4d:6169:6c63:6f77::d]:587
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:4190 to:[fd4d:6169:6c63:6f77::e]:4190
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:993 to:[fd4d:6169:6c63:6f77::e]:993
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:995 to:[fd4d:6169:6c63:6f77::e]:995
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:110 to:[fd4d:6169:6c63:6f77::e]:110
    0     0 DNAT       tcp      !br-mailcow *       ::/0                 ::/0                 tcp dpt:143 to:[fd4d:6169:6c63:6f77::e]:143

Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all      *      *       ::/0                 ::/0                 ADDRTYPE match dst-type LOCAL

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all      *      *       ::/0                !::1                  ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all      *      br-mailcow  ::/0                 ::/0                 ADDRTYPE match dst-type LOCAL
    0     0 MASQUERADE  all      *      !br-mailcow  fd4d:6169:6c63:6f77::  ::/0                
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::f  fd4d:6169:6c63:6f77::f  tcp dpt:587
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::f  fd4d:6169:6c63:6f77::f  tcp dpt:25
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::f  fd4d:6169:6c63:6f77::f  tcp dpt:465
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::b  fd4d:6169:6c63:6f77::b  tcp dpt:25
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::b  fd4d:6169:6c63:6f77::b  tcp dpt:465
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::b  fd4d:6169:6c63:6f77::b  tcp dpt:587
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::d  fd4d:6169:6c63:6f77::d  tcp dpt:143
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::d  fd4d:6169:6c63:6f77::d  tcp dpt:4190
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::d  fd4d:6169:6c63:6f77::d  tcp dpt:993
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::d  fd4d:6169:6c63:6f77::d  tcp dpt:995
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::d  fd4d:6169:6c63:6f77::d  tcp dpt:110
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::e  fd4d:6169:6c63:6f77::e  tcp dpt:110
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::e  fd4d:6169:6c63:6f77::e  tcp dpt:143
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::e  fd4d:6169:6c63:6f77::e  tcp dpt:4190
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::e  fd4d:6169:6c63:6f77::e  tcp dpt:993
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::e  fd4d:6169:6c63:6f77::e  tcp dpt:995
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::d  fd4d:6169:6c63:6f77::d  tcp dpt:25
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::d  fd4d:6169:6c63:6f77::d  tcp dpt:465
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::d  fd4d:6169:6c63:6f77::d  tcp dpt:587
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::e  fd4d:6169:6c63:6f77::e  tcp dpt:587
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::e  fd4d:6169:6c63:6f77::e  tcp dpt:25
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::e  fd4d:6169:6c63:6f77::e  tcp dpt:465
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::f  fd4d:6169:6c63:6f77::f  tcp dpt:4190
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::f  fd4d:6169:6c63:6f77::f  tcp dpt:993
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::f  fd4d:6169:6c63:6f77::f  tcp dpt:995
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::f  fd4d:6169:6c63:6f77::f  tcp dpt:110
    0     0 MASQUERADE  tcp      *      *       fd4d:6169:6c63:6f77::f  fd4d:6169:6c63:6f77::f  tcp dpt:143

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them

I've made no network modifications, but there are DNS problems.

docker exec -it $(docker ps -qf name=acme-mailcow) dig +short stackoverflow.com @172.22.1.254`
;; connection timed out; no servers could be reached

Output of systemctl --type=service | grep running

  acpid.service                      loaded active     running      ACPI event daemon                                                 
  atd.service                        loaded active     running      Deferred execution scheduler                                      
  containerd.service                 loaded active     running      containerd container runtime                                      
  cron.service                       loaded active     running      Regular background program processing daemon                      
  dbus.service                       loaded active     running      D-Bus System Message Bus                                          
  docker.service                     loaded active     running      Docker Application Container Engine                               
  getty@tty1.service                 loaded active     running      Getty on tty1                                                     
  gitlab-runsvdir.service            loaded active     running      GitLab Runit supervision process                                  
  irqbalance.service                 loaded active     running      irqbalance daemon                                                 
  mariadb.service                    loaded active     running      MariaDB 10.3.29 database server                                   
  open-vm-tools.service              loaded active     running      Service for virtual machines hosted on VMware                     
  php7-fpm.service                   loaded active     running      LSB: starts php7-fpm                                              
  php7.0-fpm.service                 loaded active     running      The PHP 7.0 FastCGI Process Manager                               
  php7.4-fpm.service                 loaded active     running      The PHP 7.4 FastCGI Process Manager                               
  polkit.service                     loaded active     running      Authorization Manager                                             
  rsyslog.service                    loaded active     running      System Logging Service                                            
  ssh.service                        loaded active     running      OpenBSD Secure Shell server                                       
  systemd-journald.service           loaded active     running      Journal Service                                                   
  systemd-logind.service             loaded active     running      Login Service                                                     
  systemd-timesyncd.service          loaded active     running      Network Time Synchronization                                      
  systemd-udevd.service              loaded active     running      udev Kernel Device Manager                                        
  user@1001.service                  loaded active     running      User Manager for UID 1001                                         
  vgauth.service                     loaded active     running      Authentication service for virtual machines hosted on VMware      
  vsftpd.service                     loaded active     running      vsftpd FTP server

Output of netstat -tulpn

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:13306         0.0.0.0:*               LISTEN      28138/docker-proxy  
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      28949/docker-proxy  
tcp        0      0 127.0.0.1:8828          0.0.0.0:*               LISTEN      18678/node          
tcp        0      0 0.0.0.0:4190            0.0.0.0:*               LISTEN      28786/docker-proxy  
tcp        0      0 0.0.0.0:993             0.0.0.0:*               LISTEN      28981/docker-proxy  
tcp        0      0 0.0.0.0:995             0.0.0.0:*               LISTEN      28863/docker-proxy  
tcp        0      0 127.0.0.1:7654          0.0.0.0:*               LISTEN      27808/docker-proxy  
tcp        0      0 127.0.0.1:18983         0.0.0.0:*               LISTEN      27610/docker-proxy  
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      4187/mysqld         
tcp        0      0 0.0.0.0:587             0.0.0.0:*               LISTEN      28708/docker-proxy  
tcp        0      0 0.0.0.0:110             0.0.0.0:*               LISTEN      29165/docker-proxy  
tcp        0      0 0.0.0.0:143             0.0.0.0:*               LISTEN      29118/docker-proxy  
tcp        0      0 127.0.0.1:9007          0.0.0.0:*               LISTEN      885/php-fpm: master 
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      29023/docker-proxy  
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      1150/unicorn master 
tcp        0      0 0.0.0.0:465             0.0.0.0:*               LISTEN      28824/docker-proxy  
tcp        0      0 127.0.0.1:34197         0.0.0.0:*               LISTEN      18599/node          
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      3351/sshd           
tcp        0      0 127.0.0.1:19991         0.0.0.0:*               LISTEN      28758/docker-proxy  
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      28890/docker-proxy  
tcp6       0      0 :::443                  :::*                    LISTEN      28956/docker-proxy  
tcp6       0      0 :::4190                 :::*                    LISTEN      28794/docker-proxy  
tcp6       0      0 :::993                  :::*                    LISTEN      28988/docker-proxy  
tcp6       0      0 :::995                  :::*                    LISTEN      28873/docker-proxy  
tcp6       0      0 :::587                  :::*                    LISTEN      28713/docker-proxy  
tcp6       0      0 :::110                  :::*                    LISTEN      29172/docker-proxy  
tcp6       0      0 :::143                  :::*                    LISTEN      29129/docker-proxy  
tcp6       0      0 :::80                   :::*                    LISTEN      29029/docker-proxy  
tcp6       0      0 :::465                  :::*                    LISTEN      28829/docker-proxy  
tcp6       0      0 :::21                   :::*                    LISTEN      19926/vsftpd        
tcp6       0      0 :::22                   :::*                    LISTEN      3351/sshd           
tcp6       0      0 :::25                   :::*                    LISTEN      28895/docker-proxy
milkmaker commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

SoulofAkuma commented 2 years ago

Am I missing the obvious here or is it quite the contrary? Does anyone have an idea what could cause all these errors?

milkmaker commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

eddiewebb commented 1 year ago

uhhh, anyone want to respond instead of marking it stale?

dodedodo commented 1 year ago

Given the failed writes/missing file I'd say take a look at permissions and check if you're not out of diskspace. Maybe you're running docker as non-root?

Try asking around on the mailcow telegram, there are a lot of smart people active there.

eddiewebb commented 1 year ago

Thanks @dodedodo - it was actually Ubunutu snap's installation of docker/Lxc, tore that out and things were fine.

XVII commented 1 year ago

@eddiewebb did you have any more insight into your solution? I saw lxd was installed via snap and removed it (not used), but still getting dns resolution issues inside by containers...

sudo snap remove lxd
eddiewebb commented 1 year ago

@ShadowXVII it was a prior box i dont have that command history,but I currently have 0 snap packages and know docker was installed direct per their docs.

root@localhost:~# snap list
No snaps are installed yet. Try 'snap install hello-world'.

apt-get update
apt-get install     ca-certificates     curl     gnupg     lsb-release
mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo   "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin