Issues after docker update to version 19.03.3

MElhagaly commented 4 years ago

Prior to placing the issue, please check following: (fill out each checkbox with a X once done)

[X] I understand that not following below instructions might result in immediate closing and deletion of my issue.
[X] I have understood that answers are voluntary and community-driven, and not commercial support.
[X] I have verified that my issue has not been already answered in the past. I also checked previous issues.

Description of the bug: What kind of issue have you exactly come across?

Last night I updated docker to a newer version using apt update & apt upgrade

Since the update I am not able to send and receive emails. Running df -h results in:

Filesystem      Size  Used Avail Use% Mounted on
udev            7.4G     0  7.4G   0% /dev
tmpfs           1.5G  173M  1.3G  12% /run
/dev/vda1        46G   32G   13G  73% /
tmpfs           7.4G  1.8M  7.4G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.4G     0  7.4G   0% /sys/fs/cgroup
/dev/vda15       99M  120K   99M   1% /boot/efi

Usually (at least before updating) the docker containers would show up as mounted filesystems too.

Running docker container ls --all results in:

CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS                    PORTS                                                                                                                                        NAMES
fa3bcf73b156        mailcow/solr:1.6         "/docker-entrypoint.…"   12 minutes ago      Created                   127.0.0.1:18983->8983/tcp                                                                                                                    mailcowdockerized_solr-mailcow_1
a99ae1096d2e        redis:5-alpine           "docker-entrypoint.s…"   12 minutes ago      Created                   6379/tcp                                                                                                                                     mailcowdockerized_redis-mailcow_1
7d2eb6d545b7        mailcow/dovecot:1.89     "/docker-entrypoint.…"   12 minutes ago      Created                   0.0.0.0:110->110/tcp, 0.0.0.0:143->143/tcp, 0.0.0.0:993->993/tcp, 0.0.0.0:995->995/tcp, 0.0.0.0:4190->4190/tcp, 127.0.0.1:19991->12345/tcp   mailcowdockerized_dovecot-mailcow_1
bc730adc27cb        mailcow/watchdog:1.59    "/bin/sh -c '/watchd…"   12 minutes ago      Created                                                                                                                                                                mailcowdockerized_watchdog-mailcow_1
1e6f669aabef        mailcow/sogo:1.61        "/bin/sh -c 'exec /u…"   12 minutes ago      Created                                                                                                                                                                mailcowdockerized_sogo-mailcow_1
a6d9dbe85866        mariadb:10.3             "docker-entrypoint.s…"   12 minutes ago      Created                   127.0.0.1:13306->3306/tcp                                                                                                                    mailcowdockerized_mysql-mailcow_1
a7227c6a20ed        mailcow/olefy:1.1        "python3 -u /app/ole…"   12 minutes ago      Created                                                                                                                                                                mailcowdockerized_olefy-mailcow_1
0a20c4684042        mailcow/unbound:1.8      "/docker-entrypoint.…"   12 minutes ago      Created                   53/tcp, 53/udp                                                                                                                               mailcowdockerized_unbound-mailcow_1
211c8cadccad        mailcow/dockerapi:1.32   "python3 -u /app/ser…"   12 minutes ago      Created                                                                                                                                                                mailcowdockerized_dockerapi-mailcow_1
0854ed03d024        mailcow/clamd:1.28       "/sbin/tini -g -- /b…"   12 minutes ago      Created                                                                                                                                                                mailcowdockerized_clamd-mailcow_1
24e3e3b4bfb3        mailcow/postfix:1.40     "/bin/sh -c 'exec /u…"   12 minutes ago      Created                   0.0.0.0:25->25/tcp, 0.0.0.0:465->465/tcp, 0.0.0.0:587->587/tcp, 588/tcp                                                                      mailcowdockerized_postfix-mailcow_1
ca7455c09ed2        memcached:alpine         "docker-entrypoint.s…"   12 minutes ago      Created                   11211/tcp                                                                                                                                    mailcowdockerized_memcached-mailcow_1

This, I guess, means that the containers exist somewhere in my system but they are simply not mounted.

I have also tried restarting (using docker-compose down && docker-compose up -d) the mailcow instance but that didn't help either.

Results at restart:

~/mailcow-dockerized# docker-compose down && docker-compose up -d
WARNING: The WATCHDOG_NOTIFY_EMAIL variable is not set. Defaulting to a blank string.
Removing mailcowdockerized_solr-mailcow_1      ... done
Removing mailcowdockerized_redis-mailcow_1     ... done
Removing mailcowdockerized_dovecot-mailcow_1   ... done
Removing mailcowdockerized_watchdog-mailcow_1  ... done
Removing mailcowdockerized_sogo-mailcow_1      ... done
Removing mailcowdockerized_mysql-mailcow_1     ... done
Removing mailcowdockerized_olefy-mailcow_1     ... done
Removing mailcowdockerized_unbound-mailcow_1   ... done
Removing mailcowdockerized_dockerapi-mailcow_1 ... done
Removing mailcowdockerized_clamd-mailcow_1     ... done
Removing mailcowdockerized_postfix-mailcow_1   ... done
Removing mailcowdockerized_memcached-mailcow_1 ... done
Removing network mailcowdockerized_mailcow-network
WARNING: The WATCHDOG_NOTIFY_EMAIL variable is not set. Defaulting to a blank string.
Creating network "mailcowdockerized_mailcow-network" with driver "bridge"
Creating mailcowdockerized_clamd-mailcow_1     ... error
Creating mailcowdockerized_dockerapi-mailcow_1 ... 
Creating mailcowdockerized_memcached-mailcow_1 ... 
Creating mailcowdockerized_postfix-mailcow_1   ... 
Creating mailcowdockerized_unbound-mailcow_1   ... 
Creating mailcowdockerized_olefy-mailcow_1     ... 
Creating mailcowdockerized_mysql-mailcow_1     ... 
Creating mailcowdockerized_redis-mailcow_1     ... 
Creating mailcowdockerized_memcached-mailcow_1 ... error
Creating mailcowdockerized_watchdog-mailcow_1  ... 
Creating mailcowdockerized_solr-mailcow_1      ... 
Creating mailcowdockerized_dovecot-mailcow_1   ... 
Creating mailcowdockerized_unbound-mailcow_1   ... error
ERROR: for mailcowdockerized_clamd-mailcow_1  Cannot start service clamd-mailcow: no status provided on response: unknown
Creating mailcowdockerized_solr-mailcow_1      ... error
ERROR: for mailcowdockerized_mysql-mailcow_1  Cannot start service mysql-mailcow: no status provided on response: unknown

Creating mailcowdockerized_postfix-mailcow_1   ... error

Creating mailcowdockerized_dockerapi-mailcow_1 ... error
Creating mailcowdockerized_sogo-mailcow_1      ... error
Creating mailcowdockerized_dovecot-mailcow_1   ... error

Creating mailcowdockerized_redis-mailcow_1     ... error

ERROR: for mailcowdockerized_sogo-mailcow_1  Cannot start service sogo-mailcow: no status provided on response: unknown

Creating mailcowdockerized_olefy-mailcow_1     ... error

Creating mailcowdockerized_watchdog-mailcow_1  ... error

ERROR: for mailcowdockerized_dockerapi-mailcow_1  Cannot start service dockerapi-mailcow: no status provided on response: unknown

ERROR: for mailcowdockerized_watchdog-mailcow_1  Cannot start service watchdog-mailcow: no status provided on response: unknown

ERROR: for mailcowdockerized_olefy-mailcow_1  Cannot start service olefy-mailcow: no status provided on response: unknown

ERROR: for clamd-mailcow  Cannot start service clamd-mailcow: no status provided on response: unknown

ERROR: for mysql-mailcow  Cannot start service mysql-mailcow: no status provided on response: unknown

ERROR: for solr-mailcow  Cannot start service solr-mailcow: no status provided on response: unknown

ERROR: for memcached-mailcow  Cannot start service memcached-mailcow: no status provided on response: unknown

ERROR: for unbound-mailcow  Cannot start service unbound-mailcow: no status provided on response: unknown

ERROR: for dovecot-mailcow  Cannot start service dovecot-mailcow: no status provided on response: unknown

ERROR: for sogo-mailcow  Cannot start service sogo-mailcow: no status provided on response: unknown

ERROR: for postfix-mailcow  Cannot start service postfix-mailcow: no status provided on response: unknown

ERROR: for redis-mailcow  Cannot start service redis-mailcow: no status provided on response: unknown

ERROR: for dockerapi-mailcow  Cannot start service dockerapi-mailcow: no status provided on response: unknown

ERROR: for watchdog-mailcow  Cannot start service watchdog-mailcow: no status provided on response: unknown

ERROR: for olefy-mailcow  Cannot start service olefy-mailcow: no status provided on response: unknown
ERROR: Encountered errors while bringing up the project.
~/mailcow-dockerized#

Results of docker status:

sudo service docker status
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-10-11 07:18:43 UTC; 23min ago
     Docs: https://docs.docker.com
 Main PID: 19750 (dockerd)
   CGroup: /system.slice/docker.service
           └─19750 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Oct 11 07:40:41 dockerd[19750]: time="2019-10-11T07:40:41.836770354Z" level=error msg="Handler for POST /v1.24/containers/21bf691661f699eec889383d5d0c982306ebed95a65c02f9d50b549c2174ee07/start returned error: no status provided 
Oct 11 07:40:42 dockerd[19750]: time="2019-10-11T07:40:42.275616307Z" level=warning msg="auplink flush failed: " error="exec: \"auplink\": executable file not found in $PATH" method=Unmount storage-driver=aufs
Oct 11 07:40:42 dockerd[19750]: time="2019-10-11T07:40:42.336625206Z" level=error msg="a4b4c137fb97177595bffcb3d4a6ba9317946631f73534bf44b18e6936878d22 cleanup: failed to delete container from containerd: no such container"
Oct 11 07:40:42 dockerd[19750]: time="2019-10-11T07:40:42.336694431Z" level=error msg="Handler for POST /v1.24/containers/a4b4c137fb97177595bffcb3d4a6ba9317946631f73534bf44b18e6936878d22/start returned error: no status provided 
Oct 11 07:40:42 dockerd[19750]: time="2019-10-11T07:40:42.355566612Z" level=warning msg="auplink flush failed: " error="exec: \"auplink\": executable file not found in $PATH" method=Unmount storage-driver=aufs
Oct 11 07:40:42 dockerd[19750]: time="2019-10-11T07:40:42.397113610Z" level=error msg="506b90aa2e17289a77d64296d025b05a0e2693fc7ddc96c4c072110c0f6cbf21 cleanup: failed to delete container from containerd: no such container"
Oct 11 07:40:42 dockerd[19750]: time="2019-10-11T07:40:42.397806593Z" level=error msg="Handler for POST /v1.24/containers/506b90aa2e17289a77d64296d025b05a0e2693fc7ddc96c4c072110c0f6cbf21/start returned error: no status provided 
Oct 11 07:40:42 dockerd[19750]: time="2019-10-11T07:40:42.475625058Z" level=warning msg="auplink flush failed: " error="exec: \"auplink\": executable file not found in $PATH" method=Unmount storage-driver=aufs
Oct 11 07:40:42 dockerd[19750]: time="2019-10-11T07:40:42.517418671Z" level=error msg="c15b56b4a7247bd974af88ef731afe54bef81d80aa99da333c75fe59c7bd8557 cleanup: failed to delete container from containerd: no such container"
Oct 11 07:40:42 dockerd[19750]: time="2019-10-11T07:40:42.517556004Z" level=error msg="Handler for POST /v1.24/containers/c15b56b4a7247bd974af88ef731afe54bef81d80aa99da333c75fe59c7bd8557/start returned error: no status provided 
lines 1-18/18 (END)

My issue is...

Reproduction of said bug: How exactly do you reproduce the bug?

Update docker/system to latest using apt update && apt upgrade

I have tried restarting (using docker-compose down && docker-compose up -d) the mailcow instance but that didn't help either.

[ ] In case of WebUI issue, I have tried clearing the browser cache and the issue persists.
[ ] I do run mailcow on a Synology, QNAP or any other sort of NAS. Results at restart:

~/mailcow-dockerized# docker-compose down && docker-compose up -d
WARNING: The WATCHDOG_NOTIFY_EMAIL variable is not set. Defaulting to a blank string.
Removing mailcowdockerized_solr-mailcow_1      ... done
Removing mailcowdockerized_redis-mailcow_1     ... done
Removing mailcowdockerized_dovecot-mailcow_1   ... done
Removing mailcowdockerized_watchdog-mailcow_1  ... done
Removing mailcowdockerized_sogo-mailcow_1      ... done
Removing mailcowdockerized_mysql-mailcow_1     ... done
Removing mailcowdockerized_olefy-mailcow_1     ... done
Removing mailcowdockerized_unbound-mailcow_1   ... done
Removing mailcowdockerized_dockerapi-mailcow_1 ... done
Removing mailcowdockerized_clamd-mailcow_1     ... done
Removing mailcowdockerized_postfix-mailcow_1   ... done
Removing mailcowdockerized_memcached-mailcow_1 ... done
Removing network mailcowdockerized_mailcow-network
WARNING: The WATCHDOG_NOTIFY_EMAIL variable is not set. Defaulting to a blank string.
Creating network "mailcowdockerized_mailcow-network" with driver "bridge"
Creating mailcowdockerized_clamd-mailcow_1     ... error
Creating mailcowdockerized_dockerapi-mailcow_1 ... 
Creating mailcowdockerized_memcached-mailcow_1 ... 
Creating mailcowdockerized_postfix-mailcow_1   ... 
Creating mailcowdockerized_unbound-mailcow_1   ... 
Creating mailcowdockerized_olefy-mailcow_1     ... 
Creating mailcowdockerized_mysql-mailcow_1     ... 
Creating mailcowdockerized_redis-mailcow_1     ... 
Creating mailcowdockerized_memcached-mailcow_1 ... error
Creating mailcowdockerized_watchdog-mailcow_1  ... 
Creating mailcowdockerized_solr-mailcow_1      ... 
Creating mailcowdockerized_dovecot-mailcow_1   ... 
Creating mailcowdockerized_unbound-mailcow_1   ... error
ERROR: for mailcowdockerized_clamd-mailcow_1  Cannot start service clamd-mailcow: no status provided on response: unknown
Creating mailcowdockerized_solr-mailcow_1      ... error
ERROR: for mailcowdockerized_mysql-mailcow_1  Cannot start service mysql-mailcow: no status provided on response: unknown

Creating mailcowdockerized_postfix-mailcow_1   ... error

Creating mailcowdockerized_dockerapi-mailcow_1 ... error
Creating mailcowdockerized_sogo-mailcow_1      ... error
Creating mailcowdockerized_dovecot-mailcow_1   ... error

Creating mailcowdockerized_redis-mailcow_1     ... error

ERROR: for mailcowdockerized_sogo-mailcow_1  Cannot start service sogo-mailcow: no status provided on response: unknown

Creating mailcowdockerized_olefy-mailcow_1     ... error

Creating mailcowdockerized_watchdog-mailcow_1  ... error

ERROR: for mailcowdockerized_dockerapi-mailcow_1  Cannot start service dockerapi-mailcow: no status provided on response: unknown

ERROR: for mailcowdockerized_watchdog-mailcow_1  Cannot start service watchdog-mailcow: no status provided on response: unknown

ERROR: for mailcowdockerized_olefy-mailcow_1  Cannot start service olefy-mailcow: no status provided on response: unknown

ERROR: for clamd-mailcow  Cannot start service clamd-mailcow: no status provided on response: unknown

ERROR: for mysql-mailcow  Cannot start service mysql-mailcow: no status provided on response: unknown

ERROR: for solr-mailcow  Cannot start service solr-mailcow: no status provided on response: unknown

ERROR: for memcached-mailcow  Cannot start service memcached-mailcow: no status provided on response: unknown

ERROR: for unbound-mailcow  Cannot start service unbound-mailcow: no status provided on response: unknown

ERROR: for dovecot-mailcow  Cannot start service dovecot-mailcow: no status provided on response: unknown

ERROR: for sogo-mailcow  Cannot start service sogo-mailcow: no status provided on response: unknown

ERROR: for postfix-mailcow  Cannot start service postfix-mailcow: no status provided on response: unknown

ERROR: for redis-mailcow  Cannot start service redis-mailcow: no status provided on response: unknown

ERROR: for dockerapi-mailcow  Cannot start service dockerapi-mailcow: no status provided on response: unknown

ERROR: for watchdog-mailcow  Cannot start service watchdog-mailcow: no status provided on response: unknown

ERROR: for olefy-mailcow  Cannot start service olefy-mailcow: no status provided on response: unknown
ERROR: Encountered errors while bringing up the project.
~/mailcow-dockerized#

System information

Further information (where applicable):

Question	Answer
My operating system	Ubuntu 16.04.6 LTS (Xenial Xerus)
Is Apparmor, SELinux or similar active?	-
Virtualization technlogy (KVM, VMware, Xen, etc)	-
Server/VM specifications (Memory, CPU Cores)	16GB/ 6 cores
Docker Version (`docker version`)	`Docker version 19.03.3, build a872fc2`
Docker-Compose Version (`docker-compose version`)	`docker-compose version 1.24.1, build 4667896b`
Reverse proxy (custom solution)	-

My observation is: updating docker results in breakage of software. Recommendation: Do not update to Docker 19.03.3

Further notes:

Output of git diff origin/master, any other changes to the code? If so, please post them.
All third-party firewalls and custom iptables rules are unsupported. Please check the Docker docs about how to use Docker with your own ruleset. Nevertheless, iptabels output can help us to help you: iptables -L -vn, ip6tables -L -vn, iptables -L -vn -t nat and ip6tables -L -vn -t nat
Check docker exec -it $(docker ps -qf name=acme-mailcow) dig +short stackoverflow.com @172.22.1.254 (set the IP accordingly, if you changed the internal mailcow network) and docker exec -it $(docker ps -qf name=acme-mailcow) dig +short stackoverflow.com @1.1.1.1 - output? Timeout?

General logs:
- Please take a look at the official documentation.

ntimo commented 4 years ago

Can you maybe check if you have a /etc/docker/daemon.json file and what it contains? And also run ‘docker info | grep "Storage Driver"’ it seems like Docker wants to use the aufs storage driver but can’t find it.

MElhagaly commented 4 years ago

@ntimo I unfortunately cannot find the deamon.json file. Executing ls in /etc/docker only shows: key.json

running docker info | grep "Storage Driver" outputs

 Storage Driver: aufs
WARNING: the aufs storage-driver is deprecated, and will be removed in a future release.

Shadesh commented 4 years ago

Same Probleme here, occured after updating mailcow via update.sh You can create / start the container by hand, they are up and running but running into error 128 after OS restart. Some kind of failing dependency?

Ubuntu 16.04.6 LTS

Docker version 19.03.3, build a872fc2

docker info | grep "Storage Driver"
 Storage Driver: overlay2
WARNING: No swap limit support

ll /etc/docker/
insgesamt 12
drwx------  2 root root 4096 Mär 18  2018 ./
drwxr-xr-x 96 root root 4096 Okt 10 17:46 ../
-rw-------  1 root root  244 Mär  2  2018 key.json

andryyy commented 4 years ago

Anyone created a report at moby or Ubuntu?

My recommendation is to "down" the stack and upgrade to 18.04.

I don't think that we can fix it here.

If someone is fine with me checking their 16.04 setup, mail me. I will try to upgrade and fix the stack.

MElhagaly commented 4 years ago

@andryyy so no problems with docker version 19.03.3 and ubuntu 18.04?

from my apt update logs I get the following

Log started: 2019-10-09  22:53:17
(Reading database ... 136039 files and directories currently installed.)
Preparing to unpack .../libpython3.5_3.5.2-2ubuntu0~16.04.9_amd64.deb ...
Unpacking libpython3.5:amd64 (3.5.2-2ubuntu0~16.04.9) over (3.5.2-2ubuntu0~16.04.8) ...
Preparing to unpack .../python3.5_3.5.2-2ubuntu0~16.04.9_amd64.deb ...
Unpacking python3.5 (3.5.2-2ubuntu0~16.04.9) over (3.5.2-2ubuntu0~16.04.8) ...
Preparing to unpack .../libpython3.5-stdlib_3.5.2-2ubuntu0~16.04.9_amd64.deb ...
Unpacking libpython3.5-stdlib:amd64 (3.5.2-2ubuntu0~16.04.9) over (3.5.2-2ubuntu0~16.04.8) ...
Preparing to unpack .../python3.5-minimal_3.5.2-2ubuntu0~16.04.9_amd64.deb ...
Unpacking python3.5-minimal (3.5.2-2ubuntu0~16.04.9) over (3.5.2-2ubuntu0~16.04.8) ...
Preparing to unpack .../libpython3.5-minimal_3.5.2-2ubuntu0~16.04.9_amd64.deb ...
Unpacking libpython3.5-minimal:amd64 (3.5.2-2ubuntu0~16.04.9) over (3.5.2-2ubuntu0~16.04.8) ...
Preparing to unpack .../libpython2.7_2.7.12-1ubuntu0~16.04.9_amd64.deb ...
Unpacking libpython2.7:amd64 (2.7.12-1ubuntu0~16.04.9) over (2.7.12-1ubuntu0~16.04.8) ...
Preparing to unpack .../python2.7_2.7.12-1ubuntu0~16.04.9_amd64.deb ...
Unpacking python2.7 (2.7.12-1ubuntu0~16.04.9) over (2.7.12-1ubuntu0~16.04.8) ...
Preparing to unpack .../libpython2.7-stdlib_2.7.12-1ubuntu0~16.04.9_amd64.deb ...
Unpacking libpython2.7-stdlib:amd64 (2.7.12-1ubuntu0~16.04.9) over (2.7.12-1ubuntu0~16.04.8) ...
Preparing to unpack .../python2.7-minimal_2.7.12-1ubuntu0~16.04.9_amd64.deb ...
Unpacking python2.7-minimal (2.7.12-1ubuntu0~16.04.9) over (2.7.12-1ubuntu0~16.04.8) ...
Preparing to unpack .../libpython2.7-minimal_2.7.12-1ubuntu0~16.04.9_amd64.deb ...
Unpacking libpython2.7-minimal:amd64 (2.7.12-1ubuntu0~16.04.9) over (2.7.12-1ubuntu0~16.04.8) ...
Preparing to unpack .../docker-ce-cli_5%3a19.03.3~3-0~ubuntu-xenial_amd64.deb ...
Unpacking docker-ce-cli (5:19.03.3~3-0~ubuntu-xenial) over (5:19.03.2~3-0~ubuntu-xenial) ...
Preparing to unpack .../docker-ce_5%3a19.03.3~3-0~ubuntu-xenial_amd64.deb ...
Unpacking docker-ce (5:19.03.3~3-0~ubuntu-xenial) over (5:19.03.2~3-0~ubuntu-xenial) ...
Preparing to unpack .../cloud-init_19.2-36-g059d049c-0ubuntu2~16.04.1_all.deb ...
 Unpacking cloud-init (19.2-36-g059d049c-0ubuntu2~16.04.1) over (19.2-36-g059d049c-0ubuntu1~16.04.1) ...
Preparing to unpack .../grafana_6.4.2_amd64.deb ...
Unpacking grafana (6.4.2) over (6.4.1) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Processing triggers for mime-support (3.59ubuntu1) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for systemd (229-4ubuntu21.22) ...
Setting up libpython3.5-minimal:amd64 (3.5.2-2ubuntu0~16.04.9) ...
Setting up libpython3.5-stdlib:amd64 (3.5.2-2ubuntu0~16.04.9) ...
Setting up libpython3.5:amd64 (3.5.2-2ubuntu0~16.04.9) ...
Setting up python3.5-minimal (3.5.2-2ubuntu0~16.04.9) ...
Setting up python3.5 (3.5.2-2ubuntu0~16.04.9) ...
Setting up libpython2.7-minimal:amd64 (2.7.12-1ubuntu0~16.04.9) ...
Setting up libpython2.7-stdlib:amd64 (2.7.12-1ubuntu0~16.04.9) ...
Setting up libpython2.7:amd64 (2.7.12-1ubuntu0~16.04.9) ...
Setting up python2.7-minimal (2.7.12-1ubuntu0~16.04.9) ...
Setting up python2.7 (2.7.12-1ubuntu0~16.04.9) ...
Setting up docker-ce-cli (5:19.03.3~3-0~ubuntu-xenial) ...
Setting up docker-ce (5:19.03.3~3-0~ubuntu-xenial) ...
Setting up cloud-init (19.2-36-g059d049c-0ubuntu2~16.04.1) ...
Leaving 'diversion of /etc/init/ureadahead.conf to /etc/init/ureadahead.conf.disabled by cloud-init'
Setting up grafana (6.4.2) ...
Restarting grafana-server service... OK
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Log ended: 2019-10-09  22:53:58

Log started: 2019-10-10  20:15:20
(Reading database ... 136039 files and directories currently installed.)
Preparing to unpack .../base-files_9.4ubuntu4.11_amd64.deb ...
Unpacking base-files (9.4ubuntu4.11) over (9.4ubuntu4.8) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up base-files (9.4ubuntu4.11) ...
(Reading database ... 136043 files and directories currently installed.)
Preparing to unpack .../containerd.io_1.2.10-2_amd64.deb ...
Unpacking containerd.io (1.2.10-2) over (1.2.6-3) ...
Setting up containerd.io (1.2.10-2) ...
Log ended: 2019-10-10  20:15:26

Seeing this I think that the problems started occurring after updating containerd.io from 1.2.6-3 to 1.2.10-2. I came to this conclusion because I was able to receive emails yesterday afternoon & evening.

I also found this issue at the docker for-linux repos: https://github.com/docker/for-linux/issues/813

Would downgrading the version of containerd.io help solving the issue?

unze commented 4 years ago

ubuntu 18.04 containerd.io: containerd.io/bionic,now 1.2.6-3 amd64 docker: 19.03.3, build a872fc2f86

no issues

MElhagaly commented 4 years ago

@andryyy A technical question while solving the issue. People have reached out to me asking about email I supposedly received. My question is: will I receive said emails when I am back up and running or are they lost in void?

mkuron commented 4 years ago

will I receive said emails when I am back up and running or are they lost in void?

Email servers are expected to keep trying to deliver a message for at least three days before they give up and notify the sender. So you will eventually receive your missed messages since you fixed your server quickly enough.

MElhagaly commented 4 years ago

I fixed the issue by downgrading containerd.io to 1.2.6-3 and so far I am back up and running.

pjv commented 4 years ago

so.... is containerd.io=1.2.6-3 a hard version dependency for now? Should everyone running mailcow on ubuntu 18.04 be pinning to that version?

at the moment, apt upgrade wants to install containerd.io=1.2.10-2.

MElhagaly commented 4 years ago

@pjv containerd.io=1.2.6-3 is the latest version of containerd.io that works for me. Upgrading to containerd.io=1.2.10-2 broke everything using docker on my system. I am not sure if anyone has containerd.io=1.2.10-2 working with mailcow.

My recommendation for the time being is remain using containerd.io=1.2.6-3 until newer version of containerd are tested approved by the community.

cklabautermann commented 4 years ago

I am running containerd.io 1.2.10-2 on my ubuntu 18.04 mailcow box running smoothly

rockhouse commented 4 years ago

I too had to revert to version containerd.io=1.2.6-3

andryyy commented 4 years ago

You guys need to create issues at moby or Ubuntu :(

pgassmann commented 4 years ago

Summary from the upstream issue: When the package containerd.io was updated, it did not correctly restart the containerd service in the postinstall steps.

Just restart containerd service: service containerd restart

andryyy commented 4 years ago

Thank you! :)

Am 18.10.2019 um 11:53 schrieb Philipp Gassmann notifications@github.com:

Summary from the other issue: When the package containerd.io was updated, it did not correctly restart the containerd service in the postinstall steps.

Just restart containerd service: service containerd restart

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

choffman5718 commented 4 years ago

Anyone created a report at moby or Ubuntu?

My recommendation is to "down" the stack and upgrade to 18.04.

I don't think that we can fix it here.

If someone is fine with me checking their 16.04 setup, mail me. I will try to upgrade and fix the stack.

Andryyy, I am 100% fine with you trying to do that on mine. I am on 16.04. Since I did an update today, Dovecot is having many errors. I emailed them to the info email address. Im begging for help. I would purchase the support but I am a poor college student who is still learning computer science. Please help,

MElhagaly commented 4 years ago

@choffman5718 I think you should consider looking at the latest post on the mailcow website.

andryyy commented 4 years ago

I already helped him and we upgraded the kernel. :)

choffman5718 commented 4 years ago

You both are the greatest, thank you so so much.

emilianobaum commented 4 years ago

Hi all, i use:

Distributor ID: neon Description: KDE neon User Edition 5.18 Release: 18.04 Codename: bionic

and i have the same issue, the service stay loaded but never run. When i check the installation status never finish the dpkgof docker-ce, and obviously never start the installation of docker-ce-cli or containerd.io. For the installation i followed the instructions of the official site ([https://docs.docker.com/engine/install/ubuntu/]). I made an upgrade/update of the SO, and tried to repeat the process without results.

Finally i need to remove from dpkg --remove and then install a previously version:

● docker.service - Docker Application Container Engine Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enable Drop-In: /etc/systemd/system/docker.service.d └─override.conf Active: active (running) since Tue 2020-04-28 11:08:19 -03; 37s ago Docs: https://docs.docker.com Main PID: 9433 (dockerd) Tasks: 24 CGroup: /system.slice/docker.service └─9433 /usr/bin/dockerd -H fd:// -H tcp://127.0.0.1:2375

andryyy commented 4 years ago

Have you tried the moby issue tracker?

mailcow / mailcow-dockerized

Issues after docker update to version 19.03.3 #3028