search

mailcow / mailcow-dockerized

mailcow: dockerized - 🐮 + 🐋 = 💕
https://mailcow.email
GNU General Public License v3.0
8.94k stars 1.17k forks source link

Kernel Panic on Ubuntu 18.04.03 with kernel 4.15.0-60 #2904

Closed FingerlessGlov3s closed 5 years ago

FingerlessGlov3s commented 5 years ago

Prior to placing the issue, please check following: (fill out each checkbox with a X once done)

Description of the bug: What kind of issue have you exactly come across? When using the latest kernel on Ubuntu 18.04 "kernel 4.15.0-60" soon as you start the mailcow docker enviroment which 10-30 seconds the machine kernels panics. I've been sitting in the IRC last 24 hours and someone tried to setup mailcow on Vultr and Hetzner and got kernel panics, he left before I could ask which kernel but I bet they would of been fully up to date images,

IRC message from the guy before he left, he also joined after I message the room about mine kernel panicing asking if anyone else has had the issue. ": any current known issues around spinning up mailcow instances? have had two fail consecutively today. ubuntu 18.04, fresh docker/compose install - everything seems to go ok, then lose connectivity. kvm shows kernel panic. have tried two vps's - vultr and hetzner, plenty of ram, vcpus, etc. have installed on vultr before. i'm probably being dumb somewhere - but wanted to check in/ask before i beat my head into it for a while "

At the time of writing this Roger1 on IRC is having the same issue on AWS, who use their own kernel but I bet probably bases on 4.15.0-60, theirs is 4.15.0-1047-aws

Roger1 started up each service by hand and it crashed after starting mailcow/dockerapi

I will post more information later, and Roger1 said he would post logs too.

Further information (where applicable):

Question Answer
My operating system Ubuntu 18.04.3
Is Apparmor, SELinux or similar active? shall check later
Virtualization technlogy (KVM, VMware, Xen, etc) I believe Scaleway use VMware
Server/VM specifications (Memory, CPU Cores) 3 cores, 4GB Ram
Docker Version (docker version) 19.03
Docker-Compose Version (docker-compose version) v1.24.1
Reverse proxy (custom solution) NOPE
RogerSik commented 5 years ago

Switch the linux kernel from 4.15.0-1047-aws x86_64 to 5.0.0-1014-aws solved my issue. It was a mailcow instances which was longer and in use and the problem for the first time today began.

When starting all docker container from mailcow no TCP/HTTP/Ping Communition was more possible. I assumed the ip blocker was going in rage but the crash/black happens when I started all container manually and than started "mailcow/dockerapi:1.32"

patschi commented 5 years ago

I'd more likely guess that this is an issue from Docker, and not from mailcow. Mailcow is "just" a collection out of various containers to build a mailserver - it doesn't have enough kernel privileges to let the host kernel panic.

c0by85 commented 5 years ago

Got the same Problem, had to go one kernel back.

Problem kernel was Linux mail 4.15.0-60

currently working kernel:

uname -a Linux mail 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.3 LTS Release: 18.04 Codename: bionic

FingerlessGlov3s commented 5 years ago

Got the same Problem, had to go one kernel back.

Problem kernel was Linux mail 4.15.0-60

uname -a Linux mail 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.3 LTS Release: 18.04 Codename: bionic

There's something with this this latest kernel.

Anyone running any other docker contains with this kernel on different host? Web servers, databases or something?

andryyy commented 5 years ago

Checking it in a bit.

What's the panic message?

andryyy commented 5 years ago

Is AppArmor active?

RogerSik commented 5 years ago

AppArmor not active. And no else container running.

Shineson1001 commented 5 years ago

the last syslog entries:

`root@smtprelay02:~# tail -f /var/log/syslog Sep 3 12:23:26 smtprelay02 kernel: [ 16.850172] eth0: renamed from vethe4dfad1 Sep 3 12:23:27 smtprelay02 systemd-networkd[1000]: br-mailcow: Gained IPv6LL Sep 3 12:23:27 smtprelay02 kernel: [ 17.083272] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready Sep 3 12:23:27 smtprelay02 systemd-networkd[1000]: vethfb645ce: Gained carrier Sep 3 12:23:27 smtprelay02 kernel: [ 17.089237] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready Sep 3 12:23:27 smtprelay02 kernel: [ 17.089383] br-mailcow: port 16(vethce06c3c) entered disabled state Sep 3 12:23:27 smtprelay02 kernel: [ 17.089494] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Sep 3 12:23:27 smtprelay02 kernel: [ 17.089546] IPv6: ADDRCONF(NETDEV_CHANGE): vethfb645ce: link becomes ready

Sep 3 12:23:27 smtprelay02 kernel: [ 17.089575] br-mailcow: port 3(vethfb645ce) entered blocking state

Sep 3 12:23:27 smtprelay02 kernel: [ 17.089576] br-mailcow: port 3(vethfb645ce) entered forwarding state

Sep 3 12:23:27 smtprelay02 kernel: [ 17.318859] eth0: renamed from veth2c9e04f

Sep 3 12:23:27 smtprelay02 systemd-networkd[1000]: veth2cf0374: Gained carrier

Sep 3 12:23:27 smtprelay02 kernel: [ 17.396235] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:27 smtprelay02 kernel: [ 17.398722] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:27 smtprelay02 kernel: [ 17.398730] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:27 smtprelay02 kernel: [ 17.398774] IPv6: ADDRCONF(NETDEV_CHANGE): veth2cf0374: link becomes ready

Sep 3 12:23:27 smtprelay02 kernel: [ 17.398872] br-mailcow: port 5(veth2cf0374) entered blocking state

Sep 3 12:23:27 smtprelay02 kernel: [ 17.398875] br-mailcow: port 5(veth2cf0374) entered forwarding state

Sep 3 12:23:27 smtprelay02 containerd[1335]: time="2019-09-03T12:23:27.836730866+02:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/8e927e58c109d4ae3b255521b792bb213e8a089db04149765bb8d0c2fef41c89/shim.sock" debug=false pid=4181

Sep 3 12:23:27 smtprelay02 systemd-networkd[1000]: vethd87bee6: Gained IPv6LL

Sep 3 12:23:27 smtprelay02 containerd[1335]: time="2019-09-03T12:23:27.920245956+02:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/9ea6823344f4d8cda454f2281b33cace2d471cb362577293eafb7dd44bd502fa/shim.sock" debug=false pid=4225

Sep 3 12:23:27 smtprelay02 containerd[1335]: time="2019-09-03T12:23:27.985258651+02:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/1558911a3f198704efdf17c9260b8ce3ef4dc1898edc42cca992c6f5b6738711/shim.sock" debug=false pid=4253

Sep 3 12:23:27 smtprelay02 containerd[1335]: time="2019-09-03T12:23:27.987555563+02:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/261a1fa502c54e630d9d1831e6afe42653228e950f4daf97959025e200688990/shim.sock" debug=false pid=4257

Sep 3 12:23:28 smtprelay02 kernel: [ 18.014537] eth0: renamed from veth8c2f06e

Sep 3 12:23:28 smtprelay02 kernel: [ 18.151543] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:28 smtprelay02 systemd-networkd[1000]: veth3cdfa48: Gained carrier

Sep 3 12:23:28 smtprelay02 kernel: [ 18.156377] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:28 smtprelay02 kernel: [ 18.156389] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:28 smtprelay02 kernel: [ 18.156463] IPv6: ADDRCONF(NETDEV_CHANGE): veth3cdfa48: link becomes ready

Sep 3 12:23:28 smtprelay02 kernel: [ 18.156538] br-mailcow: port 4(veth3cdfa48) entered blocking state

Sep 3 12:23:28 smtprelay02 kernel: [ 18.156553] br-mailcow: port 4(veth3cdfa48) entered forwarding state

Sep 3 12:23:28 smtprelay02 kernel: [ 18.501021] eth0: renamed from veth17b6626

Sep 3 12:23:28 smtprelay02 kernel: [ 18.574858] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:28 smtprelay02 systemd-networkd[1000]: veth9eddcff: Gained carrier

Sep 3 12:23:28 smtprelay02 kernel: [ 18.581918] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:28 smtprelay02 kernel: [ 18.581930] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:28 smtprelay02 kernel: [ 18.582221] IPv6: ADDRCONF(NETDEV_CHANGE): veth9eddcff: link becomes ready

Sep 3 12:23:28 smtprelay02 kernel: [ 18.582416] br-mailcow: port 8(veth9eddcff) entered blocking state

Sep 3 12:23:28 smtprelay02 kernel: [ 18.582419] br-mailcow: port 8(veth9eddcff) entered forwarding state

Sep 3 12:23:29 smtprelay02 systemd-networkd[1000]: vethfb645ce: Gained IPv6LL

Sep 3 12:23:29 smtprelay02 systemd-networkd[1000]: veth2cf0374: Gained IPv6LL

Sep 3 12:23:29 smtprelay02 kernel: [ 19.070157] eth0: renamed from vetha9b3771

Sep 3 12:23:29 smtprelay02 kernel: [ 19.134323] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:29 smtprelay02 systemd-networkd[1000]: vethe7fa3e7: Gained carrier

Sep 3 12:23:29 smtprelay02 kernel: [ 19.136915] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:29 smtprelay02 kernel: [ 19.136927] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:29 smtprelay02 kernel: [ 19.136988] IPv6: ADDRCONF(NETDEV_CHANGE): vethe7fa3e7: link becomes ready

Sep 3 12:23:29 smtprelay02 kernel: [ 19.137027] br-mailcow: port 2(vethe7fa3e7) entered blocking state

Sep 3 12:23:29 smtprelay02 kernel: [ 19.137031] br-mailcow: port 2(vethe7fa3e7) entered forwarding state

Sep 3 12:23:29 smtprelay02 systemd-networkd[1000]: veth3cdfa48: Gained IPv6LL

Sep 3 12:23:29 smtprelay02 kernel: [ 19.414803] eth0: renamed from veth7fb8648

Sep 3 12:23:29 smtprelay02 kernel: [ 19.483509] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:29 smtprelay02 systemd-networkd[1000]: vetha939816: Gained carrier

Sep 3 12:23:29 smtprelay02 kernel: [ 19.486320] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:29 smtprelay02 kernel: [ 19.486339] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:29 smtprelay02 kernel: [ 19.486380] IPv6: ADDRCONF(NETDEV_CHANGE): vetha939816: link becomes ready

Sep 3 12:23:29 smtprelay02 kernel: [ 19.486489] br-mailcow: port 10(vetha939816) entered blocking state

Sep 3 12:23:29 smtprelay02 kernel: [ 19.486491] br-mailcow: port 10(vetha939816) entered forwarding state

Sep 3 12:23:30 smtprelay02 systemd-networkd[1000]: vethe7fa3e7: Gained IPv6LL

Sep 3 12:23:30 smtprelay02 kernel: [ 20.274464] eth0: renamed from veth0a9fbfc

Sep 3 12:23:30 smtprelay02 kernel: [ 20.342475] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:30 smtprelay02 kernel: [ 20.344759] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:30 smtprelay02 kernel: [ 20.344779] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:30 smtprelay02 kernel: [ 20.344825] IPv6: ADDRCONF(NETDEV_CHANGE): vethf7a040a: link becomes ready

Sep 3 12:23:30 smtprelay02 kernel: [ 20.344933] br-mailcow: port 7(vethf7a040a) entered blocking state

Sep 3 12:23:30 smtprelay02 kernel: [ 20.344937] br-mailcow: port 7(vethf7a040a) entered forwarding state

Sep 3 12:23:30 smtprelay02 systemd-networkd[1000]: vethf7a040a: Gained carrier

Sep 3 12:23:30 smtprelay02 systemd-networkd[1000]: veth9eddcff: Gained IPv6LL

Sep 3 12:23:30 smtprelay02 systemd-networkd[1000]: vetha939816: Gained IPv6LL

Sep 3 12:23:30 smtprelay02 kernel: [ 20.683061] eth0: renamed from veth7950e6f

Sep 3 12:23:30 smtprelay02 kernel: [ 20.759578] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:30 smtprelay02 systemd-networkd[1000]: veth70a186e: Gained carrier

Sep 3 12:23:30 smtprelay02 kernel: [ 20.762019] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:30 smtprelay02 kernel: [ 20.762101] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:30 smtprelay02 kernel: [ 20.762192] IPv6: ADDRCONF(NETDEV_CHANGE): veth70a186e: link becomes ready

Sep 3 12:23:30 smtprelay02 kernel: [ 20.762353] br-mailcow: port 14(veth70a186e) entered blocking state

Sep 3 12:23:30 smtprelay02 kernel: [ 20.762354] br-mailcow: port 14(veth70a186e) entered forwarding state

Sep 3 12:23:31 smtprelay02 kernel: [ 21.784033] eth0: renamed from vethfeb6cd8

Sep 3 12:23:31 smtprelay02 kernel: [ 21.847287] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:31 smtprelay02 kernel: [ 21.849767] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:31 smtprelay02 kernel: [ 21.849787] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:31 smtprelay02 kernel: [ 21.849828] IPv6: ADDRCONF(NETDEV_CHANGE): vethaf5af12: link becomes ready

Sep 3 12:23:31 smtprelay02 kernel: [ 21.849913] br-mailcow: port 13(vethaf5af12) entered blocking state

Sep 3 12:23:31 smtprelay02 kernel: [ 21.849915] br-mailcow: port 13(vethaf5af12) entered forwarding state

Sep 3 12:23:31 smtprelay02 systemd-networkd[1000]: vethaf5af12: Gained carrier

Sep 3 12:23:32 smtprelay02 systemd-networkd[1000]: vethf7a040a: Gained IPv6LL

Sep 3 12:23:32 smtprelay02 dockerd[1395]: time="2019-09-03T12:23:32+02:00" level=warning msg="xtables contention detected while running [--wait -t nat -I POSTROUTING -d 127.0.0.11 -j DOCKER_POSTROUTING]: Waited for 4.01 seconds and received \"\""

Sep 3 12:23:32 smtprelay02 kernel: [ 22.351044] eth0: renamed from vethd203b78

Sep 3 12:23:32 smtprelay02 systemd-networkd[1000]: veth70a186e: Gained IPv6LL

Sep 3 12:23:32 smtprelay02 kernel: [ 22.426306] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:32 smtprelay02 systemd-networkd[1000]: veth2958443: Gained carrier

Sep 3 12:23:32 smtprelay02 kernel: [ 22.429602] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:32 smtprelay02 kernel: [ 22.429623] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:32 smtprelay02 kernel: [ 22.429695] IPv6: ADDRCONF(NETDEV_CHANGE): veth2958443: link becomes ready

Sep 3 12:23:32 smtprelay02 kernel: [ 22.429771] br-mailcow: port 6(veth2958443) entered blocking state

Sep 3 12:23:32 smtprelay02 kernel: [ 22.429777] br-mailcow: port 6(veth2958443) entered forwarding state

Sep 3 12:23:33 smtprelay02 systemd-networkd[1000]: vethaf5af12: Gained IPv6LL

Sep 3 12:23:33 smtprelay02 kernel: [ 23.540965] eth0: renamed from vethbfda38c

Sep 3 12:23:33 smtprelay02 kernel: [ 23.626221] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:33 smtprelay02 kernel: [ 23.634258] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:33 smtprelay02 systemd-networkd[1000]: veth1b3a578: Gained carrier

Sep 3 12:23:33 smtprelay02 kernel: [ 23.639386] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:33 smtprelay02 kernel: [ 23.639522] IPv6: ADDRCONF(NETDEV_CHANGE): veth1b3a578: link becomes ready

Sep 3 12:23:33 smtprelay02 kernel: [ 23.639603] br-mailcow: port 9(veth1b3a578) entered blocking state

Sep 3 12:23:33 smtprelay02 kernel: [ 23.639608] br-mailcow: port 9(veth1b3a578) entered forwarding state

Sep 3 12:23:34 smtprelay02 systemd-networkd[1000]: veth2958443: Gained IPv6LL

Sep 3 12:23:34 smtprelay02 kernel: [ 24.603020] eth0: renamed from veth5fb449d

Sep 3 12:23:34 smtprelay02 kernel: [ 24.687067] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:34 smtprelay02 kernel: [ 24.690037] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:34 smtprelay02 kernel: [ 24.691929] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Sep 3 12:23:34 smtprelay02 kernel: [ 24.692031] IPv6: ADDRCONF(NETDEV_CHANGE): veth58f4bcc: link becomes ready

Sep 3 12:23:34 smtprelay02 kernel: [ 24.692104] br-mailcow: port 12(veth58f4bcc) entered blocking state

Sep 3 12:23:34 smtprelay02 kernel: [ 24.692108] br-mailcow: port 12(veth58f4bcc) entered forwarding state

Sep 3 12:23:34 smtprelay02 systemd-networkd[1000]: veth58f4bcc: Gained carrier

Sep 3 12:23:34 smtprelay02 systemd-networkd[1000]: veth1b3a578: Gained IPv6LL

Sep 3 12:23:35 smtprelay02 kernel: [ 25.645392] eth0: renamed from veth964a240

Sep 3 12:23:35 smtprelay02 systemd-networkd[1000]: veth58f4bcc: Gained IPv6LL

Sep 3 12:23:35 smtprelay02 kernel: [ 25.739583] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Sep 3 12:23:35 smtprelay02 kernel: [ 25.742687] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready Sep 3 12:23:35 smtprelay02 kernel: [ 25.742697] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Sep 3 12:23:35 smtprelay02 kernel: [ 25.742748] IPv6: ADDRCONF(NETDEV_CHANGE): vethdd28ca3: link becomes ready Sep 3 12:23:35 smtprelay02 kernel: [ 25.742794] br-mailcow: port 11(vethdd28ca3) entered blocking state Sep 3 12:23:35 smtprelay02 kernel: [ 25.742796] br-mailcow: port 11(vethdd28ca3) entered forwarding state Sep 3 12:23:35 smtprelay02 systemd-networkd[1000]: vethdd28ca3: Gained carrier Sep 3 12:23:36 smtprelay02 kernel: [ 26.491201] eth0: renamed from veth073401d Sep 3 12:23:36 smtprelay02 kernel: [ 26.592834] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready Sep 3 12:23:36 smtprelay02 kernel: [ 26.598836] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready Sep 3 12:23:36 smtprelay02 kernel: [ 26.598846] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Sep 3 12:23:36 smtprelay02 kernel: [ 26.598891] IPv6: ADDRCONF(NETDEV_CHANGE): vethe06f939: link becomes ready Sep 3 12:23:36 smtprelay02 kernel: [ 26.598933] br-mailcow: port 15(vethe06f939) entered blocking state Sep 3 12:23:36 smtprelay02 kernel: [ 26.598934] br-mailcow: port 15(vethe06f939) entered forwarding state Sep 3 12:23:36 smtprelay02 systemd-networkd[1000]: vethe06f939: Gained carrier Sep 3 12:23:37 smtprelay02 systemd-networkd[1000]: vethdd28ca3: Gained IPv6LL Sep 3 12:23:37 smtprelay02 kernel: [ 27.668102] eth0: renamed from veth04ee854 Sep 3 12:23:37 smtprelay02 kernel: [ 27.770486] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready Sep 3 12:23:37 smtprelay02 systemd-networkd[1000]: vethce06c3c: Gained carrier Sep 3 12:23:37 smtprelay02 kernel: [ 27.774643] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready Sep 3 12:23:37 smtprelay02 kernel: [ 27.774674] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Sep 3 12:23:37 smtprelay02 kernel: [ 27.774746] IPv6: ADDRCONF(NETDEV_CHANGE): vethce06c3c: link becomes ready Sep 3 12:23:37 smtprelay02 kernel: [ 27.774816] br-mailcow: port 16(vethce06c3c) entered blocking state Sep 3 12:23:37 smtprelay02 kernel: [ 27.774816] br-mailcow: port 16(vethce06c3c) entered forwarding state Sep 3 12:23:38 smtprelay02 dockerd[1395]: time="2019-09-03T12:23:38.534496356+02:00" level=info msg="Loading containers: done." Sep 3 12:23:38 smtprelay02 systemd-networkd[1000]: vethe06f939: Gained IPv6LL Sep 3 12:23:38 smtprelay02 dockerd[1395]: time="2019-09-03T12:23:38.735362740+02:00" level=info msg="Docker daemon" commit=74b1e89 graphdriver(s)=overlay2 version=19.03.1 Sep 3 12:23:38 smtprelay02 dockerd[1395]: time="2019-09-03T12:23:38.735785526+02:00" level=info msg="Daemon has completed initialization" Sep 3 12:23:38 smtprelay02 dockerd[1395]: time="2019-09-03T12:23:38.806077067+02:00" level=info msg="API listen on /var/run/docker.sock"

Sep 3 12:23:38 smtprelay02 systemd[1]: Started Docker Application Container Engine.

Sep 3 12:23:38 smtprelay02 systemd[1]: Reached target Multi-User System.

Sep 3 12:23:38 smtprelay02 systemd[1]: Reached target Graphical Interface. Sep 3 12:23:38 smtprelay02 systemd[1]: Starting Execute cloud user/final scripts... Sep 3 12:23:38 smtprelay02 systemd[1]: Starting Update UTMP about System Runlevel Changes... Sep 3 12:23:38 smtprelay02 systemd[1]: Started Update UTMP about System Runlevel Changes. Sep 3 12:23:39 smtprelay02 cloud-init[6530]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 running 'modules:final' at Tue, 03 Sep 2019 10:23:39 +0000. Up 29.12 seconds. Sep 3 12:23:39 smtprelay02 cloud-init[6530]: Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~18.04.1 finished at Tue, 03 Sep 2019 10:23:39 +0000. Datasource DataSourceNoCloud [seed=/var/lib/cloud/seed/nocloud-net][dsmode=net]. Up 29.26 seconds Sep 3 12:23:39 smtprelay02 systemd[1]: Started Execute cloud user/final scripts. Sep 3 12:23:39 smtprelay02 systemd[1]: Reached target Cloud-init target. Sep 3 12:23:39 smtprelay02 systemd[1]: Startup finished in 3.747s (kernel) + 25.605s (userspace) = 29.352s. Sep 3 12:23:39 smtprelay02 systemd-networkd[1000]: vethce06c3c: Gained IPv6LL Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 6 br-mailcow 172.22.1.1:123 Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 7 br-mailcow [fd4d:6169:6c63:6f77::1]:123 Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 8 br-mailcow [fe80::42:b2ff:fe4e:1f7b%3]:123 Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 9 br-mailcow [fe80::1%3]:123 Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 10 vethd87bee6 [fe80::fc96:cbff:fe0b:2f6d%6]:123 Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 11 vethe7fa3e7 [fe80::d81f:4fff:fe50:3357%8]:123 Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 12 vethfb645ce [fe80::d052:93ff:fe5a:83e6%10]:123 Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 13 veth3cdfa48 [fe80::10ba:ffff:fea3:71f1%12]:123 Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 14 veth2cf0374 [fe80::5082:36ff:feb9:6ca5%14]:123 Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 15 veth2958443 [fe80::7ce3:3dff:fe9b:8337%16]:123 Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 16 vethf7a040a [fe80::c4d2:77ff:febd:3af2%18]:123 Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 17 veth9eddcff [fe80::c03:5aff:feb4:b57a%20]:123 Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 18 veth1b3a578 [fe80::4c6f:dfff:fef6:956a%22]:123 Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 19 vetha939816 [fe80::d0ce:2cff:feea:113e%24]:123 Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 20 vethdd28ca3 [fe80::dcb4:15ff:fe68:b219%26]:123 Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 21 veth58f4bcc [fe80::10a0:39ff:fe51:e2db%28]:123 Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 22 vethaf5af12 [fe80::c49a:edff:fe0b:ced1%30]:123 Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 23 veth70a186e [fe80::6cbe:adff:fec2:2c76%32]:123 Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 24 vethe06f939 [fe80::78ab:caff:fe70:501d%34]:123 Sep 3 12:23:40 smtprelay02 ntpd[1258]: Listen normally on 25 vethce06c3c [fe80::1cad:79ff:fe3e:f3a8%36]:123 Sep 3 12:23:40 smtprelay02 ntpd[1258]: new interface(s) found: waking up resolver

Sep 3 12:23:48 smtprelay02 kernel: [ 38.658727] ------------[ cut here ]------------ Sep 3 12:23:48 smtprelay02 kernel: [ 38.658731] kernel BUG at /build/linux-5mCauq/linux-4.15.0/net/ipv4/ip_output.c:636! Sep 3 12:23:48 smtprelay02 kernel: [ 38.658792] invalid opcode: 0000 [#1] SMP PTI Sep 3 12:23:48 smtprelay02 kernel: [ 38.658817] Modules linked in: ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_security ip6table_raw ip6table_mangle iptable_security iptable_raw iptable_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_filter ip6_tables nf_tables_ipv4 nf_tables veth xt_nat xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc aufs overlay vmw_vsock_vmci_transport vsock vmw_balloon snd_ens1371 intel_rapl_perf snd_ac97_codec input_leds joydev serio_raw gameport snd_rawmidi snd_seq_device ac97_bus snd_pcm snd_timer snd soundcore vmw_vmci shpchp mac_hid sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp

Sep 3 12:23:48 smtprelay02 kernel: [ 38.659034] libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd psmouse vmwgfx ttm drm_kms_helper syscopyarea mptspi sysfillrect sysimgblt mptscsih fb_sys_fops i2c_piix4 mptbase ahci e1000 drm libahci scsi_transport_spi pata_acpi Sep 3 12:23:48 smtprelay02 kernel: [ 38.659170] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.15.0-60-generic #67-Ubuntu Sep 3 12:23:48 smtprelay02 kernel: [ 38.659185] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018 Sep 3 12:23:48 smtprelay02 kernel: [ 38.659210] RIP: 0010:ip_do_fragment+0x482/0x820 Sep 3 12:23:48 smtprelay02 kernel: [ 38.659220] RSP: 0018:ffff9458796c3a18 EFLAGS: 00010202 Sep 3 12:23:48 smtprelay02 kernel: [ 38.659231] RAX: 0000000000000001 RBX: ffff9457be2e3000 RCX: ffffffffa104cdf0 Sep 3 12:23:48 smtprelay02 kernel: [ 38.659245] RDX: 0000000000000024 RSI: 00000000000005c8 RDI: ffff9457b7be1500 Sep 3 12:23:48 smtprelay02 kernel: [ 38.659260] RBP: ffff9458796c3a80 R08: ffff945873cacb80 R09: 00000000000005dc Sep 3 12:23:48 smtprelay02 kernel: [ 38.659274] R10: 0000000000000000 R11: ffff94586b977400 R12: 0000000000000014 Sep 3 12:23:48 smtprelay02 kernel: [ 38.659288] R13: ffff9457b7be0f00 R14: 0000000000000d17 R15: ffff945873cacbce Sep 3 12:23:48 smtprelay02 kernel: [ 38.659303] FS: 0000000000000000(0000) GS:ffff9458796c0000(0000) knlGS:0000000000000000`

Shineson1001 commented 5 years ago

I get the kernel error, if i start the container manually ... the rspamd container crashes the system.

Sep 3 12:59:30 smtprelay02 kernel: [ 173.123882] br-mailcow: port 14(veth9540b17) entered blocking state Sep 3 12:59:30 smtprelay02 kernel: [ 173.123884] br-mailcow: port 14(veth9540b17) entered disabled state Sep 3 12:59:30 smtprelay02 kernel: [ 173.123952] device veth9540b17 entered promiscuous mode Sep 3 12:59:30 smtprelay02 kernel: [ 173.124056] IPv6: ADDRCONF(NETDEV_UP): veth9540b17: link is not ready Sep 3 12:59:30 smtprelay02 systemd-udevd[12772]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable. Sep 3 12:59:30 smtprelay02 systemd-udevd[12773]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable. Sep 3 12:59:30 smtprelay02 systemd-udevd[12772]: Could not generate persistent MAC address for vethc7f56a5: No such file or directory Sep 3 12:59:30 smtprelay02 systemd-udevd[12773]: Could not generate persistent MAC address for veth9540b17: No such file or directory Sep 3 12:59:30 smtprelay02 networkd-dispatcher[1001]: WARNING:Unknown index 87 seen, reloading interface list Sep 3 12:59:30 smtprelay02 containerd[1062]: time="2019-09-03T12:59:30.469138265+02:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/2cf2287d059960f91f96ebc7346a24c46f3244b60ede7a908a891f714a9bc47d/shim.sock" debug=false pid=12780 Sep 3 12:59:31 smtprelay02 kernel: [ 173.757655] eth0: renamed from vethc7f56a5 Sep 3 12:59:31 smtprelay02 systemd-networkd[790]: veth9540b17: Gained carrier Sep 3 12:59:31 smtprelay02 kernel: [ 173.832872] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready Sep 3 12:59:31 smtprelay02 kernel: [ 173.834984] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready Sep 3 12:59:31 smtprelay02 kernel: [ 173.835007] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Sep 3 12:59:31 smtprelay02 kernel: [ 173.835100] IPv6: ADDRCONF(NETDEV_CHANGE): veth9540b17: link becomes ready Sep 3 12:59:31 smtprelay02 kernel: [ 173.835273] br-mailcow: port 14(veth9540b17) entered blocking state Sep 3 12:59:31 smtprelay02 kernel: [ 173.835276] br-mailcow: port 14(veth9540b17) entered forwarding state Sep 3 12:59:32 smtprelay02 systemd-networkd[790]: veth9540b17: Gained IPv6LL Sep 3 12:59:33 smtprelay02 ntpd[1016]: Listen normally on 43 veth9540b17 [fe80::fc01:94ff:fe4b:fb21%87]:123 Sep 3 12:59:33 smtprelay02 ntpd[1016]: new interface(s) found: waking up resolver Sep 3 12:59:45 smtprelay02 kernel: [ 187.775225] ------------[ cut here ]------------ Sep 3 12:59:45 smtprelay02 kernel: [ 187.775241] kernel BUG at /build/linux-5mCauq/linux-4.15.0/net/ipv4/ip_output.c:636! Sep 3 12:59:45 smtprelay02 kernel: [ 187.776628] invalid opcode: 0000 [#1] SMP PTI

andryyy commented 5 years ago

No, I removed it, it still crashes.

I removed almost all containers manually. It always crashes.

Someone should file a bug at Ubuntu or Moby (but I think it is a Ubuntu issue).

FingerlessGlov3s commented 5 years ago

No, I removed it, it still crashes.

I removed almost all containers manually. It always crashes.

Someone should file a bug at Ubuntu or Moby (but I think it is a Ubuntu issue).

"almost all", which ones left?

andryyy commented 5 years ago

You cannot just stop containers, some containers may depend on each other in scripts and don't start before service X is available.

I removed Redis and Unbound and it works. But this does not mean much, as Redis is a dependency in many start scripts.

andryyy commented 5 years ago

Remember to stop watchdog...

I removed unbound and all dns: flags - no crash so far.

But we need to report that anyway... if it breaks with a kernel upgrade and runs fine on all other kernels on Ubuntu 18.04 and even other OSes, it is probably not our fault. :/

FingerlessGlov3s commented 5 years ago

No you can't just stop containers, as some are dependent on each other MYSQL etc.

Just wondered if you had go through the compose and taken out the dependencies, although they might just keep restarting as they'll error to due other services not being there.

FingerlessGlov3s commented 5 years ago

Shall I create a bug on Ubuntu Launch Pad Kernel? or you guys thinking something else.

andryyy commented 5 years ago

I mean, it works on other Kernels within Ubuntu. And all other distros.

I recommend to update to Kernel 5 as long as it isn't addressed.

andryyy commented 5 years ago

It works without the...

      dns:
        - ${IPV4_NETWORK:-172.22.1}.254

...parameters.

Can you confirm?

It is important to remove all existing containers before trying this.

FingerlessGlov3s commented 5 years ago

It works without the...

      dns:
        - ${IPV4_NETWORK:-172.22.1}.254

...parameters.

Can you confirm?

It is important to remove all existing containers before trying this.

Whats the - before the IP address for?

andryyy commented 5 years ago

It's yml. key => value. :)

FingerlessGlov3s commented 5 years ago

It's yml. key => value. :)

Ah yes, silly me. Was seeing it like a foreign character.

I'll test it when I get home and build a test instance but wont be for like 4-5 hours. So if someone else can test it before the go for it!

andryyy commented 5 years ago

Thanks. :) Just try with update.sh, please. :)

FingerlessGlov3s commented 5 years ago

Thanks. :) Just try with update.sh, please. :)

It'll be a fresh install, so shall be fully up to date!

Ah so RSPAMD and postfix will only use unbound.

Why not reference the DNS server by the container hostname? or just its ip? without the {$

andryyy commented 5 years ago

I just updated the compose file, you need to update again.

It does not work like that. You cannot resolve a name without a resolver.

fbcd4f50ea5350f051450323b5abfd0b commented 5 years ago

I can confirm that the updated docker-compose works with the latest kernel. Thanks for the quick fix!

andryyy commented 5 years ago

It is strange as hell... can someone test which container crashes with the dns flag?

fbcd4f50ea5350f051450323b5abfd0b commented 5 years ago

Sorry, after a few minutes it crashed again. Also with the updated compose file.

Shineson1001 commented 5 years ago

it does not work with the new docker-compose

`Sep 3 15:43:40 smtprelay kernel: [ 106.505952] ------------[ cut here ]------------ Sep 3 15:43:40 smtprelay kernel: [ 106.506032] kernel BUG at /build/linux-5mCauq/linux-4.15.0/net/ipv4/ip_output.c:636! Sep 3 15:43:40 smtprelay kernel: [ 106.506152] invalid opcode: 0000 [#1] SMP PTI Sep 3 15:43:40 smtprelay kernel: [ 106.506231] Modules linked in: ip6t_MASQUERADE nf_nat_masquerade_ipv6 nf_tables_ipv4 nf_tables ip6table_security ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_raw ip6table_mangle iptable_security iptable_raw iptable_mangle ip6table_filter ip6_tables xt_nat xt_tcpudp veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc ipt_REJECT nf_reject_ipv4 aufs xt_multiport iptable_filter overlay vmw_vsock_vmci_transport vsock coretemp ppdev intel_rapl_perf joydev input_leds serio_raw vmw_balloon parport_pc parport shpchp vmw_vmci mac_hid sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi Sep 3 15:43:40 smtprelay kernel: [ 106.506794] ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc vmwgfx ttm aesni_intel aes_x86_64 drm_kms_helper crypto_simd glue_helper syscopyarea cryptd sysfillrect sysimgblt fb_sys_fops mptspi psmouse mptscsih mptbase drm pata_acpi vmxnet3 scsi_transport_spi i2c_piix4 floppy Sep 3 15:43:40 smtprelay kernel: [ 106.507116] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.15.0-60-generic #67-Ubuntu Sep 3 15:43:40 smtprelay kernel: [ 106.507175] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015 Sep 3 15:43:40 smtprelay kernel: [ 106.507266] RIP: 0010:ip_do_fragment+0x482/0x820 Sep 3 15:43:40 smtprelay kernel: [ 106.507305] RSP: 0018:ffff93837fd83a98 EFLAGS: 00010202 Sep 3 15:43:40 smtprelay kernel: [ 106.507348] RAX: 0000000000000001 RBX: ffff93836efd4200 RCX: ffffffffa204cdf0 Sep 3 15:43:40 smtprelay kernel: [ 106.507403] RDX: 0000000000000024 RSI: 00000000000005c8 RDI: ffff93836fe4b500 Sep 3 15:43:40 smtprelay kernel: [ 106.507458] RBP: ffff93837fd83b00 R08: ffff93836fe9e800 R09: 00000000000005dc Sep 3 15:43:40 smtprelay kernel: [ 106.507537] R10: 0000000000000000 R11: ffff93836a9bc000 R12: 0000000000000014 Sep 3 15:43:40 smtprelay kernel: [ 106.507634] R13: ffff93836fe4b900 R14: 0000000000000681 R15: ffff93836fe9e84e Sep 3 15:43:40 smtprelay kernel: [ 106.507690] FS: 0000000000000000(0000) GS:ffff93837fd80000(0000) knlGS:0000000000000000 Sep 3 15:43:40 smtprelay kernel: [ 106.509527] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Sep 3 15:43:40 smtprelay kernel: [ 106.511380] CR2: 00007fb34b201c52 CR3: 00000001c6d92000 CR4: 00000000000406e0 Sep 3 15:43:40 smtprelay kernel: [ 106.513337] Call Trace: Sep 3 15:43:40 smtprelay kernel: [ 106.515161] Sep 3 15:43:40 smtprelay kernel: [ 106.516934] ? ip_copy_metadata+0x220/0x220 Sep 3 15:43:40 smtprelay kernel: [ 106.518794] ip_fragment.constprop.45+0x43/0x80 Sep 3 15:43:40 smtprelay kernel: [ 106.520622] ip_finish_output+0x182/0x270 Sep 3 15:43:40 smtprelay kernel: [ 106.522299] ? nf_hook_slow+0x48/0xc0 Sep 3 15:43:40 smtprelay kernel: [ 106.523922] ip_output+0x70/0xe0 Sep 3 15:43:40 smtprelay kernel: [ 106.525515] ? ip_fragment.constprop.45+0x80/0x80 Sep 3 15:43:40 smtprelay kernel: [ 106.527090] ip_forward_finish+0x51/0x80 Sep 3 15:43:40 smtprelay kernel: [ 106.528629] ip_forward+0x376/0x470 Sep 3 15:43:40 smtprelay kernel: [ 106.530138] ? ip4_key_hashfn+0xc0/0xc0 Sep 3 15:43:40 smtprelay kernel: [ 106.531614] ip_rcv_finish+0x129/0x430 Sep 3 15:43:40 smtprelay kernel: [ 106.533050] ip_rcv+0x296/0x360 Sep 3 15:43:40 smtprelay kernel: [ 106.534458] ? inet_del_offload+0x40/0x40 Sep 3 15:43:40 smtprelay kernel: [ 106.535862] netif_receive_skb_core+0x432/0xb80 Sep 3 15:43:40 smtprelay kernel: [ 106.537316] netif_receive_skb+0x18/0x60 Sep 3 15:43:40 smtprelay kernel: [ 106.538671] ? netif_receive_skb+0x18/0x60 Sep 3 15:43:40 smtprelay kernel: [ 106.540096] netif_receive_skb_internal+0x45/0xe0 Sep 3 15:43:40 smtprelay kernel: [ 106.541353] napi_gro_receive+0xc5/0xf0 Sep 3 15:43:40 smtprelay kernel: [ 106.542560] vmxnet3_rq_rx_complete+0x85b/0xeb0 [vmxnet3] Sep 3 15:43:40 smtprelay kernel: [ 106.543745] vmxnet3_poll_rx_only+0x36/0xa0 [vmxnet3] Sep 3 15:43:40 smtprelay kernel: [ 106.544856] net_rx_action+0x140/0x3a0 Sep 3 15:43:40 smtprelay kernel: [ 106.545943] do_softirq+0xe4/0x2d4 Sep 3 15:43:40 smtprelay kernel: [ 106.546973] irq_exit+0xc5/0xd0 Sep 3 15:43:40 smtprelay kernel: [ 106.547980] do_IRQ+0x86/0xe0 Sep 3 15:43:40 smtprelay kernel: [ 106.549301] common_interrupt+0x8c/0x8c Sep 3 15:43:40 smtprelay kernel: [ 106.550222] Sep 3 15:43:40 smtprelay kernel: [ 106.551135] RIP: 0010:native_safe_halt+0x12/0x20 Sep 3 15:43:40 smtprelay kernel: [ 106.552303] RSP: 0018:ffffb1fac0ccfe80 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd3 Sep 3 15:43:40 smtprelay kernel: [ 106.553245] RAX: ffffffffa21bb0f0 RBX: 0000000000000003 RCX: 0000000000000000 Sep 3 15:43:40 smtprelay kernel: [ 106.554212] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 Sep 3 15:43:40 smtprelay kernel: [ 106.555143] RBP: ffffb1fac0ccfe80 R08: 00000018cf5bf95f R09: ffff9383712aa100 Sep 3 15:43:40 smtprelay kernel: [ 106.556083] R10: 0000000000000000 R11: 7fffffffffffffff R12: 0000000000000003 Sep 3 15:43:40 smtprelay kernel: [ 106.557046] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 Sep 3 15:43:40 smtprelay kernel: [ 106.558169] ? __sched_text_end+0x1/0x1 Sep 3 15:43:40 smtprelay kernel: [ 106.559068] default_idle+0x20/0x100 Sep 3 15:43:40 smtprelay kernel: [ 106.560061] arch_cpu_idle+0x15/0x20 Sep 3 15:43:40 smtprelay kernel: [ 106.560912] default_idle_call+0x23/0x30 Sep 3 15:43:40 smtprelay kernel: [ 106.561789] do_idle+0x172/0x1f0 Sep 3 15:43:40 smtprelay kernel: [ 106.562668] cpu_startup_entry+0x73/0x80 Sep 3 15:43:40 smtprelay kernel: [ 106.563525] start_secondary+0x1ab/0x200 Sep 3 15:43:40 smtprelay kernel: [ 106.564393] secondary_startup_64+0xa5/0xb0 Sep 3 15:43:40 smtprelay kernel: [ 106.565246] Code: 8b 87 d8 00 00 00 48 2b 87 d0 00 00 00 39 c2 0f 87 f4 00 00 00 8b 87 e4 00 00 00 83 f8 01 0f 85 e5 00 00 00 48 83 7f 18 00 74 8e <0f> 0b 8b 0a 89 08 44 89 e1 8b 54 0a fc 89 54 08 fc e9 00 fd ff Sep 3 15:43:40 smtprelay kernel: [ 106.568090] RIP: ip_do_fragment+0x482/0x820 RSP: ffff93837fd83a98 Sep 3 15:43:40 smtprelay kernel: [ 106.569165] ---[ end trace fc721464731e6cec ]---

`

andryyy commented 5 years ago

You need to make sure it removed all old containers.

It works here.

Please report that to Ubuntu!! Don't use their new kernel, report it, upgrade to kernel 5 (within Ubuntu).

FingerlessGlov3s commented 5 years ago

How come you've done

Just wondering

andryyy commented 5 years ago

IPV4_NETWORK is a var.

FingerlessGlov3s commented 5 years ago

So what does the var expand as then in the end?

andryyy commented 5 years ago

Please don't do that here. Ask on IRC. The variable expansion has nothing to do with this issue. See mailcow.conf => $IPV4_NETWORK => ${IPV4_NETWORK:-172.22.1}.254 => 172.22.1.254

patschi commented 5 years ago

Filed a ubuntu bug report: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842447 If you have a Launchpad account and when you CAN reproduce the issue reliably, please consider clicking "bug affects me too" on above link.

FingerlessGlov3s commented 5 years ago

@patschi

I'm going to try and replicate tonight on a fresh install.

Should I use the old docker compose yml? Or the fixed one?

patschi commented 5 years ago

@FingerlessGlov3s I'd say both. I've tried the new one and didn't got any crashes. Once @andryyy replaced the docker-compose.yml file on my test VM to the previous one, the kernel crashes were reproducible every time.

FingerlessGlov3s commented 5 years ago

@patschi I shall fully update the VM, Snapshot it and then test both.

What environment/provider was your test VM?

patschi commented 5 years ago

@FingerlessGlov3s No need for a snapshot IMO. Just a tip: Disable docker from autostart. Else you might end up in an endless kernel crash & reboot-loop. (Even the container starts a bit delayed once system booted up).

I've tested it with a Hetzner Cloud VM, and they're using KVM underneath. According to posted logs by @Shineson1001 this also happens on the VMware ESXi hypervisor.

Shineson1001 commented 5 years ago

Test on VMWare Workstation 15:

Result with "old" docker-compose.yml: Kernel crashes Result with "new" docker-compose.yml: no crash and all containers works fine

Test on production system with VMWare ESXi:

Result: kernel panic with "old" and "new" docker-compose.yml

Kernel 4.15.0.-58 works fine on both systems with "old" and "new" docker-compose.yml

rpasing commented 5 years ago

Test on production system with VMWare ESXi:

* Ubuntu 18.04.3 LTS with latest updates (Kernel 4.15.0.-60)

* Mailcow with data (domains and mailboxes ...)

Result: kernel panic with "old" and "new" docker-compose.yml

Kernel 4.15.0.-58 works fine on both systems with "old" and "new" docker-compose.yml

Are you sure you have recreated the containers after changing the docker-compose.yml file? E.g. by doing docker-compose down && docker-compose up -d after changing the file.

andryyy commented 5 years ago

Remove all existing containers with docker rm -f first. Check them with docker ps -qa

Shineson1001 commented 5 years ago

yes, but it crashed some minutes after all container started.

remove all container grafik

reboot with Kernel Kernel 4.15.0.-60 and start mailcow grafik

5-10 minutes later the system stuckes

cyanPaperclip commented 5 years ago

Same here. Seems to work, but it crashes ~5mins after reboot.

patschi commented 5 years ago

For the records: There's also a blog post about this issue at https://mailcow.email/2019/09/03/attention-for-ubuntu-18-04-users-kernel-4-15-0-60-causes-kernel-panic/.

andryyy commented 5 years ago

We cannot change this. Docker and Ubuntu need to fix this.

FingerlessGlov3s commented 5 years ago

Woops

FingerlessGlov3s commented 5 years ago

I've +1 the bug on Launchpad

Also posted my test results, plus crash log. https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842447/comments/3

andryyy commented 5 years ago

I will add a check for that kernel, when I get home.

tmowbray commented 5 years ago

Experienced similar issue with 4.15.0-60. Server would crash ~5 after a boot with no indication as to why. Have reverted back to 4.15.0-58 for now to keep client happy.

EDIT: This was an existing mailcow deployment that started crashing after updating Ubuntu with the latest kernel. I have also confirmed the bug report over on Ubuntu's launchpad

andryyy commented 5 years ago

Can you guys confirm the report?

Saalbuerger commented 5 years ago

Experienced similar issue with 4.15.0-60. Server would crash ~5 after a boot with no indication as to why. Have reverted back to 4.15.0-58 for now to keep client happy.

+1

Workaround

  1. make a backup copy of /etc/default/grub sudo cp /etc/default/grub /etc/default/grub.bak

  2. find 4.15.0-58 in /boot/grub/grub.cfg sudo view /boot/grub/grub.cfg e.g. line menuentry 'Ubuntu, mit Linux 4.15.0-58-generic' --> 'gnulinux-4.15.0-58-generic-advanced-0473a811-2081-47ab-ac48-DUMMY-ID'

  3. edit /etc/default/grub (backup first!) and change GRUB_DEFAULT=0 to GRUB_DEFAULT='gnulinux-advanced-0473a811-2081-47ab-ac48-DUMMY-ID>gnulinux-4.15.0-58-generic-advanced-0473a811-2081-47ab-ac48-DUMMY-ID'

  4. create and check grup config sudo update-grub

  5. sudo reboot now

  6. check kernel Version uname -r --> 4.15.0-58-generic

  7. start mailcow

https://askubuntu.com/questions/100232/how-do-i-change-the-grub-boot-order/110738#110738