Open codiflow opened 6 months ago
Maybe thats helpful too:
These updates have been installed today alongside with mailcow:
Start-Date: 2024-02-10 14:14:30
Commandline: /usr/bin/apt-get upgrade -y
Upgrade: containerd.io:amd64 (1.6.27-1, 1.6.28-1), libisccfg163:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), libirs161:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), bind9-host:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), dnsutils:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), sudo:amd64 (1.8.27-1+deb10u5, 1.8.27-1+deb10u6), libisc-export1100:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), libisc1100:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), man-db:amd64 (2.8.5-2, 2.8.5-2+deb10u1), liblwres161:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), libdns-export1104:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), libisccc161:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), libbind9-161:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), libdns1104:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), docker-compose-plugin:amd64 (2.24.2-1~debian.10~buster, 2.24.5-1~debian.10~buster), docker-ce:amd64 (5:25.0.1-1~debian.10~buster, 5:25.0.2-1~debian.10~buster), docker-ce-cli:amd64 (5:25.0.1-1~debian.10~buster, 5:25.0.2-1~debian.10~buster)
End-Date: 2024-02-10 14:15:08
Start-Date: 2024-02-10 14:42:57
Commandline: apt upgrade
Install: debsuryorg-archive-keyring:amd64 (2024.02.05+0~20240205.1+debian10~1.gbp343037, automatic)
Upgrade: php-common:amd64 (2:94+0~20240121.50+debian10~1.gbpe06825, 2:94+0~20240205.51+debian10~1.gbp6faa2e), docker-ce:amd64 (5:25.0.2-1~debian.10~buster, 5:25.0.3-1~debian.10~buster), docker-ce-cli:amd64 (5:25.0.2-1~debian.10~buster, 5:25.0.3-1~debian.10~buster)
End-Date: 2024-02-10 14:43:17
Had a similar problem...
What i did:
Tried your approach step by step – unfortunately this did not change anything for me. The issue persists. 😥
Maybe your problem is ufw, i don't use it in my setup...
I don't think so as UFW uses its own chains.
But I could find some other hint while playing around with docker daemon and mailcow. After shutting down mailcow, reloading UFW and restarting the docker daemon the docker daemon was not coming up again.
I found this on the log:
dockerd[3154]: failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to register "bridge" driver: failed to create FILTER chain DOCKER: iptables failed: iptables -t filter -N DOCKER: iptables v1.8.2 (nf_tables): Chain already exists
I could "fix" it temporarily by setting DISABLE_NETFILTER_ISOLATION_RULE
to y
in .env
and restarting the machine / docker daemon and the firewall. Now UFW and iptables work like before. So the cause most likely is within recent netfilter changes like this one: https://github.com/mailcow/mailcow-dockerized/commit/087481ac12bfa5dd715f3630f0b1697be94f7e88#diff-1765fe97b183010a3dae8d7c2051c1e1ec96926170cdf72e980c893e37027072R218
I've had a similar case. My mailcow currently is on 2024-01d
but I noticed the outstanding update of docker to 25.0.3
. I've performed the upgrade to the new docker version with apt-get dist-upgrade
and then had the problem, that the docker engine could not start, resulting in the following
Do you want to continue? [Y/n] y
Reading changelogs... Done
(Reading database ... 37766 files and directories currently installed.)
Preparing to unpack .../docker-ce-cli_5%3a25.0.3-1~debian.10~buster_amd64.deb ...
Unpacking docker-ce-cli (5:25.0.3-1~debian.10~buster) over (5:25.0.2-1~debian.10~buster) ...
Preparing to unpack .../docker-ce_5%3a25.0.3-1~debian.10~buster_amd64.deb ...
Unpacking docker-ce (5:25.0.3-1~debian.10~buster) over (5:25.0.2-1~debian.10~buster) ...
Preparing to unpack .../docker-ce-rootless-extras_5%3a25.0.3-1~debian.10~buster_amd64.deb ...
Unpacking docker-ce-rootless-extras (5:25.0.3-1~debian.10~buster) over (5:25.0.2-1~debian.10~buster) ...
Setting up docker-ce-cli (5:25.0.3-1~debian.10~buster) ...
Setting up docker-ce-rootless-extras (5:25.0.3-1~debian.10~buster) ...
Setting up docker-ce (5:25.0.3-1~debian.10~buster) ...
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
After some research in the logs I found out the following:
Feb 11 13:08:23 systemd[1]: Starting Docker Application Container Engine...
Feb 11 13:08:23 dockerd[9790]: time="2024-02-11T13:08:23.354218510+01:00" level=info msg="Starting up"
Feb 11 13:08:23 dockerd[9790]: time="2024-02-11T13:08:23.354365090+01:00" level=warning msg="Running experimental build"
Feb 11 13:08:23 dockerd[9790]: time="2024-02-11T13:08:23.355628755+01:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
Feb 11 13:08:23 dockerd[9790]: time="2024-02-11T13:08:23.449964795+01:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Feb 11 13:08:23 dockerd[9790]: time="2024-02-11T13:08:23.473845925+01:00" level=info msg="Loading containers: start."
Feb 11 13:08:23 dockerd[9790]: time="2024-02-11T13:08:23.556482277+01:00" level=info msg="unable to detect if iptables supports xlock: 'iptables --wait -L -n': `iptables v1.8.2 (nf_tables): table `filter' is incompatible, use 'nft' tool.`" error="exit status 1"
Feb 11 13:08:23 dockerd[9790]: time="2024-02-11T13:08:23.630432993+01:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Feb 11 13:08:23 dockerd[9790]: failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to register "bridge" driver: failed to create FILTER chain DOCKER: iptables failed: iptables -t filter -N DOCKER: iptables v1.8.2 (nf_tables): Chain already exists
Feb 11 13:08:23 dockerd[9790]: (exit status 1)
After verifying the rule DOCKER
already exists I've removed it (iptables -D …
) and mailcow afterwards starts without any problem.
I think that moby/moby#47303 has something to do with it. That's the pull request regarding to the change log entry Fix a bug where containers are unable to communicate over an internal network.
in the docker engine version 25.0.3.
The error message from the logs come from here https://github.com/akerouanton/docker/blob/990e95dcf08af03ada217eac6109c09063a935a5/libnetwork/drivers/bridge/setup_ip_tables_linux.go#L59 and is part of the initialization progress of docker.
Upgrade of mailcow to 2024-01e
without any problem.
If you encounter Problems, please try to set DISABLE_NETFILTER_ISOLATION_RULE=y
in mailcow.conf and do a docker compose up -d
. But beware that this might open a Vulnerability. To be on the safe side, restrict the access to the mailcow docker network with something like:
iptables:
iptables -I DOCKER-USER ! -i br-mailcow -o br-mailcow -p tcp -m multiport --dport 3306,6379,8983,12345 -j DROP
nftables:
nft insert rule ip "filter" "DOCKER-USER" iifname != "br-mailcow" oifname "br-mailcow" tcp dport {3306, 6379, 8983, 12345} counter packets 0 bytes 0 drop
Thanks for clarifying this.
So does this mean there will be no "fix" for this issue (is it an issue within mailcow then?) and the only solution to this problem is to disable the netfilter isolation rule?
I'm experiencing the same/similar (?) issue, but don't have ufw installed, only nftables. netfilter container is also restarting every x seconds
netfilter-mailcow-1 | # Warning: table ip filter is managed by iptables-nft, do not touch!
netfilter-mailcow-1 | # Warning: table ip nat is managed by iptables-nft, do not touch!
netfilter-mailcow-1 | # Warning: table ip6 filter is managed by iptables-nft, do not touch!
netfilter-mailcow-1 | # Warning: table ip6 nat is managed by iptables-nft, do not touch!
netfilter-mailcow-1 | Using NFTables backend
netfilter-mailcow-1 | Clearing all bans
netfilter-mailcow-1 | Initializing mailcow netfilter chain
netfilter-mailcow-1 | Setting MAILCOW isolation
netfilter-mailcow-1 | Watching Redis channel F2B_CHANNEL
netfilter-mailcow-1 | MAILCOW target is in position 7 in the ip forward table, restarting container to fix it...
I've not made changes to /etc/nftables.conf
Can reproduce on Debian 10 and Ubuntu 22.04 using the steps described in original issue. The usage of ufw does not affect the end result.
Suggested workaround of settings DISABLE_NETFILTER_ISOLATION_RULE to Y has an effect as long as no SNAT has been set. If SNAT has been set it appears that there is also a incompatability on the nat table.
iptables v1.8.7 (nf_tables): table `nat' is incompatible, use 'nft' tool.
Mailcow's Netfilter seems to be breaking compatibility with nftables to iptables translation layer both for filters and network address translation.
I had the same issue #5798 but was able on one of my servers, to fix it, with reoving old rules from /etc/rules.v{4,6}
not the best option, but now i have build up all again and without the problems.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
If you encounter Problems, please try to set
DISABLE_NETFILTER_ISOLATION_RULE=y
in mailcow.conf and do adocker compose up -d
. But beware that this might open a Vulnerability. To be on the safe side, restrict the access to the mailcow docker network with something like:iptables:
iptables -I DOCKER-USER ! -i br-mailcow -o br-mailcow -p tcp -m multiport --dport 3306,6379,8983,12345 -j DROP
nftables:
nft insert rule ip "filter" "DOCKER-USER" iifname != "br-mailcow" oifname "br-mailcow" tcp dport {3306, 6379, 8983, 12345} counter packets 0 bytes 0 drop
Is this the official solution for this issue or is there something in the pipeline for an upcoming release?
Can a maintainer please clarify this?
I have the same issue on debian bullseye (11). The workaround suggested by FreddleSpl0it did not work for me.
Experiencing the same issue on Ubuntu 24.04. The iptables rules duplicate due to the restarts and the server gets unresponsive due to the many rules after a few hours (~10 hours)
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
It would also be really nice if a maintainer would comment on this.
Sitting it out and automatically closing existing problems will not solve them.
Sitting it out and automatically closing existing problems will not solve them.
I'd also like to see an end to the automatic closing of issues.
Contribution guidelines
I've found a bug and checked that ...
Description
Updating my machine like usual to the newest mailcow 2024-01e (and I think there was also a docker update) broke my UFW / iptables setup on one of my servers which is using Debian 10.
If I shutdown the mailcow container and reboot the machine everything is fine and I get an output from
iptables -L
and also fromufw status
I have two other docker containers on the same machine and they are working fine. I tried several combinations like shutting down all containers, reboot, check ufw/iptables and starting only other containers and not mailcow, reboot, check ufw/iptables.
The result was clear:
As soon as I start the mailcow docker containers with
docker compose up
both outputs break and also the firewall functionality. I can only bring it back by shutting down mailcow, reenabling UFW withufw enable
, restarting the machine and reenabling UFW again withufw enable
.Maybe the issue has to do with the Netfilter Enhancements like stated here?
Logs:
docker compose logs -t -f
https://paste.armbian.com/izeqepuzim.yaml
(text was too long)
Steps to reproduce:
Shutdown mailcow with
docker compose down
Reboot machine Checkufw status
Output:
iptables -L
Output: The usual IP tables entries
Now start mailcow with
docker compose up -d
Check
ufw status
Output:
Start
ufw enable
Output:
Check
ufw reload
Output:
Check
iptables -L
Output:
Shutdown mailcow with
docker compose down
Check commands again like above: Same result
Only way to bring everything up again is to either disable ufw with
ufw disable
and reboot the machine or to shutdown mailcow, reenabling UFW withufw enable
, restarting the machine and reenabling UFW again withufw enable
.But I would really like to use both like I did for years now 😅
Which branch are you using?
master
Which architecture are you using?
x86
Operating System:
Debian GNU/Linux 10 (buster)
Server/VM specifications:
32GB RAM, 6 Cores
Is Apparmor, SELinux or similar active?
no
Virtualization technology:
KVM
Docker version:
25.0.3
docker-compose version or docker compose version:
v2.24.5
mailcow version:
2024-01e
Reverse proxy:
nginx
Logs of git diff:
Logs of iptables -L -vn:
Logs of ip6tables -L -vn:
Logs of iptables -L -vn -t nat:
Logs of ip6tables -L -vn -t nat:
DNS check: