Open codiflow opened 9 months ago
Maybe thats helpful too:
These updates have been installed today alongside with mailcow:
Start-Date: 2024-02-10 14:14:30
Commandline: /usr/bin/apt-get upgrade -y
Upgrade: containerd.io:amd64 (1.6.27-1, 1.6.28-1), libisccfg163:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), libirs161:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), bind9-host:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), dnsutils:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), sudo:amd64 (1.8.27-1+deb10u5, 1.8.27-1+deb10u6), libisc-export1100:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), libisc1100:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), man-db:amd64 (2.8.5-2, 2.8.5-2+deb10u1), liblwres161:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), libdns-export1104:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), libisccc161:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), libbind9-161:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), libdns1104:amd64 (1:9.11.5.P4+dfsg-5.1+deb10u9, 1:9.11.5.P4+dfsg-5.1+deb10u10), docker-compose-plugin:amd64 (2.24.2-1~debian.10~buster, 2.24.5-1~debian.10~buster), docker-ce:amd64 (5:25.0.1-1~debian.10~buster, 5:25.0.2-1~debian.10~buster), docker-ce-cli:amd64 (5:25.0.1-1~debian.10~buster, 5:25.0.2-1~debian.10~buster)
End-Date: 2024-02-10 14:15:08
Start-Date: 2024-02-10 14:42:57
Commandline: apt upgrade
Install: debsuryorg-archive-keyring:amd64 (2024.02.05+0~20240205.1+debian10~1.gbp343037, automatic)
Upgrade: php-common:amd64 (2:94+0~20240121.50+debian10~1.gbpe06825, 2:94+0~20240205.51+debian10~1.gbp6faa2e), docker-ce:amd64 (5:25.0.2-1~debian.10~buster, 5:25.0.3-1~debian.10~buster), docker-ce-cli:amd64 (5:25.0.2-1~debian.10~buster, 5:25.0.3-1~debian.10~buster)
End-Date: 2024-02-10 14:43:17
Had a similar problem...
What i did:
Tried your approach step by step – unfortunately this did not change anything for me. The issue persists. 😥
Maybe your problem is ufw, i don't use it in my setup...
I don't think so as UFW uses its own chains.
But I could find some other hint while playing around with docker daemon and mailcow. After shutting down mailcow, reloading UFW and restarting the docker daemon the docker daemon was not coming up again.
I found this on the log:
dockerd[3154]: failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to register "bridge" driver: failed to create FILTER chain DOCKER: iptables failed: iptables -t filter -N DOCKER: iptables v1.8.2 (nf_tables): Chain already exists
I could "fix" it temporarily by setting DISABLE_NETFILTER_ISOLATION_RULE
to y
in .env
and restarting the machine / docker daemon and the firewall. Now UFW and iptables work like before. So the cause most likely is within recent netfilter changes like this one: https://github.com/mailcow/mailcow-dockerized/commit/087481ac12bfa5dd715f3630f0b1697be94f7e88#diff-1765fe97b183010a3dae8d7c2051c1e1ec96926170cdf72e980c893e37027072R218
I've had a similar case. My mailcow currently is on 2024-01d
but I noticed the outstanding update of docker to 25.0.3
. I've performed the upgrade to the new docker version with apt-get dist-upgrade
and then had the problem, that the docker engine could not start, resulting in the following
Do you want to continue? [Y/n] y
Reading changelogs... Done
(Reading database ... 37766 files and directories currently installed.)
Preparing to unpack .../docker-ce-cli_5%3a25.0.3-1~debian.10~buster_amd64.deb ...
Unpacking docker-ce-cli (5:25.0.3-1~debian.10~buster) over (5:25.0.2-1~debian.10~buster) ...
Preparing to unpack .../docker-ce_5%3a25.0.3-1~debian.10~buster_amd64.deb ...
Unpacking docker-ce (5:25.0.3-1~debian.10~buster) over (5:25.0.2-1~debian.10~buster) ...
Preparing to unpack .../docker-ce-rootless-extras_5%3a25.0.3-1~debian.10~buster_amd64.deb ...
Unpacking docker-ce-rootless-extras (5:25.0.3-1~debian.10~buster) over (5:25.0.2-1~debian.10~buster) ...
Setting up docker-ce-cli (5:25.0.3-1~debian.10~buster) ...
Setting up docker-ce-rootless-extras (5:25.0.3-1~debian.10~buster) ...
Setting up docker-ce (5:25.0.3-1~debian.10~buster) ...
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
After some research in the logs I found out the following:
Feb 11 13:08:23 systemd[1]: Starting Docker Application Container Engine...
Feb 11 13:08:23 dockerd[9790]: time="2024-02-11T13:08:23.354218510+01:00" level=info msg="Starting up"
Feb 11 13:08:23 dockerd[9790]: time="2024-02-11T13:08:23.354365090+01:00" level=warning msg="Running experimental build"
Feb 11 13:08:23 dockerd[9790]: time="2024-02-11T13:08:23.355628755+01:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
Feb 11 13:08:23 dockerd[9790]: time="2024-02-11T13:08:23.449964795+01:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Feb 11 13:08:23 dockerd[9790]: time="2024-02-11T13:08:23.473845925+01:00" level=info msg="Loading containers: start."
Feb 11 13:08:23 dockerd[9790]: time="2024-02-11T13:08:23.556482277+01:00" level=info msg="unable to detect if iptables supports xlock: 'iptables --wait -L -n': `iptables v1.8.2 (nf_tables): table `filter' is incompatible, use 'nft' tool.`" error="exit status 1"
Feb 11 13:08:23 dockerd[9790]: time="2024-02-11T13:08:23.630432993+01:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Feb 11 13:08:23 dockerd[9790]: failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to register "bridge" driver: failed to create FILTER chain DOCKER: iptables failed: iptables -t filter -N DOCKER: iptables v1.8.2 (nf_tables): Chain already exists
Feb 11 13:08:23 dockerd[9790]: (exit status 1)
After verifying the rule DOCKER
already exists I've removed it (iptables -D …
) and mailcow afterwards starts without any problem.
I think that moby/moby#47303 has something to do with it. That's the pull request regarding to the change log entry Fix a bug where containers are unable to communicate over an internal network.
in the docker engine version 25.0.3.
The error message from the logs come from here https://github.com/akerouanton/docker/blob/990e95dcf08af03ada217eac6109c09063a935a5/libnetwork/drivers/bridge/setup_ip_tables_linux.go#L59 and is part of the initialization progress of docker.
Upgrade of mailcow to 2024-01e
without any problem.
If you encounter Problems, please try to set DISABLE_NETFILTER_ISOLATION_RULE=y
in mailcow.conf and do a docker compose up -d
. But beware that this might open a Vulnerability. To be on the safe side, restrict the access to the mailcow docker network with something like:
iptables:
iptables -I DOCKER-USER ! -i br-mailcow -o br-mailcow -p tcp -m multiport --dport 3306,6379,8983,12345 -j DROP
nftables:
nft insert rule ip "filter" "DOCKER-USER" iifname != "br-mailcow" oifname "br-mailcow" tcp dport {3306, 6379, 8983, 12345} counter packets 0 bytes 0 drop
Thanks for clarifying this.
So does this mean there will be no "fix" for this issue (is it an issue within mailcow then?) and the only solution to this problem is to disable the netfilter isolation rule?
I'm experiencing the same/similar (?) issue, but don't have ufw installed, only nftables. netfilter container is also restarting every x seconds
netfilter-mailcow-1 | # Warning: table ip filter is managed by iptables-nft, do not touch!
netfilter-mailcow-1 | # Warning: table ip nat is managed by iptables-nft, do not touch!
netfilter-mailcow-1 | # Warning: table ip6 filter is managed by iptables-nft, do not touch!
netfilter-mailcow-1 | # Warning: table ip6 nat is managed by iptables-nft, do not touch!
netfilter-mailcow-1 | Using NFTables backend
netfilter-mailcow-1 | Clearing all bans
netfilter-mailcow-1 | Initializing mailcow netfilter chain
netfilter-mailcow-1 | Setting MAILCOW isolation
netfilter-mailcow-1 | Watching Redis channel F2B_CHANNEL
netfilter-mailcow-1 | MAILCOW target is in position 7 in the ip forward table, restarting container to fix it...
I've not made changes to /etc/nftables.conf
Can reproduce on Debian 10 and Ubuntu 22.04 using the steps described in original issue. The usage of ufw does not affect the end result.
Suggested workaround of settings DISABLE_NETFILTER_ISOLATION_RULE to Y has an effect as long as no SNAT has been set. If SNAT has been set it appears that there is also a incompatability on the nat table.
iptables v1.8.7 (nf_tables): table `nat' is incompatible, use 'nft' tool.
Mailcow's Netfilter seems to be breaking compatibility with nftables to iptables translation layer both for filters and network address translation.
I had the same issue #5798 but was able on one of my servers, to fix it, with reoving old rules from /etc/rules.v{4,6}
not the best option, but now i have build up all again and without the problems.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
If you encounter Problems, please try to set
DISABLE_NETFILTER_ISOLATION_RULE=y
in mailcow.conf and do adocker compose up -d
. But beware that this might open a Vulnerability. To be on the safe side, restrict the access to the mailcow docker network with something like:iptables:
iptables -I DOCKER-USER ! -i br-mailcow -o br-mailcow -p tcp -m multiport --dport 3306,6379,8983,12345 -j DROP
nftables:
nft insert rule ip "filter" "DOCKER-USER" iifname != "br-mailcow" oifname "br-mailcow" tcp dport {3306, 6379, 8983, 12345} counter packets 0 bytes 0 drop
Is this the official solution for this issue or is there something in the pipeline for an upcoming release?
Can a maintainer please clarify this?
I have the same issue on debian bullseye (11). The workaround suggested by FreddleSpl0it did not work for me.
Experiencing the same issue on Ubuntu 24.04. The iptables rules duplicate due to the restarts and the server gets unresponsive due to the many rules after a few hours (~10 hours)
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
It would also be really nice if a maintainer would comment on this.
Sitting it out and automatically closing existing problems will not solve them.
Sitting it out and automatically closing existing problems will not solve them.
I'd also like to see an end to the automatic closing of issues.
Is there anything new here? I have exactly the same problem ...
Had a similar problem...
What i did:
- stop the mailcow
- stop docker service
- install nftables (apt install nftables), this installs the nft tool
- nft -f /etc/nftables.conf
- start docker
- start mailcow
That helped me on Ubuntu 24.04 too with even nftables was installed.
@codiflow Debian 10 is EOL after 5 years being in production. You should really consider upgrading your OS (to 11 then 12) to be able to run modern software on it smoothly and keep your system secure.
This is not a mailcow issue and can be closed I think @DerLinkman .
This is not a mailcow issue and can be closed I think @DerLinkman .
I am using the latest Debian 12 (bookworm) and have the same problem. I think it is a mailcow problem ...
This is not a mailcow issue and can be closed I think @DerLinkman .
I am using the latest Debian 12 (bookworm) and have the same problem. I think it is a mailcow problem ...
I would confirm your assumption.
Saying that this is not a Mailcow problem without investigating is just as strange as automatically closing the issue, @apio-sys .
I don't agree, it's a setup/configuration issue of your OS not a mailcow issue. In Debian 12, nftables is used as the default UFW backend. So maybe you don't have a default setup and changed some stuff before installing mailcow. See prereqs here https://docs.mailcow.email/getstarted/prerequisite-system/ .
@Fighter456 I'm not just saying this, I can confirm that with a default Ubuntu 22.04 and 24.04 and Debian 12 setup, all works as expected hence it is not a mailcow issue but simply that some people missed the prereqs. The initial post here talks about Debioan 10 which is EOL since and I don't think we want to waste any time on EOL setups. I suggest anybody experiencing this on a default setup and following prereqs as described could open a new question. The answer of @erichk4 solve the issue here.
The problem was proven to have occurred with the initial update mentioned and has never been investigated since then.
I am against closing the issue. It doesn't matter whether it concerns Debian 10 or Debian 11 or Debian 12. A problem is a problem. Moving it to another issue with the same information doesn't change anything. And I have the problem on a standard Debian 12.
I don't think we want to waste any time on EOL setups
As you can read a few words ago, it is also a current problem on non-EOL systems. But: What is your position with the Mailcow organization to make such a decision?
I don't agree, it's a setup/configuration issue of your OS not a mailcow issue. In Debian 12, nftables is used as the default UFW backend. So maybe you don't have a default setup and changed some stuff before installing mailcow. See prereqs here https://docs.mailcow.email/getstarted/prerequisite-system/ .
The prereqs in my system are implemented or apply as described there and yet there is still a problem ... I don't think it's a problem with the OS
I don't make any decisions, I just try to contribute to a great FOSS project. Hence I SUGGESTED to close the issue with an IMHO.
If you want to keep it open or investigate, then please feel free to contribute @codiflow and @Fighter456 . And of course it does matter which version of the OS is used since Debian 10 is out of scope as stated here: https://docs.mailcow.email/getstarted/prerequisite-system/#supported-os .
I don't make any decisions, I just try to contribute to a great FOSS project. Hence I SUGGESTED to close the issue with an IMHO.
Instead of making useless suggestions or being an issue sheriff, use your time, contribute to the project and solve the problem. 😉
As has been said to you several times, the problem continues to occur on non-EOL systems.
How would you like to help me and the others with the problem, @apio-sys?
If you want to help, try and reproduce it on a clean system. Saying it doesn't work on Debian 12 or Ubuntu 24.04 doesn't help nobody and is totally useless. I have (re-)confirmed manually that it does on a fresh installation. Hence you have a local/conf issue and that becomes a support issue which this is not the place for. You can go to commercial support or the community forum for that.
My suggestion to close this still holds since this is no longer relevant and there have been a lot of releases since. But don't worry, I won't bother you any longer.
Saying it doesn't work on Debian 12 or Ubuntu 24.04 doesn't help nobody and is totally useless.
Why it is useless? It does not work properly after upgrading MailCow to 2024-01e
. The previous release does not show the problem.
reproduce it on a clean system.
Please clarify what you unterstand unter a clean system. Why do you assume that those involved with the problem do not have a clean system?
How does it help those (if this is done and it works) who have the problem on their unclean system?
I never said that the problem is not on my side and I am willing to spend money. But as long as no one offers to help, I can only throw money away.
However, since several people have reported the problem, your statement is more than questionable. We cannot always set up a fresh system to get around a problem when there are problems.
Would you, dear @apio-sys , like to earn money and solve the problem? Without reinstalling, of course.
No I would not like to earn money to solve the problem. A solution has been given by @erichk4 and confirmed working for others also. Hence case closed. IMHO.
Guy, the solution written down by @erichk4 does not work on my side and @tmwtnbg0.
What do you want to achieve with always writing "case closed"? The problem isn't solved!
Then why do you write in your feedback from 11 Feb
Upgrade of mailcow to 2024-01e without any problem.
?
You've read the sentences above this one and understood them?
Apart from unnecessary notifications, I don't see any progress here through your comments. You skillfully avoid questions and don't answer them.
This clean system talk does not bring us one step forward or close to a solution ... my system is a standard system, there are no fundamental changes to it; moreover, nothing is mentioned in the requirements about a clean system, my system fulfills all requirements there > so mailcow must run completely and without errors or there is a problem with mailcow
Apart from Debian 10 being EOL (which indeed is true) the initial problem is most likely related to changes within the mailcow update mentioned in my first post. And the steps mentioned by @erichk4 to fix this did not resolve my issue.
I had to disabled the netfilter rules with the ENV variable. As the server will be decomissioned soon for several reasons (Debian 10 being EOL is only one of them) I will not dig deeper into it.
But I'm backing @Fighter456 's opinion that closing an issue which is not really solved should not be the way to go here.
So better leave this open and add a flag like "information needed" or "tests needed" to make clear that this issue still persists even if two possible workarounds exist.
Maybe this issue should concentrate on fixing the issues with the Debian 12 users from now on?
Contribution guidelines
I've found a bug and checked that ...
Description
Updating my machine like usual to the newest mailcow 2024-01e (and I think there was also a docker update) broke my UFW / iptables setup on one of my servers which is using Debian 10.
If I shutdown the mailcow container and reboot the machine everything is fine and I get an output from
iptables -L
and also fromufw status
I have two other docker containers on the same machine and they are working fine. I tried several combinations like shutting down all containers, reboot, check ufw/iptables and starting only other containers and not mailcow, reboot, check ufw/iptables.
The result was clear:
As soon as I start the mailcow docker containers with
docker compose up
both outputs break and also the firewall functionality. I can only bring it back by shutting down mailcow, reenabling UFW withufw enable
, restarting the machine and reenabling UFW again withufw enable
.Maybe the issue has to do with the Netfilter Enhancements like stated here?
Logs:
docker compose logs -t -f
https://paste.armbian.com/izeqepuzim.yaml
(text was too long)
Steps to reproduce:
Shutdown mailcow with
docker compose down
Reboot machine Checkufw status
Output:
iptables -L
Output: The usual IP tables entries
Now start mailcow with
docker compose up -d
Check
ufw status
Output:
Start
ufw enable
Output:
Check
ufw reload
Output:
Check
iptables -L
Output:
Shutdown mailcow with
docker compose down
Check commands again like above: Same result
Only way to bring everything up again is to either disable ufw with
ufw disable
and reboot the machine or to shutdown mailcow, reenabling UFW withufw enable
, restarting the machine and reenabling UFW again withufw enable
.But I would really like to use both like I did for years now 😅
Which branch are you using?
master
Which architecture are you using?
x86
Operating System:
Debian GNU/Linux 10 (buster)
Server/VM specifications:
32GB RAM, 6 Cores
Is Apparmor, SELinux or similar active?
no
Virtualization technology:
KVM
Docker version:
25.0.3
docker-compose version or docker compose version:
v2.24.5
mailcow version:
2024-01e
Reverse proxy:
nginx
Logs of git diff:
Logs of iptables -L -vn:
Logs of ip6tables -L -vn:
Logs of iptables -L -vn -t nat:
Logs of ip6tables -L -vn -t nat:
DNS check: