Incompatibilty between iptables (legacy) and nftables for Debian 11 (hosts) and Alpine (v3.16?) in netfilter-mailcow-container

jfossheim-skyfritt commented 1 year ago

Contribution guidelines

[X] I've read the contribution guidelines and wholeheartedly agree

I've found a bug and checked that ...

[X] ... I understand that not following the below instructions will result in immediate closure and/or deletion of my issue.
[X] ... I have understood that this bug report is dedicated for bugs, and not for support-related inquiries.
[X] ... I have understood that answers are voluntary and community-driven, and not commercial support.
[X] ... I have verified that my issue has not been already answered in the past. I also checked previous issues.

Description

After updating to address forever-looping SNAT-issue, addressed here:

https://github.com/mailcow/mailcow-dockerized/issues/4658
https://github.com/mailcow/mailcow-dockerized/issues/4697
https://github.com/mailcow/mailcow-dockerized/pull/4724 

I noticed that SNAT was not added to the netfilter-mailcow-container, and started a thorough debug-session together with @mnin. 

@mnin changed only the condition to prevent adding an existing rule to the POSTROUTING chain in his PR: https://github.com/mailcow/mailcow-dockerized/pull/4724

He didn't touched the way how the rule are inserted into the chain. We started to suspect compatibility problems between iptables and nftables, since the host was running iptables v1.8.7 (nf_tables) and the docker-container is running iptables v1.8.8 (legacy).

Logs:

docker-compose logs netfilter-mailcow: 

mailcowdockerized-netfilter-mailcow-1  | Clearing all bans
mailcowdockerized-netfilter-mailcow-1  | Initializing mailcow netfilter chain
mailcowdockerized-netfilter-mailcow-1  | Watching Redis channel F2B_CHANNEL

From inside the netfilter-mailcow container:

# iptables -t nat -L POSTROUTING -nvx
Chain POSTROUTING (policy ACCEPT 454 packets, 29564 bytes)
    pkts      bytes target     prot opt in     out     source               destination
/ #

Steps to reproduce:

Updating to latest mailcow release (2022-10a), after november 26th 2022, on a Debian 11 host. 

Everything will seem to work straight after the update, but when you reboot/restart the host/docker, no SNAT rules are added.

Which branch are you using?

master

Operating System:

Debian 11, Buster

Server/VM specifications:

6GB / 2 Cores

Is Apparmor, SELinux or similar active?

yes

Virtualization technology:

KVM (proxmox)

Docker version:

20.10.18

docker-compose version or docker compose version:

v2.6.0

mailcow version:

2022-10a

Reverse proxy:

NO

Logs of git diff:

nothing

Logs of iptables -L -vn:

cannot reproduce

Logs of ip6tables -L -vn:

cannot reproduce

Logs of iptables -L -vn -t nat:

empty

Logs of ip6tables -L -vn -t nat:

empty

DNS check:

151.101.1.69
151.101.129.69
151.101.65.69
151.101.193.69

jfossheim-skyfritt commented 1 year ago

Workaround fix, first explained by me here:

https://github.com/mailcow/mailcow-dockerized/pull/4724#issuecomment-1330571686

Change the default firewall (from nftables to iptables-legacy) on the Debian host in the following way, will make the 'netfilter-mailcow'-container work as expected:

# update-alternatives --set iptables /usr/sbin/iptables-legacy
# update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

Explained here: https://wiki.debian.org/nftables

Then reboot the system, and everything behaves as one should expect:

root@mail:/opt/mailcow-dockerized# iptables -V
iptables v1.8.7 (legacy)
root@mail:/opt/mailcow-dockerized# docker-compose exec netfilter-mailcow sh
/ # iptables -V
iptables v1.8.8 (legacy)
/ # iptables -t nat -L POSTROUTING -nvx
Chain POSTROUTING (policy ACCEPT 328 packets, 22456 bytes)
    pkts      bytes target     prot opt in     out     source               destination
     688    52158 SNAT       all  --  *      *       172.22.1.0/24       !172.22.1.0/24        /* 1669724878 */ to:XXX.XXX.XXX.XXX
       0        0 MASQUERADE  all  --  *      !docker0  172.18.0.0/16        0.0.0.0/0
      59     4176 MASQUERADE  all  --  *      !br-mailcow  172.22.1.0/24        0.0.0.0/0
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:12345
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:4190
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:995
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:993
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:143
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:110
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.249         172.22.1.249         tcp dpt:6379
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.253         172.22.1.253         tcp dpt:587
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.253         172.22.1.253         tcp dpt:465
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.253         172.22.1.253         tcp dpt:25
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.10          172.22.1.10          tcp dpt:8983
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.13          172.22.1.13          tcp dpt:3306
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.7           172.22.1.7           tcp dpt:443
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.7           172.22.1.7           tcp dpt:80

jfossheim-skyfritt commented 1 year ago

Testing another hypothesis with @mnin by rolling back the 'netfilter-mailcow'-container to version 1.47 (Alpine 3.15.2) from 1.49 (Alpine 3.16.2) via docker-compose.yml:

  netfilter-mailcow:
      image: mailcow/netfilter:1.47
      stop_grace_period: 30s

And switch back to nftables on the Debian host:

# update-alternatives --set iptables /usr/sbin/iptables-nft
# update-alternatives --set ip6tables /usr/sbin/ip6tables-nft

Then rebooted the host, produced the following nat /POSTRUTING table:

root@mail:/opt/mailcow-dockerized# ip6tables -V
ip6tables v1.8.7 (nf_tables)
root@mail:/opt/mailcow-dockerized# docker-compose exec netfilter-mailcow sh
/ # ip6tables -V
ip6tables v1.8.7 (legacy)
/ # iptables -t nat -L POSTROUTING -nvx
Chain POSTROUTING (policy ACCEPT 353 packets, 21682 bytes)
    pkts      bytes target     prot opt in     out     source               destination
     118     8471 SNAT       all  --  *      *       172.22.1.0/24       !172.22.1.0/24        to:XXX.XXX.XXX.XXX
/ #

See https://github.com/mailcow/mailcow-dockerized/issues/4697 for further information, regarding Alpine-version and the SNAT-issues.

jfossheim-skyfritt commented 1 year ago

Testing 'netfilter-mailcow'-container to version 1.47 (Alpine 3.15.2) with 'iptables-legacy' on the host produces the following nat /POSTRUTING table:

root@mail:/opt/mailcow-dockerized# iptables -V
iptables v1.8.7 (legacy)
root@mail:/opt/mailcow-dockerized# docker-compose exec netfilter-mailcow sh
/ # iptables -V
iptables v1.8.7 (legacy)
/ # iptables -t nat -L POSTROUTING -nvx
Chain POSTROUTING (policy ACCEPT 416 packets, 25425 bytes)
    pkts      bytes target     prot opt in     out     source               destination
      90     6390 SNAT       all  --  *      *       172.22.1.0/24       !172.22.1.0/24        to:XXX.XXX.XXX.XXX
       0        0 MASQUERADE  all  --  *      !docker0  172.18.0.0/16        0.0.0.0/0
     107     7690 MASQUERADE  all  --  *      !br-mailcow  172.22.1.0/24        0.0.0.0/0
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.253         172.22.1.253         tcp dpt:587
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.253         172.22.1.253         tcp dpt:465
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.253         172.22.1.253         tcp dpt:25
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.8           172.22.1.8           tcp dpt:3306
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:12345
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:4190
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:995
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:993
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:143
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.250         172.22.1.250         tcp dpt:110
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.249         172.22.1.249         tcp dpt:6379
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.12          172.22.1.12          tcp dpt:8983
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.14          172.22.1.14          tcp dpt:443
       0        0 MASQUERADE  tcp  --  *      *       172.22.1.14          172.22.1.14          tcp dpt:80
/ #

Veske commented 1 year ago

Probably the same issue as: https://github.com/mailcow/mailcow-dockerized/issues/4802

milkmaker commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

SimonFischer04 commented 1 year ago

Same issue with 1.50 for me.

milkmaker commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

pfoo commented 1 year ago

On debian 12 host with fresh mailcow 2023-08 :

SNAT rule is added in iptables-legacy
other rules are added in iptables-nft

Mixing iptables-nft and iptables-legacy for the same ruleset feels like a bad idea

meinradr commented 9 months ago

On a fresh Debian 12 installation with mailcow 2023-12 the following iptables-legacy rules are added: -P INPUT ACCEPT -P FORWARD ACCEPT -P OUTPUT ACCEPT

Everything else is added in iptables-nft.

mailcow / mailcow-dockerized