Different behavior in portforwarding between Kali Linux 2022.4 vs. Ubuntu 22.04.2

[gfarrasb]

Hi!

We have this scenario:

pc1[0]="A" (internal ip: 1.2.3.4 with ssh service in open state) firewall[0]="A" firewall[1]="B" firewall[bridged]="true" (this firewall has three interfaces). firewall[sysctl]="net.ipv4.conf.all.forwarding=1"

On the firewalll.startup we have these lines to portforward 22 TCP port into an internal PC (pc1):

iptables -t nat -A PREROUTING -p tcp --dport 22 -j DNAT --to 1.2.3.4 iptables -t nat -A POSTROUTING -d 1.2.3.4 -j SNAT --to-source 172.17.0.2 -o eth2

With exactly the same scenario and with the same configuration in two different operative systems, we have different results when nmap external firewall interface: ubuntu filters and kali permits the connection to the internal port.

.

Any idea to solve this issue ? Thank you!

[gfarrasb]

In Ubuntu I have Docker version 23.0.1 installed. In Kali, 20.10.23.

[tcaiazzi]

Hi @gfarrasb,

Thank you for opening the issue!

Do you have an active firewall on your Ubuntu host?

Please, can you attach your lab so I can do some testing?

Thanks, Tommaso

[gfarrasb]

I have exactly the same scenario and configuration in Ubuntu than in Kali.

Of course! testkat.zip

I'm from the Open University of Catalonia (Barcelona). We are testing Kathará. Thank you very much!

When I try to nmap the external ip I'm unable to portforward ports (only in Ubuntu, not in Kali):

[tcaiazzi]

Hi @gfarrasb,

We are very happy that you are testing Kathará!

Sorry for the late reply! Your lab triggers a very tricky scenario :smile:

TL; DR; Skip to the end for the fix! :innocent:

Here is what I understood.

I found that the problem is related to the usage of the Docker network (by default on Ubuntu, 172.17.0.0/16) inside a network scenario. Currently, Kathará uses Linux bridges to implement collision domains, deploying one Linux bridge for each collision domain. So, the host's iptables rules installed by Docker are used also inside the Kathará bridges. Among them there is the following rule:

-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE

that masquerades all the packets that have the source IP belonging to the Docker network.

For the sake of completeness, these are all the iptables rules on my Ubuntu host, while running the network scenario:

# Generated by iptables-save v1.8.7 on Tue May  2 22:04:24 2023
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
:nat - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-d9d15e015e46 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-d9d15e015e46 -j DOCKER
-A FORWARD -i br-d9d15e015e46 ! -o br-d9d15e015e46 -j ACCEPT
-A FORWARD -i br-d9d15e015e46 -o br-d9d15e015e46 -j ACCEPT
-A FORWARD -i kt-368ca7bf38b9 -o kt-368ca7bf38b9 -j ACCEPT
-A FORWARD -i kt-9b93dfc788bc -o kt-9b93dfc788bc -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-d9d15e015e46 ! -o br-d9d15e015e46 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-d9d15e015e46 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Tue May  2 22:04:24 2023
# Generated by iptables-save v1.8.7 on Tue May  2 22:04:24 2023
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 192.168.49.0/24 ! -o br-d9d15e015e46 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i br-d9d15e015e46 -j RETURN
COMMIT
# Completed on Tue May  2 22:04:24 2023

Now, let's consider your configuration line:

iptables -t nat -A POSTROUTING -d 1.2.3.4 -j SNAT --to-source 172.17.0.2 

This changes the source IP of the packets destinated to 1.2.3.4 with 172.17.0.2 that belongs to the 172.17.0.0/16 network. At this point, when the packet traverses the Kathará bridge of the collision domain A, it will trigger the following rule:

-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE

So, pc1 receives packets that have as source IP the one of your physical interface on the host that has the default route, and when the answer is sent back the packet is dropped (probably by the bridge of the collision domain A). I have to figure out exactly why, if you have any idea, please let me know!

However, to fix the problem you have two options.

First, you can change the iptables rule with ANY other IP as source (outside 172.17.0.0/16):

iptables -t nat -A POSTROUTING -d 1.2.3.4 -j SNAT --to-source <ANY-IP>

The following is a reasonable example:

iptables -t nat -A POSTROUTING -d 1.2.3.4 -j SNAT --to-source 1.2.3.1

\<ANY-IP> works since the SNAT rule properly changes the IP addresses on both incoming and outcoming packets.

I attached you a fixed copy of the lab: testkat.zip.

Second, you can prevent the host from calling iptables rules when a packet is processed by a bridge changing the kernel parameter net.bridge.bridge-nf-call-iptables to zero. To do this you can execute this command:

sysctl net.bridge.bridge-nf-call-iptables=0

Can you check if on your Kali host the value of this parameter is zero? This would explain the different behaviours :smile:

Let me know if I can help any further!

Thanks, Tommaso

[gfarrasb]

Hi Tommaso! Thank you very much for your answer!

Kali has also this kernel value set to 1:

sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-iptables = 1

(I attach here an screenshoot).

That's the Kali iptables:

filter :INPUT ACCEPT [0:0] :FORWARD DROP [0:0] :OUTPUT ACCEPT [0:0] :DOCKER - [0:0] :DOCKER-ISOLATION-STAGE-1 - [0:0] :DOCKER-ISOLATION-STAGE-2 - [0:0] :DOCKER-USER - [0:0] -A FORWARD -j DOCKER-USER -A FORWARD -j DOCKER-ISOLATION-STAGE-1 -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A FORWARD -o docker0 -j DOCKER -A FORWARD -i docker0 ! -o docker0 -j ACCEPT -A FORWARD -i docker0 -o docker0 -j ACCEPT -A FORWARD -i kt-f325bf463017 -o kt-f325bf463017 -j ACCEPT -A FORWARD -i kt-4a6559d69350 -o kt-4a6559d69350 -j ACCEPT -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2 -A DOCKER-ISOLATION-STAGE-1 -j RETURN -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP -A DOCKER-ISOLATION-STAGE-2 -j RETURN -A DOCKER-USER -j RETURN COMMIT nat :PREROUTING ACCEPT [0:0] :INPUT ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :POSTROUTING ACCEPT [0:0] :DOCKER - [0:0] -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE -A DOCKER -i docker0 -j RETURN COMMIT

[tcaiazzi]

Hi @gfarrasb,

I tested the lab also on a Kali host and effectively it works.

Checking the iptables rules' counter, I spotted that this rule is triggered only on the Kali host, even if the set of rules is the same for Ubuntu and Kali:

Kali: Ubuntu:

I'll investigate further to come up with a proper explanation :innocent: Maybe there are some differences in the kernel parameters that cause the different behaviours.

If you have any idea, please let me know.

Many thanks, Tommaso

[tcaiazzi]

Hi @gfarrasb,

With the new version of Kathará (3.7.0) we released a new Docker Network Plugin based on VDE.

The VDE plugin is set as default. With this plugin when Docker creates networks, a new VDE switch process is created for each required LAN. In this way, we overcome many limits of the old plugin (which uses Linux Bridges) and there should be no differences in the behavior among different Linux distros.

I'm closing the issue since the problem should be resolved, but feel free to re-open it, if needed.

Tommaso.