NICMx / Jool

SIIT and NAT64 for Linux
GNU General Public License v2.0
326 stars 66 forks source link

ipv4 transfer to ipv6 failed under flannel network #324

Closed GeorgeGuo2018 closed 4 years ago

GeorgeGuo2018 commented 4 years ago

Hi, I deployed Jool in a pod within a k8s cluster with cni flannel. My config of Jool is as follows(using jool as nat64 mode):

jool instance add 'default' --iptables --pool6 fdff:ffff:ffff:ffff:3:0::/96
jool pool4 add 172.31.1.2  1-65535 --tcp

My client's IP is fe::10/96 ,and from it I start up a request fe::10 --> fdff:ffff:ffff:ffff:3:0::a06:73f4. Using tcpdump, within the Jool pod,I can see the request is transferred to 172.31.1.2 --> 10.6.115.244. And I can also get reply from 10.6.115.244, the response packet I get within jool pod is 10.6.115.244 --> 10.6.115.244. Using iptables -t mangle -L -n -v, I am sure the responsed ipv4 packages go to the JOOL module, because the packets num target by the -j JOOL rule is increased. But the those ipv4 packagets did not cameout as ipv6 packaget. something went wrong for the responsed ipv4 packet to be transferred back to ipv6.

I deploy the same Jool pod in a k8s cluster under calico network, and all works well. Seems that some thing was wrong when k8s's network is flannel.

Is there any logs I can check to find the reason? The following infos seems all right.

[root@nat64-7f984bb7bf-tjs2l /]# jool stats display
JSTAT_SUCCESS: 228
JSTAT_BIB_ENTRIES: 8
JSTAT_SESSIONS: 8
JSTAT_BIB4_NOT_FOUND: 8
JSTAT_SYN4_EXPECTED: 9
[root@nat64-7f984bb7bf-tjs2l /]# jool bib display  
[Dynamic TCP] 172.30.1.2#11896 - fe::10#41800
[Dynamic TCP] 172.30.1.2#11897 - fe::10#43694
[Dynamic TCP] 172.30.1.2#11898 - fe::10#57688
[root@nat64-7f984bb7bf-tjs2l /]# jool pool4 display
+------------+-------+--------------------+-----------------+-------------+
|       Mark | Proto |     Max iterations |         Address |       Ports |
+------------+-------+--------------------+-----------------+-------------+
|          0 |   TCP |       1024 ( auto) |      172.30.1.2 |     1-65535 |
+------------+-------+--------------------+-----------------+-------------+
[root@nat64-7f984bb7bf-tjs2l /]# jool session display
---------------------------------
(TRANS) Expires in 0:00:20.364
Remote: 10.6.115.244#41343  fe::10#41800
Local: 172.30.1.2#11896 fdff:ffff:ffff:ffff:3:0:a06:73f4#41343
---------------------------------
(TRANS) Expires in 0:03:20.724
Remote: 10.6.115.244#41343  fe::10#43694
Local: 172.30.1.2#11897 fdff:ffff:ffff:ffff:3:0:a06:73f4#41343
---------------------------------
(ESTABLISHED) Expires in 1:52:05.642
Remote: 10.6.115.244#41343  fe::10#57688
Local: 172.30.1.2#11898 fdff:ffff:ffff:ffff:3:0:a06:73f4#41343
---------------------------------
(ESTABLISHED) Expires in 1:54:22.858
Remote: 10.6.115.244#41343  fe::10#59204
Local: 172.30.1.2#11899 fdff:ffff:ffff:ffff:3:0:a06:73f4#41343
---------------------------------
(V6_INIT) Expires in 0:01:10.473
Remote: 10.6.115.244#41343  fe::10#33050
[root@nat64-7f984bb7bf-tjs2l /]#  ip6tables -t mangle -L -n
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
JOOL       all      ::/0                 fdff:ffff:ffff:ffff:3:0:a06:73f4  instance:default
JOOL       all      ::/0                 fdff:ffff:ffff:ffff:3:0:a06:73f5  instance:default

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
[root@nat64-7f984bb7bf-tjs2l /]#iptables -t mangle -L -n
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
JOOL       all  --  0.0.0.0/0            172.30.1.2           instance:default
GeorgeGuo2018 commented 4 years ago

It is clear that, the incoming ipv6 packets are transferred successfully, but not vice versa through the bib and session still exist. Is there any log or other commands i can check to figure out what was wrong during the 4-to-6 process?

my Envrionment: Host: centos 7.6 3.10.0-957.1.3.el7.x86_64 Jool: v4.0.7

ydahhrk commented 4 years ago

Disclaimer: I'm not familiar with "k8s clusters" nor "cni flannels."

Using tcpdump, within the Jool pod,I can see the request is transferred to 172.31.1.2 --> 10.6.115.244. And I can also get reply from 10.6.115.244, the response packet I get within jool pod is 10.6.115.244 --> 10.6.115.244.

This seems wrong to me.

If Jool translates a packet into 172.31.1.2 --> 10.6.115.244, the response needs to be 10.6.115.244 --> 172.31.1.2, otherwise Jool doesn't recognize it.

10.6.115.244 --> 10.6.115.244

Are you sure about this?

Is there any logs I can check to find the reason?

Yes, but you have to recompile Jool first.

In any case, your stats output already says a lot to me:

The following infos seems all right.

[root@nat64-7f984bb7bf-tjs2l /]# jool stats display JSTAT_SUCCESS: 228 JSTAT_BIB_ENTRIES: 8 JSTAT_SESSIONS: 8 JSTAT_BIB4_NOT_FOUND: 8 JSTAT_SYN4_EXPECTED: 9

BIB4_NOT_FOUND and SYN4_EXPECTED mean that it's receiving IPv4 packets for which it has no state. This is the same as my observation above: In the 6->4 direction, the packet became 172.31.1.2 --> 10.6.115.244, not 10.6.115.244 --> 10.6.115.244. Therefore, the BIB Jool has expects packet 10.6.115.244 --> 172.31.1.2, not 10.6.115.244 --> 10.6.115.244.

It is clear that, the incoming ipv6 packets are transferred successfully, but not vice versa through the bib and session still exist.

There is a serviceable BIB, but no session. If you want Jool to translate 10.6.115.244 --> 10.6.115.244, then you need a session that looks somewhat like this:

(ESTABLISHED) Expires in x:xx:xx.xxx
Remote: 10.6.115.244#xxxxx  xx::xx#xxxxx
Local: 10.6.115.244#xxxxx   xx::xx#xxxxx
GeorgeGuo2018 commented 4 years ago

Disclaimer: I'm not familiar with "k8s clusters" nor "cni flannels."

Using tcpdump, within the Jool pod,I can see the request is transferred to 172.31.1.2 --> 10.6.115.244. And I can also get reply from 10.6.115.244, the response packet I get within jool pod is 10.6.115.244 --> 10.6.115.244.

This seems wrong to me.

If Jool translates a packet into 172.31.1.2 --> 10.6.115.244, the response needs to be 10.6.115.244 --> 172.31.1.2, otherwise Jool doesn't recognize it.

10.6.115.244 --> 10.6.115.244

Are you sure about this?

Is there any logs I can check to find the reason?

Yes, but you have to recompile Jool first.

In any case, your stats output already says a lot to me:

The following infos seems all right. [root@nat64-7f984bb7bf-tjs2l /]# jool stats display JSTAT_SUCCESS: 228 JSTAT_BIB_ENTRIES: 8 JSTAT_SESSIONS: 8 JSTAT_BIB4_NOT_FOUND: 8 JSTAT_SYN4_EXPECTED: 9

BIB4_NOT_FOUND and SYN4_EXPECTED mean that it's receiving IPv4 packets for which it has no state. This is the same as my observation above: In the 6->4 direction, the packet became 172.31.1.2 --> 10.6.115.244, not 10.6.115.244 --> 10.6.115.244. Therefore, the BIB Jool has expects packet 10.6.115.244 --> 172.31.1.2, not 10.6.115.244 --> 10.6.115.244.

It is clear that, the incoming ipv6 packets are transferred successfully, but not vice versa through the bib and session still exist.

There is a serviceable BIB, but no session. If you want Jool to translate 10.6.115.244 --> 10.6.115.244, then you need a session that looks somewhat like this:

(ESTABLISHED) Expires in x:xx:xx.xxx
Remote: 10.6.115.244#41343    xx::xx#xxxxx
Local: 10.6.115.244#xxxxx xx::xx#xxxxx

Hi, ydahhrk. Sorry, my mistake. The response packet I get within jool pod is correct: 10.6.115.244 --> 172.31.1.2. It was a clerical error .

GeorgeGuo2018 commented 4 years ago

It seems that the 4-->6 direction didnot workout as expected. But how can i figure out what is going wrong. exec jool bib dispaly, jool stats dispaly, jool session display, did not get any exception info.

ydahhrk commented 4 years ago

Sorry, my mistake. The response packet I get within jool pod is correct: 10.6.115.244 --> 172.31.1.2. It was a clerical error .

But you iptables match address is 172.30.1.2, not 172.31.1.2. If your destination address is really 172.31.1.2, then iptables will never send it to Jool.

It seems that the 4-->6 direction didnot workout as expected. But how can i figure out what is going wrong. exec jool bib dispaly, jool stats dispaly, jool session display, did not get any exception info.

You can enable debug by following these instructions.

GeorgeGuo2018 commented 4 years ago

will try jool's debug mode. Thanks a lot.