Not all MAP-E assigned ports are being used in the 4-to-6 tunnel

[fakemanhk]

Discussed in https://github.com/fakemanhk/openwrt-jp-ipoe/discussions/7

^{Originally posted by **cmspam** February 22, 2023} If you just do the setup according to your current instructions, there is a missing piece. Although it may vary depending on MAP-E provider, in the case of JPNE (v6Plus) 240 ports are available for usage. However, with your instructions, only the first set of ports will be used. In the case of JPNE/v6plus, this is only 15 ports. Therefore, once the ports are used up, it will be impossible to make connections because it doesn't SNAT to the other sets of ports. This is the reason that sites like https://www.nichiban.co.jp/ which open a lot of sessions and use a lot of ports will have issues. The Japanese community has worked around this by implementing iptables /nftables rules which mark round-robin, a fwmark between 0xa and 0x18 in the prerouting table, then implement in the map.sh script a mark checking and routing using the corresponding port set. I have personally noticed it makes a big difference and solves the above-mentioned issues. I think it would be good to add information and instructions to this, if this is to be the English resource for using OpenWRT with Japanese Map-E. I can provide the following map.sh file and firewall.user file which are compatible with fw4, and fix the issues with both not SNAT-ing to all the ports, and also fix ping on fw4. It works well with JPNE / v6plus, but I haven't an opportunity to use it with other Map-E providers, which may have different port count, in which case the firewall.user script would need modifications. It would also be necessary to run the firewall.user file which requires the addition of the following to /etc/config/firewall config include option path '/etc/firewall.user' option fw4_compatible '1' [map.sh.txt](https://github.com/fakemanhk/openwrt-jp-ipoe/files/10801112/map.sh.txt) [firewall.user.txt](https://github.com/fakemanhk/openwrt-jp-ipoe/files/10801113/firewall.user.txt)

[cmspam]

Here are new files which fix everything. We are just using fwmark masks instead of straight fwmarks, that is the difference from above. Policy based routing and mwan3 confirmed to work. Ping also working.

map.sh.txt firewall.user.txt

(A small note. It should be OK with anything like mwan3 or pbr because neither use the last two bytes, but if you need to use firewall mark masks, you need to avoid the last two bytes, as they are used to control which portset packets are routed on. If you are going to manually use firewall marks for something else, make sure you use masks and not change the whole mark, and also avoid the last two bytes.)

(Another note, I am by no means an expert so consider this a hack, not a solution.)

[fakemanhk]

For easy viewing I put the diff between the stock map.sh and your edited version (map.test) here:

root@OpenWrt:/lib/netifd/proto# diff -c map.sh map.test
*** map.sh      Thu Aug 20 04:14:00 2020
--- map.test    Thu Feb 23 15:03:59 2023
***************
*** 140,145 ****
--- 140,148 ----
              json_add_string snat_ip $(eval "echo \$RULE_${k}_IPV4ADDR")
            json_close_object
          else
+ local mark=10
+ nft add table inet nathack
+ nft add chain inet nathack srcnat {type nat hook postrouting priority 0\; policy accept\; }
            for portset in $(eval "echo \$RULE_${k}_PORTSETS"); do
                for proto in icmp tcp udp; do
                json_add_object ""
***************
*** 147,157 ****
--- 150,165 ----
                  json_add_string target SNAT
                  json_add_string family inet
                  json_add_string proto "$proto"
+                 json_add_string mark "$mark/0x0000ff"
                    json_add_boolean connlimit_ports 1
                    json_add_string snat_ip $(eval "echo \$RULE_${k}_IPV4ADDR")
                    json_add_string snat_port "$portset"
                json_close_object
                done
+             nft add rule inet nathack srcnat meta nfproto ipv4 meta l4proto icmp oifname "map-$cfg" meta mark and 0x0000ff ==  $mark counter packets 0 bytes 0 snat ip to $(eval "echo \$RULE_${k}_IPV4ADDR"):$portset
+ 
+             mark=`expr $mark + 1`
+ 
            done
          fi
          if [ "$maptype" = "map-t" ]; then

Here is the content of firewall.user:

root@OpenWrt:/etc# cat firewall.user 
nft add table ip nat
nft add chain ip nat PREROUTING { type filter hook prerouting priority 0 \; }
nft add chain ip nat OUTPUT { type filter hook output priority 0 \; }
nft add rule ip nat PREROUTING numgen inc mod 15 0 counter meta mark set meta mark and 0xffffff00 or 0xa 
nft add rule ip nat PREROUTING numgen inc mod 15 1 counter meta mark set meta mark and 0xffffff00 or 0xb 
nft add rule ip nat PREROUTING numgen inc mod 15 2 counter meta mark set meta mark and 0xffffff00 or 0xc 
nft add rule ip nat PREROUTING numgen inc mod 15 3 counter meta mark set meta mark and 0xffffff00 or 0xd 
nft add rule ip nat PREROUTING numgen inc mod 15 4 counter meta mark set meta mark and 0xffffff00 or 0xe 
nft add rule ip nat PREROUTING numgen inc mod 15 5 counter meta mark set meta mark and 0xffffff00 or 0xf 
nft add rule ip nat PREROUTING numgen inc mod 15 6 counter meta mark set meta mark and 0xffffff00 or 0x10 
nft add rule ip nat PREROUTING numgen inc mod 15 7 counter meta mark set meta mark and 0xffffff00 or 0x11 
nft add rule ip nat PREROUTING numgen inc mod 15 8 counter meta mark set meta mark and 0xffffff00 or 0x12 
nft add rule ip nat PREROUTING numgen inc mod 15 9 counter meta mark set meta mark and 0xffffff00 or 0x13 
nft add rule ip nat PREROUTING numgen inc mod 15 10 counter meta mark set meta mark and 0xffffff00 or 0x14 
nft add rule ip nat PREROUTING numgen inc mod 15 11 counter meta mark set meta mark and 0xffffff00 or 0x15 
nft add rule ip nat PREROUTING numgen inc mod 15 12 counter meta mark set meta mark and 0xffffff00 or 0x16 
nft add rule ip nat PREROUTING numgen inc mod 15 13 counter meta mark set meta mark and 0xffffff00 or 0x17 
nft add rule ip nat PREROUTING numgen inc mod 15 14 counter meta mark set meta mark and 0xffffff00 or 0x18 
nft add rule ip nat OUTPUT numgen inc mod 15 0 counter meta mark set meta mark and 0xffffff00 or 0xa 
nft add rule ip nat OUTPUT numgen inc mod 15 1 counter meta mark set meta mark and 0xffffff00 or 0xb 
nft add rule ip nat OUTPUT numgen inc mod 15 2 counter meta mark set meta mark and 0xffffff00 or 0xc 
nft add rule ip nat OUTPUT numgen inc mod 15 3 counter meta mark set meta mark and 0xffffff00 or 0xd 
nft add rule ip nat OUTPUT numgen inc mod 15 4 counter meta mark set meta mark and 0xffffff00 or 0xe 
nft add rule ip nat OUTPUT numgen inc mod 15 5 counter meta mark set meta mark and 0xffffff00 or 0xf 
nft add rule ip nat OUTPUT numgen inc mod 15 6 counter meta mark set meta mark and 0xffffff00 or 0x10 
nft add rule ip nat OUTPUT numgen inc mod 15 7 counter meta mark set meta mark and 0xffffff00 or 0x11 
nft add rule ip nat OUTPUT numgen inc mod 15 8 counter meta mark set meta mark and 0xffffff00 or 0x12 
nft add rule ip nat OUTPUT numgen inc mod 15 9 counter meta mark set meta mark and 0xffffff00 or 0x13 
nft add rule ip nat OUTPUT numgen inc mod 15 10 counter meta mark set meta mark and 0xffffff00 or 0x14 
nft add rule ip nat OUTPUT numgen inc mod 15 11 counter meta mark set meta mark and 0xffffff00 or 0x15 
nft add rule ip nat OUTPUT numgen inc mod 15 12 counter meta mark set meta mark and 0xffffff00 or 0x16 
nft add rule ip nat OUTPUT numgen inc mod 15 13 counter meta mark set meta mark and 0xffffff00 or 0x17 
nft add rule ip nat OUTPUT numgen inc mod 15 14 counter meta mark set meta mark and 0xffffff00 or 0x18 

[fakemanhk]

The discussion thread here has an integrated solution by changing /lib/netifd/proto/map.sh.

Repost about the diff between old and new one:

root@OpenWrt:/lib/netifd/proto# mv map.sh.bak map.old
root@OpenWrt:/lib/netifd/proto# 
root@OpenWrt:/lib/netifd/proto# 
root@OpenWrt:/lib/netifd/proto# diff -c map.old map.sh 
*** map.old     Thu Feb 23 15:03:03 2023
--- map.sh      Fri Feb 24 19:57:14 2023
***************
*** 140,145 ****
--- 140,148 ----
              json_add_string snat_ip $(eval "echo \$RULE_${k}_IPV4ADDR")
            json_close_object
          else
+ local mark=10
+ nft add table inet nathack
+ nft add chain inet nathack srcnat {type nat hook postrouting priority 0\; policy accept\; }
            for portset in $(eval "echo \$RULE_${k}_PORTSETS"); do
                for proto in icmp tcp udp; do
                json_add_object ""
***************
*** 147,158 ****
--- 150,177 ----
                  json_add_string target SNAT
                  json_add_string family inet
                  json_add_string proto "$proto"
+                 json_add_string mark "$mark/0x0000ff"
                    json_add_boolean connlimit_ports 1
                    json_add_string snat_ip $(eval "echo \$RULE_${k}_IPV4ADDR")
                    json_add_string snat_port "$portset"
                json_close_object
                done
+             nft add rule inet nathack srcnat meta nfproto ipv4 meta l4proto icmp oifname "map-$cfg" meta mark and 0x0000ff ==  $mark counter packets 0 bytes 0 snat ip to $(eval "echo \$RULE_${k}_IPV4ADDR"):$portset
+ 
+             mark=`expr $mark + 1`
            done
+ nft add table ip nathack
+ nft add chain ip nathack PREROUTING { type filter hook prerouting priority 0 \; }
+ nft add chain ip nathack OUTPUT { type filter hook output priority 0 \; }
+ local mcount=0
+ local max=`expr $mark - 10`
+ mark=10
+               for portset in $(eval "echo \$RULE_${k}_PORTSETS"); do
+                       nft add rule ip nathack PREROUTING numgen inc mod $max $mcount counter meta mark set meta mark and 0xffffff00 or $mark
+                       nft add rule ip nathack OUTPUT numgen inc mod $max $mcount counter meta mark set meta mark and 0xffffff00 or $mark
+                       mark=`expr $mark + 1`
+                       mcount=`expr $mcount + 1`
+               done
          fi
          if [ "$maptype" = "map-t" ]; then
                [ -z "$zone" ] && zone=$(fw3 -q network $iface 2>/dev/null)

Tested by visiting https://www.nichiban.co.jp/ and the speed becomes a lot faster due to more port groups are being used instead of only single port group (16 ports).