search

NICMx / Jool

SIIT and NAT64 for Linux
GNU General Public License v2.0
326 stars 66 forks source link

Unable to connect via HTTPS to host www.cpc.ncep.noaa.gov using 464xlat #394

Closed gtxaspec closed 1 year ago

gtxaspec commented 1 year ago

Host T: Stateful NAT64 Translator Host V: ipv6 server using node based translation clat ( directions followed from https://nicmx.github.io/Jool/en/node-based-translation.html, using pool6 )

All running jool 4.1.8.0

host V ifconfig output:

clat: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.0.0.2  netmask 255.255.255.252  broadcast 0.0.0.0
        inet6 fe80::30a7:cfff:fe27:d868  prefixlen 64  scopeid 0x20<link>
        ether 32:a7:cf:27:d8:68  txqueuelen 1000  (Ethernet)
        RX packets 4463  bytes 1445327 (1.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4453  bytes 1439659 (1.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 2001:db8:ba03:3800::1  prefixlen 128  scopeid 0x0<global>
        inet6 fe80::216:3eff:fe08:ca32  prefixlen 64  scopeid 0x20<link>
        ether 00:16:3e:08:ca:32  txqueuelen 1000  (Ethernet)
        RX packets 102210  bytes 87118723 (83.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 120461  bytes 817555570 (779.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 48  bytes 29256 (28.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 48  bytes 29256 (28.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

host V: 

curl -4 https://www.cpc.ncep.noaa.gov/products/predictions/30day/ -vv
*   Trying 140.90.101.19:443...
* Connected to www.cpc.ncep.noaa.gov (140.90.101.19) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* * Operation timed out after 300958 milliseconds with 0 out of 0 bytes received
* Closing connection 0
curl: (28) Operation timed out after 300958 milliseconds with 0 out of 0 bytes received

host V tcpdump of the above curl:

root@http:/# tcpdump -i clat
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on clat, link-type EN10MB (Ethernet), snapshot length 262144 bytes
05:04:07.849981 IP 192.0.0.2.39136 > vm-lnx-wwwdm9_pub.ncep.noaa.gov.https: Flags [S], seq 2093154993, win 64240, options [mss 1460,sackOK,TS val 3015688984 ecr 0,nop,wscale 7], length 0
05:04:07.850168 IP6 2001:db8:b0a3:3800:38c1:67ff:feb3:1307.39136 > vm-lnx-wwwdm9_pub.ncep.noaa.gov.https: Flags [S], seq 2093154993, win 64240, options [mss 1460,sackOK,TS val 3015688984 ecr 0,nop,wscale 7], length 0
05:04:07.869068 IP6 vm-lnx-wwwdm9_pub.ncep.noaa.gov.https > 2001:db8:b0a3:3800:38c1:67ff:feb3:1307.39136: Flags [S.], seq 745200438, ack 2093154994, win 32768, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
05:04:07.869085 IP vm-lnx-wwwdm9_pub.ncep.noaa.gov.https > 192.0.0.2.39136: Flags [S.], seq 745200438, ack 2093154994, win 32768, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
05:04:07.869111 IP 192.0.0.2.39136 > vm-lnx-wwwdm9_pub.ncep.noaa.gov.https: Flags [.], ack 1, win 502, length 0
05:04:07.869118 IP6 2001:db8:b0a3:3800:38c1:67ff:feb3:1307.39136 > vm-lnx-wwwdm9_pub.ncep.noaa.gov.https: Flags [.], ack 1, win 502, length 0
05:04:07.875606 IP 192.0.0.2.39136 > vm-lnx-wwwdm9_pub.ncep.noaa.gov.https: Flags [P.], seq 1:518, ack 1, win 502, length 517
05:04:07.875661 IP6 2001:db8:b0a3:3800:38c1:67ff:feb3:1307.39136 > vm-lnx-wwwdm9_pub.ncep.noaa.gov.https: Flags [P.], seq 1:518, ack 1, win 502, length 517
05:04:07.894997 IP6 vm-lnx-wwwdm9_pub.ncep.noaa.gov.https > 2001:db8:b0a3:3800:38c1:67ff:feb3:1307.39136: Flags [P.], seq 2921:4170, ack 518, win 508, length 1249
05:04:07.895016 IP vm-lnx-wwwdm9_pub.ncep.noaa.gov.https > 192.0.0.2.39136: Flags [P.], seq 2921:4170, ack 518, win 508, length 1249
05:04:07.895029 IP 192.0.0.2.39136 > vm-lnx-wwwdm9_pub.ncep.noaa.gov.https: Flags [.], ack 1, win 502, options [nop,nop,sack 1 {2921:4170}], length 0
05:04:07.895035 IP6 2001:db8:b0a3:3800:38c1:67ff:feb3:1307.39136 > vm-lnx-wwwdm9_pub.ncep.noaa.gov.https: Flags [.], ack 1, win 502, options [nop,nop,sack 1 {2921:4170}], length 0
05:04:07.912097 IP6 vm-lnx-wwwdm9_pub.ncep.noaa.gov.https > 2001:db8:b0a3:3800:38c1:67ff:feb3:1307.39136: Flags [P.], seq 4170:4768, ack 518, win 508, length 598
05:04:07.912116 IP vm-lnx-wwwdm9_pub.ncep.noaa.gov.https > 192.0.0.2.39136: Flags [P.], seq 4170:4768, ack 518, win 508, length 598
05:04:07.912128 IP 192.0.0.2.39136 > vm-lnx-wwwdm9_pub.ncep.noaa.gov.https: Flags [.], ack 1, win 502, options [nop,nop,sack 1 {2921:4768}], length 0
05:04:07.912134 IP6 2001:db8:b0a3:3800:38c1:67ff:feb3:1307.39136 > vm-lnx-wwwdm9_pub.ncep.noaa.gov.https: Flags [.], ack 1, win 502, options [nop,nop,sack 1 {2921:4768}], length 0
05:04:08.957767 IP6 vm-lnx-wwwdm9_pub.ncep.noaa.gov.https > 2001:db8:b0a3:3800:38c1:67ff:feb3:1307.39136: Flags [P.], seq 2921:4170, ack 518, win 508, length 1249
05:04:08.957770 IP6 vm-lnx-wwwdm9_pub.ncep.noaa.gov.https > 2001:db8:b0a3:3800:38c1:67ff:feb3:1307.39136: Flags [P.], seq 4170:4768, ack 518, win 508, length 598
05:04:08.957784 IP vm-lnx-wwwdm9_pub.ncep.noaa.gov.https > 192.0.0.2.39136: Flags [P.], seq 2921:4170, ack 518, win 508, length 1249
05:04:08.957799 IP 192.0.0.2.39136 > vm-lnx-wwwdm9_pub.ncep.noaa.gov.https: Flags [.], ack 1, win 502, options [nop,nop,sack 2 {2921:4170}{2921:4768}], length 0
05:04:08.957786 IP vm-lnx-wwwdm9_pub.ncep.noaa.gov.https > 192.0.0.2.39136: Flags [P.], seq 4170:4768, ack 518, win 508, length 598
05:04:08.957805 IP6 2001:db8:b0a3:3800:38c1:67ff:feb3:1307.39136 > vm-lnx-wwwdm9_pub.ncep.noaa.gov.https: Flags [.], ack 1, win 502, options [nop,nop,sack 2 {2921:4170}{2921:4768}], length 0
05:04:08.957930 IP 192.0.0.2.39136 > vm-lnx-wwwdm9_pub.ncep.noaa.gov.https: Flags [.], ack 1, win 502, options [nop,nop,sack 2 {4170:4768}{2921:4768}], length 0
05:04:08.957935 IP6 2001:db8:b0a3:3800:38c1:67ff:feb3:1307.39136 > vm-lnx-wwwdm9_pub.ncep.noaa.gov.https: Flags [.], ack 1, win 502, options [nop,nop,sack 2 {4170:4768}{2921:4768}], length 0
05:04:10.967704 IP6 vm-lnx-wwwdm9_pub.ncep.noaa.gov.https > 2001:db8:b0a3:3800:38c1:67ff:feb3:1307.39136: Flags [P.], seq 2921:4170, ack 518, win 508, length 1249
05:04:10.967708 IP6 vm-lnx-wwwdm9_pub.ncep.noaa.gov.https > 2001:db8:b0a3:3800:38c1:67ff:feb3:1307.39136: Flags [P.], seq 4170:4768, ack 518, win 508, length 598
05:04:10.967720 IP vm-lnx-wwwdm9_pub.ncep.noaa.gov.https > 192.0.0.2.39136: Flags [P.], seq 2921:4170, ack 518, win 508, length 1249
05:04:10.967721 IP vm-lnx-wwwdm9_pub.ncep.noaa.gov.https > 192.0.0.2.39136: Flags [P.], seq 4170:4768, ack 518, win 508, length 598
05:04:10.967866 IP 192.0.0.2.39136 > vm-lnx-wwwdm9_pub.ncep.noaa.gov.https: Flags [.], ack 1, win 502, options [nop,nop,sack 2 {4170:4768}{2921:4768}], length 0
05:04:10.967873 IP6 2001:db8:b0a3:3800:38c1:67ff:feb3:1307.39136 > vm-lnx-wwwdm9_pub.ncep.noaa.gov.https: Flags [.], ack 1, win 502, options [nop,nop,sack 2 {4170:4768}{2921:4768}], length 0
05:04:14.977719 IP6 vm-lnx-wwwdm9_pub.ncep.noaa.gov.https > 2001:db8:b0a3:3800:38c1:67ff:feb3:1307.39136: Flags [P.], seq 2921:4170, ack 518, win 508, length 1249
05:04:14.977723 IP6 vm-lnx-wwwdm9_pub.ncep.noaa.gov.https > 2001:db8:b0a3:3800:38c1:67ff:feb3:1307.39136: Flags [P.], seq 4170:4768, ack 518, win 508, length 598
05:04:14.977737 IP vm-lnx-wwwdm9_pub.ncep.noaa.gov.https > 192.0.0.2.39136: Flags [P.], seq 2921:4170, ack 518, win 508, length 1249
05:04:14.977739 IP vm-lnx-wwwdm9_pub.ncep.noaa.gov.https > 192.0.0.2.39136: Flags [P.], seq 4170:4768, ack 518, win 508, length 598
05:04:14.977856 IP 192.0.0.2.39136 > vm-lnx-wwwdm9_pub.ncep.noaa.gov.https: Flags [.], ack 1, win 502, options [nop,nop,sack 2 {4170:4768}{2921:4768}], length 0
05:04:14.977871 IP6 2001:db8:b0a3:3800:38c1:67ff:feb3:1307.39136 > vm-lnx-wwwdm9_pub.ncep.noaa.gov.https: Flags [.], ack 1, win 502, options [nop,nop,sack 2 {4170:4768}{2921:4768}], length 0

works as expected if i curl it from T directly... also works fine from V using DNS64 w/NAT64 from T also works using clatd + tayga... but I like jool =D

curl https://www.cpc.ncep.noaa.gov -vvvv
*   Trying  2001:db8:b0a3:4::8c5a:6513:443...
* Connected to www.cpc.ncep.noaa.gov ( 2001:db8:b0a3:4::8c5a:6513) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=ncep.noaa.gov
*  start date: Nov 29 06:13:07 2022 GMT
*  expire date: Feb 27 06:13:06 2023 GMT
*  subjectAltName: host "www.cpc.ncep.noaa.gov" matched cert's "www.cpc.ncep.noaa.gov"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: www.cpc.ncep.noaa.gov
> User-Agent: curl/7.74.0
> Accept: */*
***proper html output cut***

What would be preventing the TLS handshake from working properly while using jool?

ydahhrk commented 1 year ago

Ok, forget everything I said before; I brainfarted badly.

So the way I see it, your network is like this, right?

+--------+  IPv6  +---------+  IPv4  +-----------------------+
| V      |--------| T       |--------| www.cpc.ncep.noaa.gov |
| (CLAT) |        | (NAT64) |        |                       |
+--------+        +---------+        +-----------------------+

You're telling me a curl from T to cpc works. IPv6 from V to cpc works as well. IPv4 from V to cpc doesn't.

Ok. I'm currently abroad, and lack access to my hardware. I'll be back on Monday, though I will probably have to postpone the actual testing until Tuesday.

But I can ask some preliminary questions:

  1. What's your distro? It seems odd that you're using ifconfig instead of ip address. Is this because of the distro?
  2. What happens if you ping from V's outer namespace to V's inner namespace (using IPv4)? What happens if you ping from V's inner namespace to T (using IPv6)?
  3. Did you try reaching other IPv4 servers, aside from cpc?

Actually, your tcpdump implies the endpoints can reach each other. Guess I really can't solve this without hitting the lab.

ydahhrk commented 1 year ago

Comment above heavily edited; please disregard the email version.

gtxaspec commented 1 year ago

@ydahhrk thank you for looking at this.

You're telling me a curl from T to cpc works. IPv6 from V to cpc works as well. IPv4 from V to cpc doesn't.

cpc is ipv4 only T to cpc works IPv6 from V to cpc works (dns64) IPv4 from V to cpc doesn't work

  1. Debian bullseye, just my stubbornness to avoid using ip a =D
  2. See below:
  3. The problem happens with https://tgftp.nws.noaa.gov ,https://tgftp.op.ncep.noaa.gov, as well as https://tgftp.bldr.ncep.noaa.gov. Other services seem to work.

Ping from V's outer namespace to V's inner namespace:

root@V:~# traceroute 192.0.0.1
traceroute to 192.0.0.1 (192.0.0.1), 30 hops max, 60 byte packets
 1  192.0.0.1 (192.0.0.1)  0.035 ms  0.005 ms  0.004 ms
root@V:~# ping 192.0.0.1
PING 192.0.0.1 (192.0.0.1) 56(84) bytes of data.
64 bytes from 192.0.0.1: icmp_seq=1 ttl=64 time=0.034 ms
64 bytes from 192.0.0.1: icmp_seq=2 ttl=64 time=0.047 ms
64 bytes from 192.0.0.1: icmp_seq=3 ttl=64 time=0.049 ms
--- 192.0.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2034ms
rtt min/avg/max/mdev = 0.034/0.043/0.049/0.006 ms

From V inner namespace to T using ipv6:

root@V:~# ip netns exec jool ping 2001:db7:b0a3:3000::1
PING 2001:db7:b0a3:3000::1(2001:db7:b0a3:3000::1) 56 data bytes
64 bytes from 2001:db7:b0a3:3000::1: icmp_seq=1 ttl=62 time=0.095 ms
64 bytes from 2001:db7:b0a3:3000::1: icmp_seq=2 ttl=62 time=0.098 ms
--- 2001:db7:b0a3:3000::1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1030ms
rtt min/avg/max/mdev = 0.095/0.096/0.098/0.001 ms
hagodoy commented 1 year ago

Hi @gtxaspec and @ydahhrk

At the university, I have an environment with a Wi-Fi network using 464XLAT. I have noticed the same behavior reported on some sites, only when the client machine only uses IPv4 on its interface.

Windows client machine IP:

Network Adapter Wi-Fi:

Sufixo DNS específico de conexão. . . . . . : fca.unicamp.br Endereço IPv4. . . . . . . . . . . . . . . : 192.168.200.111 Máscara de Sub-rede . . . . . . . . . . . . : 255.255.255.0 Gateway Padrão. . . . . . . . . . . . . . . : 192.168.200.1

The mapping in PLAT is proceeding normally:

Jool: default 2022/12/14 17:45:25 (GMT) - Added session 2801:8a:c040:200:6f00::#59753|64:ff9b::c8a0:406#443|143.106.230.240#47222|200.160.4.6#443|TCP

However, running a test in curl, stops when exchanging SSL certificates and does not transfer site data.

C:\Users\Henri\curl https://www.nic.br -vvvv

Using Wireshark, the exchange of SSL certificates appears, but then there is no data exchange. Interesting that several [TCP Dup ACK] appear, and there is no more response.

I am still trying to figure out what could be going on.

When the client machine receives the double stack, there is no problem because the path is made by IPv6, not involving the translation in Jool PLAT.

Thanks

emilylange commented 1 year ago

@gtxaspec can you try what happens after you run

ip link set clat mtu 1400

or

ip6tables -t mangle -A FORWARD -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
gtxaspec commented 1 year ago

@IndeedNotJames confirmed, connections now working with ip link set clat mtu 1400 or ip6tables -t mangle -A FORWARD -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu completely forgot about MTU in this situation... lol thank you!

hagodoy commented 1 year ago

Hi @gtxaspec and @IndeedNotJames

Wowwww very good, I confirm that changing the MTU of the network interface to 1400 of the client machines connected to WiFi, several sites worked. Great :-))

Now could someone explain the reason for this behavior only in this specific case, I was curious.

Thanks all !

ydahhrk commented 1 year ago

It's probably either this, or there's a router somewhere in the path blocking ICMP traffic.

You can't comfortably do PMTU discovery if some dumbass in the way is dropping PMTU messages.