mwan3: Google (selectively) blocking pings on their IPv6 servers?

[brianjmurrell]

Maintainer: @feckert Environment: N/A

Description: In the last day or two my IPv6 pings to the two Google IPv6 addresses (2001:4860:4860::8844 and 2001:4860:4860::8888) have started to fail, but only from my mwan3 running gateway. They have been failing on both WAN interfaces on that machine. Pings from other hosts in my network seem to be working fine.

I wonder if anyone else is seeing this behaviour. I wonder if those upstream servers are starting to block hosts that ping (or even otherwise use) them too frequently.

[brianjmurrell]

I should have made it clear, that pings to other IPv6 hosts (i.e. 2620:0:ccd::2) work fine, so the IPv6 stack on the host is in working order.

[efahl]

Can confirm, various google v6 addresses (including gstatic.com) no longer respond to pings. When I use some other host, e.g., dns.cloudflare.com in the mixed v4/v6 example below, it works fine.

$ speedtest-netperf.sh -H 10.1.1.1 -p 2606:4700::6810:85e5
2022-09-22 15:11:29 Starting speedtest for 60 seconds per transfer session.
Measure speed to 10.1.1.1 (IPv4) while pinging 2606:4700::6810:85e5.
Download and upload sessions are sequential, each with 5 simultaneous streams.
............................................................
 Download: 941.36 Mbps
  Latency: [in msec, 60 pings, 0.00% packet loss]
      Min:  10.500
    10pct:  11.100
   Median:  14.900
      Avg:  15.047
    90pct:  19.000
      Max:  25.100
...

All the DNS services that I've checked seem to respond to ping (including google), so maybe change 'gstatic.com' to 'dns.google.com'?

[efahl]

Oops, I see that you are using dns.google.com for pings... I was seeing issues with google.com and gstatic.com, all of which appear to be discarding pings.

[jamesmacwhite]

@brianjmurrell I am seeing some ping failures for Google DNS both IPv4 and IPv6, other DNS resolvers set on the interfaces i.e. OpenDNS do not appear to be failing.

Sat Sep 24 10:19:05 2022 user.info mwan3track[2005]: Check (ping) failed for target "2001:4860:4860::8844" on interface wan6 (6in4-wan6)
Sat Sep 24 10:26:40 2022 user.info mwan3track[11253]: Check (ping) failed for target "2001:4860:4860::8888" on interface wg6 (wg)
Sat Sep 24 10:35:33 2022 user.info mwan3track[2005]: Check (ping) failed for target "2001:4860:4860::8844" on interface wan6 (6in4-wan6)
Sat Sep 24 10:38:48 2022 user.info mwan3track[2005]: Check (ping) failed for target "2001:4860:4860::8888" on interface wan6 (6in4-wan6)
Sat Sep 24 11:21:35 2022 user.info mwan3track[9714]: Check (ping) failed for target "8.8.8.8" on interface wg (wg)
Sat Sep 24 11:26:48 2022 user.info mwan3track[15673]: Check (ping) failed for target "2001:4860:4860::8888" on interface wgb6 (wgb)
Sat Sep 24 12:01:10 2022 user.info mwan3track[21028]: Check (ping) failed for target "2001:4860:4860::8888" on interface wgc6 (wgc)
Sat Sep 24 12:25:38 2022 user.info mwan3track[9714]: Check (ping) failed for target "8.8.8.8" on interface wg (wg)
Sat Sep 24 13:21:01 2022 user.info mwan3track[9714]: Check (ping) failed for target "8.8.8.8" on interface wg (wg)
Sat Sep 24 13:30:03 2022 user.info mwan3track[11253]: Check (ping) failed for target "2001:4860:4860::8888" on interface wg6 (wg)
Sat Sep 24 14:16:19 2022 user.info mwan3track[2005]: Check (ping) failed for target "2001:4860:4860::8888" on interface wan6 (6in4-wan6)
Sat Sep 24 14:35:07 2022 user.info mwan3track[11253]: Check (ping) failed for target "2001:4860:4860::8888" on interface wg6 (wg)
Sat Sep 24 14:38:09 2022 user.info mwan3track[2005]: Check (ping) failed for target "2001:4860:4860::8844" on interface wan6 (6in4-wan6)
Sat Sep 24 14:42:22 2022 user.info mwan3track[11253]: Check (ping) failed for target "2001:4860:4860::8888" on interface wg6 (wg)
Sat Sep 24 14:44:24 2022 user.info mwan3track[2005]: Check (ping) failed for target "2001:4860:4860::8888" on interface wan6 (6in4-wan6)
Sat Sep 24 14:44:49 2022 user.info mwan3track[9714]: Check (ping) failed for target "8.8.8.8" on interface wg (wg)
Sat Sep 24 14:51:34 2022 user.info mwan3track[9714]: Check (ping) failed for target "8.8.8.8" on interface wg (wg)
Sat Sep 24 14:51:48 2022 user.info mwan3track[9714]: Check (ping) failed for target "8.8.8.8" on interface wg (wg)
Sat Sep 24 14:55:39 2022 user.info mwan3track[11253]: Check (ping) failed for target "2001:4860:4860::8888" on interface

As you say maybe the frequency of the pings is triggering some throttling behaviour from Google.

It doesn't appear to be causing an issue as the failure count threshold is not met, therefore the interface remains up.

[brianjmurrell]

@jamesmacwhite Indeed. Just fills the logs with all of the failures though.

I did switch my ping targets away from Google DNS servers and haven't looked back.

[jamesmacwhite]

I'll add a note to the docs on potential throttling of icmp traffic with Google DNS. I know there is potential rate limiting on DNS requests, but unless you're an ISP, I doubt you'll ever hit it. I've not seen any public around icmp throttling though

[jamesmacwhite]

@brianjmurrell Out of interest, what did you replace Google DNS IPs with?

[brianjmurrell]

2606:4700:4700::1111 2606:4700:4700::1001

[jamesmacwhite]

Ah Cloudflare IPv6. Maybe we should potentially switch the default config to Cloudflare IPv4 and IPv6 to avoid this issue.

[jamesmacwhite]

PR submitted to update the default configuration of interfaces to use Cloudflare and OpenDNS as the track IP defaults. I have added a note to the mwan3 docs around ICMP filtering on the public DNS section. I'll amend the configuration examples to also replace references to Google DNS.

[feckert]

PR #19446 merged Thanks