search

socallinuxexpo / scale-network

SCaLE's on-site expo network configurations, wifi, tooling, and scripts
https://www.socallinuxexpo.org/
BSD 3-Clause "New" or "Revised" License
40 stars 16 forks source link

resolve PMTU discovery issues on IPv6 #752

Open davidelang opened 3 months ago

davidelang commented 3 months ago

Description

Google, linkedin posting, and other sites were inacceassable during scale 21x via IPv6

Owen identified this as a PMTU discovery incompatibility between Google and the HE tunnel

Acceptance Criteria

the HE IPv6 tunnel works with PMTU, including Google's implementation, or we find a way to override PMTU or we disable IPv6

owendelong commented 3 months ago

Disabling IPv6 is not an option.

On Mar 22, 2024, at 13:34, David Lang @.***> wrote:

Description

Google, linkedin posting, and other sites were inacceassable during scale 21x via IPv6

Owen identified this as a PMTU discovery incompatibility between Google and the HE tunnel

Acceptance Criteria

the HE IPv6 tunnel works with PMTU, including Google's implementation, or we find a way to override PMTU or we disable IPv6

— Reply to this email directly, view it on GitHub https://github.com/socallinuxexpo/scale-network/issues/752, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAK6GTVR3ZYZHBZBM6HVL5LYZSIVPAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIYDGMRWG42TCNI. You are receiving this because you are subscribed to this thread.

MrHamel commented 3 months ago

It will be if a solution is not found by next year.

I don't think you understand the impact it has to Android devices, which is a popular subset of the Linux population, at a.. Linux conference.

From: Owen DeLong @.> Sent: Monday, March 25, 2024 10:39:15 AM To: socallinuxexpo/scale-network @.> Cc: Subscribed @.***> Subject: Re: [socallinuxexpo/scale-network] resolve PMTU discovery issues on IPv6 (Issue #752)

Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.

Disabling IPv6 is not an option.

On Mar 22, 2024, at 13:34, David Lang @.***> wrote:

Description

Google, linkedin posting, and other sites were inacceassable during scale 21x via IPv6

Owen identified this as a PMTU discovery incompatibility between Google and the HE tunnel

Acceptance Criteria

the HE IPv6 tunnel works with PMTU, including Google's implementation, or we find a way to override PMTU or we disable IPv6

— Reply to this email directly, view it on GitHub https://github.com/socallinuxexpo/scale-network/issues/752, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAK6GTVR3ZYZHBZBM6HVL5LYZSIVPAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIYDGMRWG42TCNI. You are receiving this because you are subscribed to this thread.

— Reply to this email directly, view it on GitHubhttps://github.com/socallinuxexpo/scale-network/issues/752#issuecomment-2018549600, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AACLU665JXQZSWYPZ4EXRU3Y2BOMHAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJYGU2DSNRQGA. You are receiving this because you are subscribed to this thread.Message ID: @.***>

hriday commented 3 months ago

Guys guys, we have a year. Let's use that time to pacify both the android camp and the IPv6 camp.

On Mon, Mar 25, 2024 at 11:44 AM Ryan Hamel @.***> wrote:

It will be if a solution is not found by next year.

I don't think you understand the impact it has to Android devices, which is a popular subset of the Linux population, at a.. Linux conference.

From: Owen DeLong @.> Sent: Monday, March 25, 2024 10:39:15 AM To: socallinuxexpo/scale-network @.> Cc: Subscribed @.***> Subject: Re: [socallinuxexpo/scale-network] resolve PMTU discovery issues on IPv6 (Issue #752)

Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.

Disabling IPv6 is not an option.

On Mar 22, 2024, at 13:34, David Lang @.***> wrote:

Description

Google, linkedin posting, and other sites were inacceassable during scale 21x via IPv6

Owen identified this as a PMTU discovery incompatibility between Google and the HE tunnel

Acceptance Criteria

the HE IPv6 tunnel works with PMTU, including Google's implementation, or we find a way to override PMTU or we disable IPv6

— Reply to this email directly, view it on GitHub < https://github.com/socallinuxexpo/scale-network/issues/752>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAK6GTVR3ZYZHBZBM6HVL5LYZSIVPAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIYDGMRWG42TCNI>.

You are receiving this because you are subscribed to this thread.

— Reply to this email directly, view it on GitHub< https://github.com/socallinuxexpo/scale-network/issues/752#issuecomment-2018549600>, or unsubscribe< https://github.com/notifications/unsubscribe-auth/AACLU665JXQZSWYPZ4EXRU3Y2BOMHAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJYGU2DSNRQGA>.

You are receiving this because you are subscribed to this thread.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/socallinuxexpo/scale-network/issues/752#issuecomment-2018557555, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFDVQDTJRGUCVEJJEG4NATY2BO6FAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJYGU2TONJVGU . You are receiving this because you are subscribed to this thread.Message ID: @.***>

irabinovitch commented 3 months ago

If we continue to prioritize broken IPv6 over usable internet, all we are doing is reinforcing attendee perception that IPv6 isn't ready for prime time and that the first thing one should do when they notice network issues is: disable ipv6.

If we want to drive ipv6 adoption and education through SCALE, we have to make sure ipv6 connectivity actually works and offers an equivalent or better experience to ipv4 only. if we can't then we just need to disable it. id hate to see that be the outcome, but with the current implementation we aren't meeting our attendees, speakers or sponsors needs.

hriday commented 3 months ago

So, there's a helpful theory on this : https://twitter.com/MyzaRed/status/1671835975801253889

It's been happening to people for over 10 years now and it appears to be isolated to one of HE's range being blocked at Google's end. Either we work with someone other than HE, or we get a different block from them.

H.

On Tue, Mar 26, 2024 at 4:45 PM Ilan Rabinovitch @.***> wrote:

If we continue to prioritize broken IPv6 over usable internet, all we are doing is reinforcing attendee perception that IPv6 isn't ready for prime time and that the first thing one should do when they notice network issues is: disable ipv6.

If we want to drive ipv6 adoption and education through SCALE, we have to make sure ipv6 connectivity actually works and offers an equivalent or better experience to ipv4 only. if we can't then we just need to disable it. id hate to see that be the outcome, but with the current implementation we aren't meeting our attendees, speakers or sponsors needs.

— Reply to this email directly, view it on GitHub https://github.com/socallinuxexpo/scale-network/issues/752#issuecomment-2021593879, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFDVQEOUZPT765XBZI66XLY2HT53AVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRRGU4TGOBXHE . You are receiving this because you commented.Message ID: @.***>

owendelong commented 3 months ago

This will probably solve it. Performance hit to everyone that can do proper PMTU-D, but hey, by all means, let's cater to Google our corporate overlords above all else: https://supportportal.juniper.net/s/article/Configuring-TCP-MSS-clamping-on-SRX-devices-to-avoid-unnecessary-fragmentation?language=en_US

MrHamel commented 3 months ago

I don't like the solution either, but it is what it is, Owen. My phone did not function on the Wi-Fi at all this year, compared to previous, and including the Hilton. Add on many attendees and expo hall vendor phones, tablets, and various other gadgets. The experience needs to be good for all OS vendors.

This change needs to be tested before next year, or we'll be back to square one resulting in disabling V6 or using another provider like me.

Ryan Hamel

From: Owen DeLong @.> Sent: Thursday, March 28, 2024 7:05 PM To: socallinuxexpo/scale-network @.> Cc: Ryan Hamel @.>; Comment @.> Subject: Re: [socallinuxexpo/scale-network] resolve PMTU discovery issues on IPv6 (Issue #752)

Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.

This will probably solve it. Performance hit to everyone that can do proper PMTU-D, but hey, by all means, let's cater to Google our corporate overlords above all else: https://supportportal.juniper.net/s/article/Configuring-TCP-MSS-clamping-on-SRX-devices-to-avoid-unnecessary-fragmentation?language=en_US

— Reply to this email directly, view it on GitHubhttps://github.com/socallinuxexpo/scale-network/issues/752#issuecomment-2026473100, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AACLU6ZOUOGOAAXXALXE2ETY2TD7PAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRWGQ3TGMJQGA. You are receiving this because you commented.Message ID: @.***>

davidelang commented 3 months ago

If the problem was all apple devices not working would you be just as willing to break them?

It's not that we are big fans of Google or that they control us, it's that a large percentage of our users are dependent on them.

David Lang

On Thu, 28 Mar 2024, Owen DeLong wrote:

This will probably solve it. Performance hit to everyone that can do proper PMTU-D, but hey, by all means, let's cater to Google our corporate overlords above all else: https://supportportal.juniper.net/s/article/Configuring-TCP-MSS-clamping-on-SRX-devices-to-avoid-unnecessary-fragmentation?language=en_US

davidelang commented 3 months ago

since we will have the tunnel up during the year, we can test it.

David Lang

On Thu, 28 Mar 2024, Owen DeLong wrote:

This will probably solve it. Performance hit to everyone that can do proper PMTU-D, but hey, by all means, let's cater to Google our corporate overlords above all else: https://supportportal.juniper.net/s/article/Configuring-TCP-MSS-clamping-on-SRX-devices-to-avoid-unnecessary-fragmentation?language=en_US

owendelong commented 3 months ago

Yes, I would, as a matter of fact, but it turns out that Apple does PMTU-D correctly.

Further, Ryan, if your phone wasn't working on the WiFi at the Hilton, this had NOTHING to do with IPv6 or problems on our network. We don't extend our network to the Hilton and the Hilton has ZERO IPv6 capability. Perhaps your phone just suffers from Android.

Another provider won't help because we still won't be able to get a 1500 octet MTU through you, GRE is GRE and 6in4 is 6in4 and both have a certain amount of overhead that you can't get around. The MTU on the ethernet interface facing the convention center is limited to 1500 octets. They won't do jumbo frames (not like I didn't ask, but the response was something between a blank stare and "what's an MTU" or "what's a frame", or "jumbo what?"). This is not a surprise given the level of training I've observed among their on-site people. They're nice, they try to be helpful, but they really have very minimal training and understanding of networking.

I'm actually less concerned about Android than I am about non-Android users trying to get to Google services from Linux devices, which was the problem we were able to observe and trace in the NOC.

Android would be even harder to troubleshoot since it has a complete lack of troubleshooting tools (e.g. tcpdump or any other libpcap based capture tool) last I heard.

If we want to test it, we'll need to add some equipment behind the tunnel and get a little creative. Doable, but not currently deployed. Right now, the tunnel is just idling on an interface on one of my MX-240s just to keep HE from deleting it. It's not actually moving real traffic or anything and I don't have an easy way to so without adding hardware. I can probably pull a spare SRX I have here into service rather than needing someone to ship our SRX devices. I have the replacement ex4200-48px from Hula already (same day replacement, no questions asked). It's probably a good idea to deploy that and get it tested anyway.

I don't have anything that pretends to be Android, but I can probably through a pi at it and we can at least do some testing with that. Problem is Pi only fails on Google stuff some times and works mostly. Making it an IPv6-only subnet will probably help make the Pi fail more consistently.

owendelong commented 3 months ago

On another note, I have good paths into Apple for getting bugs this serious resolved. Google, OTOH, is a black hole of uselessness when it comes to this sort of issue.

MrHamel commented 3 months ago

Owen,

Further, Ryan, if your phone wasn't working on the WiFi at the Hilton, this had NOTHING to do with IPv6 or problems on our network. We don't extend our network to the Hilton and the Hilton has ZERO IPv6 capability. Perhaps your phone just suffers from Android.

That's not what I said at all... I said that my phone did not like the network this year, compared to the previous years which worked just fine (including the Hilton).

Anyway, if someone could meet me at the storage facility, then tell me what cart and bin where the firewall is, I can bring it to the colo in DTLA, and get it operational temporarily. When your tweaks are done, I can test with a USB Ethernet adapter connected to my phone, if that works for you. Please let me know your thoughts.

From: Owen DeLong @.> Sent: Saturday, March 30, 2024 11:13:43 PM To: socallinuxexpo/scale-network @.> Cc: Ryan Hamel @.>; Comment @.> Subject: Re: [socallinuxexpo/scale-network] resolve PMTU discovery issues on IPv6 (Issue #752)

Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.

On another note, I have good paths into Apple for getting bugs this serious resolved. Google, OTOH, is a black hole of uselessness when it comes to this sort of issue.

— Reply to this email directly, view it on GitHubhttps://github.com/socallinuxexpo/scale-network/issues/752#issuecomment-2028571029, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AACLU63CMR7ELV3FIFTRHILY26SRPAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRYGU3TCMBSHE. You are receiving this because you commented.Message ID: @.***>

owendelong commented 3 months ago

If you weren’t having problems in years previous, including the Hilton, it wasn’t a change we made this year, because we didn’t make any IPv6 changes that would have affected that. We did have some problems with Google users in years past (though it wasn’t android phones so much as people trying to use Google apps on other systems).

On Mar 31, 2024, at 05:11, Ryan Hamel @.***> wrote:

Owen,

Further, Ryan, if your phone wasn't working on the WiFi at the Hilton, this had NOTHING to do with IPv6 or problems on our network. We don't extend our network to the Hilton and the Hilton has ZERO IPv6 capability. Perhaps your phone just suffers from Android.

That's not what I said at all... I said that my phone did not like the network this year, compared to the previous years which worked just fine (including the Hilton).

Anyway, if someone could meet me at the storage facility, then tell me what cart and bin where the firewall is, I can bring it to the colo in DTLA, and get it operational temporarily. When your tweaks are done, I can test with a USB Ethernet adapter connected to my phone, if that works for you. Please let me know your thoughts.

From: Owen DeLong @.> Sent: Saturday, March 30, 2024 11:13:43 PM To: socallinuxexpo/scale-network @.> Cc: Ryan Hamel @.>; Comment @.> Subject: Re: [socallinuxexpo/scale-network] resolve PMTU discovery issues on IPv6 (Issue #752)

Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.

On another note, I have good paths into Apple for getting bugs this serious resolved. Google, OTOH, is a black hole of uselessness when it comes to this sort of issue.

— Reply to this email directly, view it on GitHubhttps://github.com/socallinuxexpo/scale-network/issues/752#issuecomment-2028571029, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AACLU63CMR7ELV3FIFTRHILY26SRPAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRYGU3TCMBSHE. You are receiving this because you commented.Message ID: @.***> — Reply to this email directly, view it on GitHub https://github.com/socallinuxexpo/scale-network/issues/752#issuecomment-2028685962, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAK6GTVI4NHL77TTAQ3NTEDY274QHAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRYGY4DKOJWGI. You are receiving this because you commented.

MrHamel commented 3 months ago

Well, when someone opens Chrome (default browser on non-modified Android installs), then wants to search a topic from an ongoing talk and stares at a loading bar, it doesn't look good either way.

From: Owen DeLong @.> Sent: Monday, April 1, 2024 2:23:14 PM To: socallinuxexpo/scale-network @.> Cc: Ryan Hamel @.>; Comment @.> Subject: Re: [socallinuxexpo/scale-network] resolve PMTU discovery issues on IPv6 (Issue #752)

Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.

If you weren’t having problems in years previous, including the Hilton, it wasn’t a change we made this year, because we didn’t make any IPv6 changes that would have affected that. We did have some problems with Google users in years past (though it wasn’t android phones so much as people trying to use Google apps on other systems).

On Mar 31, 2024, at 05:11, Ryan Hamel @.***> wrote:

Owen,

Further, Ryan, if your phone wasn't working on the WiFi at the Hilton, this had NOTHING to do with IPv6 or problems on our network. We don't extend our network to the Hilton and the Hilton has ZERO IPv6 capability. Perhaps your phone just suffers from Android.

That's not what I said at all... I said that my phone did not like the network this year, compared to the previous years which worked just fine (including the Hilton).

Anyway, if someone could meet me at the storage facility, then tell me what cart and bin where the firewall is, I can bring it to the colo in DTLA, and get it operational temporarily. When your tweaks are done, I can test with a USB Ethernet adapter connected to my phone, if that works for you. Please let me know your thoughts.

From: Owen DeLong @.> Sent: Saturday, March 30, 2024 11:13:43 PM To: socallinuxexpo/scale-network @.> Cc: Ryan Hamel @.>; Comment @.> Subject: Re: [socallinuxexpo/scale-network] resolve PMTU discovery issues on IPv6 (Issue #752)

Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.

On another note, I have good paths into Apple for getting bugs this serious resolved. Google, OTOH, is a black hole of uselessness when it comes to this sort of issue.

— Reply to this email directly, view it on GitHubhttps://github.com/socallinuxexpo/scale-network/issues/752#issuecomment-2028571029, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AACLU63CMR7ELV3FIFTRHILY26SRPAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRYGU3TCMBSHE. You are receiving this because you commented.Message ID: @.***> — Reply to this email directly, view it on GitHub https://github.com/socallinuxexpo/scale-network/issues/752#issuecomment-2028685962, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAK6GTVI4NHL77TTAQ3NTEDY274QHAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRYGY4DKOJWGI. You are receiving this because you commented.

— Reply to this email directly, view it on GitHubhttps://github.com/socallinuxexpo/scale-network/issues/752#issuecomment-2030580124, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AACLU66QTKDHKTS4C7E456LY3HF4FAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZQGU4DAMJSGQ. You are receiving this because you commented.Message ID: @.***>

owendelong commented 3 months ago

That won’t usually be a result of the PMTU problem we’ve observed. A partial page of results until the window size expands is far more likely. We mostly saw the problem in past years with apps (docs, sheets, etc. )If it’s more widespread this year, that’s interesting data, but not the result of any changes, since the only change we made this year was to reduce interface MTUs. On Apr 1, 2024, at 14:40, Ryan Hamel @.***> wrote: Well, when someone opens Chrome (default browser on non-modified Android installs), then wants to search a topic from an ongoing talk and stares at a loading bar, it doesn't look good either way.

From: Owen DeLong @.***>

Sent: Monday, April 1, 2024 2:23:14 PM

To: socallinuxexpo/scale-network @.***>

Cc: Ryan Hamel @.>; Comment @.>

Subject: Re: [socallinuxexpo/scale-network] resolve PMTU discovery issues on IPv6 (Issue #752)

Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.

If you weren’t having problems in years previous, including the Hilton, it wasn’t a change we made this year, because we didn’t make any IPv6 changes that would have affected that. We did have some problems with Google users in years past (though it wasn’t android phones so much as people trying to use Google apps on other systems).

On Mar 31, 2024, at 05:11, Ryan Hamel @.***> wrote:

Owen,

Further, Ryan, if your phone wasn't working on the WiFi at the Hilton, this had NOTHING to do with IPv6 or problems on our network. We don't extend our network to the Hilton and the Hilton has ZERO IPv6 capability. Perhaps your phone just suffers from Android.

That's not what I said at all... I said that my phone did not like the network this year, compared to the previous years which worked just fine (including the Hilton).

Anyway, if someone could meet me at the storage facility, then tell me what cart and bin where the firewall is, I can bring it to the colo in DTLA, and get it operational temporarily. When your tweaks are done, I can test with a USB Ethernet adapter connected to my phone, if that works for you. Please let me know your thoughts.

From: Owen DeLong @.***>

Sent: Saturday, March 30, 2024 11:13:43 PM

To: socallinuxexpo/scale-network @.***>

Cc: Ryan Hamel @.>; Comment @.>

Subject: Re: [socallinuxexpo/scale-network] resolve PMTU discovery issues on IPv6 (Issue #752)

Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.

On another note, I have good paths into Apple for getting bugs this serious resolved. Google, OTOH, is a black hole of uselessness when it comes to this sort of issue.

—

Reply to this email directly, view it on GitHubhttps://github.com/socallinuxexpo/scale-network/issues/752#issuecomment-2028571029, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AACLU63CMR7ELV3FIFTRHILY26SRPAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRYGU3TCMBSHE.

You are receiving this because you commented.Message ID: @.***>

—

Reply to this email directly, view it on GitHub https://github.com/socallinuxexpo/scale-network/issues/752#issuecomment-2028685962, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAK6GTVI4NHL77TTAQ3NTEDY274QHAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRYGY4DKOJWGI.

You are receiving this because you commented.

—

Reply to this email directly, view it on GitHubhttps://github.com/socallinuxexpo/scale-network/issues/752#issuecomment-2030580124, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AACLU66QTKDHKTS4C7E456LY3HF4FAVCNFSM6AAAAABFD6NLRWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZQGU4DAMJSGQ.

You are receiving this because you commented.Message ID: @.***>

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>