TcpClient ConnectAsync timeout when using Wireshark to observe traffic over VPN connection

[SteveSyfuhs]

Description

Might be environmental, but can't figure out what is special here aside from the remote target is on the other side of VPN. I can repro this with just a single call to ConnectAsync(addr, port). I can share a more specific scenario and wireshark log offline.

var client = new TcpClient(AddressFamily.InterNetwork);

await client.ConnectAsync("remote-ip", 80); // doesn't matter

1. Start Wireshark
2. Launch above code
3. Observe error: System.Net.Internals.SocketExceptionFactory.ExtendedSocketException: 'A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. remote-ip:80'
4. Stop Wireshark
5. Launch above code
6. Observe connect succeeds

Configuration

❯ dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   3.1.401
 Commit:    5b6f5e5005

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19041
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.1.401\

Host (useful for support):
  Version: 3.1.7
  Commit:  fcfdef8d6b

.NET Core SDKs installed:
  2.1.516 [C:\Program Files\dotnet\sdk]
  2.1.517 [C:\Program Files\dotnet\sdk]
  2.2.110 [C:\Program Files\dotnet\sdk]
  3.1.201 [C:\Program Files\dotnet\sdk]
  3.1.300 [C:\Program Files\dotnet\sdk]
  3.1.401 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Regression?

Doesn't appear to be a regression.

Other information

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[SteveSyfuhs]

Slightly more specific repro for Microsoft folk. Code effectively calls the repro above client.ConnectAsync(...).

1. Connect to VPN
2. dotnet tool install bruce -g
3. bruce.exe
4. Start Wireshark listening on VPN connection
5. bruce> kinit
6. Enter password
7. Observe timeout (TCP Connect failed)
8. Stop Wireshark
9. Retry from (4) without wireshark
10. Observe non-timeout response

You should see your username show up, otherwise try kinit user@your.corp.domain.com replacing domain as appropriate.

[poizan42]

Might be relevant for reproduction to know whether you are using WinPcap or Npcap.

[SteveSyfuhs]

Capture driver is npcap.

[ghost]

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

[ghost]

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

[antonfirsov]

@SteveSyfuhs any chance you can try this with 5.0 RC1?

[wfurt]

.NET really just relay on OS functions so this is interesting. with "Doesn't appear to be a regression" I assume this fails also with 3.1? We have seen some odd behavior with VPN in the past - but not quite like this.

Can you please try same scenario when VPN is not involved @SteveSyfuhs? Also I assume other applications do work? (like browser or "openssl s_client -connect IP:port)

Also any log entries for Windows firewall - if it is enabled?

[SteveSyfuhs]

Tried .NET 5 RC1. Same behavior.

Also tried without VPN. This does work.

The symptoms:

TcpClient Connect fails when

• Connecting to a target over VPN, and
• Wireshark is listening on the VPN interface

TcpClient Connect succeeds when

• Connecting to a target over VPN, and
• Wireshark is not listening on the VPN interface OR
• Connecting to a target over any other interface, and
• Wireshark is listening on any/all interfaces.

The failure is consistently the timeout reported above. Wireshark does indicate something is going on. It can detect the DNS queries right before and observes something on the TCP level:

I can provide a full wireshark capture offline if you'd like.

Edit: also nothing logged by Firewall.

[wfurt]

I could look at the capture but I'm afraid we will not see much there. From the trace it looks like SYN is going out but the server does not answer. If you can, could you do packet capture on the server @SteveSyfuhs ? (or any node behind the VPN) If you do not see the same SYN, that would probably point finger to the VPN client. Depending on how the VPN is implemented, it hooks to network stack at various levels and it "steals" packets and injects new one (with VPN encapsulation) (e.g. you should see encrypted traffic on your primary interface)

From .NET prospective the interesting question is if other applications have same problem or this behavior is specific to .NET. In the former case we can possibly involve Windows networking or open ticket for VPN. In the latter, we would need to investigate further - perhaps enabling Windows network tracing. So test with nc, Netcat, telnet or openssl s_client would be interesting.

[SteveSyfuhs]

I'll see if I can capture a trace on the far side of the VPN, but that may take a while since its currently production traffic.

I did verify that the TCP socket call Windows makes when doing the equivalent command as the Bruce kinit call does succeed. However, that happens in LSA, so running as SYSTEM, so it doesn't entirely rule out firewall permission issues.

Telnet interestingly also reproduces the behavior on both the success and failure scenarios so I'm leaning towards this looking like it's more likely a Windows networking issue and not .NET itself. Any instructions on how to capture the networking traces?

[wfurt]

You can start with https://community.progress.com/s/article/How-to-run-a-NETSH-Trace but this is really not area of my expertize so I don't know what to look for.

As I mentioned we have seen some off behavior in the past and that may point to 3rd party application - unless you use built-in L2TP (or similar)

[karelz]

Closing as there is nothing actionable at this moment. Feel free to reopen if there is more info / traces for us to look at.