Persistent Dev Tunnel Stops Responding

[david-maw]

I've been trying out Dev Tunnels as a replacement for ngrok but I've run into a problem that persistent tunnels work fine for hours but stop responding after a while. I'm not sure what triggers this, it could be a hibernate/resume on my dev machine, could be that the tunnel is idle overnight, might be something else altogether. The current tunnel I'm trying is at https://[zllwmpxn-7190.usw3.devtunnels.ms](https://zllwmpxn-7190.usw3.devtunnels.ms/)/, the simplest way to test is just browse to it but if I do that it times out. The DNS entry looks ok:

C:\>nslookup zllwmpxn-7190.usw3.devtunnels.ms
Server:  Linksys05100
Address:  192.168.7.1

Non-authoritative answer:
Name:    tunnels-prod-rel-usw3-v3-cluster.westus3.cloudapp.azure.com
Address:  20.125.70.28
Aliases:  zllwmpxn-7190.usw3.devtunnels.ms
          tunnels-prod-rel-usw3-live-tm.trafficmanager.net
          v3-usw3.cluster.rel.tunnels.api.visualstudio.com

The tunnel inspection URI works but shows no traffic. It's a public persistent tunnel (see image), but for security I'll recreate it once this problem is resolved.

The tunnel was created using Visual Studio and I am new to this so I may be missing something obvious.

[bcrosbcorp]

Im also experiencing a similar issue running a .net core application

[derekbekoe]

Do you have an idea of how many hours the tunnel is running for? Are you able to try the devtunnel CLI? Does the issue reproduce with that?

devtunnel create mytunnel
devtunnel port create mytunnel -p PORT
devtunnel access create mytunnel -p PORT --anonymous
devtunnel host mytunnel

Docs links:

• https://learn.microsoft.com/en-us/azure/developer/dev-tunnels/get-started?tabs=windows#install
• https://learn.microsoft.com/en-us/azure/developer/dev-tunnels/cli-commands#advanced-manage-dev-tunnels

[david-maw]

No, sorry, my scenario is I run the test web service on my test system using VS and specify a tunnel, the web service comes up and the tunnel works. I repeatedly start and stop a test client and sleep or hibernate the system. The tunnel survives a sleep/hibernate/resume cycle but somewhere it stops, typically after more than 12 hours.

I just tried again using the tunnel I created for this test and it seems to work fine again, so that's news - the failure isn't permanent. I'll have a bash with the command line once it fails again.

Incidentally, It's a giant PITA trying to find the URL for a persistent tunnel from the UI, especially an inactive one. Is it easier from the command line? I realize that a temporary tunnel probably does not even have a defined URL until it is started, but that is not the case for persistent tunnels.

[david-maw]

ok @derekbekoe , it failed again which is good news I suppose in that it's at least reproducible (or at any rate, recurring). The tunnel ran for about 4 hours then I put the system to sleep with the web service and devtunnel still running but the test client app stopped. After resuming the system a couple of hours later, the tunnel was no longer responsive. a 'show' command on the tunnel returns:

PS C:\Users\david> devtunnel show zllwmpxn.usw3
Tunnel ID             : zllwmpxn.usw3
Description           : Ws
Labels                : VisualStudioCreatedTunnel
Access control        : {+Anonymous [connect]}
Host connections      : 1
Client connections    : 0
Current upload rate   : 0 MB/s (limit: 20 MB/s)
Current download rate : 0 MB/s (limit: 20 MB/s)
Upload total          : 322 KB
Download total        : 964 KB
Ports                 : 1
  7190  http  https://zllwmpxn-7190.usw3.devtunnels.ms/
Tunnel Expiration     : 30 days

Having seen tha failure again, I'll go ahead and try one using the CLI as you suggested, stay tuned...

[david-maw]

I tried a tunnel created and initiated via the CLI as in your example above, ran it for a few hours and it worked fine. I suspended the system for a couple of hours with the tunnel and test web service still running, it worked without an issue. Tried hibernate overnight and likewise, the following morning ithe same tunnel still worked. So the only failure I've seen was with VS initiated tunnels. Of course it might be an intermittent problem or my test was somehow insufficient so I'll keep this tunnel going for a while longer and see what happens.

[david-maw]

Aha @derekbekoe the CLI initiated tunnel failed, here's what I saw in the console:

PS C:\Users\david> devtunnel host mytunnel
Hosting port: 7190
Connect via browser: https://ztlnq9kt-7190.usw3.devtunnels.ms
Inspect network activity: https://ztlnq9kt-7190-inspect.usw3.devtunnels.ms

Ready to accept connections for tunnel: mytunnel
ClientSSH: PortForwardingService connection  to localhost:7190 failed: No connection could be made because the target machine actively refused it.
ClientSSH: PortForwardingService connection  to localhost:7190 failed: No connection could be made because the target machine actively refused it.
Connection to host tunnel relay closed. Connection lost.. Reconnecting.
ClientSSH: Session closed unexpectedly due to ConnectionLost, "Connection lost.
Reason: ConnectionLost"
Microsoft.DevTunnels.Ssh.SshConnectionException: Connection lost.
Reason: ConnectionLost
 ---> Microsoft.DevTunnels.Ssh.SshConnectionException: Connection lost.
Reason: ConnectionLost
   at Microsoft.DevTunnels.Ssh.SshStream.DequeueBuffer()
   at Microsoft.DevTunnels.Ssh.SshStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellation)
   at Microsoft.DevTunnels.Ssh.IO.SshProtocol.ReadAsync(Buffer buffer, CancellationToken cancellation)
   --- End of inner exception stack trace ---
   at Microsoft.DevTunnels.Ssh.IO.SshProtocol.ReadAsync(Buffer buffer, CancellationToken cancellation)
   at Microsoft.DevTunnels.Ssh.IO.SshProtocol.ReceiveMessageAsync(CancellationToken cancellation)
   at Microsoft.DevTunnels.Ssh.SshSession.ReceiveAndHandleOneMessageAsync(CancellationToken cancellation)
ClientSSH: Session closed unexpectedly due to ConnectionLost, "Connection lost.
Reason: ConnectionLost"
Microsoft.DevTunnels.Ssh.SshConnectionException: Connection lost.
Reason: ConnectionLost
 ---> Microsoft.DevTunnels.Ssh.SshConnectionException: Connection lost.
Reason: ConnectionLost
   at Microsoft.DevTunnels.Ssh.SshStream.DequeueBuffer()
   at Microsoft.DevTunnels.Ssh.SshStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellation)
   at Microsoft.DevTunnels.Ssh.IO.SshProtocol.ReadAsync(Buffer buffer, CancellationToken cancellation)
   --- End of inner exception stack trace ---
   at Microsoft.DevTunnels.Ssh.IO.SshProtocol.ReadAsync(Buffer buffer, CancellationToken cancellation)
   at Microsoft.DevTunnels.Ssh.IO.SshProtocol.ReceiveMessageAsync(CancellationToken cancellation)
   at Microsoft.DevTunnels.Ssh.SshSession.ReceiveAndHandleOneMessageAsync(CancellationToken cancellation)
...
ClientSSH: Session closed unexpectedly due to ConnectionLost, "Connection lost.
Reason: ConnectionLost"
Microsoft.DevTunnels.Ssh.SshConnectionException: Connection lost.
Reason: ConnectionLost
 ---> Microsoft.DevTunnels.Ssh.SshConnectionException: Connection lost.
Reason: ConnectionLost
   at Microsoft.DevTunnels.Ssh.SshStream.DequeueBuffer()
   at Microsoft.DevTunnels.Ssh.SshStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellation)
   at Microsoft.DevTunnels.Ssh.IO.SshProtocol.ReadAsync(Buffer buffer, CancellationToken cancellation)
   --- End of inner exception stack trace ---
   at Microsoft.DevTunnels.Ssh.IO.SshProtocol.ReadAsync(Buffer buffer, CancellationToken cancellation)
   at Microsoft.DevTunnels.Ssh.IO.SshProtocol.ReceiveMessageAsync(CancellationToken cancellation)
   at Microsoft.DevTunnels.Ssh.SshSession.ReceiveAndHandleOneMessageAsync(CancellationToken cancellation)
Connection to host tunnel relay closed. Connection lost.. Reconnecting.
Connection to host tunnel relay closed. Connection lost.. Reconnecting.
Refreshing tunnel.
devtunnel: Error connecting host tunnel session: Not authorized0. Refreshed tunnel access token is not valid. Tunnel service response status code: Unauthorized
Request ID: f736c417-5d74-43be-a329-7183b678f45c

So, I used ctrl-c to kill it and these lines were added.

devtunnel: Error disposing host: System.UnauthorizedAccessException: Tunnel service response status code: Unauthorized
Request ID: 23ef9f9e-f22d-4848-8f6e-eff727705d05
 ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 401 (Unauthorized).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Microsoft.DevTunnels.Management.TunnelManagementClient.ConvertResponseAsync[T](HttpMethod method, HttpResponseMessage response, CancellationToken cancellation)
   --- End of inner exception stack trace ---
   at Microsoft.DevTunnels.Management.TunnelManagementClient.ConvertResponseAsync[T](HttpMethod method, HttpResponseMessage response, CancellationToken cancellation)
   at Microsoft.DevTunnels.Management.TunnelManagementClient.SendRequestAsync[TRequest,TResult](HttpMethod method, Uri uri, TunnelRequestOptions options, AuthenticationHeaderValue authHeader, TRequest body, CancellationToken cancellation)
   at Microsoft.DevTunnels.Management.TunnelManagementClient.SendTunnelRequestAsync[TRequest,TResult](HttpMethod method, Tunnel tunnel, String[] accessTokenScopes, String path, String query, TunnelRequestOptions options, TRequest body, CancellationToken cancellation, Boolean isCreate)
   at Microsoft.DevTunnels.Management.TunnelManagementClient.DeleteTunnelEndpointsAsync(Tunnel tunnel, String id, TunnelRequestOptions options, CancellationToken cancellation)
   at Microsoft.DevTunnels.Connections.TunnelRelayTunnelHost.DisposeConnectionAsync()
   at Microsoft.DevTunnels.Connections.TunnelConnection.DisposeAsync()

After that I was able to enter devtunnel host mytunnel at the command prompt and the tunnel was back to working again.

[derekbekoe]

Thanks for the detailed response.

When attempting to reconnect, if an access token is expired, we do attempt to refresh this. However, this may not be successful in all cases depending on many factors (e.g. when the tunnel disconnected, when the token expires, configurable token lifetimes, conditional access, etc.).

With that said, similar to the devtunnel CLI, in Visual Studio you should be able to stop/start your persistent tunnel and still maintain the same url to connect to it.

[david-maw]

ok, possibley the token refresh fails if the system isn't active (meaning it's hibernated or suspended) at the time it needs refreshing, of copurse that might just be coincidence. What's the default token refresh period? Is the refresh process expected to survive hibernate/resume?

Should I report VS issues against this repo and if not then where? I'll report them here for now. I''s not clear to me how to stop/start a tunnel using the VS window, also, while the window shows my directly created tunnel it fails to start it, failing with:

Configurating the dev tunnel to host the port failed. Tunnel service error: Invalid arguments. The tunnel port protocol cannot be changed.
Request ID: 51aa054f-971e-40f6-ab98-6704b05c786a Response status code does not indicate success: 400 (Bad Request).

The tunnel starts fine with the WebTunnel CLI, though that displays a strange set of update messages, which might be fine, since I don't see them any more so the update may have happened.

PS C:\Users\david> devtunnel host

You are using an older version of the devtunnel CLI. Please upgrade to the latest version by using one of the method(s):
- Direct download: https://aka.ms/TunnelsCliDownload/win-x64
- Package manager command: 'winget install Microsoft.devtunnel'

Hosting port: 7190
Connect via browser: https://ztlnq9kt-7190.usw3.devtunnels.ms
Inspect network activity: https://ztlnq9kt-7190-inspect.usw3.devtunnels.ms

Ready to accept connections for tunnel: mytunnel
PS C:\Users\david> winget install Microsoft.devtunnel
Found an existing package already installed. Trying to upgrade the installed package...
No available upgrade found.
No newer package versions are available from the configured sources.

But it's now working fine witout the messages so maybe the alarming "No newer package versions are available from the configured sources" is just a routine message.

[derekbekoe]

Tunnel service error: Invalid arguments. The tunnel port protocol cannot be changed.

Were you intentionally trying to change the protocol of the tunnel?

But it's now working fine witout the messages so maybe the alarming "No newer package versions are available from the configured sources" is just a routine message.

Those winget messages are expected.

[david-maw]

No, I only use TCP and I didn't realize I even could attempt to change the tunnel protocol!

Thanks, I'll (continue to) ignore that Winget message.

You didn't answer my question "Should I report VS issues against this repo and if not then where?", I'm fine reporting them here, just don't want to waste anyone's time by reporting them in the wrong place.

[derekbekoe]

The relevant VS team is already aware and looking to see if there's something better that could be done here for long-running tunnels.

In the future though, you can also report it at https://developercommunity.visualstudio.com/VisualStudio if it's related to Visual Studio.

[david-maw]

Thanks, I wasn't sure whether the VS people owned the VS Dev Tunnels GUI or not., I'll report GUI issues to them.

The other odd thing about the original problem within VS was that stopping the web service and restarting it did not seem to help and I'd expected it would stop and restart the associated tunnel as well. That brings up a related question - is there an orderly way to stop a tunnel using the CLI? Ctrl-C on the devtunnel host mytunnel process certainly stops it, but that seems a bit brutal!

[derekbekoe]

is there an orderly way to stop a tunnel using the CLI? Ctrl-C on the devtunnel host mytunnel process certainly stops it, but that seems a bit brutal!

Ctrl-C is the best way and when you do that, the CLI does clean-up appropriately before it shuts down.

[david-maw]

ok, thanks, Ctrl-C and restart via CLI whenever a long-running tunnel stops working seems sufficient until the VS automated version starts working reliably.

[plppp2001]

I'm also having the same issues with my Persistent Dev Tunnel (using it as .net CORE API for my apps) today.

[derekbekoe]

Closing the thread as topics were discussed. @plppp2001 if you are still seeing an issue related to dev tunnels or the CLI, please file a new issue.

[david-maw]

@derekbekoe I'm not sure what this closure means - is something corrected or is this "works as designed"?

[derekbekoe]

From the thread, the devtunnel CLI is working as expected and is an appropriate workaround to the issue you had in VS. For the VS issue, if that still persists, that can be tracked on the developercommunity site.

[david-maw]

ok, @derekbekoe what you wrote was "When attempting to reconnect, if an access token is expired, we do attempt to refresh this. However, this may not be successful in all cases depending on many factors...you should be able to stop/start your persistent tunnel and still maintain the same url to connect to it".

I took that to mean that intermittent tunnel failures, even when using the CLI, are to be expected, and I should use the CLI to stop/start the tunnel when one happens. Is that correct?

[derekbekoe]

Hi @david-maw - If it's been connected for several hours, I would recommend stopping and re-hosting the tunnel (e.g. after hibernating/resuming your machine).

[david-maw]

@derekbekoe ok, I'll treat disconnections under such circumstances should they happen as being expected and do as you recommend, thanks.