`az network bastion tunnel` quietly drops connection

tillahoffmann commented 8 months ago

Describe the bug

Having tunneled to an Azure instance, the az network bastion tunnel command drops the connection. Rather than raising an exception or reconnecting, it hangs. It is only possible to identify that there was a problem when debug mode is enabled. Without debug mode, the connection is dropped silently. The interval varies between a few minutes to several hours.

Related command

az network bastion tunnel \
--debug \
--name <redacted> \
--resource-group <redacted> \
--resource-port 22 \
--port 2222 \
--target-resource-id "/subscriptions/<redacted>/resourceGroups/<redacted>/providers/Microsoft.Compute/virtualMachines/<redacted>" \
2> az-network-bastion-tunnel.log

Contents of az-network-bastion-tunnel.log are shown below.

Errors

No errors raised.

Issue script & Debug output

Logs

``` DEBUG: cli.knack.cli: Command arguments: ['network', 'bastion', 'tunnel', '--debug', '--name', '', '--resource-group', '', '--resource-port', '22', '--port', '2222', '--target-resource-id', '/subscriptions//resourceGroups//providers/Microsoft.Compute/virtualMachines/'] DEBUG: cli.knack.cli: __init__ debug log: Cannot enable color. DEBUG: cli.knack.cli: Event: Cli.PreExecute [] DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [>, >, >] DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate [] DEBUG: cli.azure.cli.core: Modules found from index for 'network': ['azure.cli.command_modules.network', 'azure.cli.command_modules.privatedns', 'azext_bastion'] DEBUG: cli.azure.cli.core: Loading command modules: DEBUG: cli.azure.cli.core: Name Load Time Groups Commands DEBUG: cli.azure.cli.core: network 0.349 115 355 DEBUG: cli.azure.cli.core: privatedns 0.009 14 60 DEBUG: cli.azure.cli.core: Total (2) 0.358 129 415 DEBUG: cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next'] DEBUG: cli.azure.cli.core: Loading extensions: DEBUG: cli.azure.cli.core: Name Load Time Groups Commands Directory DEBUG: cli.azure.cli.core: bastion 0.004 2 9 /cliextensions/bastion DEBUG: cli.azure.cli.core: Total (1) 0.004 2 9 DEBUG: cli.azure.cli.core: Loaded 129 groups, 424 commands. DEBUG: cli.azure.cli.core: Found a match in the command table. DEBUG: cli.azure.cli.core: Raw command : network bastion tunnel DEBUG: cli.azure.cli.core: Command table: network bastion tunnel DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [>] DEBUG: cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/commands/2024-02-13.12-43-30.network_bastion_tunnel.51973.log'. INFO: az_command_data_logger: command args: network bastion tunnel --debug --name {} --resource-group {} --resource-port {} --port {} --target-resource-id {} DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [.add_subscription_parameter at 0x>] DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad [] DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [.add_ids_arguments at 0x>, .add_cache_arguments at 0x>] DEBUG: cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded [] DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreParseArgs [] DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [>, >, .parse_ids_arguments at 0x>] INFO: az_command_data_logger: extension name: bastion INFO: az_command_data_logger: extension version: 0.2.3 WARNING: Command group 'az network' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus DEBUG: cli.azure.cli.core.auth.persistence: build_persistence: location='/msal_token_cache.json', encrypt=False DEBUG: cli.azure.cli.core.auth.binary_cache: load: /msal_http_cache.bin DEBUG: urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None) DEBUG: msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com//oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com//discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com//v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com//oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com//oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com//oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com//kerberos', 'tenant_region_scope': 'NA', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'} DEBUG: msal.application: Broker enabled? False DEBUG: cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={} DEBUG: cli.azure.cli.core.auth.msal_authentication: UserCredential.get_token: scopes=('https://management.core.windows.net//.default',), claims=None, kwargs={} DEBUG: msal.application: Cache hit an AT DEBUG: msal.telemetry: Generate or reuse correlation_id: DEBUG: cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions//resourceGroups//providers/Microsoft.Network/bastionHosts/?api-version=2022-01-01' DEBUG: cli.azure.cli.core.sdk.policies: Request method: 'GET' DEBUG: cli.azure.cli.core.sdk.policies: Request headers: DEBUG: cli.azure.cli.core.sdk.policies: 'Accept': 'application/json' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': '' DEBUG: cli.azure.cli.core.sdk.policies: 'CommandName': 'network bastion tunnel' DEBUG: cli.azure.cli.core.sdk.policies: 'ParameterSetName': '--debug --name --resource-group --resource-port --port --target-resource-id' DEBUG: cli.azure.cli.core.sdk.policies: 'User-Agent': 'AZURECLI/2.57.0 (HOMEBREW) (AAZ) azsdk-python-core/1.28.0 Python/3.11.7 (macOS-14.3-arm64-arm-64bit)' DEBUG: cli.azure.cli.core.sdk.policies: 'Authorization': '*****' DEBUG: cli.azure.cli.core.sdk.policies: Request body: DEBUG: cli.azure.cli.core.sdk.policies: This request has no body DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443 DEBUG: urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions//resourceGroups//providers/Microsoft.Network/bastionHosts/?api-version=2022-01-01 HTTP/1.1" 200 1701 DEBUG: cli.azure.cli.core.sdk.policies: Response status: 200 DEBUG: cli.azure.cli.core.sdk.policies: Response headers: DEBUG: cli.azure.cli.core.sdk.policies: 'Cache-Control': 'no-cache' DEBUG: cli.azure.cli.core.sdk.policies: 'Pragma': 'no-cache' DEBUG: cli.azure.cli.core.sdk.policies: 'Content-Length': '1701' DEBUG: cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json; charset=utf-8' DEBUG: cli.azure.cli.core.sdk.policies: 'Expires': '-1' DEBUG: cli.azure.cli.core.sdk.policies: 'ETag': '' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-correlation-request-id': '' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-arm-service-request-id': '' DEBUG: cli.azure.cli.core.sdk.policies: 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-subscription-reads': '11999' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-routing-request-id': 'EASTUS2:20240213T174330Z:' DEBUG: cli.azure.cli.core.sdk.policies: 'X-Content-Type-Options': 'nosniff' DEBUG: cli.azure.cli.core.sdk.policies: 'X-Cache': 'CONFIG_NOCACHE' DEBUG: cli.azure.cli.core.sdk.policies: 'X-MSEdge-Ref': 'Ref A: Ref B: Ref C: 2024-02-13T17:43:30Z' DEBUG: cli.azure.cli.core.sdk.policies: 'Date': 'Tue, 13 Feb 2024 17:43:30 GMT' DEBUG: cli.azure.cli.core.sdk.policies: Response content: DEBUG: cli.azure.cli.core.sdk.policies: { "name": "", "id": "/subscriptions//resourceGroups//providers/Microsoft.Network/bastionHosts/", "etag": "W/\"\"", "type": "Microsoft.Network/bastionHosts", "location": "eastus2", "tags": { "User": "" }, "properties": { "provisioningState": "Succeeded", "dnsName": "bst-.bastion.azure.com", "scaleUnits": 2, "enableTunneling": true, "enableIpConnect": false, "disableCopyPaste": false, "enableShareableLink": false, "enableKerberos": false, "ipConfigurations": [ { "name": "IpConf", "id": "/subscriptions//resourceGroups//providers/Microsoft.Network/bastionHosts//bastionHostIpConfigurations/IpConf", "etag": "", "type": "Microsoft.Network/bastionHosts/bastionHostIpConfigurations", "properties": { "provisioningState": "Succeeded", "privateIPAllocationMethod": "Dynamic", "publicIPAddress": { "id": "/subscriptions//resourceGroups//providers/Microsoft.Network/publicIPAddresses/" }, "subnet": { "id": "/subscriptions//resourceGroups//providers/Microsoft.Network/virtualNetworks//subnets/AzureBastionSubnet" } } } ] }, "sku": { "name": "Standard" } } INFO: cli.azext_bastion.tunnel: Port 2222 is open INFO: cli.azext_bastion.tunnel: Creating a socket on port: 2222 INFO: cli.azext_bastion.tunnel: Setting socket options INFO: cli.azext_bastion.tunnel: Binding to socket on local address and port INFO: cli.azext_bastion.tunnel: Finished initialization WARNING: cli.azext_bastion.custom: Opening tunnel on port: 2222 WARNING: cli.azext_bastion.custom: Tunnel is ready, connect on port 2222 WARNING: cli.azext_bastion.custom: Ctrl + C to close DEBUG: urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None) DEBUG: msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com//oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com//discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com//v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com//oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com//oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com//oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com//kerberos', 'tenant_region_scope': 'NA', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'} DEBUG: msal.application: Broker enabled? False DEBUG: cli.azure.cli.core.auth.msal_authentication: UserCredential.get_token: scopes=('https://management.core.windows.net//.default',), claims=None, kwargs={} DEBUG: msal.application: Cache hit an AT DEBUG: msal.telemetry: Generate or reuse correlation_id: DEBUG: cli.azext_bastion.tunnel: Content: {'resourceId': '/subscriptions//resourceGroups//providers/Microsoft.Compute/virtualMachines/', 'protocol': 'tcptunnel', 'workloadHostPort': '22', 'aztoken': '', 'token': None} DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): bst-.bastion.azure.com:443 DEBUG: urllib3.connectionpool: https://bst-.bastion.azure.com:443 "POST /api/tokens HTTP/1.1" 200 None INFO: cli.azext_bastion.tunnel: Websocket, connected status: True INFO: cli.azext_bastion.tunnel: Got debugger connection... index: 1 INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 1 INFO: cli.azext_bastion.tunnel: Both debugger and websocket threads started... INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 21, index: 1 INFO: cli.azext_bastion.tunnel: Successfully connected to local server.. INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1 INFO: cli.azext_bastion.tunnel: Received websocket index: 1 INFO: cli.azext_bastion.tunnel: Sending to debugger, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to debugger, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 1 INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 1504, index: 1 INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1 INFO: cli.azext_bastion.tunnel: Received websocket index: 1 INFO: cli.azext_bastion.tunnel: Sending to debugger, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to debugger, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 1 INFO: cli.azext_bastion.tunnel: Received websocket index: 1 INFO: cli.azext_bastion.tunnel: Sending to debugger, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to debugger, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 1 INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 48, index: 1 INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1 INFO: cli.azext_bastion.tunnel: Received websocket index: 1 INFO: cli.azext_bastion.tunnel: Sending to debugger, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to debugger, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 1 INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 16, index: 1 INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1 INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 44, index: 1 INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1 INFO: cli.azext_bastion.tunnel: Received websocket index: 1 INFO: cli.azext_bastion.tunnel: Sending to debugger, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to debugger, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 1 INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 68, index: 1 INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1 INFO: cli.azext_bastion.tunnel: Received websocket index: 1 INFO: cli.azext_bastion.tunnel: Sending to debugger, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to debugger, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 1 INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 140, index: 1 INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1 INFO: cli.azext_bastion.tunnel: Received websocket index: 1 <100k lines redacted all having the same pattern> INFO: cli.azext_bastion.tunnel: Sending to debugger, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to debugger, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 1 INFO: cli.azext_bastion.tunnel: Received websocket index: 1 INFO: cli.azext_bastion.tunnel: Sending to debugger, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to debugger, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 1 INFO: cli.azext_bastion.tunnel: Received websocket index: 1 INFO: cli.azext_bastion.tunnel: Sending to debugger, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to debugger, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for websocket data, connection status: True, index: 1 INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 80, index: 1 INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1 INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 84, index: 1 INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1 INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 84, index: 1 INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1 INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 84, index: 1 INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1 INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 84, index: 1 INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1 INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 84, index: 1 INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1 INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 84, index: 1 INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1 INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 100, index: 1 INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1 INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 84, index: 1 INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1 INFO: cli.azext_bastion.tunnel: Received debugger data, nbytes: 84, index: 1 INFO: cli.azext_bastion.tunnel: Sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Done sending to websocket, index: 1 INFO: cli.azext_bastion.tunnel: Waiting for debugger data, index: 1 INFO: cli.azext_bastion.tunnel: Connection to remote host was lost. INFO: cli.azext_bastion.tunnel: Client disconnected!, index: 1 INFO: cli.azext_bastion.tunnel: [Errno 9] Bad file descriptor INFO: cli.azext_bastion.tunnel: Client disconnected 1 INFO: cli.azext_bastion.tunnel: Cleaning up session DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): bst-.bastion.azure.com:443 DEBUG: urllib3.connectionpool: https://bst-.bastion.azure.com:443 "DELETE /api/tokens/ HTTP/1.1" 204 0 INFO: cli.azext_bastion.tunnel: Both debugger and websocket threads stopped... INFO: cli.azext_bastion.tunnel: Stopped local server.. [hangs here] ```

Expected behavior

The connection remains stable, the connection is re-established, or an error is raised if reconnection attempts fail.

Environment Summary

azure-cli                         2.57.0

core                              2.57.0
telemetry                          1.1.0

Extensions:
bastion                            0.2.3
ssh                                1.1.3

Dependencies:
msal                              1.26.0
azure-mgmt-resource             23.1.0b2

Python location '/opt/homebrew/Cellar/azure-cli/2.57.0/libexec/bin/python'
Extensions directory '/Users/till/.azure/cliextensions'

Python (Darwin) 3.11.7 (main, Dec  4 2023, 18:10:11) [Clang 15.0.0 (clang-1500.1.0.2.5)]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

Additional context

Running on M1 MacBook Pro (2020), macOS 14.3 (23D56).

yonzhan commented 8 months ago

Thank you for opening this issue, we will look into it.

microsoft-github-policy-service[bot] commented 8 months ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @aznetsuppgithub.

microsoft-github-policy-service[bot] commented 8 months ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @bastionsuppgithub.

jjournet commented 2 weeks ago

Hello,

I have almost the same issue, also on Silicon Mac: I'm opening a tunnel to a VM, and uses the tunnel to ssh to the VM.

at random internal, the az bastion tunnel drops, the ssh connection dies. The main difference is that I have an additional error line in the logs:

cli.azext_bastion.tunnel: Waiting for debugger data, index: 0                                                                                                                                                                                                                                                                                                                                   
cli.azext_bastion.tunnel: [SSL] record layer failure (_ssl.c:2580)                                                                                                                                                                                                                                                                                                                              
cli.azext_bastion.tunnel: Client disconnected!, index: 0                                                                                                                                                                                                                                                                                                                                        
cli.azext_bastion.tunnel: [Errno 9] Bad file descriptor                                                                                                                                                                                                                                                                                                                                         
cli.azext_bastion.tunnel: Client disconnected 0                                                                                                                                                                                                                                                                                                                                                 
cli.azext_bastion.tunnel: Cleaning up session                                                                                                                                                                                                                                                                                                                                                   
urllib3.connectionpool: Starting new HTTPS connection (1): bst-92d64564-079c-489b-bce5-4522582a7c1e.bastion.azure.com:443                                                                                                                                                                                                                                                                       
urllib3.connectionpool: https://bst-92d64564-079c-489b-bce5-4522582a7c1e.bastion.azure.com:443 "DELETE /api/tokens/D5EAD29EF05D3B3FA86B622CC39968D448D6AE86DD8D31E41B999E009A787DAB HTTP/1.1" 204 0                                                                                                                                                                                             
cli.azext_bastion.tunnel: Both debugger and websocket threads stopped...                                                                                                                                                                                                                                                                                                                        
cli.azext_bastion.tunnel: Stopped local server.

error: cli.azext_bastion.tunnel: [SSL] record layer failure (_ssl.c:2580)

Strangely enough:

I have several colleagues also on Mac, using the same bastions, and they don't seem to have the issue.
a few weeks back, before I upgraded the modules ssh and tunnel, I didn't have the issue (bastion 0.3.0 and ssh 2.0.3)

I don't think current issue is related to #24600 because I'm not creating several connections through the tunnel (can't comment if it's the case for @tillahoffmann )

Note that az bastion ssh is dying the same way at random interval

❯ az --version
azure-cli                         2.65.0

core                              2.65.0
telemetry                          1.1.0

Extensions:
account                            0.2.5
azure-devops                      0.26.0
bastion                            1.3.0
ssh                                2.0.5

Dependencies:
msal                              1.31.0
azure-mgmt-resource               23.1.1

Python location '/opt/homebrew/Cellar/azure-cli/2.65.0/libexec/bin/python'
Extensions directory '/Users/jjournet@healthequity.com/.azure/cliextensions'

Python (Darwin) 3.11.10 (main, Sep  7 2024, 01:03:31) [Clang 15.0.0 (clang-1500.3.9.4)]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

MacBook Pro M2 Max, MacOS Sequoia 15.0.1 Connecting using VPN (Cisco secure client version 5.16.103)

Azure / azure-cli