search

aws / aws-cli

Universal Command Line Interface for Amazon Web Services
Other
15.34k stars 4.09k forks source link

ec2 instance connect fails sporadically on WebSocket (NO_PROXY=1 does not help) #8880

Open asterkin opened 3 weeks ago

asterkin commented 3 weeks ago

Describe the bug

aws ec2-instance-connect open-tunnel

Fails with Exception in WebSocket on_connection_setup callback

Traceback (most recent call last): File "awscrt\websocket.py", line 459, in _on_connection_setup File "awscli\customizations\ec2instanceconnect\websocket.py", line 239, in _on_connection TypeError: 'NoneType' object is not iterable

and is stuck.

I was unable to detect any recurring pattern. It could happen on the first connection or reconnection attempt in case of a temporal network disconnect.

Expected Behavior

Either do not fail at all, or fail quickly allowing an automatic recover.

Current Behavior

It is stuck and I need to restart VSCode to get rid of it.

Reproduction Steps

It happens within my custom PowrShell script presented below:

Initiate an SSH session over AWS EC2 Instance Connect Endpoint using a temporary Public Key.

param ( [string]$HostName, # Assume $input is the input string "/" [int]$PortNumber )

Global configuration.

$PROFILE_NAME = $HostName.Split('/')[0] $INSTANCE_NAME_TAG = $HostName.Split('/')[1] $SSH_DIR = "$env:USERPROFILE.ssh" $SSH_TMP_KEY = "$SSH_DIR\ssh-tmp" $SSH_USER = "ec2-user" $env:NO_PROXY="1"

Remove expired ssh private and public keys

function Clear-SSHKeyCache { Write-Host "Cleaning up the SSH Keys Cache" Remove-Item "$SSH_TMP_KEY*" -Force & ssh-add -D }

Log in to AWS SSO, if the session expired

function Connect-AWSSSOLogin {

Attempt to get the AWS caller identity to check if the SSO session is valid.

aws sts get-caller-identity --profile $PROFILE_NAME *>&1
if ($LASTEXITCODE -ne 0) {        
    Write-Host "SSO session expired or not valid"
    Clear-SSHKeyCache
    Write-Host "Logging in ..."
    if (aws sso login --profile $PROFILE_NAME) {
        Write-Host "SSO login successful."
    } else {
        Write-Host "SSO login failed"
        exit 1
    }
} else {
    Write-Host "SSO session is still valid, using existing credentials."
}

}

Find EC2 Instance ID by name

function Find-InstanceID { $INSTANCE_ID = aws ec2 describe-instances --profile $PROFILE_NAME --filters "Name=tag:Name,Values=$INSTANCE_NAME_TAG" "Name=instance-state-name,Values=pending,running,stopping,stopped" ` --query "Reservations[].Instances[].InstanceId" --output text

if (-not $INSTANCE_ID) {
    Write-Host "No non-terminated instances found with the Name tag: $INSTANCE_NAME_TAG"
    exit 1
}

Write-Host "Instance ID: $INSTANCE_ID"
return $INSTANCE_ID

}

Wait for a particular instance state

function Wait-ForInstanceState($INSTANCE_ID, $STATE) { $state = "instance-$STATE" Write-Host "Waiting for instance to enter $state state..." aws ec2 wait $state --profile $PROFILE_NAME --instance-ids $INSTANCE_ID }

Start the instance and wait for it to become running

function Start-AndWaitInstance($INSTANCE_ID) { Write-Host "Starting instance $INSTANCE_ID ..." aws ec2 start-instances --profile $PROFILE_NAME --instance-ids $INSTANCE_ID Wait-ForInstanceState $INSTANCE_ID "running" }

Checks and manages instance state.

function Test-AndManageInstanceState($INSTANCE_ID) { $INSTANCE_STATE = aws ec2 describe-instance-status --profile $PROFILE_NAME --instance-id $INSTANCE_ID --include-all-instances --query InstanceStatuses[0].InstanceState.Name --output text

switch ($INSTANCE_STATE) {
    "stopped" {
        Write-Host "Instance is stopped."
        Start-AndWaitInstance $INSTANCE_ID
    }
    "pending" {
        Write-Host "Instance is pending"
        Wait-ForInstanceState $INSTANCE_ID "running"
    }
    "stopping" {
        Write-Host "Instance is stopping..."
        Wait-ForInstanceState $INSTANCE_ID "stopped"
        Start-AndWaitInstance $INSTANCE_ID
    }
    "running" {
        Write-Host "Instance $INSTANCE_ID is already running."
    }
    default {
        Write-Host "Unexpected instance state: $INSTANCE_STATE"
        exit 1
    }
}

}

Generates a temporary SSH key.

function New-SSHKey { if (-not (Test-Path $SSH_TMP_KEY)) { Write-Host "Generating a temporary SSH key..." ssh-keygen -t rsa -b 2048 -f $SSH_TMP_KEY -N '""' *> $null 2>&1 Set-ItemProperty -Path $SSH_TMP_KEY -Name IsReadOnly -Value $true } $SSH_PUB_KEY = Get-Content "$SSH_TMP_KEY.pub" -Raw return $SSH_PUB_KEY }

Sends the public key to the instance via EIC.

function Send-PublicKey($INSTANCE_ID, $SSH_PUB_KEY) { Write-Host "Sending public key to the instance $INSTANCE_ID ..." $response = aws ec2-instance-connect send-ssh-public-key --instance-id $INSTANCE_ID --profile $PROFILE_NAME --instance-os-user $SSH_USER --ssh-public-key $SSH_PUB_KEY Write-Host "send-ssh-public-key response $response" }

Starts SSH session.

function Open-Tunnel($INSTANCE_ID) { Write-Host "Opening tunnel over EIC..." aws ec2-instance-connect open-tunnel --instance-id $INSTANCE_ID --profile $PROFILE_NAME }

Main script execution.

Connect-AWSSSOLogin $INSTANCE_ID = Find-InstanceID Test-AndManageInstanceState $INSTANCE_ID $SSH_PUB_KEY = New-SSHKey Send-PublicKey $INSTANCE_ID $SSH_PUB_KEY Open-Tunnel $INSTANCE_ID

I normally use it from VSCode, but the same problem happens with ssh command. I never experienced such a problem on my Linux client desktop, only Windows.

Possible Solution

One possible explanation is a time delay after an automatic instance restart. If I had a way to whether EIC is available and to wait until it is, that would solve the problem. Alternatively, fail fast in order to retry.

Additional Information/Context

No response

CLI version used

aws-cli/2.17.27 Python/3.11.9 Windows/10 exe/AMD64

Environment details (OS name and version, etc.)

Windows 10, PowerShell 7.4.4

tim-finnigan commented 2 weeks ago

Thanks for reaching out. Do you have proxy or firewall settings configured that could be affecting the connection here? I would also look into issues with your certificate or TLS version. It's strange that this happens sporadically though and you do not see a recurring pattern. Could you share your debug logs (with any sensitive info redacted) by adding --debug to your command?

asterkin commented 1 week ago

Hi, Tim!

Thanks for your guidelines. I spent some time on experimenting with the system setups increasing the timeout tolerance and also switched on --debug. So far the problem did not show up. May be increased tolerance or --debug switch, or the both stabilized the system and this specific problem disappeared.

I will continue monitoring how it works on daily basis and if it comes back will let you know, hopefully with more details.

Best regards, Asher

On Tue, Aug 27, 2024 at 2:59 AM Tim Finnigan @.***> wrote:

Thanks for reaching out. Do you have proxy or firewall settings configured that could be affecting the connection here? I would also look into issues with your certificate or TLS version. It's strange that this happens sporadically though and you do not see a recurring pattern. Could you share your debug logs (with any sensitive info redacted) by adding --debug to your command?

— Reply to this email directly, view it on GitHub https://github.com/aws/aws-cli/issues/8880#issuecomment-2311312287, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQBQQCNXXHNDB3OHNUXY6TZTO6OJAVCNFSM6AAAAABM7VFTXCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJRGMYTEMRYG4 . You are receiving this because you authored the thread.Message ID: @.***>

asterkin commented 1 week ago

I managed to catch at least one connectivity problem. Here is the debug out:

[DEBUG] [2024-09-05T14:16:00Z] [00005548] [dns] - static: resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com Exception in WebSocket on_connection_setup callback [ERROR] [2024-09-05T14:16:11Z] [00005548] [dns] - static: getaddrinfo failed with error_code 11002 Traceback (most recent call last): [WARN] [2024-09-05T14:16:11Z] [00005548] [dns] - static, resolving host eice-01b1 File "awscrt\websocket.py", line 459, in _on_connection_setup 18b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com failed, ec 1058 (aws-c-io: AWS_IO_DNS_QUERY_FAILED, A query to dns failed to reso File "awscli\customizations\ec2instanceconnect\websocket.py", line 239, in _on_connection lve.) TypeError: 'NoneType' object is not iterable [DEBUG] [2024-09-05T14:16:11Z] [00005548] [dns] - static, invoking resolution callback for host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com with failure [ERROR] [2024-09-05T14:16:11Z] [00005548] [channel-bootstrap] - id=000001C41380D990: dns resolution failed, or all socket connections to the endpoint failed. [ERROR] [2024-09-05T14:16:11Z] [00005548] [http-connection] - static: Client connection failed with error 1058 (AWS_IO_DNS_QUERY_FAILED). [ERROR] [2024-09-05T14:16:11Z] [00005548] [websocket-setup] - id=000001C4137E0D00: Websocket setup failed to establish HTTP connection, error 1058 (AWS_IO_DNS_QUERY_FAILED). [DEBUG] [2024-09-05T14:16:11Z] [00005548] [channel-bootstrap] - id=000001C41380D990: releasing bootstrap reference [DEBUG] [2024-09-05T14:16:12Z] [00005548] [dns] - static: resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com [ERROR] [2024-09-05T14:16:12Z] [00005548] [dns] - static: getaddrinfo failed with error_code 11002 [WARN] [2024-09-05T14:16:12Z] [00005548] [dns] - static, resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com failed, ec 1058 (aws-c-io: AWS_IO_DNS_QUERY_FAILED, A query to dns failed to resolve.) [DEBUG] [2024-09-05T14:16:13Z] [00005548] [dns] - static: resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com [ERROR] [2024-09-05T14:16:13Z] [00005548] [dns] - static: getaddrinfo failed with error_code 11002 [WARN] [2024-09-05T14:16:13Z] [00005548] [dns] - static, resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com failed, ec 1058 (aws-c-io: AWS_IO_DNS_QUERY_FAILED, A query to dns failed to resolve.) [DEBUG] [2024-09-05T14:16:14Z] [00005548] [dns] - static: resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com [ERROR] [2024-09-05T14:16:14Z] [00005548] [dns] - static: getaddrinfo failed with error_code 11002 [WARN] [2024-09-05T14:16:14Z] [00005548] [dns] - static, resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com failed, ec 1058 (aws-c-io: AWS_IO_DNS_QUERY_FAILED, A query to dns failed to resolve.) [DEBUG] [2024-09-05T14:16:16Z] [00005548] [dns] - static: resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com [ERROR] [2024-09-05T14:16:16Z] [00005548] [dns] - static: getaddrinfo failed with error_code 11002 [WARN] [2024-09-05T14:16:16Z] [00005548] [dns] - static, resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com failed, ec 1058 (aws-c-io: AWS_IO_DNS_QUERY_FAILED, A query to dns failed to resolve.) [DEBUG] [2024-09-05T14:16:17Z] [00005548] [dns] - static: resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com [DEBUG] [2024-09-05T14:16:17Z] [00005548] [dns] - static: resolved record: 51.17.89.43 [DEBUG] [2024-09-05T14:16:17Z] [00005548] [dns] - static: resolved record: 51.17.234.48 [DEBUG] [2024-09-05T14:16:17Z] [00005548] [dns] - static: resolved record: 51.17.15.97 [DEBUG] [2024-09-05T14:16:17Z] [00005548] [dns] - static, resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com successful, returned 3 addresses [DEBUG] [2024-09-05T14:16:17Z] [00005548] [dns] - static: new address resolved 51.17.89.43 for host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com caching [DEBUG] [2024-09-05T14:16:17Z] [00005548] [dns] - static: new address resolved 51.17.234.48 for host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com caching [DEBUG] [2024-09-05T14:16:17Z] [00005548] [dns] - static: new address resolved 51.17.15.97 for host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com caching [DEBUG] [2024-09-05T14:16:18Z] [00005548] [dns] - static: resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com [DEBUG] [2024-09-05T14:16:18Z] [00005548] [dns] - static: resolved record: 51.17.89.43 [DEBUG] [2024-09-05T14:16:18Z] [00005548] [dns] - static: resolved record: 51.17.234.48 [DEBUG] [2024-09-05T14:16:18Z] [00005548] [dns] - static: resolved record: 51.17.15.97 [DEBUG] [2024-09-05T14:16:18Z] [00005548] [dns] - static, resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com successful, returned 3 addresses [DEBUG] [2024-09-05T14:16:19Z] [00005548] [dns] - static: resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com [DEBUG] [2024-09-05T14:16:19Z] [00005548] [dns] - static: resolved record: 51.17.89.43 [DEBUG] [2024-09-05T14:16:19Z] [00005548] [dns] - static: resolved record: 51.17.234.48 [DEBUG] [2024-09-05T14:16:19Z] [00005548] [dns] - static: resolved record: 51.17.15.97 [DEBUG] [2024-09-05T14:16:19Z] [00005548] [dns] - static, resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com successful, returned 3 addresses [DEBUG] [2024-09-05T14:16:20Z] [00005548] [dns] - static: resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com [DEBUG] [2024-09-05T14:16:20Z] [00005548] [dns] - static: resolved record: 51.17.89.43 [DEBUG] [2024-09-05T14:16:20Z] [00005548] [dns] - static: resolved record: 51.17.234.48 [DEBUG] [2024-09-05T14:16:20Z] [00005548] [dns] - static: resolved record: 51.17.15.97 [DEBUG] [2024-09-05T14:16:20Z] [00005548] [dns] - static, resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com successful, returned 3 addresses [DEBUG] [2024-09-05T14:16:21Z] [00005548] [dns] - static: resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com [DEBUG] [2024-09-05T14:16:21Z] [00005548] [dns] - static: resolved record: 51.17.89.43 [DEBUG] [2024-09-05T14:16:21Z] [00005548] [dns] - static: resolved record: 51.17.234.48 [DEBUG] [2024-09-05T14:16:21Z] [00005548] [dns] - static: resolved record: 51.17.15.97 [DEBUG] [2024-09-05T14:16:21Z] [00005548] [dns] - static, resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com successful, returned 3 addresses [DEBUG] [2024-09-05T14:16:22Z] [00005548] [dns] - static: resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com [DEBUG] [2024-09-05T14:16:22Z] [00005548] [dns] - static: resolved record: 51.17.89.43 [DEBUG] [2024-09-05T14:16:22Z] [00005548] [dns] - static: resolved record: 51.17.234.48 [DEBUG] [2024-09-05T14:16:22Z] [00005548] [dns] - static: resolved record: 51.17.15.97 [DEBUG] [2024-09-05T14:16:22Z] [00005548] [dns] - static, resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com successful, returned 3 addresses [DEBUG] [2024-09-05T14:16:23Z] [00005548] [dns] - static: resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com [DEBUG] [2024-09-05T14:16:23Z] [00005548] [dns] - static: resolved record: 51.17.89.43 [DEBUG] [2024-09-05T14:16:23Z] [00005548] [dns] - static: resolved record: 51.17.234.48 [DEBUG] [2024-09-05T14:16:23Z] [00005548] [dns] - static: resolved record: 51.17.15.97 [DEBUG] [2024-09-05T14:16:23Z] [00005548] [dns] - static, resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com successful, returned 3 addresses [DEBUG] [2024-09-05T14:16:24Z] [00005548] [dns] - static: resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com [DEBUG] [2024-09-05T14:16:24Z] [00005548] [dns] - static: resolved record: 51.17.89.43 [DEBUG] [2024-09-05T14:16:24Z] [00005548] [dns] - static: resolved record: 51.17.234.48 [DEBUG] [2024-09-05T14:16:24Z] [00005548] [dns] - static: resolved record: 51.17.15.97 [DEBUG] [2024-09-05T14:16:24Z] [00005548] [dns] - static, resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com successful, returned 3 addresses [DEBUG] [2024-09-05T14:16:25Z] [00005548] [dns] - static: resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com [DEBUG] [2024-09-05T14:16:25Z] [00005548] [dns] - static: resolved record: 51.17.89.43 [DEBUG] [2024-09-05T14:16:25Z] [00005548] [dns] - static: resolved record: 51.17.234.48 [DEBUG] [2024-09-05T14:16:25Z] [00005548] [dns] - static: resolved record: 51.17.15.97 [DEBUG] [2024-09-05T14:16:25Z] [00005548] [dns] - static, resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com successful, returned 3 addresses [DEBUG] [2024-09-05T14:16:26Z] [00005548] [dns] - static: resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com [DEBUG] [2024-09-05T14:16:26Z] [00005548] [dns] - static: resolved record: 51.17.89.43 [DEBUG] [2024-09-05T14:16:26Z] [00005548] [dns] - static: resolved record: 51.17.234.48 [DEBUG] [2024-09-05T14:16:26Z] [00005548] [dns] - static: resolved record: 51.17.15.97 [DEBUG] [2024-09-05T14:16:26Z] [00005548] [dns] - static, resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com successful, returned 3 addresses [DEBUG] [2024-09-05T14:16:27Z] [00005548] [dns] - static: resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com [DEBUG] [2024-09-05T14:16:27Z] [00005548] [dns] - static: resolved record: 51.17.89.43 [DEBUG] [2024-09-05T14:16:27Z] [00005548] [dns] - static: resolved record: 51.17.234.48 [DEBUG] [2024-09-05T14:16:27Z] [00005548] [dns] - static: resolved record: 51.17.15.97 [DEBUG] [2024-09-05T14:16:27Z] [00005548] [dns] - static, resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com successful, returned 3 addresses [DEBUG] [2024-09-05T14:16:28Z] [00005548] [dns] - static: resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com [DEBUG] [2024-09-05T14:16:28Z] [00005548] [dns] - static: resolved record: 51.17.89.43 [DEBUG] [2024-09-05T14:16:28Z] [00005548] [dns] - static: resolved record: 51.17.234.48 [DEBUG] [2024-09-05T14:16:28Z] [00005548] [dns] - static: resolved record: 51.17.15.97 [DEBUG] [2024-09-05T14:16:28Z] [00005548] [dns] - static, resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com successful, returned 3 addresses [DEBUG] [2024-09-05T14:16:29Z] [00005548] [dns] - static: resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com [DEBUG] [2024-09-05T14:16:29Z] [00005548] [dns] - static: resolved record: 51.17.89.43 [DEBUG] [2024-09-05T14:16:29Z] [00005548] [dns] - static: resolved record: 51.17.234.48 [DEBUG] [2024-09-05T14:16:29Z] [00005548] [dns] - static: resolved record: 51.17.15.97 [DEBUG] [2024-09-05T14:16:29Z] [00005548] [dns] - static, resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com successful, returned 3 addresses [DEBUG] [2024-09-05T14:16:30Z] [00005548] [dns] - static: resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com [DEBUG] [2024-09-05T14:16:30Z] [00005548] [dns] - static: resolved record: 51.17.89.43 [DEBUG] [2024-09-05T14:16:30Z] [00005548] [dns] - static: resolved record: 51.17.234.48 [DEBUG] [2024-09-05T14:16:30Z] [00005548] [dns] - static: resolved record: 51.17.15.97 [DEBUG] [2024-09-05T14:16:30Z] [00005548] [dns] - static, resolving host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com successful, returned 3 addresses [DEBUG] [2024-09-05T14:16:31Z] [00005548] [dns] - static: Either no requests have been made for an address for eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com for the duration of the ttl, or this thread is being forcibly shutdown. Killing thread. 15.97 for host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com from the cache due to cache eviction or shutdown [DEBUG] [2024-09-05T14:16:31Z] [00005548] [dns] - static: purging address 51.17.89.43 for host eice-01b118b0a95468120.22ad3d15.ec2-instance-connect-endpoint.il-central-1.amazonaws.com from the cache due to cache eviction or shutdown Connection timed out during banner exchange Connection to UNKNOWN port 65535 timed out

It's not the same kind of problem, but it already did happen to me once when VSCode tried to reconnect. Any ideas or guidelines will be highly appreciated.

Best regards, Asher

On Tue, Aug 27, 2024 at 2:59 AM Tim Finnigan @.***> wrote:

Thanks for reaching out. Do you have proxy or firewall settings configured that could be affecting the connection here? I would also look into issues with your certificate or TLS version. It's strange that this happens sporadically though and you do not see a recurring pattern. Could you share your debug logs (with any sensitive info redacted) by adding --debug to your command?

— Reply to this email directly, view it on GitHub https://github.com/aws/aws-cli/issues/8880#issuecomment-2311312287, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQBQQCNXXHNDB3OHNUXY6TZTO6OJAVCNFSM6AAAAABM7VFTXCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJRGMYTEMRYG4 . You are receiving this because you authored the thread.Message ID: @.***>

tim-finnigan commented 4 days ago

Thanks for following up. It looks like this is some kind of network issue involving the AWS CRT. I found a few other issues referencing that error. You could try opting out of the crt by running aws configure set s3.preferred_transfer_client classic. In the meantime we can reach out to the CRT team internally to try and get more insight on this.

asterkin commented 3 days ago

Thanks, Tim!

In my case, S3 was not involved, only SSH session tunneling over EIC. The problem with DNS cache does not happen often. When it does happen, it's when the VSCode is trying to reconnect after a network glitch. My setup assumes SSH session running for a couple of hours which might be a relatively rare occurrence. If there is other information I can collect and share with you, let me know.

Best regards, Asher

On Thu, Sep 12, 2024 at 8:56 PM Tim Finnigan @.***> wrote:

Thanks for following up. It looks like this is some kind of network issue involving the AWS CRT. I found a few other issues https://github.com/issues?q=is%3Aissue+%22AWS_IO_DNS_QUERY_FAILED%22 referencing that error. You could try opting out of the crt by running aws configure set s3.preferred_transfer_client classic. In the meantime we can reach out to the CRT team internally to try and get more insight on this.

— Reply to this email directly, view it on GitHub https://github.com/aws/aws-cli/issues/8880#issuecomment-2346912014, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQBQQFNQIZ33WJSPM6CNP3ZWHITVAVCNFSM6AAAAABM7VFTXCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNBWHEYTEMBRGQ . You are receiving this because you authored the thread.Message ID: @.***>