search

aws-samples / aws-iot-securetunneling-localproxy

AWS Iot Secure Tunneling local proxy reference C++ implementation
https://docs.aws.amazon.com/iot/latest/developerguide/what-is-secure-tunneling.html
Apache License 2.0
74 stars 71 forks source link

Secure Tunnel Connection Using Username and Password Fails from the AWS Console #152

Closed Adesanya-Toba closed 2 months ago

Adesanya-Toba commented 4 months ago

Describe the bug After starting the local proxy on my destination device, I attempt to open an SSH connection to the device using the username and password field from the AWS console. This results in an error in the local proxy running on the destination device, causing the device to disconnect.

To Reproduce

  1. Create a new tunnel from the AWS console
  2. Obtain the destination access token and start the local proxy on the destination device. I use a config file to store credentials.
  3. Attempt to connect from the AWS console using username and password.

Expected behavior

Actual behavior

Logs

root@iot-gate-imx8plus:/home/dependencies/aws-iot-securetunneling-localproxy/build/bin# ./localproxy --config /home/local_proxy/config.ini -v 6
[2024-05-08 12:11:45.565299] (0x0000ffffa9a24000) [debug] Detect port mapping configuration provided through CLI in destination mode:
[2024-05-08 12:11:45.565450] (0x0000ffffa9a24000) [debug] ----------------------------------------------------------
[2024-05-08 12:11:45.565491] (0x0000ffffa9a24000) [debug] SSH = 22
[2024-05-08 12:11:45.565525] (0x0000ffffa9a24000) [debug] ----------------------------------------------------------
[2024-05-08 12:11:45.565619] (0x0000ffffa9a24000) [debug] /home/dependencies/aws-iot-securetunneling-localproxy/build/bin/config does not exist!
[2024-05-08 12:11:45.565747] (0x0000ffffa9a24000) [info] Starting proxy in destination mode
[2024-05-08 12:11:45.565803] (0x0000ffffa9a24000) [trace] Setting up web socket...
[2024-05-08 12:11:45.575563] (0x0000ffffa9a24000) [trace] Calling control_callback with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.575760] (0x0000ffffa9a24000) [info] Attempting to establish web socket connection with endpoint wss://data.tunneling.iot.eu-central-1.amazonaws.com:443
[2024-05-08 12:11:45.575787] (0x0000ffffa9a24000) [trace] Resolving proxy server host: data.tunneling.iot.eu-central-1.amazonaws.com
[2024-05-08 12:11:45.577803] (0x0000ffffa9a24000) [debug] Resolved proxy server IP: 18.192.167.159
[2024-05-08 12:11:45.577899] (0x0000ffffa9a24000) [trace] Calling lowest_layer with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.605751] (0x0000ffffa9a24000) [debug] Connected successfully with proxy server
[2024-05-08 12:11:45.605872] (0x0000ffffa9a24000) [trace] Calling lowest_layer with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.605940] (0x0000ffffa9a24000) [trace] Calling lowest_layer with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.606070] (0x0000ffffa9a24000) [trace] Performing SSL handshake with proxy server
[2024-05-08 12:11:45.606114] (0x0000ffffa9a24000) [trace] Calling set_verify_mode with type: single_ssl_stream
[2024-05-08 12:11:45.606195] (0x0000ffffa9a24000) [trace] Calling set_verify_callback with type: single_ssl_stream
[2024-05-08 12:11:45.606247] (0x0000ffffa9a24000) [trace] Calling next_layer().async_handshake with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.606292] (0x0000ffffa9a24000) [trace] SSL next_layer() SNI is set : data.tunneling.iot.eu-central-1.amazonaws.com
[2024-05-08 12:11:45.644859] (0x0000ffffa9a24000) [debug] Successfully completed SSL handshake with proxy server
[2024-05-08 12:11:45.645008] (0x0000ffffa9a24000) [trace] Performing websocket handshake with proxy server
[2024-05-08 12:11:45.645088] (0x0000ffffa9a24000) [trace] Calling async_handshake with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.645335] (0x0000ffffa9a24000) [trace] Web socket ugprade request(*not entirely final):
GET /tunnel?local-proxy-mode=destination HTTP/1.1
Host: data.tunneling.iot.eu-central-1.amazonaws.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: XK9zgz2STllsD3Kvh1eJAw==
Sec-WebSocket-Version: 13
Sec-WebSocket-Protocol: aws.iot.securetunneling-3.0
access-token: ***ACCESS_TOKEN_REMOVED***
User-Agent: localproxy linux 64-bit/boost-1.84.0/openssl-3.0.0/protobuf-3.17.3

[2024-05-08 12:11:45.731195] (0x0000ffffa9a24000) [trace] Web socket upgrade response:
HTTP/1.1 101 Switching Protocols
Date: Wed, 08 May 2024 12:11:45 GMT
Content-Length: 0
Connection: upgrade
channel-id: 0af1b7fffe8a147d-000016e8-00059ea4-039142acadf0a697-ad499a24
upgrade: websocket
sec-websocket-accept: zGXDkON73pbICVGndgm6tOfJ46g=
sec-websocket-protocol: aws.iot.securetunneling-3.0

[2024-05-08 12:11:45.731637] (0x0000ffffa9a24000) [trace] Calling binary with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.731684] (0x0000ffffa9a24000) [trace] Calling auto_fragment with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.731711] (0x0000ffffa9a24000) [info] Web socket session ID: 0af1b7fffe8a147d-000016e8-00059ea4-039142acadf0a697-ad499a24
[2024-05-08 12:11:45.731742] (0x0000ffffa9a24000) [debug] Web socket subprotocol selected: aws.iot.securetunneling-3.0
[2024-05-08 12:11:45.731768] (0x0000ffffa9a24000) [info] Successfully established websocket connection with proxy server: wss://data.tunneling.iot.eu-central-1.amazonaws.com:443
[2024-05-08 12:11:45.731834] (0x0000ffffa9a24000) [debug] Seting up web socket pings for every 20000 milliseconds
[2024-05-08 12:11:45.731863] (0x0000ffffa9a24000) [trace] Calling async_ping with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.732061] (0x0000ffffa9a24000) [trace] Waiting for service ids...
[2024-05-08 12:11:45.732119] (0x0000ffffa9a24000) [trace] async_web_socket_read_loop_for_service_ids
[2024-05-08 12:11:45.732145] (0x0000ffffa9a24000) [debug] Scheduled next read:
[2024-05-08 12:11:45.732168] (0x0000ffffa9a24000) [trace] Calling async_read_some with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.732472] (0x0000ffffa9a24000) [trace] Processing control message
[2024-05-08 12:11:45.732565] (0x0000ffffa9a24000) [trace] Using global control message handler
[2024-05-08 12:11:45.732596] (0x0000ffffa9a24000) [debug] Extracting service Ids from control message 5
[2024-05-08 12:11:45.732629] (0x0000ffffa9a24000) [trace] Service id received: 
[2024-05-08 12:11:45.732653] (0x0000ffffa9a24000) [trace] SSH
[2024-05-08 12:11:45.732679] (0x0000ffffa9a24000) [trace] Validating service ids configuration
[2024-05-08 12:11:45.732706] (0x0000ffffa9a24000) [trace] Setting up tcp sockets 
[2024-05-08 12:11:45.732729] (0x0000ffffa9a24000) [trace] Clearing all ws data buffers
[2024-05-08 12:11:45.732751] (0x0000ffffa9a24000) [trace] Finished Clearing all ws data buffers
[2024-05-08 12:11:45.732774] (0x0000ffffa9a24000) [trace] Initializing tcp clients ...
[2024-05-08 12:11:45.732815] (0x0000ffffa9a24000) [trace] Setting up tcp socket for service id: SSH
[2024-05-08 12:11:45.732845] (0x0000ffffa9a24000) [trace] Waiting for stream start...
[2024-05-08 12:11:45.732870] (0x0000ffffa9a24000) [debug] Starting web socket read loop continue reading...
[2024-05-08 12:11:45.732893] (0x0000ffffa9a24000) [trace] Calling async_read_some with type: websocket_stream_single_ssl_type
[2024-05-08 12:11:45.732972] (0x0000ffffa9a24000) [trace] return continue_reading true
[2024-05-08 12:11:45.733013] (0x0000ffffa9a24000) [debug] Starting web socket read loop while web socket is already reading. Ignoring...
[2024-05-08 12:12:05.732449] (0x0000ffffa9a24000) [trace] Sent ping data: 1715170325732
[2024-05-08 12:12:05.732546] (0x0000ffffa9a24000) [trace] Calling async_ping with type: websocket_stream_single_ssl_type
[2024-05-08 12:12:12.251923] (0x0000ffffa9a24000) [trace] Pong reply latency: 26520 ms
[2024-05-08 12:12:12.252044] (0x0000ffffa9a24000) [trace] Pong reply latency: 6520 ms
[2024-05-08 12:12:12.281726] (0x0000ffffa9a24000) [trace] Processing control message
[2024-05-08 12:12:12.281827] (0x0000ffffa9a24000) [trace] Using global control message handler
--------------------------------- ERROR HERE -------------------------------------------------
[2024-05-08 12:12:12.282461] (0x0000ffffa9a24000) [error] Error from io_ctx::run(): Receive stream start before receiving service ids. Cannot forward data.
[2024-05-08 12:12:12.282553] (0x0000ffffa9a24000) [error] Failed web socket session ID: 0af1b7fffe8a147d-000016e8-00059ea4-039142acadf0a697-ad499a24
--------------------------------- ERROR HERE -------------------------------------------------
[2024-05-08 12:12:12.283583] (0x0000ffffa9a24000) [trace] Setting up web socket...
[2024-05-08 12:12:12.285685] (0x0000ffffa9a24000) [trace] Calling control_callback with type: websocket_stream_single_ssl_type
[2024-05-08 12:12:12.285806] (0x0000ffffa9a24000) [info] Attempting to establish web socket connection with endpoint wss://data.tunneling.iot.eu-central-1.amazonaws.com:443
[2024-05-08 12:12:12.285856] (0x0000ffffa9a24000) [trace] Resolving proxy server host: data.tunneling.iot.eu-central-1.amazonaws.com
[2024-05-08 12:12:12.329890] (0x0000ffffa9a24000) [debug] Resolved proxy server IP: 35.157.95.122
[2024-05-08 12:12:12.330002] (0x0000ffffa9a24000) [trace] Calling lowest_layer with type: websocket_stream_single_ssl_type
[2024-05-08 12:12:12.356690] (0x0000ffffa9a24000) [debug] Connected successfully with proxy server
[2024-05-08 12:12:12.356806] (0x0000ffffa9a24000) [trace] Calling lowest_layer with type: websocket_stream_single_ssl_type
[2024-05-08 12:12:12.356850] (0x0000ffffa9a24000) [trace] Calling lowest_layer with type: websocket_stream_single_ssl_type
[2024-05-08 12:12:12.356888] (0x0000ffffa9a24000) [trace] Performing SSL handshake with proxy server
[2024-05-08 12:12:12.356920] (0x0000ffffa9a24000) [trace] Calling set_verify_mode with type: single_ssl_stream
[2024-05-08 12:12:12.356953] (0x0000ffffa9a24000) [trace] Calling set_verify_callback with type: single_ssl_stream
[2024-05-08 12:12:12.356993] (0x0000ffffa9a24000) [trace] Calling next_layer().async_handshake with type: websocket_stream_single_ssl_type
[2024-05-08 12:12:12.357030] (0x0000ffffa9a24000) [trace] SSL next_layer() SNI is set : data.tunneling.iot.eu-central-1.amazonaws.com
[2024-05-08 12:12:12.395437] (0x0000ffffa9a24000) [debug] Successfully completed SSL handshake with proxy server
[2024-05-08 12:12:12.395525] (0x0000ffffa9a24000) [trace] Performing websocket handshake with proxy server
[2024-05-08 12:12:12.395570] (0x0000ffffa9a24000) [trace] Calling async_handshake with type: websocket_stream_single_ssl_type
[2024-05-08 12:12:12.395615] (0x0000ffffa9a24000) [trace] Web socket ugprade request(*not entirely final):
GET /tunnel?local-proxy-mode=destination HTTP/1.1
Host: data.tunneling.iot.eu-central-1.amazonaws.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: vp9sMsZfS4kzKMjRbwuuig==
Sec-WebSocket-Version: 13
Sec-WebSocket-Protocol: aws.iot.securetunneling-3.0
access-token: ***ACCESS_TOKEN_REMOVED***
User-Agent: localproxy linux 64-bit/boost-1.84.0/openssl-3.0.0/protobuf-3.17.3

[2024-05-08 12:12:12.479306] (0x0000ffffa9a24000) [trace] Web socket upgrade response:
HTTP/1.1 400 Bad Request
Date: Wed, 08 May 2024 12:12:12 GMT
Content-Type: text/plain
Content-Length: 85
Connection: keep-alive
channel-id: 029836fffe37c32d-00001778-0005a26f-dd8f498194830f15-5371459f
X-Status-Reason: Invalid access token: The access token was previously used and can not be used again.

Invalid access token: The access token was previously used and can not be used again.
[2024-05-08 12:12:12.479451] (0x0000ffffa9a24000) [error] Proxy server rejected web socket upgrade request: (HTTP/1.1 400 Bad Request) "Invalid access token: The access token was previously used and can not be used again."
[2024-05-08 12:12:12.479528] (0x0000ffffa9a24000) [trace] Calling is_open with type: websocket_stream_single_ssl_type
[2024-05-08 12:12:12.479565] (0x0000ffffa9a24000) [trace] Calling lowest_layer with type: websocket_stream_single_ssl_type
[2024-05-08 12:12:12.479600] (0x0000ffffa9a24000) [trace] Calling lowest_layer with type: websocket_stream_single_ssl_type

AWS Console Screenshot Error appears as soon as I click connect from the Cosole. image

Console connection fails (Connect button is greyed out) and requires a token rotation. image

Environment (please complete the following information):

Additional context NOTE: I am able to successfully connect when using the local proxy on my machine as the source device! The failure only happens when I try to connect via the AWS console.

I pulled the latest Debian base image and built the local proxy inside Docker for my platform. I ran it with the binary image and obtained the same error.

Adesanya-Toba commented 4 months ago

Hi all, any thoughts on this, as I think it might be a small thing I'm missing?

As I said, it works when I use my local machine (i.e., laptop running the local proxy) as the source device but fails when I use the AWS console as the source device.

RogerZhongAWS commented 4 months ago

Hey @Adesanya-Toba thanks for opening this issue. I was able to quickly reproduce the same error you are seeing. Do you know if this happening as a result of a recent update? (if you used a previous version of the localproxy, was it working before?)

Adesanya-Toba commented 4 months ago

Hi @RogerZhongAWS, thanks for checking this out. No, I haven't tested earlier versions of the local proxy.

RogerZhongAWS commented 4 months ago

@Adesanya-Toba , In a recent update, we actually added a new CLI option that the user needs to pass when connecting the localproxy to tunnel clients that only support older versions of the protocol. Because the console uses an older version of the protocol (V1) you will need to pass in --destination-client-type V1 into your localproxy run command (or add it within your config)

I also realize that the existing documentation may be a little confusing to understand, will take a moment to fix that.

Adesanya-Toba commented 4 months ago

Oh, alright. Thanks, I'll give this a spin and feedback.

Adesanya-Toba commented 4 months ago

Hi @RogerZhongAWS, yeah that did the trick! 🎉

Any plans to update the local proxy protocol on the console though?

RogerZhongAWS commented 4 months ago

Can't answer that yet by myself, will need to consult with various people internally.