Unable to connect to secure tunnel via localproxy

Describe the bug

I have implemented a secure tunnel feature based on the SDK. I was able to connect to the destination through the browser's SSH, but I was unable to connect to the destination through the local proxy. How should I solve this problem? Please refer to the attachment for the SDK log. log.txt

The terminal printing error is as follows: Send Message failed with error code 13320(libaws-c-iotdevice: AWS_ERROR_IOTDEVICE_SECURE_TUNNELING_INVALID_STREAM_ID, Secure Tunnel invalid stream id.) Message received on service id:'SSH' with connection id: (1) with payload: 'SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.7

Expected Behavior

After running localproxy, it is possible to connect to the destination machine through SSH.

Current Behavior

Unable to connect to the destination machine through SSH.

Reproduction Steps

Reference code: https://github.com/iotlabtpe/aws-iot-device-sdk-cpp-v2/pull/1/commits

Possible Solution

No response

Additional Information/Context

No response

SDK version used

latest

Environment details (OS name and version, etc.)

Ubuntu 18.04

The problem cannot be duplicated by using V1 protocol.

We discovered if the source connection is from the AWS console, V1 protocol will be used by the SSH connection in browser. Then the tunnel worked well.

But if we executed the localproxy in the Ubuntu laptop, and the source connection is from localproxy. Due to the localproxy support V3 protocol, the current SDK cannot make the connection with V3 protocol.

Hello @zhangzz0413 ,

Thank you very much for your submission.

Have you reviewed the documentation provided in aws-samples/aws-iot-securetunneling-localproxy? The repository provides the V3WebSocketProtocoleGuide which goes over Establishing tunnel connection and causes of failure. This documentation is referred to as part of the Secure Tunnel sample readme along side the Secure Tunnel Userguide.

I see that you have previously opened an issue regarding this matter within our repository. I will link it here for reference.

Let me know if this helps or if this documentation has already been reviewed/is helpful in your use case. I will look into this behavior.

Best regards,

Yasmine

HI @yasminetalby,

Thanks for providing the useful info. I have reviewed the failure cases of local proxy. However, The problem was not the local proxy failed to start. Local proxy can start well on my Ubuntu laptop by the following command.

OptiPlex-7090:~/aws/localproxy/aws-iot-securetunneling-localproxy$ ./build/bin/localproxy -r us-east-1 -s 2222 -t AQGAAXiNBbzCpbuyTbWjLnDTEayiABsAAgABQQAMODUxMTMyODIzOTMwAAFUAANDQVQAAQAHYXdzLWttcwBLYXJuOmF3czprbXM6dXMtZWFzdC0xOjcwMTU0NTg5ODcwNzprZXkvMmU4ZTAxMDEtYzE3YS00NjU1LTlhYWQtNjA2N2I2NGVhZWQyALgBAgEAeCnVDgZO1Yc3KS8Kfg4y6CpZokCmL0C4Py6NJTdp3XngAcZ4JwU3rEUGqnC8bC6R0U8AAAB-MHwGCSqGSIb3DQEHBqBvMG0CAQAwaAYJKoZIhvcNAQcBMB4GCWCGSAFlAwQBLjARBAzr6r9qM9S67xHMkaMCARCAOznHr2z0uGNEjMur7Y6a3LwTcBktqHwPCkwl95B0c5thE-U5d4FMGN12SLejSrGBpMrPZV1zZSlU7NYWAgAAAAAMAAAQAAAAAAAAAAAAAAAAAJ_gwx-5FPezl0WMEZEBWyz_____AAAAAQAAAAAAAAAAAAAAAQAAAC86Zjzxk9Rt1Rfk5mbeY0KuK6G8sQ9p4WOJtuK9ECYIQ58I6BXVLdT6ausVS_XlV7Ui5x-CfngoYVZeOfOOgLg=
[2023-08-15 11:56:24.658232] (0x00007f0132065f00) [warning] Found access token supplied via CLI arg. Consider using environment variable AWSIOT_TUNNEL_ACCESS_TOKEN instead
[2023-08-15 11:56:24.658407] (0x00007f0132065f00) [info] Starting proxy in source mode
[2023-08-15 11:56:24.659640] (0x00007f0132065f00) [info] Attempting to establish web socket connection with endpoint wss://data.tunneling.iot.us-east-1.amazonaws.com:443
[2023-08-15 11:56:25.817078] (0x00007f0132065f00) [info] Web socket session ID: 1243b7fffe30d985-00006770-00005daf-299021c3e63ab4d3-efe256ad
[2023-08-15 11:56:25.817168] (0x00007f0132065f00) [info] Successfully established websocket connection with proxy server: wss://data.tunneling.iot.us-east-1.amazonaws.com:443
[2023-08-15 11:56:25.817393] (0x00007f0132065f00) [info] Updated port mapping for v1 format: 
[2023-08-15 11:56:25.817427] (0x00007f0132065f00) [info] SSH = 2222
[2023-08-15 11:56:25.817451] (0x00007f0132065f00) [info] calling setup from loop
[2023-08-15 11:56:25.817780] (0x00007f0132065f00) [info] Listening for new connection on port 2222
[2023-08-15 11:56:34.004920] (0x00007f0132065f00) [info] creating tcp connection id 1
[2023-08-15 11:56:34.005013] (0x00007f0132065f00) [info] Accepted tcp connection on port 2222 from 127.0.0.1:45594

What's I did was the data forwarding on top of tunnel established. The websocket is rely on C++ SDK I guess fully V1 to V3 protocol supported.

I did the quick testing with following situation

SSH connection from AWS console - SDK recognized V1 protocol and the connection established well.
SSH connection via Device Client from Ubuntu laptop - due to the device client used the old version of C++ SDK, which only support V1 protocol. In this case, connection established well.
SSH connection via local proxy from Ubuntu laptop - Failed to establish SSH connection. The difference from previous 2 cases is the version of websocket protocol. Local proxy used V3 protocol.

The problem here was secure tunnel can establish well with V3 protocol, but the error happens at the SSH service made the connection. I see the error log that indicate websocket shutdown.

[DEBUG] [2023-08-13T02:06:48Z] [00007f9fdffff700] [iotdevice-st] - id=0x7f9ffc0176b0: secure tunnel websocket shutdown invoked with error code 13334(libaws-c-iotdevice: AWS_ERROR_IOTDEVICE_SECURE_TUNNELING_USER_REQUESTED_STOP, Secure Tunnel connection interrupted by user request.)

Let me know if you need more information. I am looking for any sample application running on top of secure tunnel with V3 protocol. If you have, I can do a quick test as well.

Reviewing the logs you provided and it appears as though a V3 connection is established with the Source device sending a V3 Stream Start message with a service id 'SSH' and a connection id 1.

[DEBUG] [2023-08-13T02:06:41Z] [00007f9fdffff700] [iotdevice-st] - id=0x7f9fdfffe050: aws_secure_tunnel_message_view service_id set to 'SSH'
[DEBUG] [2023-08-13T02:06:41Z] [00007f9fdffff700] [iotdevice-st] - id=0x7f9fdfffe050: aws_secure_tunnel_message_view stream_id set to 1
[DEBUG] [2023-08-13T02:06:41Z] [00007f9fdffff700] [iotdevice-st] - id=0x7f9fdfffe050: aws_secure_tunnel_message_view connection_id set to 1
[INFO] [2023-08-13T02:06:41Z] [00007f9fdffff700] [iotdevice-st] - id=0x7f9ffc0176b0: Secure tunnel client Protocol set to V3 based on received STREAM START
[INFO] [2023-08-13T02:06:41Z] [00007f9fdffff700] [iotdevice-st] - id=0x7f9ffc0176b0: Secure Tunnel service id 'SSH' set to stream id (1) with active connection_id (1)```

The following log indicates a Data message is attempting to be sent but has not been provided a service id.

[DEBUG] [2023-08-13T02:06:41Z] [00007f9fdffff700] [iotdevice-st] - id=0x7f9ffc0176b0: No active stream to assign outbound DATA message a stream id```

This log is created on line 390 in crt->aws-c-iot->source->secure_tunneling_operations.c and can only be reached via the absence of a service id in an outbound secure tunnel message and is what is creating the error you mentioned: Send Message failed with error code 13320(libaws-c-iotdevice: AWS_ERROR_IOTDEVICE_SECURE_TUNNELING_INVALID_STREAM_ID, Secure Tunnel invalid stream id.)

A little further down in the logs, we can see that a Data message has been successfully received by the client. The message contains a matching service id and connection id to the stream that was established.

[DEBUG] [2023-08-13T02:06:41Z] [00007f9fdffff700] [iotdevice-st] - id=0x7f9ffc0176b0: deserializing message from cursor of size 54.
[DEBUG] [2023-08-13T02:06:41Z] [00007f9fdffff700] [iotdevice-st] - id=0x7f9fdfffe050: aws_secure_tunnel_message_view type 'DATA'
[DEBUG] [2023-08-13T02:06:41Z] [00007f9fdffff700] [iotdevice-st] - id=0x7f9fdfffe050: aws_secure_tunnel_message_view service_id set to 'SSH'
[DEBUG] [2023-08-13T02:06:41Z] [00007f9fdffff700] [iotdevice-st] - id=0x7f9fdfffe050: aws_secure_tunnel_message_view stream_id set to 1
[DEBUG] [2023-08-13T02:06:41Z] [00007f9fdffff700] [iotdevice-st] - id=0x7f9fdfffe050: aws_secure_tunnel_message_view connection_id set to 1
[DEBUG] [2023-08-13T02:06:41Z] [00007f9fdffff700] [iotdevice-st] - id=0x7f9fdfffe050: aws_secure_tunnel_message_view payload set containing 41 bytes```

This received message is what is printing this log: Message received on service id:'SSH' with connection id: (1) with payload: 'SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.7 indicating a successfully received message on stream id 'SSH' with connection id 1 with the payload "SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.7"

When you send a data message, you must provide a service id and a connection id to send a V3 message. You can see how to build a message with a service id and connection id here in the sample: https://github.com/aws/aws-iot-device-sdk-cpp-v2/blob/11855fc2601df396cbf1a36b1ae208b1b0ed8154/samples/secure_tunneling/secure_tunnel/main.cpp#L366C29-L366C29

                std::shared_ptr<Message> message = std::make_shared<Message>(ByteCursorFromCString(toSend.c_str()));

                /* If the secure tunnel has service ids, we will use one for our messages. */
                if (m_serviceId.has_value())
                {
                   // HERE WE ARE ADDING A SERVICE ID TO THE MESSAGE
                    message->WithServiceId(m_serviceId.value()); 
                }

               // HERE WE ARE ADDING A CONNECTION ID TO THE MESSAGE
                message->WithConnectionId(connectionId);

                secureTunnel->SendMessage(message);



Feel free to reach out and/or reopen the issue if you have further questions.

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

aws / aws-iot-device-sdk-cpp-v2