[QUESTION] Issue with kvs Viewer running on EC2 server

[suraj-vantigodi]

We are working on a use case in which we are using kinesis signalling channel to stream the data from the front end to backend. The front end is a webapp which uses Kinesis WebRTC JS sdk to establish a connection to a signalling channel and stream the data. The backend is an EC2 server on which a viewer is running. This viewer is making use of kinesis webrtc c SDK with slight modifications on top of the sample code provided by the sdk.

When the master from the sdk sample code is also run on the same EC2 server as the viewer, the viewer runs fine, and we are able to receive the h264 video frames.

However when the webapp acts as the master that streams the data to the signalling channel and we try to run the viewer, we get errors wrt to Turn connection. The webapp is running on a elastic beanstalk within the same vpc as the EC2 instance.

Please find the error logs below. Any help or support would be appreciated.

`[KVS Viewer] Offer creation successful

[KVS Viewer] Generating JSON of session description....Completed 2020-10-07 08:51:25 WARN receiveLwsMessage(): Signaling received an empty message 2020-10-07 08:51:25 WARN turnConnectionHandleStunError(): Received STUN error response. Error type: 0x00, Error Code: 403. attribute len 16, Error detail: Forbidden IP. 2020-10-07 08:51:25 WARN turnConnectionHandleStunError(): Received STUN error response. Error type: 0x00, Error Code: 403. attribute len 16, Error detail: Forbidden IP. 2020-10-07 08:51:25 ERROR turnConnectionSendData(): operation returned status code: 0x5a000027 2020-10-07 08:51:25 ERROR iceUtilsSendData(): operation returned status code: 0x5a000027 2020-10-07 08:51:25 ERROR iceUtilsSendStunPacket(): operation returned status code: 0x5a000027 2020-10-07 08:51:25 WARN iceAgentSendStunPacket(): iceUtilsSendStunPacket failed with 0x5a000027 2020-10-07 08:51:25 ERROR turnConnectionSendData(): operation returned status code: 0x5a000027 2020-10-07 08:51:25 ERROR iceUtilsSendData(): operation returned status code: 0x5a000027 2020-10-07 08:51:25 ERROR iceUtilsSendStunPacket(): operation returned status code: 0x5a000027 2020-10-07 08:51:25 WARN iceAgentSendStunPacket(): iceUtilsSendStunPacket failed with 0x5a000027 2020-10-07 08:51:25 ERROR iceAgentAddRemoteCandidate(): operation returned status code: 0x5a000005 2020-10-07 08:51:27 ERROR turnConnectionStepState(): operation returned status code: 0x5a000021 2020-10-07 08:51:27 ERROR turnConnectionStepState(): operation returned status code: 0x5a000021 2020-10-07 08:51:27 WARN turnConnectionStepState(): TurnConnection in TURN_STATE_FAILED due to 0x5a000021. Aborting TurnConnection 2020-10-07 08:51:27 WARN turnConnectionStepState(): TurnConnection in TURN_STATE_FAILED due to 0x5a000021. Aborting TurnConnection 2020-10-07 08:51:27 WARN turnConnectionStepState(): TurnConnection in TURN_STATE_FAILED due to 0x5a000021. Aborting TurnConnection 2020-10-07 08:51:27 WARN turnConnectionStepState(): TurnConnection in TURN_STATE_FAILED due to 0x5a000021. Aborting TurnConnection 2020-10-07 08:51:35 ERROR executeFailedIceAgentState(): IceAgent failed with 0x5a00000d [KVS Viewer] Cleaning up.... [KVS Viewer] Cleanup done`

[MushMal]

I assume you are using the viewer app to initiate the session with the master correct? Whether it's JS or C sample.

The information you've provided contains no actionable items. Try to capture the debug logs from the C SDK sample viewer application and the logs from the JS master sample. Run without any modifications as we won't be able to help you with your code changes. Try to debug your application and get as much data as possible to include with the scenario.

Include the settings - such as whether trickle ice is enabled, Stun/Turn, P2P.

For your debugging, you could also try running discoverNatBehavior API. Read through the existing issues to get more context: https://github.com/awslabs/amazon-kinesis-video-streams-webrtc-sdk-c/search?q=discoverNatBehavior&type=issues

[suraj-vantigodi]

@MushMal The viewer is the C sample. Removed all the changes and ran the sample as is and below are the logs when I run the Viewer sample ./kvsWebrtcClientViewer AI_test

Please enote that the AI_test is a sample channel created and I tried running the webcam demo and its running. My idea was to see if the webcam demo acts as the master and the viewer should be able to consume the data.

`KVS Master] Using trickleICE by default

[KVS Viewer] Created signaling channel AI_test [KVS Viewer] KVS WebRTC initialization completed successfully [KVS Viewer] Signaling client created successfully [KVS Viewer] Signaling client connection to socket established [KVS Viewer] Creating streaming session...completed [KVS Viewer] Completed setting local description [KVS Viewer] Offer creation successful [KVS Viewer] Generating JSON of session description....Completed 2020-10-07 18:51:31 WARN receiveLwsMessage(): Signaling received an empty message 2020-10-07 18:51:31 WARN turnConnectionHandleStunError(): Received STUN error response. Error type: 0x00, Error Code: 403. attribute len 16, Error detail: Forbidden IP. 2020-10-07 18:51:31 WARN turnConnectionHandleStunError(): Received STUN error response. Error type: 0x00, Error Code: 403. attribute len 16, Error detail: Forbidden IP. 2020-10-07 18:51:31 ERROR turnConnectionSendData(): operation returned status code: 0x5a000027 2020-10-07 18:51:31 ERROR iceUtilsSendData(): operation returned status code: 0x5a000027 2020-10-07 18:51:31 ERROR iceUtilsSendStunPacket(): operation returned status code: 0x5a000027 2020-10-07 18:51:31 WARN iceAgentSendStunPacket(): iceUtilsSendStunPacket failed with 0x5a000027 2020-10-07 18:51:31 ERROR turnConnectionSendData(): operation returned status code: 0x5a000027 2020-10-07 18:51:31 ERROR iceUtilsSendData(): operation returned status code: 0x5a000027 2020-10-07 18:51:31 ERROR iceUtilsSendStunPacket(): operation returned status code: 0x5a000027 2020-10-07 18:51:31 WARN iceAgentSendStunPacket(): iceUtilsSendStunPacket failed with 0x5a000027 2020-10-07 18:51:31 ERROR iceAgentAddRemoteCandidate(): operation returned status code: 0x5a000005 2020-10-07 18:51:41 ERROR executeFailedIceAgentState(): IceAgent failed with 0x5a00000d [KVS Viewer] Cleaning up.... 2020-10-07 18:51:42 WARN iceAgentShutdown(): TurnConnection shutdown did not complete within 1 seconds 2020-10-07 18:51:43 ERROR iceUtilsSendData(): operation returned status code: 0x58000022 2020-10-07 18:51:43 ERROR iceUtilsSendStunPacket(): operation returned status code: 0x58000022 2020-10-07 18:51:43 ERROR turnConnectionTimerCallback(): TurnConnection socket 11 closed unexpectedly 2020-10-07 18:51:43 WARN turnConnectionStepState(): TurnConnection in TURN_STATE_FAILED due to 0x58000022. Aborting TurnConnection 2020-10-07 18:51:43 ERROR iceUtilsSendData(): operation returned status code: 0x58000022 2020-10-07 18:51:43 ERROR iceUtilsSendStunPacket(): operation returned status code: 0x58000022 2020-10-07 18:51:43 ERROR turnConnectionTimerCallback(): TurnConnection socket 12 closed unexpectedly 2020-10-07 18:51:43 WARN turnConnectionStepState(): TurnConnection in TURN_STATE_FAILED due to 0x58000022. Aborting TurnConnection [KVS Viewer] Cleanup done `

[MushMal]

As you don't have verbose or debug level logs, it's impossible to tell what's going on.

From the look of it, it seems that you don't have a P2P connection and it fails to establish TURN as well. Perhaps configured with IPv6?

I would recommend stabilizing your solution on local machine with your network and iron out any issues there. Next, run it in VPC with debug logs that would give you some clues as to what's going on.

Again, I am guessing, but it looks like there is a network routining issue from the look of it but impossible to tell without debug logs

[suraj-vantigodi]

Can @MushMal please tell me how do I get you those additional logs. The ones I've shared now are from the print statements of the viewer code. I also checked the logger files, they don't have any additional information.

I am trying to run the viewer sample code and that itself is giving me errors. The solution would be only built after this sample code runs.

locally if I run, I am able to get data in the viewer, but the issue only appears when the view is run on an EC2 instance.

[MushMal]

https://github.com/awslabs/amazon-kinesis-video-streams-webrtc-sdk-c/blob/17fa3a7234944ee7d5d9ee8124ecfbc510ce0f98/README.md#setup-desired-log-level

Not sure what you mean by giving me errors.

Again, in EC2 you might want to configure your network to use IPv4 only

[suraj-vantigodi]

@MushMal the errors are already highlighted in the logs shared earlier in this thread. They are with respect to TURN and ICE connection. The EC2 is configured to use IPv4 only but still the same issue persists.

[MushMal]

2020-10-07 18:51:31 WARN turnConnectionHandleStunError(): Received STUN error response. Error type: 0x00, Error Code: 403. attribute len 16, Error detail: Forbidden IP. - this is benign. Should try to mask out the interfaces that you are not using.

0x5a000027 - this can happen if your socket fails resolving to a usable connection (STATUS_TURN_CONNECTION_PEER_NOT_USABLE). This is OK as we have a highly asynchronous exchange.

0x5a00000d - the resulting state error from not being able to nominate/select a pair

0x58000022 - STATUS_SOCKET_CONNECTION_CLOSED_ALREADY this can happen on a socket timeout during ICE processing.

All of these status codes are directly defined in the public header file so you can look at them.

I feel like we are not making any progress on this. Please work on your EC2 instance and the networking to ensure it has usable interfaces. Namely, it should have at least one IPv4 interface to at least use TURN.

[MushMal]

Any further update on this issue? Could you please close if the networking issue is resolved?

[disa6302]

@suraj-vantigodi ,

Please also make sure your security group rules are appropriately configured for the EC2 instance. See if you are limiting the subnets in any way. Closing this for now because it seems to have gone stale. Feel free to reopen/create a new issue if you have any questions.

[suraj-vantigodi]

@disa6302 and @MushMal We managed to get in touch with the AWS team and they suggested a change in the code within samples/kvsWebRTCClientViewer.c, line 23, change:

createSampleConfiguration(argc > 1 ? argv[1] : SAMPLE_CHANNEL_NAME, SIGNALING_CHANNEL_ROLE_TYPE_VIEWER, TRUE, TRUE, &pSampleConfiguration); to createSampleConfiguration(argc > 1 ? argv[1] : SAMPLE_CHANNEL_NAME, SIGNALING_CHANNEL_ROLE_TYPE_VIEWER, FALSE, TRUE, &pSampleConfiguration);

This disables a feature called “trickle ICE” and seems to have fixed the issue for us.