TestReceiveAV example doesn't work on AWS EC2 Windows Instance

[davyloots]

The TestReceiveAV example works on my development machine, but on a T3.Medium EC2 Instance running Windows Server 2019, I'm getting the following exception when a connection is established: System.InvalidOperationException: Cannot invoke native method with invalid peer connection handle. at Microsoft.MixedReality.WebRTC.PeerConnection.ThrowIfConnectionNotOpen() at Microsoft.MixedReality.WebRTC.PeerConnection.SetRemoteDescription(String type, String sdp) at TestReceiveAV.Program.<>c__DisplayClass3_0.<MessageReceived>b__3(Task t) at System.Threading.Tasks.ContinuationTaskFromTask.InnerInvoke() at System.Threading.Tasks.Task.<>c.<.cctor>b__274_0(Object obj) at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location where exception was thrown --- at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location where exception was thrown --- at TestReceiveAV.Program.MessageReceived(WebRtcSession session, String msg) at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__139_1(Object state) at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute() at System.Threading.ThreadPoolWorkQueue.Dispatch() at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

It looks like the Initialize call doesn't complete successfully, but I can't figure out why not.

SDP message:

{"type":"sdp","offer":"v=0 o=- 5845073552973451558 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE 0 a=msid-semantic: WMS WltuB8UORYdm7Lgs9ap6UwJO7PrfyTa5JSYZ m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 102 122 127 121 125 107 108 109 124 120 123 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:R4li a=ice-pwd:i+1mhVrMsDTb4ERXovcg2WT6 a=ice-options:trickle a=fingerprint:sha-256 CF:76:FC:20:14:B7:C8:B8:F9:A5:09:D3:85:40:04:EB:55:7C:B9:37:A5:48:68:C5:AC:51:FC:82:B3:58:3D:14 a=setup:actpass a=mid:0 a=extmap:14 urn:ietf:params:rtp-hdrext:toffset a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:13 urn:3gpp:video-orientation a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01 a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay a=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing a=extmap:8 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07 a=extmap:9 http://www.webrtc.org/experiments/rtp-hdrext/color-space a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id a=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id a=sendrecv a=msid:WltuB8UORYdm7Lgs9ap6UwJO7PrfyTa5JSYZ 87458a0d-0abc-4efa-91de-ab4a38ee031f a=rtcp-mux a=rtcp-rsize a=rtpmap:96 VP8/90000 a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=rtpmap:97 rtx/90000 a=fmtp:97 apt=96 a=rtpmap:98 VP9/90000 a=rtcp-fb:98 goog-remb a=rtcp-fb:98 transport-cc a=rtcp-fb:98 ccm fir a=rtcp-fb:98 nack a=rtcp-fb:98 nack pli a=fmtp:98 profile-id=0 a=rtpmap:99 rtx/90000 a=fmtp:99 apt=98 a=rtpmap:100 VP9/90000 a=rtcp-fb:100 goog-remb a=rtcp-fb:100 transport-cc a=rtcp-fb:100 ccm fir a=rtcp-fb:100 nack a=rtcp-fb:100 nack pli a=fmtp:100 profile-id=2 a=rtpmap:101 rtx/90000 a=fmtp:101 apt=100 a=rtpmap:102 H264/90000 a=rtcp-fb:102 goog-remb a=rtcp-fb:102 transport-cc a=rtcp-fb:102 ccm fir a=rtcp-fb:102 nack a=rtcp-fb:102 nack pli a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f a=rtpmap:122 rtx/90000 a=fmtp:122 apt=102 a=rtpmap:127 H264/90000 a=rtcp-fb:127 goog-remb a=rtcp-fb:127 transport-cc a=rtcp-fb:127 ccm fir a=rtcp-fb:127 nack a=rtcp-fb:127 nack pli a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f a=rtpmap:121 rtx/90000 a=fmtp:121 apt=127 a=rtpmap:125 H264/90000 a=rtcp-fb:125 goog-remb a=rtcp-fb:125 transport-cc a=rtcp-fb:125 ccm fir a=rtcp-fb:125 nack a=rtcp-fb:125 nack pli a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f a=rtpmap:107 rtx/90000 a=fmtp:107 apt=125 a=rtpmap:108 H264/90000 a=rtcp-fb:108 goog-remb a=rtcp-fb:108 transport-cc a=rtcp-fb:108 ccm fir a=rtcp-fb:108 nack a=rtcp-fb:108 nack pli a=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f a=rtpmap:109 rtx/90000 a=fmtp:109 apt=108 a=rtpmap:124 red/90000 a=rtpmap:120 rtx/90000 a=fmtp:120 apt=124 a=rtpmap:123 ulpfec/90000 a=ssrc-group:FID 92661118 435232488 a=ssrc:92661118 cname:b/Z5f8eZiYNoB8DH a=ssrc:92661118 msid:WltuB8UORYdm7Lgs9ap6UwJO7PrfyTa5JSYZ 87458a0d-0abc-4efa-91de-ab4a38ee031f a=ssrc:92661118 mslabel:WltuB8UORYdm7Lgs9ap6UwJO7PrfyTa5JSYZ a=ssrc:92661118 label:87458a0d-0abc-4efa-91de-ab4a38ee031f a=ssrc:435232488 cname:b/Z5f8eZiYNoB8DH a=ssrc:435232488 msid:WltuB8UORYdm7Lgs9ap6UwJO7PrfyTa5JSYZ 87458a0d-0abc-4efa-91de-ab4a38ee031f a=ssrc:435232488 mslabel:WltuB8UORYdm7Lgs9ap6UwJO7PrfyTa5JSYZ a=ssrc:435232488 label:87458a0d-0abc-4efa-91de-ab4a38ee031f "}

[djee-ms]

Hi @davyloots,

I had a look at the TestReceiveAV code and I see that the continuation task does not check if the initialization succeeded:

https://github.com/microsoft/MixedReality-WebRTC/blob/843c7dbc28cb7f2f6aa1354d4368f7ddf044b099/examples/TestReceiveAV/Program.cs#L122-L132

I'd try to check for t.Exception inside the continuation task, or add some TaskContinuationOptions.OnlyOnRanToCompletion option to that continuation task. To be fair since there's an await already there's no reason to use a continuation task.

As for why it's failing, does that EC2 instance have access to both a webcam and microphone? The video module is somewhat resilient but the default Google audio module is very picky and will fail the entire WebRTC initializing if any audio device fails to initialize, and I think also if there's no audio device (can't remember that one for sure on top of my head).

[davyloots]

Thank you for the quick response!

Yes, the Initialize call not succeeding is the actual problem. Thank you for the tip on the webcam/microphone. The server doesn't actually have an audio recording device so that's probably why it fails. I didn't expect it to need one. I've been looking for hours for ways to create a virtual microphone on the EC2 instance, but it appears virtually impossible. But at least now I why it doesn't work.

Edit: found out that https://www.vb-audio.com/Cable/ does work on a server and creates a virtual audio recording device (you have to disable the remote audio on your RDP session for it to show up). Unfortunately it doesn't solve the Initialization issue of webRTC.

If I find a way to create a virtual microphone on the server, I'll update the issue. Thanks again!

[djee-ms]

Yes this is a problem many users have and we want to fix, the code shouldn't require an audio device to be present. Unfortunately that's deep inside Google's code so not easy for us to make a change, as we are consuming WebRTC UWP which is in turn consuming the webrtc.org repo.

[djee-ms]

The virtual microphone might not be necessary, I think only the output audio device is.

[davyloots]

There actually is an audio input and output device now - albeit a virtual one. Unfortunately still the same error. I wish there was a way to get some logging out of the Google webRTC library to at least have an idea of what's going on.

[djee-ms]

You can maybe use DebugView from SysInternals, if that runs on the instance. It's a small tool that registers as a debugger (but is not a debugger) and allows you to see the debug logs that are usually sent when a debugger is attached. WebRTC should output some logs through that.

[davyloots]

Fixed! I can't believe I'm so stupid. Before copying the app to the server, I did a dotnet publish. Of course this doesn't copy over the native dll's. So I was simply missing Microsoft.MixedReality.WebRTC.Native.dll. Thanks again for your responses. I'm sorry to bother you with this. Maybe somebody in the future will make the same mistake and come across this post.

[djee-ms]

Hi @davyloots, no problem, glad it's fixed. It's unfortunate our logging is not better. Did you manage to use DebugView? How did you catch the issue? Just for future reference, and to see how we could improve things to catch this error more easily.

[djee-ms]

I logged #267 to improve DLL not found logging.

[davyloots]

Hi @davyloots, no problem, glad it's fixed. It's unfortunate our logging is not better. Did you manage to use DebugView? How did you catch the issue? Just for future reference, and to see how we could improve things to catch this error more easily.

DebugView didn't give me any logging for this app (not on my development machine either). I had already tried ProcMon earlier, which gives at least an idea on registry keys being read, and I did notice that on my dev pc it started reading a lot of registry keys related to audiodevices, which was not the case on the server, but I couldn't figure out why.

I only found out by accident when I started thinking about changing out the native dll's for another version, noticing there wasn't one to begin with. I've had missing native libraries before, but they've always resulted into uncaught exceptions. This one didn't, so improved error reporting would definitely help.

Thanks again!