C# UWP NuGet v1.0.2: System.Exception when specifying videoProfileKind in LocalVideoTrackSettings

[drejx]

EDIT: updating because I was a dingbat and it's actually videoProfileKind that causes the exception, not width/height.

Summary I'm getting a generic System.Exception when calling PeerConnection.AddLocalVideoTrackAsync(LocalVideoTrackSettings localVideoTrackSettings) where videoProfileKind is specified within the localVideoTrackSettings parameter.

Platform/Build

• C# UWP app
• Microsoft.MixedReality.WebRTC.UWP v1.0.2 NuGet package

Details Below is the code snippet I'm using:

PeerConnection.LocalVideoTrackSettings localVideoTrackSettings = new PeerConnection.LocalVideoTrackSettings()
{
    enableMrc = false,
    framerate = 30.0,
    videoDevice = captureDevice,
    width = 1920,
    height = 1080,
    videoProfileKind = PeerConnection.VideoProfileKind.VideoConferencing
};

await Peer.AddLocalVideoTrackAsync(localVideoTrackSettings); ===> System.Exception thrown

• If I don't specify videoProfileKind or set it to PeerConnection.VideoProfileKind.Unspecified, the above works and I receive frame events via Peer.I420LocalVideoFrameReady with a frame resolution of 1920x1080.
• I can confirm that the above framerate, width and height are available for my camera as I enumerate all its capabilities (the above hardcoded values are for simplicity).

Exception Info

"Exception of type 'System.Exception' was thrown."

-   Data    {System.Collections.ListDictionaryInternal} System.Collections.IDictionary {System.Collections.ListDictionaryInternal}
        Count   0   int

There was no inner exception.

Stack Trace

   at Microsoft.MixedReality.WebRTC.Interop.Utils.ThrowOnErrorCode(UInt32 res)
   at Microsoft.MixedReality.WebRTC.PeerConnection.<>c__DisplayClass74_0.<AddLocalVideoTrackAsync>b__0()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)

Even if there is something bad I was passing in, which it doesn't seem like, it would be useful to have some more details on what the error could be. Luckily this is not a blocker as I should be able to specify the desired widthxheight I need.

Cheers, Andrej

[djee-ms]

Yes I agree with you in principle on error details. Unfortunately when WebRTC fails to create some object often it does not return any info, just a nullptr, so there is not a lot we know about the failure. And for video capture init, there are multiple calls like this. Also because of the interop we return just error codes, which is poorer than an actual message.

In this case I still suspect the video format is mismatching, since it works in the default case. Since I see you specify both the profile kind AND the resolution/framerate, are you 100% sure that the webcam supports that format (resolution + framerate) when that particular profile kind is selected? Can you please try to specify either the profile kind OR the resolution/framerate, and see if that works?

[djee-ms]

Try to see if 83119150dbce530a281c0be12bb231a8fadbc7f2 helps also, I added some logging that should show up in Debug builds in the VS Output window (or any other debugger).

[drejx]

Hi,

I did a bit more testing and created a blank UWP demo app and it still crashes even if only profile kind is set. Here is my code snippet for the UWP MainPage OnLoaded event:

        private async void OnLoaded(object sender, RoutedEventArgs e)
        {
            // Request access to microphone and camera
            var settings = new MediaCaptureInitializationSettings();
            settings.StreamingCaptureMode = StreamingCaptureMode.AudioAndVideo;
            var capture = new MediaCapture();
            await capture.InitializeAsync(settings);

            // Retrieve a list of available video capture devices (webcams).
            List<VideoCaptureDevice> deviceList =
                await PeerConnection.GetVideoCaptureDevicesAsync();

            if (deviceList.Count == 0)
            {
                Log("ERROR: no video capture devices found.");
                return;
            }

            // Print out the webcam devices found.
            foreach (var device in deviceList)
            {
                // This message will show up in the Output window of Visual Studio
                Log($"Webcam: name={device.name}, id={device.id})");
            }

            _peerConnection = new PeerConnection();

            var config = new PeerConnectionConfiguration
            {
                IceServers = new List<IceServer> {
                    new IceServer{ Urls = { "stun:stun.l.google.com:19302" } }
                }
            };

            await _peerConnection.InitializeAsync(config);
            Log("Peer connection initialized successfully.");

            _peerConnection.I420LocalVideoFrameReady += OnI420LocalVideoFrameReady;

            // ##################### AREA OF INTEREST ###################
            Log("Adding local audio and video tracks");
            PeerConnection.LocalVideoTrackSettings localVideoTrackSettings = new PeerConnection.LocalVideoTrackSettings()
            {
                enableMrc = false,
                videoDevice = deviceList[0],
                videoProfileKind = PeerConnection.VideoProfileKind.VideoRecording
            };

            await _peerConnection.AddLocalVideoTrackAsync(localVideoTrackSettings);
            await _peerConnection.AddLocalAudioTrackAsync();
        }

I have 1 webcam and my output is:

Webcam: name=HD Pro Webcam C920, id=\\?\USB#VID_046D&PID_082D&MI_00#6&2053ea0a&2&0000#{e5323777-f976-4f5b-9b55-b94699c46e44}\global)
Peer connection initialized successfully.
Adding local audio and video tracks

Testing it further if I just set width, height and framerate, but not a video profile there is no exception:

PeerConnection.LocalVideoTrackSettings localVideoTrackSettings = new PeerConnection.LocalVideoTrackSettings()
{
    enableMrc = false,
    videoDevice = deviceList[0],
    framerate = 30.0,  <<==== works if setting framerate, width, height
    width = 640,
    height = 480,
    //videoProfileKind = PeerConnection.VideoProfileKind.VideoRecording
};

And just to see what happens when I set a ridiculous value, like framerate=1000.0 then I actually do get the same System.Exception as with passing in video profile kind.

Currently I'm using the NuGet package and haven't built from sources so I can't try 8311975 yet. I think at this point though it's probably worthwhile for me to get jamming on building my own DLLs :)

Cheers, Andrej

p.s. I'd be happy to provide my sample UWP app VS solution if it helps.

[djee-ms]

Yes that's weird, the easiest way would be to at least build the MixedReality-WebRTC solution, that will get you a bit more information and it is relatively easy. Compiling the Google code is more difficult, but will get you all debug info. I am not quite sure yet why the detection algorithm is failing to find a matching format.

[drejx]

Quite strange indeed. I'm going to dive into the docs for building from sources today.

One thing which would be great is to be able to toggle on verbose logging. I've built the webrtc-uwp-sdk from sources and in debug I'm able to get low level verbose debug messages, for example:

(basicportallocator.cc:922): Discarding candidate because it doesn't match filter.
(basicportallocator.cc:1347): AllocationSequence: UDPPort will be handling the STUN candidate generation.
...
(peerconnection.cc:1231): Adding video transceiver in response to a call to AddTrack.
(peerconnection.cc:1231): Adding audio transceiver in response to a call to AddTrack.

I'm guessing this is because webrtc-uwp-sdk builds the webrtc.lib (707MB) from scratch.

[drejx]

I built the debug DLL from sources and it looks like the error result is because the webcam doesn't support video profiles. The relevant log snippet:

(impl_webrtc_videocapturer.cpp:1172): Using local detection for orientation source
(impl_webrtc_videocapturer.cpp:1232): Init called for device \\?\USB#VID_046D&PID_082D&MI_00#6&2053ea0a&2&0000#{e5323777-f976-4f5b-9b55-b94699c46e44}\global
(impl_webrtc_videocapturer.cpp:1303): webrtc::VideoCapturer::init: Video capture device \\?\USB#VID_046D&PID_082D&MI_00#6&2053ea0a&2&0000#{e5323777-f976-4f5b-9b55-b94699c46e44}\global 
    does not support video profiles, cannot assign requested video profile  or kind #1.
(interop_api.cpp:178): Failed to find a local video capture device matching the capture format constraints. None of the 1 devices tested had a compatible capture format.
(interop_api.cpp:727): Failed to open video capture device.
Exception thrown: 'System.Exception' in Microsoft.MixedReality.WebRTC.dll
An exception of type 'System.Exception' occurred in Microsoft.MixedReality.WebRTC.dll but was not handled in user code
Object not found.

As a solution, if the device doesn't support video profiles, how about logging a warning and then continuing with the standard path as if no video profile was specified?

Cheers, Andrej

[djee-ms]

Ok right I didn't think about that one, sorry. I assumed you knew video profiles were available.

I already discussed a similar design question, and the rationale is that MixedReality-WebRTC is a library for developers, not an end user product. So I took the decision to always have hard errors when the implementation cannot honor the constraints passed by the caller, instead of trying a fallback alternative. This allows devs to implement any logic they want on top of this, like a fallback mechanism as you suggested, or anything else. But if we start to use hard-coded fallback mechanisms here and there, although convenient in some usage scenarios, they force a particular behavior, and devs using the library and expecting a video profile to be selected maybe want to know if something went wrong instead of getting back a video feed which is not using the profile they selected. The same approach is true for capture format : although we could have a fallback when the resolution specified is not found, this fallback is easily implemented on top of the existing library, while a developer targeting a very specific capture format will be notified of any failure instead of getting another capture format they did not request.

I agree though that we should have a better error reporting such that you don't have to build the core library from scratch, and can easily identify why an API call failed and how to make it work.

[stephenatwork]

Would it be convenient for developers to have a "bool bestEffort" (defaults to false) in the options struct? How much boilerplate is needed for the dev to emulate this?

[drejx]

+1 on having some better error capture/warnings for unexpected cases such as this. I totally understand the design approach of not doing a fallback procedure by default, but then the dev needs to know if something went wrong. A warning log message is great and/or returning an error struct that is similar to an exception:

struct
{
   bool Success;
   ErrorCode ErroCode;   // optional enum
   string FailureReason;
}

Although at that point I guess it would be simpler to just toss an exception :)

I'd like to point out that many devs using this library are likely not seasoned and don't (yet) know all the details of WebRTC/SDP/Video Profiles/... so any help they can get figuring out what went wrong is most welcome.

Cheers, Andrej