CameraStarterKit crash when recording and minimizing the application (the sequel)

[AdamSzofran]

I've got a new variation of this issue from back in 2017. The repro is the same but the result is slightly different for me.

Repro:

1. Build CameraStarterKit sample for x64 and run.
2. Start a video recording.
3. Minimize the app.

Result:

System.Runtime.InteropServices.COMException (0xC00D3E82): A media source cannot go from the stopped state to the paused state. A media source cannot go from the stopped state to the paused state.

Yes, the exception error message appears twice!?!

Exception call stack:

System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task task)
System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task)
System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.TaskAwaiter.GetResult()
CameraStarterKit.exe!CameraStarterKit.MainPage.StopPreviewAsync() Line 351
[Resuming Async Method]
System.Private.CoreLib.ni.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)
System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.RunWithDefaultContext()
System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.OutputAsyncCausalityEvents.AnonymousMethod__0()
System.Private.CoreLib.ni.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__0()
System.Private.CoreLib.ni.dll!System.Threading.Tasks.SynchronizationContextAwaitTaskContinuation.GetActionLogDelegate.AnonymousMethod__0()
System.Runtime.WindowsRuntime.dll!System.Threading.WinRTSynchronizationContext.Invoker.InvokeCore()
System.Private.CoreLib.ni.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)
System.Runtime.WindowsRuntime.dll!System.Threading.WinRTSynchronizationContext.Invoker.Invoke()
[Async Call Stack]
[Async] CameraStarterKit.exe!CameraStarterKit.MainPage.CleanupCameraAsync() Line 468
[Async] CameraStarterKit.exe!CameraStarterKit.MainPage.SetUpBasedOnStateAsync.AnonymousMethod__0() Line 526
[Async] CameraStarterKit.exe!CameraStarterKit.MainPage.SetUpBasedOnStateAsync() Line 533
[Async] CameraStarterKit.exe!CameraStarterKit.MainPage.Window_VisibilityChanged(object sender, Windows.UI.Core.VisibilityChangedEventArgs args) Line 131

[AdamSzofran]

What I'm really try to do is to get my UWP app to continue recording video from the camera even when the app is minimized. Since my app is used to record sporting events, it's critical that the recording not be interrupted if the app is accidentally minimized.

I've experimented with ExtendedExecutionSession to no avail. That's able to keep my app from being suspended but MediaCapture still sends me the CameraStreamStateChanged message saying the camera is Shutdown which kills the recording.

The sequence of events goes roughly like this:

1. Call ExtendedExecutionSession.RequestExtensionAsync() result is Allowed.
2. Start recording video.
3. Minimize the app.
4. Receive AudioStateMonitor.SoundLevelChanged notifying that the volume is now Muted.
5. Receive Window.VisibilityChanged notifying that the window is no longer visible.
6. Receive Application.EnteredBackground.
7. Receive MediaCapture.CameraStreamStateChanged notifying that the camera is now Shutdown.

At this point the recording has already been terminated by Windows or MediaCapture.

Is there a way to keep the recording going without interruption?

[BillMcCrary]

I'm in the same boat as @AdamSzofran - mine is a simple UWP desktop app. Experiencing media capture stoppage with extended execution bits. In my case, the rest of the app seems to work (i.e. UI updates, timers) but the media capture is killing the preview and/or recording stream(s).

This article (from a long time ago) mentions some creator's update broke this functionality.

https://social.msdn.microsoft.com/Forums/en-US/0e2937a1-13c2-45fa-8008-36e7b227b124/uwp-auto-continue-recording-media-capture-after-sleep-mode?forum=wpdevelop

Please help!

[oldnewthing]

Hi, Adam. This is not an issue with the sample, but rather a question about the platform. You can open a support case with our Developer Support team who can further help with the specific problem you are experiencing. Please use these steps below for further help:  

1. Visit http://aka.ms/storesupport and sign-in with your Microsoft account
2. Under the “App Development” section choose "Windows 10 universal app development", then choose the appropriate problem type and category to get engaged through a formal support channel.

[Zofware]

Hi Raymond,

I was able to fix this in my app by requesting the backgroundMediaRecording capability. It took 2 weeks to get approval but now I no longer have the issue and recordings continue even after my app is minimized. There may, in fact, be a problem with the platform lurking here but it might still be worth mentioning in the sample's readme that a restricted capability is required to avoid bad behavior when minimizing the app.

Thanks, Adam

[shinsetsu]

[image: 726306457.jpg] Wut are you recording Sports games from? Am I seeing some ting thats a windows store app? Are tings hiding in the fascade again?

With those rappers and singers? Mostly just commenting to MAKE some NOISe.... say... aHOI Zach... lol j.k

2 weeks toget approval? is this a store ap? Is there a

JenSteele says :aSmooothe Gell "Pen" is worth a breif outting where the Duo's precense != null to the wash forest for some Bird watchn around FeberMarchJilyAugsta ‾_(ツ)_/‾ lol
TG Waldrop.

he Awl, the meaning of the “the shruggie” is always two AmericanAirlineTickets Both Directions to the nest...... of inspiring rad stuff to aid in "Us Gettings Ours"

[shinsetsu]

[shinsetsu]

[BillMcCrary]

@Zofware thanks for that - it showed me some new behavior. After adding that rescap, here is what I see:

• Preview only – no recording – preview stream stops ~9-10 seconds after we minimize

• Recording in progress – while minimized, everything continues to work properly – as soon as the recording stops, the preview stream stops

I have a ticket open with Microsoft (enterprise support) so if we get to the bottom of our issue I will try and reply here.

[Zofware]

Bill,

Yeah, same here. I haven't figured out how to keep the preview going while minimized so I just restart it after the app is restored from minimized. Like you, it only stops if I'm not also recording.

Thanks, Adam

[shinsetsu]

[image: 76183116_2712713522118200_1632859867729362944_o.jpg]

On Fri, Jan 31, 2020 at 12:09 PM Adam Zofware notifications@github.com wrote:

Bill,

Yeah, same here. I haven't figured out how to keep the preview going while minimized so I just restart it after the app is restored from minimized. Like you, it only stops if I'm not also recording.

Thanks, Adam

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/microsoft/Windows-universal-samples/issues/1144?email_source=notifications&email_token=ABOOZHZZIIGKX4BT7OOCTITRARSN7A5CNFSM4I52L6DKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKPP64Y#issuecomment-580845427, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOOZHY2JW57LZWII2VU5CDRARSN7ANCNFSM4I52L6DA .

[BillMcCrary]

I wanted to follow up here after I had some discussions with Microsoft. tl;dr cameras via media foundation (what UWP is using under the hood) do NOT play nicely with managed code... YMMV but from the horse's mouth we should NOT be trying to do long running media HW access with managed code.

I will say I am having some luck with Telerik's web cam control (good old winforms for me right now). I can minimize, go to background etc and the preview and recordings seem to continue to work fine. Time will tell how stable their control is.

However the proper solution is to roll our sleeves up and implement this as a C++/Win32 native application to get the ideal HW level control over our cameras.

So personally for now I am going to get my winforms app shipped ASAP then immediately revisit it and implement a bare metal solution - should be fun!

[Zofware]

@BillMcCrary,

Thanks for following up. Wow that's some troubling feedback from Microsoft. What's your impression of what Microsoft means by "long running"? Days, hours, or minutes? The longest I go is about 2 hours. I sure hope that's ok because I've got a lot invested in UWP so far. I use C# for high-level MediaCapture control with some C++/WinRT for low-level IBasicVideoEffects.

Thanks, Adam

[BillMcCrary]

@Zofware sorry for a late reply - 'long running' is impossible for them to define. The blanket statement as I take it is 'zero guarantee of stability' of anything in managed code w/ media foundation. If you are actively recording to disk, I don't think this issue applies since some low level hardware handles are keeping the OS from killing the UWP app. But as soon as you stop recording anything can kill the app.

I threw in the YMMV bit because it MAY work out just fine, we're having a better experience on winforms (full .net framework) but still having some issues which are suspected to be related to this MF limitation w/ managed code.

For the record, we're proceeding with a C++ implementation and it's not so bad. For anyone finding this thread in the future, I wish I had seen this article before delving into the media foundation source docs: http://www.codeofview.com/2017/08/17/sudoku-solver-ar-part-5-camera-capture-windows/ - I don't think this person ever got around to finishing this tutorial but the media foundation bits are a good start. From there - I suggest going to the win7 github examples @ https://github.com/microsoft/Windows-classic-samples/tree/master/Samples/Win7Samples/multimedia/mediafoundation.

That said the rest of our app will also need to be written in C++ so definitely a time/complexity sink but I guess that is the cost for what we're trying to do...

Good luck!

[LPBourret]

Hi, @BillMcCrary @Zofware can you elaborate on why using MediaCapture WinRT APIs is not addressing your issue here (you can use these WinRT APIs from your Win32 app)? And from an app package standpoint (UWP APPX/MSIX) declaring the backgroundMediaRecording restricted capability did not solve your issue?

[BillMcCrary]

@LPBourret - Media Foundation API from C++ is working just fine for my application now. Our initial blocker was trying to use them from UWP C# apps. After multiple enterprise support calls and emails directly with Sr. Engineers from Microsoft's platform team, their answer was pretty unanimous that Media Foundation simply is not meant to run as managed code - they didn't go into too much detail on exactly why but it was more of a 'trust us' response.

Specifically, backgroundMediaRecording works if you start a recording then minimize, but if you are not actively recording (just running a preview stream for say frame inference) when you minimize then the UWP frameworks fail and kill the media foundation connection.

[LPBourret]

I see @BillMcCrary , have you tried using MediaFrameReader? I wrote a couple of apps that run in the background and process frames (i.e. using CV) from the camera (i.e. C# UWP, C# .Net Core console apps, Win32 C++from console) using MediaCapture and MediaFrameReader. Some examples are in the AI Skills github repo: https://github.com/microsoft/AISkillsForWindows for example look at the this .Net Core C# sample that runs SkeletonDetector from a console: https://github.com/microsoft/AISkillsForWindows/tree/master/samples/SkeletalDetector/cs/SkeletalDetectorSample_NetCore3 As a dev in the Windows camera pipeline team I don't think there is any issue in using managed code and MediaFroundation. From a UWP standpoint, there are some capabilities you can use to request running in the background such as extendedExecution

[BillMcCrary]

@LPBourret - no I never came across that, wish I had. I trust you if you say it will work, but at the time I was working with MS support this never came up as an option... This is the first time I'm seeing that repo too. Where would/should I have seen this stuff initially during research of the issue?

My chain of events was:

• Build our 1st prototype in UWP C# with media capture bits
• Find out it fails in the scenario we all mentioned above
• Open an enterprise support ticket (and found this github thread) with MS developer support to see what I was doing wrong
• I then get sent down the path of Media Foundation bare metal integration - my use case is "constantly running app, inference against frames + simultaneously recording video"
• Now I have a working C++ app but I would have preferred to stay in .net land due to our in-house talent being primarily .net devs

Is the issue that we're talking specifically WinRT? Our app has to run on a Windows 10 Desktop PC. Does this stuff you mentioned also work on plain old Win10 desktop machines?

[LPBourret]

@BillMcCrary WinRT APIs such as MediaCapture are accessible from a Desktop app (i.e. a C# WPF app, a console app, etc.) in Windows 10. I think the issue you faced were related to the lifetime of a UWP app (suspended upon minimizing) and not directly related to the WinRT APIs. This issue can also be circumvented by declaring special capabilities, or avoiding to be minimized altogether by replacing your title bar, etc. I think what's cool with the recent advances in developing using WinRT is you can mesh together C++/WinRT components that bridge with lower level code such as C++ frameworks and C# projects that consume them (such as your UI and app logic).

The AI Skills framework is something relatively new started last year, it was not evidently tied to the Windows sample github and we provided samples that tie with the WinRT Camera APIs to streamline the execution of AI solution using frames from the camera (btw the pre-built solutions there may be interesting for you if you do inference).

Sorry I stumbled upon this thread just today. We have another GitHub dedicated to to the Windows camera pipeline with other extra samples where you can file issues whenever you need help as well: https://github.com/Microsoft/Windows-Camera. I'll give the feedback to the team to take a look frequently at issues on the main Windows sample GitHub for anything related to Camera.

[BillMcCrary]

@LPBourret thanks for this. Now I'm torn because I'm almost done with our service in C++ using media foundation and its working well so far. I've also passed this thread to the two Microsoft engineers I worked with previously as both were fairly adamant about not using Media Foundation on managed .net...

I tend to trust MS folks' advice, but I'm getting conflicting advice here vs the two engineers I talked to - I'm sure you see where I'm coming from re. being torn about what to do for the final version... Would it be ok if I possibly added you directly to the email thread I have going with those guys? Maybe if we're all in the same conversation we can agree on the best direction?

With the rise in production ML and specifically ML for various vision inspection systems, I would love to see Microsoft come out with some definitive direction for various tech stacks for the supported platforms to accomplish this. Your repos above seem like good sources, I would think those would be better linked/marketed for example from the ONNX runtime home page/github, with some cross references or mentions on media foundation docs that Windows-Camera is another options for input.

Is there any long-term support reasons to pick one direction over the other? I.e. will media foundation (for camera access at least) be abandoned in favor the Windows-Camera pipeline work you guys are doing?

BTW - re. the UWP special capabilities - absolutely nothing I found worked to keep a preview stream alive while minimized and not recording. I wish that something did work because our UWP app was clean and easy to build quickly. I would LOVE to see a MS sample of exactly what we were trying to do...

[LPBourret]

@BillMcCrary Yes please feel free to add me to any thread concerning questions around the use of camera APIs in Windows :)

I would say that MediaCapture APIs are the preferred way to record and consume video from a camera since you can consume WinRT APIs from any app target now (Win32, ..NetCore, UWP, etc.). That said MediaFoundation is still supported, as well a DShow.

If you already have a working app that uses MF this is great, no need to rewrite it using MediaCapture. I do think the WinRT APIs demand less code to reach a workable solution, but all roads lead to Rome.

The Windows AI platform (Windows ML, AI Skills, DirectML, ONNX Runtime, Azure CustomVision services, etc.) is a good starting point if you are interested in using the Windows baked-in capabilities to run AI workload in your app: https://docs.microsoft.com/en-us/windows/ai/ As you may know, Windows ships with an inferencing engine called Windows ML, based on ONNX Runtime and DirectML which exposes hardware acceleration to run ML model operator graph.

[BillMcCrary]

@LPBourret I think we're good here. I'll stick with my current C++ integration. It's working well, very performant, and I have some degree of assurance that media foundation will be around for a good while.

For the record we also integrated with ONNX Runtime which is just a great library so far. Our app is using media foundation to grab frames, run them through ORT for scoring, then start/stop recording based on that score. I do plan to check out the windows AI repo this week.

Thanks again!