can't capture photos at 1280x720 resolution after using SpatialUnderstanding

[ForrestTrepte]

I am having a problem when I include SpatialUnderstanding in my application. This problem occurs whenever SpatialUnderstanding_Init is called, even if I am not actually using any SpatialUnderstanding features.

When I later call PhotoCapture.StartPhotoModeAsync, the following error is printed to debug output: Failed to set capture resolution (hr = 0xC00D36B4) (Filename: C:/buildslave/unity/build/Runtime/VR/HoloLens/WebCam/PhotoCapture.cpp Line: 1180)

I'll see if I can create a repro case and update this issue with more info, but I thought first I'd post the basic symptoms. Please let me know if this is a known issue or if there is a reason that SpatialUnderstanding and PhotoCapture are incompatible.

[ForrestTrepte]

REPRO CASE: 1) Open HoloToolkit-Unity in Unity 5.5.1f1. 2) Create new scene. 3) Create a game object with the atached PhotoCaptureExample.cs script. (This script is based on https://docs.unity3d.com/ScriptReference/VR.WSA.WebCam.PhotoCapture.html, but the resolution is changed to 1280x720.) 4) Add the SpatialMapping prefab to your scene. 5) Add the SpatialUnderstanding prefab to your scene. In the Spatial Understanding component, uncheck the Auto Begin Scanning property. 6) In player settings, publishing, check the WebCam checkbox. 7) Build and run the project on HoloLens.

result: The following error occurs and then the application unexpectedly exists: Failed to set capture resolution (hr = 0xC00D36B4) (Filename: C:/buildslave/unity/build/Runtime/VR/HoloLens/WebCam/PhotoCapture.cpp Line: 1180)

expected: PhotoCaptureExample script succeeds and prints "OnCapturedPhotoToMemory succeeded". Note that you can currently get the expected result either by changing the resolution to 2048x1152 or by skipping over the call to SpatialUnderstanding_Init.

PhotoCaptureExample.zip

[StephenHodgson]

Thanks for the report @ForrestTrepte.

Instead of attaching a zip file could you share the project here on GitHub? Thanks.

[ForrestTrepte]

You can access the repro case from my fork at https://github.com/ForrestTrepte/HoloToolkit-Unity/tree/issues/520_SpUndPhCap_FailedMinRepro. Use the scene Assets/Issue520.unity. Is that a good way to share the project? (I'm a new to GitHub and I don't know the best way to do things yet.)

By the way, the zip file just contains a C# script. (I couldn't figure out how to attach a .cs file to this issue.)

[StephenHodgson]

I'd say so. For instance, my work policies don't allow me to download zip files.

[StephenHodgson]

@mrbobbybobberson any ideas on what might be causing this?

[mrbobbybobberson]

Unfortunately, no insight from me on this. I'm not very familiar with the internals of the SpatialUnderstanding native dll, even though I got a little familiar with the C# spatial understanding code to make it work in editor.

[StephenHodgson]

@ForrestTrepte I'd submit a issue over on the main HoloToolkit repository. This might be something in the source for the SpatialUnderstanding library that's causing this.

[ForrestTrepte]

Entered as issue 73 in the main HoloToolkit respository.

[ForrestTrepte]

Perhaps @jevertt would have ideas about what would cause this.

[jevertt]

No, sorry - I don't have any guesses. I was thinking that you needed to also enable the microphone capability to get photo/video capture - it's worth a test, but I'm not certain of that.

Do you happen to know if spatial mapping also causes this same issue?

[ForrestTrepte]

No, spatial mapping doesn't cause the issue. If I comment out the call to SpatialUnderstanding_Init, spatial mapping and photos work fine.

[jevertt]

No, sorry - I'm not sure. The photo capture probably requests exclusive access of the camera. I wouldn't expect that anything in the C++ understanding code would be grabbing the camera, but sure sounds like that is happening. It could be some left over code - I can't think of any reason it would be using it.

[ForrestTrepte]

Thanks for taking a look, Jeff. That makes sense. I won't get a chance to come back to this problem for a while, but when I do I would see if I can step through the native code of SpatialUnderstanding_Init to see if I can find what affects camera capture. Or, with luck, perhaps someone else out there will discover the source of this issue.

[cstensonpv]

Hey ! I'm currently running into a similar problem myself. I have a unity project which will use both the RGB camera and spatial understanding. It works fine in the editor but when deploying to the hololens only one of them work. No error messages but I never get the "camera ready" photocapture object. And thus can't capture any photos. Any updates ?

[ForrestTrepte]

No updates. I have been wanting to investigate, but haven't had the chance yet. My plan would be to build the spatial understanding dll, step into SpatialUnderstanding_Init, and step through (or skip over) its internals to see if I can figure out what interferes with the camera.

[ForrestTrepte]

I took a look and this seems to occur because SpatialUnderstanding changes the floating point rounding mode.

SetFloatControlDownward in SpatialUnderstanding calls _control87 to set the floating point rounding mode to down. This seems to be done so that FTOL, which uses the fistp instruction, will round floats down instead of to the nearest int. The rounding mode isn't restored to its previous value, and Unity photo capture seems to fail when rounding is left in the down mode.

I'm thinking the fix would be to modify SpatialUnderstanding to either: 1) Avoid changing the rounding mode. or 2) Change the rounding mode on entry of every API call to SpatialUnderstanding and restore it to its previous value on every return.

Since I'm not sure of all the dependencies on rounding mode that SpatialUnderstanding may possibly have, I'm learning towards option 2 unless someone out there has alternative advice.

[ForrestTrepte]

I'm working on implementing #2. (This may take a little while if I only get the chance to work on it a little bit each day.)