hologram alignment issue in mixed reality captures

[jwawon]

Hi,

I have noticed there is an alignment issue between MRC and what I was looking through Hololens. When I see holograms with RemoteVideoPlayer in Unity, the positions of holograms are slightly offset from what it was supposed to be.

It was addressed in MRTK Github (https://github.com/microsoft/MixedRealityToolkit-Unity/issues/7380). And I found a solution (https://github.com/microsoft/MixedRealityToolkit-Unity/pull/6692). By using PV camera rendering, the alignment issue was solved in MRTK.

However, I couldn't find the way I can use the PV camera option in MixedReality-WebRTC. I am using MRTK and MR-WebRTC for Hololens user side, so I tried to set PV camera profile like this guide (https://microsoft.github.io/MixedRealityToolkit-Unity/Documentation/CameraSystem/WindowsMixedRealityCameraSettings.html#configuring-the-windows-mixed-reality-camera-settings-provider), but it didn't work for MixedReality-WebRTC. (I have attached a picture, the hand mesh wasn't aligned with my hand even though I set PV camera.)

Could you help me with this issue?

[fibann]

MR-WebRTC doesn't support PV camera rendering at the moment.

It should be reasonably easy to enable this, the relevant code is in WebRTC-UWP around external\webrtc-uwp-sdk\webrtc\xplatform\webrtc-apis\windows\wrapper\impl_webrtc_VideoCapturer.cpp. This uses what looks like an old version of MrcVideoEffectDefinition, it should probably be replaced by the version at https://github.com/microsoft/Windows-universal-samples/blob/master/Samples/HolographicMixedRealityCapture/cpp/MrcEffectDefinitions/MrcVideoEffectDefinition.h that supports the PV camera.

[kspark-scott]

I can confirm that it is in fact reasonably easy and basically involves what Filippo suggests: adding the 'PreferredHologramPerspective' property to MrcVideoEffectDefinition. We are using a lightly customized version of v1.0.3 with this change and it works well. We hard-coded the property value to 1 here. You still have to opt-in at the application level, so you can control whether the PV camera is used or not by making the opt-in conditional even when the property value is hard-coded to 1 -- so we had no need to add it to the configuration API.

[jwawon]

@fibann @kspark-scott Thank you for giving me a answer. However, I am a Unity developer and I am not good at C/C++ programming so I wasn't able to understand what you guys mentioned 100%. Can I help me in detail?

Did you mean to just replace impl_webrtc_VideoCapturer.cpp and impl_webrtc_VideoCapturer.h with MrcVideoEffectDefinition.cpp and MrcVideoEffectDefinition.h. After that, compile webrtc-uwp project for Unity libraries. Am I correct?

And could you let me know how to access to the 'PreferredHologramPerspective' property in Unity?

[kspark-scott]

@jwawon The change is simple in principle, but can be challenging because of the complexity of the build process. Have you built the dependencies yourself before? If you aren't comfortable with doing that then I'm not sure how to advise you -- this change may be out of reach.

You can't do a wholesale replacement because I believe the UWP samples project referenced above is a C++/CX project whereas the dependencies for MR-WebRTC are C++/WinRT, so the code is not compatible. Here is the diff for the change I made to this project so you will have a reference for a change that is known to work. The file names are relative to the submodule location external/webrtc-uwp-sdk/webrtc/xplatform/webrtc-apis.

pv-camera-diff.txt

There is also git submodule complexity here. The root MR-WebRTC project imports webrtc-uwp-sdk as a submodule which in turn imports many projects, including webrtc-apis -- which is where the files to be changed reside. So if you want to persist the customizations for yourself you must clone/fork and maintain two additional repos.

Finally, this change alone is not enough. The app must also opt-in for the PV camera, and then handle it when MRC is active -- which is why I just hard-code the new property to "enabled", and then use this opt-in step at the application level to decide if I really want it enabled. At the time I made this change for our own engine, I believe Unity did not yet support it. I recall seeing something since then that made me think it has since been added, but I don't use Unity myself so can't help you with this part of it.

[kspark-scott]

@fibann Would I be correct in assuming that this will all change once you complete the work to move away from webrtc-uwp-sdk? I presume WinRTC will have something equivalent to the MRC video effect definition, and am hopeful that it already uses a version that includes this property, but is there any chance the ability to make this customization will be lost?

[fibann]

Thanks for commenting above @kspark-scott. Indeed one needs to rebuild the dependencies to apply the fix above. There is a step-by-step guide for build the library, dependencies included, at https://microsoft.github.io/MixedReality-WebRTC/manual/building-windows.html, but the process assumes some of familiarity with C++ development.

Re WinRTC, as far as I can see it still uses the Windows::Media::Capture API for video capture, so porting the fix should be possible if not straightforward.

[jwawon]

@fibann Thank you so much for all your help. Unfortunately, I am not comfortable with the build process. That sounds challenging to me. I am going to study the build process first. I really appreciate it!

@kspark-scott Thank you for your comment!