Windows UWP Driver not complete - Mapping Depth to RGB

[reneschulte]

We are using the UWP Windows.Media.Capture APIs to access the RealSense D415. We need to map the depth data onto the color for rendering a point cloud.

We are doing a manual registration / mapping based on camera intrinsics and extrinsics. We can get the camera intrinsics from the UWP APIs but the coordinate mapping transform (extrinsics) are not provided. This call would normally provide a 4x4 matrix for doing the transformation and it works fine with Kinect v2 but for the RealSense this returns a null value.

var matDepthToRgb = depthSource.Info.CoordinateSystem.TryGetTransformTo(colorSource.Info.CoordinateSystem)

Got the latest driver and Windows 10 17134 but still the CoordinateSystem provided by any source regardless if marked as depth or color is NULL.

depthSource.Info.CoordinateSystem == null colorSource.Info.CoordinateSystem == null

I know there is a point cloud sample in the native RealSense SDK that is using

environment::get_instance().get_extrinsics_graph()

But we need to use the UWP Windows.Media.Capture APIs and don't see those extrinsics values exposed. Not via the coordinate system transformation nor another API. We don't use the CoordinateMapper as we need to run the mapping in a custom shader and really just need the above mentioned matrix. This is a bug in the driver that should surface those parameters to Windows.Media.Capture but it's broken since months.

I have the latest Windows 10 UWP driver and D415 firmware installed as of today. Winver 1803 / 17134 / RS4

[dorodnic]

If using librealsense is an option, we have sensor-control example that among other things offers extrinsics. @RealSense-Customer-Engineering - can we check the UWP driver and submit a bug to the relevant team if there is no fix?

[reneschulte]

Thanks for the quick response Sergey! 👍 Unfortunately we can't use the native librealsense as our app has to be an UWP app.

[MartyG-RealSense]

There has been a long discussion about the 400 Series UWP driver and the possibility of mapping depth to RGB on the Intel Support site. There has not been a good outcome so far - according to support member Jesus G (the most recent reply on 30 October 2018), work between Intel and Microsoft about this is ongoing.

https://communities.intel.com/message/539599#539599

Another user has had problems with trying to create a RealSense app in Unity that can communicate with the Microsoft HoloLens headset via UWP. During the research of a solution for this case, the existence of an open-source Microsoft mixed-reality toolkit came to light.

https://github.com/Microsoft/MixedRealityToolkit

There is also a version of the Toolkit for the Unity engine. So perhaps as a last resort if the UWP discussions between Intel and Microsoft do not provide a solution soon, the Unity version of the Microsoft toolkit could somehow be made to interact with the RealSense Unity wrapper that handles camera data processing.

https://github.com/Microsoft/MixedRealityToolkit-Unity

[reneschulte]

Yes, that is my discussion on the Intel forums but since there's no outcome since months I was trying another path here.

The MRTK (Mixed Reality Toolkit) is not related at all to this issue. The MRTK is a toolkit for Unity development targeting the HoloLens. How would that at all fix a driver issue or be a workaround...

We need a working driver that is what is needed and it seems not to be open source, so no way to fix it ourselves. :(

[MartyG-RealSense]

Mixed Reality Toolkit is not just for Unity. There are two separate versions, non-Unity (the one linked to) and Unity. The intention of providing the link was to see if there might be a means to adapt the open source code to provide your application with a UWP connection by a method other than the RealSense UWP driver.

I appreciate that the suggestion wasn't entirely what you were looking for. I presented it as an alternate option to think on, given that your problem has not been able to be solved with the RealSense UWP driver so far and there is no date for when a driver-based solution may be forthcoming.

[reneschulte]

The non-Unity MRTK is mainly companion apps and tools. I know it quite well and it's not helping us here. Appreciate you are taking the efforts to post Marty and didn't want to sound rude. 🙌

[MartyG-RealSense]

It's ok, I understand your frustrations.

Could you give some information please about why using UWP is so important to your project? When UWP first came out, its main advantage was that UWP apps would work on Windows 10, Xbox and Windows Phone. Now that Windows Phone is dead, the advantages gained from using it instead of developing a Windows-only app may be limited.

[reneschulte]

Our app needs to run on Windows 10: Desktop, HoloLens and XBox, so yes UWP. Also the universal layer with an unified API that Windows.Media.Capture offers is a big plus (if the driver is implemented fully).

[dorodnic]

I was thinking about this problem... I believe Media Capture API allows the user to read from camera XU (need to look up how, couldn't find it online, but I think I saw someone do it). If this is true, one could send hardware command like in ds5-device.cpp:271-275 and then parse the resulting table similar to how its done in librealsense.

[dorodnic]

So, expanding on my previous post -

• From MediaCapture object (of the Depth device), you can get VideoDeviceController object that exposes GetDeviceProperty and SetDeviceProperty.
• The two functions get a string param of form: "{GUID} number". For our purposes {GUID} should be string representation of depth_xu and number should be the value of DS5_HWMONITOR
• Get returns an object castable to byte[] and set gets a byte[] as its second parameter.
• The overall flow would look like command_transfer_over_xu::send_receive method.
• Preparing the command would look like ds5_device::get_raw_calibration_table called with rgb_calibration_id.
• Parsing the result would look like get_color_stream_extrinsic function.

This certainly should not be the official solution for this problem, but I want to help and I know librealsense much better than Media Capture. Let me know if this direction can be useful to you, and I can assist with more details.

[MartyG-RealSense]

Thanks so much for your method, Dorodnic. I've posted a link to it on the Intel Support forum and distributed it to the past cases in 2018 who had UWP trouble with the 400 Series cameras.

[ashikns]

I tried this in a C# sample. Short answer: it didn't work. It seems that the UWP driver simply does not provide access to the data. When trying to access it an exception is thrown saying "The property set does not exist on the device". Here is the code that I tried: https://gist.github.com/ashikns/21b93ebe94454fd1d85ed33415fe21e2. I also discovered another guy on github who has done the same but in a much more comprehensive manner: https://github.com/jubransh/SynchronicMediaCapture. The sample from this github link doesn't run without some fixes, but the gist of it is that even if you get it to run it'll still throw the same error as I mentioned before.

All experiments were done on a Intel D415 camera with firmware 5.10.6, OS is Windows 10 1809.

[dorodnic]

Hi @ashikns I'm sorry it didn't work for you. With the default UWP driver I get the same "The property does not exist" error. However, with D400 Series UWP Driver for Windows 10 installed I do manage to get data back. (The only change from the GIST is var selectedGroup = groups.First(x => x.DisplayName.Contains("Depth")); instead of groups.First();)

@teichgraf - could you please clarify if using the Intel UWP Driver is an option for you?

[ashikns]

@dorodnic Confirming that the updated usb driver works 👍 Thanks for your help

[reneschulte]

Thank you so much @dorodnic and @ashikns 👏

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment] Hi @teichgraf,

[Skavesen]

does anyone have an example of a camera working with a uwp app?

[MartyG-RealSense]

Hi @Skavesen An app called Camera Stream Correlation has previously been suggested for use with RealSense, as described at https://github.com/IntelRealSense/librealsense/issues/2722

RealSense UWP support was not used by most RealSense users however and updates of the RealSense UWP driver have ceased.

[Skavesen]

And if I need a UWP app, because there is a RealSense library, but I get an error during initialization. Any advice on fixing it? Windows 10 19044, camera d415, sdk 2.0 from github (downloaded today), camera driver 6.1.160.22, usb 3.0. Everything is according to the rules. But there is an error during initialization var pipeline = new Pipeline(); System.TypeInitializationException: 'The type initializer for 'Intel.RealSense.Context' threw an exception.' PlatformNotSupportedException: Custom marshalers are not supported

[MartyG-RealSense]

@Skavesen The RealSense UWP driver is not required to use a RealSense 400 Series camera with Windows and is only needed in very rare circumstances where the camera needs to be made compatible with a UWP-supporting application such as the Microsoft Hololens headset.

If you do need to interface with a UWP application then the PlatformNotSupportedException: Custom marshalers are not supported error message has been reported previously at https://github.com/IntelRealSense/librealsense/issues/5774

There is not any information available regarding a solution and my recommendation would be to not attempt to use a RealSense camera with UWP applications.