Open emaschino opened 10 months ago
As a side question, how are you actually locating the camera pose in OpenXR? Through the scene graph bridge? It's so poorly documented, what sequence of calls did you perform to make it give you the position in OpenXR space? I do sucessfully have the GUID and Mixed Reality Spatial pose from the camera frame...
Yes, with XR_MSFT_spatial_graph_bridge extension. You get the dynamic node ID of the PV camera from the calibrated transform retrieved from the MediaReferenceFrame properties (lookup on MFSampleExtension_CameraExtrinsics). You then create a reference space for this dynamic node (of type XR_SPATIAL_GRAPH_NODE_TYPE_DYNAMIC_MSFT). This can be later located with xrLocateSpace, thus giving you the location of the PV camera if using the correct timestamp (hence my above question).
Hi,
I want to locate with
xrLocateSpace
HoloLens 2 PV camera at given frame timestamps. But I don't know how to properly interop between camera timestamps and OpenXR runtime timestamps, such as predicted display times.For HoloLens 2 PV camera, I'm using MediaFrameReference::SystemRelativeTime timestamps, defined as 100ns interval QPC time, so a fraction of a QPC value if I understand it correctly. Thanks to xrConvertWin32PerformanceCounterToTimeKHR from XR_KHR_win32_convert_performance_counter_time extension, I was hoping to convert the camera timestamps to OpenXR runtime
XrTime
timestamps for use withxrLocateSpace
. But this doesn't work, i.e. I'm getting an invalid location.Below are the logs of a running session where I'm outputting the timestamps of interest:
MediaFrameReference::SystemRelativeTime
inMediaFrameReader::FrameArrived
event;QueryPerformanceCounter
inMediaFrameReader::FrameArrived
event;XrTime
withxrConvertWin32PerformanceCounterToTimeKHR
;XrTime
timestamp.If I pass to
xrLocateSpace
the timestamp returned byMediaFrameReference::SystemRelativeTime
, the PV camera isn't located. Expected as it's a QPC time value.Not shown in the logs, if I pass to
xrLocateSpace
the above timestamp converted toXrTime
withxrConvertWin32PerformanceCounterToTimeKHR
, the PV camera isn't located. The resultingXrTime
timestamp appears to be based on a very different time reference than OpenXR predicted display time.If I pass to
xrLocateSpace
the timestamp returned by the call toQueryPerformanceCounter
inMediaFrameReader::FrameArrived
event, the PV camera isn't located. Expected as it's a QPC time value.But if I pass to
xrLocateSpace
the above timestamp converted toXrTime
withxrConvertWin32PerformanceCounterToTimeKHR
, the PV camera is successfully located.I wish I could use the camera timestamp directly (using the appropriate adjustment and/or conversion, of course) rather than having to perform a call to
QueryPerformanceCounter
to generate a timestamp suitable for use withxrLocateSpace
(once converted toXrTime
). Any guidance on this or is callingQueryPerformanceCounter
inMediaFrameReader::FrameArrived
event the right approach? Otherwise, what am I doing wrong with the camera timestamps handling?If you carefully look at the timestamps returned by
MediaFrameReference::SystemRelativeTime
, you'll notice there are roughly 1/100th of OpenXR runtimeXrTime
timestamps. And indeed, if I pass toxrLocateSpace
the timestamps returned byMediaFrameReference::SystemRelativeTime
multiplied by 100, the PV camera is successfully located. Is this 100 multiplier in any way correlated to the 100ns interval QPC time inMediaFrameReference::SystemRelativeTime
documentation? In short, is it expected that I simply have to multiply by 100 the timestamps returned byMediaFrameReference::SystemRelativeTime
to pass toxrLocateSpace
without having to rely uponxrConvertWin32PerformanceCounterToTimeKHR
at all? Does it mean that HoloLens 2 PV camera gives already-suitable-for-use-with-OpenXR timestamps? If yes, is this guaranteed to always be the case for all the WMR devices? What's then the purpose of XR_KHR_win32_convert_performance_counter_time extension?Thanks.