search

AR-Eye-Tracking-Toolkit / ARETT

ARETT: Augmented Reality Eye Tracking Toolkit for Head Mounted Displays
MIT License
19 stars 15 forks source link

Data Access Layer for Holographic Remoting #11

Closed serhan-gul closed 2 years ago

serhan-gul commented 2 years ago

Hi, I'm trying to modify DataAccessUWP.cs to make it work with Holographic Remoting because I want to use eye tracking on a complex scene which cannot be run on HL2 due to the high rendering load.

My first attempt was to simply use DataAccessUWP.cs while running the Remoting inside Unity Play mode. However, the web app showed me that the eyes API is not available; it seems that UWP API SpatialPointerPose.Eyes cannot be used while using Remoting. Then I tried to use the MRTK interface CoreServices.InputSystem.EyeGazeProvider (see docs here) but it seems to output gaze samples at an unpredictable rate (usually less than 30 Hz) although I use the same Timer mechanism (checking each 10ms) implemented in ARETT.

Another idea was to build as a Windows Standalone app but again, it seems not to be possible to use the SpatialPointerPose API then.

Do you have any suggestions/ideas how to tackle this? Would be great to hear your opinion on whether this is feasible. Thanks.

sekapp commented 2 years ago

Hi,

sadly I don't know how MRTK accesses ET data when running Holographic Remoting. If you want to try accessing it using ARETT you could try looking at the EyeGazeProvider source and checking what APIs it uses for data access during remoting. However, even then I'm unsure if you can access the same data rate and quality as ARETT generates running directly on the device. The unreliable data rate you saw using MRTK is the reason I developed ARETT and am using the low-level UWP API. When running on the device MRTK only checks for new ET data every frame, resulting in the lower and inconsistent data rate with unprecise timing. Checking the current gaze using the EyeGazeProvider only accesses the cached gaze data and therefore even when checking it on a 10ms timer you only get the low data rate connected to the frame rate of the app.

Best regards Sebastian Kapp

serhan-gul commented 2 years ago

Thank you Sebastian, this is helpful information. It seems to be a limitation of MRTK that the ET data is only updated with every new frame. Unfortunately, it is not possible to check how exactly Holographic Remoting works internally but my guess is, requests to retrieve ET data are tied to the frame rate in Unity, so variations/drops in frame rate affect the timing of the samples.