search

MicrosoftDocs / mixed-reality

Mixed Reality documentation
Creative Commons Attribution 4.0 International
173 stars 184 forks source link

Extended Eye Tracking Provider Not Working #670

Closed JinghuiHu1018 closed 1 year ago

JinghuiHu1018 commented 1 year ago

[Enter feedback here] I'm not able to use the extended eye tracking provider on hololens 2 when following the instructions in the document. My app was doing fine when using the basic eye-tracking service but I do need the higher frame rate provided by the extended eye-tracking service.

I'm using Unity 2020.3.40f1, Mixed Reality Toolkit 2.8.2 and OpenXR plugin 1.5.1. error message: Assets\Scripts\ExtendedEyeTrackingDataProvider.cs(215,140): error CS0234: The type or namespace name 'WindowsMREnvironment' does not exist in the namespace 'UnityEngine.XR.WindowsMR' (are you missing an assembly reference?)

Document Details

⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

magdavuko commented 1 year ago

Hello, Can you please let me know what your app does? Do you need to use Combined gaze or left and Right gaze?

Bowen-MS commented 1 year ago

Hello, that Extended Eye Tracking Provider class in the doc contains some legacy dependencies for older Unity version. Please check out our latest sample here, which should work fine with your Unity version and the OpenXR plugin

JinghuiHu1018 commented 1 year ago

Hi,

I have tried both deploying using the mentioned sample code in my project and deploying the sample project alone. But, in both cases, no eye-tracking content was shown on device. It looks like _eyeGazeTracker=null all the time.

From: Bowen Zhang @.> Sent: 28 October 2022 0:17 To: MicrosoftDocs/mixed-reality @.> Cc: Jinghui Hu @.>; Author @.> Subject: Re: [MicrosoftDocs/mixed-reality] Extended Eye Tracking Provider Not Working (Issue #670)

Hello, that Extended Eye Tracking Provider class in the doc contains some legacy dependencies for older Unity version. Please check out our latest sample herehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2FMixedReality-EyeTracking-Sample%2Fblob%2Fmain%2FSampleEyeTracking%2FAssets%2FScripts%2FExtendedEyeGazeDataProvider.cs&data=05%7C01%7Cjh2265%40universityofcambridgecloud.onmicrosoft.com%7C157e3b8b42854d87257c08dab8718c3d%7C49a50445bdfa4b79ade3547b4f3986e9%7C0%7C0%7C638025095115620418%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hT%2FGj61ftpa9dd3hnYh4onIZ3ykxLJ41ce%2Bueiy3bk8%3D&reserved=0, which should work fine with your Unity version and the OpenXR plugin

— Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fmixed-reality%2Fissues%2F670%23issuecomment-1294223792&data=05%7C01%7Cjh2265%40universityofcambridgecloud.onmicrosoft.com%7C157e3b8b42854d87257c08dab8718c3d%7C49a50445bdfa4b79ade3547b4f3986e9%7C0%7C0%7C638025095115620418%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xnIu5UEA%2BONFYfiqWVjbU%2FvpaG1oR6YMxUQT6IJUULU%3D&reserved=0, or unsubscribehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FARSL6YB7WV3MUROCG7RQ7CLWFMESBANCNFSM6AAAAAARQJVQ5I&data=05%7C01%7Cjh2265%40universityofcambridgecloud.onmicrosoft.com%7C157e3b8b42854d87257c08dab8718c3d%7C49a50445bdfa4b79ade3547b4f3986e9%7C0%7C0%7C638025095115620418%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FGc69wAJr7LU%2BqVT2n2NnI6JeGdACp%2BZG9Wcteth%2FnM%3D&reserved=0. You are receiving this because you authored the thread.Message ID: @.***>

Bowen-MS commented 1 year ago

There might be several causes

  1. Could you check about your HoloLens system version and try to update it? The extended eye tracking capability becomes available in August update
  2. When launching the app from HoloLens, does the dialog asking for gaze permission pop up and permissions are granted?
JinghuiHu1018 commented 1 year ago

I believe the HL2 system is up to date and the eye gaze permission dialog popped up in the deployment.

Could it be related to the settings for basic eye tracking as I still followed the setup for using basic eye tracking?

From: Bowen Zhang @.> Sent: 03 November 2022 17:45 To: MicrosoftDocs/mixed-reality @.> Cc: Jinghui Hu @.>; Author @.> Subject: Re: [MicrosoftDocs/mixed-reality] Extended Eye Tracking Provider Not Working (Issue #670)

There might be several causes

  1. Could you check about your HoloLens system version and try to update it? The extended eye tracking capability becomes available in August update
  2. When launching the app from HoloLens, does the dialog asking for gaze permission pop up and permissions are granted?

— Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fmixed-reality%2Fissues%2F670%23issuecomment-1302463435&data=05%7C01%7Cjh2265%40universityofcambridgecloud.onmicrosoft.com%7Ce3f25de8885d4e980c5c08dabdc340b7%7C49a50445bdfa4b79ade3547b4f3986e9%7C0%7C0%7C638030943553027235%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=1z%2BSDdwuX%2BsOp4ALpRXh3v%2FD4%2BzIheiDvB90nRTTISI%3D&reserved=0, or unsubscribehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FARSL6YCJSRD6XEMEEFSD3TDWGP23BANCNFSM6AAAAAARQJVQ5I&data=05%7C01%7Cjh2265%40universityofcambridgecloud.onmicrosoft.com%7Ce3f25de8885d4e980c5c08dabdc340b7%7C49a50445bdfa4b79ade3547b4f3986e9%7C0%7C0%7C638030943553027235%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BEjJC3aGdsza40kMgRFdA1xuQpoTB2VP2c0SJFwP9Zo%3D&reserved=0. You are receiving this because you authored the thread.Message ID: @.***>

Bowen-MS commented 1 year ago

The basic and extended eye tracking should be parallel and won't influence each other. At least the sample project should work.

Could you provide your OS build number by going to Settings -> System -> About? And you may also try to use Debug mode for the sample project to see what logs are showed.

JinghuiHu1018 commented 1 year ago

My OS build number is 19043.2130

Here is logs of running the sample project.

Triggering eye gaze permission request UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[]) UnityEngine.Logger:Log(LogType, Object) UnityEngine.Debug:Log(Object)

d__18:MoveNext() System.Runtime.CompilerServices.AsyncVoidMethodBuilder:Start(d__18&) ExtendedEyeGazeDataProvider:Start() Eye gaze access status: Allowed UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[]) UnityEngine.Logger:Log(LogType, Object) UnityEngine.Debug:Log(Object) d__21:MoveNext() System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start(d__21&) ExtendedEyeGazeDataProvider:AskForEyePosePermission() d__18:MoveNext() System.Runtime.CompilerServices.AsyncVoidMethodBuilder:Start(d__18&) ExtendedEyeGazeDataProvider:Start() Exception thrown at 0x00007FFB8C6457EC in ExtendedEyeTrackingSample.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x0000003F2BCFB400. Exception thrown at 0x00007FFB8C6457EC in ExtendedEyeTrackingSample.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x0000003F2BCFB850. Exception thrown at 0x00007FFB8C6457EC in ExtendedEyeTrackingSample.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x0000003F2BCFBBB0. Exception thrown at 0x00007FFB8C6457EC in ExtendedEyeTrackingSample.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x0000003F2BCFBBB0. Exception thrown at 0x00007FFB8C6457EC in ExtendedEyeTrackingSample.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x0000003F2BCFC200. Exception thrown at 0x00007FFB8C6457EC in ExtendedEyeTrackingSample.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x0000003F2BCFC350. Exception thrown at 0x00007FFB8C6457EC in ExtendedEyeTrackingSample.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x0000003F2BCFCA80. Exception thrown at 0x00007FFB8C6457EC in ExtendedEyeTrackingSample.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x0000003F2BCFCF30. Exception thrown at 0x00007FFB8C6457EC in ExtendedEyeTrackingSample.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x0000003F2BCFDF50. COMException at System.Runtime.InteropServices.Marshal.ThrowExceptionForHR (System.Int32 errorCode) [0x00000] in <00000000000000000000000000000000>:0 at WinRT.Platform.RoGetActivationFactory (System.IntPtr runtimeClassId, System.Guid& iid) [0x00000] in <00000000000000000000000000000000>:0 at WinRT.WinrtModule.GetActivationFactory (WinRT.HString runtimeClassId) [0x00000] in <00000000000000000000000000000000>:0 at WinRT.ActivationFactory`1[T]..ctor () [0x00000] in <00000000000000000000000000000000>:0 at System.Reflection.MonoCMethod.InternalInvoke (System.Object obj, System.Object[] parameters) [0x00000] in <00000000000000000000000000000000>:0 at System.RuntimeType.CreateInstanceMono (System.Boolean nonPublic) [0x00000] in <00000000000000000000000000000000>:0 at System.RuntimeType.CreateInstanceSlow (System.Boolean publicOnly, System.Boolean skipCheckThis, System.Boolean fillCache, System.Threading.StackCrawlMark& stackMark) [0x00000] in <00000000000000000000000000000000>:0 at System.RuntimeType.CreateInstanceDefaultCtor (System.Boolean publicOnly, System.Boolean skipCheckThis, System.Boolean fillCache, System.Threading.StackCrawlMark& stackMark) [0x00000] in <00000000000000000000000000000000>:0 at System.Activator.CreateInstance[T] () [0x00000] in <00000000000000000000000000000000>:0 at WinRT.WeakLazy`1+<>c[T].<.ctor>b__2_0 () [0x00000] in <00000000000000000000000000000000>:0 at WinRT.WeakLazy`1[T].get_Value () [0x00000] in <00000000000000000000000000000000>:0 at WinRT.ActivationFactory`1[T].ActivateInstance[I] () [0x00000] in <00000000000000000000000000000000>:0 at Microsoft.MixedReality.EyeTracking.EyeGazeTrackerWatcher..ctor () [0x00000] in <00000000000000000000000000000000>:0 at ExtendedEyeGazeDataProvider+d__18.MoveNext () [0x00000] in <00000000000000000000000000000000>:0 at System.Runtime.CompilerServices.AsyncVoidMethodBuilder.Start[TStateMachine] (TStateMachine& stateMachine) [0x00000] in <00000000000000000000000000000000>:0 at ExtendedEyeGazeDataProvider.Start () [0x00000] in <00000000000000000000000000000000>:0 Rethrow as TargetInvocationException: Exception has been thrown by the target of an invocation. at System.Reflection.MonoCMethod.InternalInvoke (System.Object obj, System.Object[] parameters) [0x00000] in <00000000000000000000000000000000>:0 at System.RuntimeType.CreateInstanceMono (System.Boolean nonPublic) [0x00000] in <00000000000000000000000000000000>:0 at System.RuntimeType.CreateInstanceSlow (System.Boolean publicOnly, System.Boolean skipCheckThis, System.Boolean fillCache, System.Threading.StackCrawlMark& stackMark) [0x00000] in <00000000000000000000000000000000>:0 at System.RuntimeType.CreateInstanceDefaultCtor (System.Boolean publicOnly, System.Boolean skipCheckThis, System.Boolean fillCache, System.Threading.StackCrawlMark& stackMark) [0x00000] in <00000000000000000000000000000000>:0 at System.Activator.CreateInstance[T] () [0x00000] in <00000000000000000000000000000000>:0 at WinRT.WeakLazy`1+<>c[T].<.ctor>b__2_0 () [0x00000] in <00000000000000000000000000000000>:0 at WinRT.WeakLazy`1[T].get_Value () [0x00000] in <00000000000000000000000000000000>:0 at WinRT.ActivationFactory`1[T].ActivateInstance[I] () [0x00000] in <00000000000000000000000000000000>:0 at Microsoft.MixedReality.EyeTracking.EyeGazeTrackerWatcher..ctor () [0x00000] in <00000000000000000000000000000000>:0 at ExtendedEyeGazeDataProvider+d__18.MoveNext () [0x00000] in <00000000000000000000000000000000>:0 at System.Runtime.CompilerServices.AsyncVoidMethodBuilder.Start[TStateMachine] (TStateMachine& stateMachine) [0x00000] in <00000000000000000000000000000000>:0 at ExtendedEyeGazeDataProvider.Start () [0x00000] in <00000000000000000000000000000000>:0 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x00000] in <00000000000000000000000000000000>:0 at System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.b__6_0 (System.Object state) [0x00000] in <00000000000000000000000000000000>:0 at UnityEngine.UnitySynchronizationContext+WorkRequest.Invoke () [0x00000] in <00000000000000000000000000000000>:0 at UnityEngine.UnitySynchronizationContext.Exec () [0x00000] in <00000000000000000000000000000000>:0 at UnityEngine.UnitySynchronizationContext.ExecuteTasks () [0x00000] in <00000000000000000000000000000000>:0 UnityEngine.DebugLogHandler:LogException(Exception, Object) UnityEngine.Logger:LogException(Exception, Object) UnityEngine.Debug:LogException(Exception) UnityEngine.WorkRequest:Invoke() UnityEngine.UnitySynchronizationContext:Exec() UnityEngine.UnitySynchronizationContext:ExecuteTasks() ________________________________ From: Bowen Zhang ***@***.***> Sent: 04 November 2022 0:11 To: MicrosoftDocs/mixed-reality ***@***.***> Cc: Jinghui Hu ***@***.***>; Author ***@***.***> Subject: Re: [MicrosoftDocs/mixed-reality] Extended Eye Tracking Provider Not Working (Issue #670) The basic and extended eye tracking should be parallel and won't influence each other. At least the sample project should work. Could you provide your OS build number by going to Settings -> System -> About? And you may also try to use Debug mode for the sample project to see what logs are showed. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>
Bowen-MS commented 1 year ago

19043.2130 looks like the desktop OS version. Could you help to provide your HoloLens OS version?

JinghuiHu1018 commented 1 year ago

Sure, HoloLens OS build is 20348.1522.

From: Bowen Zhang @.> Sent: 04 November 2022 17:56 To: MicrosoftDocs/mixed-reality @.> Cc: Jinghui Hu @.>; Author @.> Subject: Re: [MicrosoftDocs/mixed-reality] Extended Eye Tracking Provider Not Working (Issue #670)

19043.2130 looks like the desktop OS version. Could you help to provide your HoloLens OS version?

— Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fmixed-reality%2Fissues%2F670%23issuecomment-1303953603&data=05%7C01%7Cjh2265%40universityofcambridgecloud.onmicrosoft.com%7Cf3c7bcab20b04519831708dabe8df5ad%7C49a50445bdfa4b79ade3547b4f3986e9%7C0%7C0%7C638031814174495981%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=yxEFYzivjgv7DC9xYoME0HqBmC%2FmT%2FpD3DHGQoDfW64%3D&reserved=0, or unsubscribehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FARSL6YAKOIOPGROQMRQY7PDWGVE6JANCNFSM6AAAAAARQJVQ5I&data=05%7C01%7Cjh2265%40universityofcambridgecloud.onmicrosoft.com%7Cf3c7bcab20b04519831708dabe8df5ad%7C49a50445bdfa4b79ade3547b4f3986e9%7C0%7C0%7C638031814174651730%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=G2LahD7PMoefHV2MByCzpfEXASZzFUc6TUCnu9ERTTA%3D&reserved=0. You are receiving this because you authored the thread.Message ID: @.***>

Bowen-MS commented 1 year ago

After investigation, we found that the latest version of SDK 1.0.1 missed to import some dlls when importing to Unity project, and we're fixing the issue and will release a new version of SDK soon.

A temporary solution is to use the 1.0.0 version of SDK at this time. You may notice that there will be some Unity build errors but you could ignore those and deploy the app to device.

JinghuiHu1018 commented 1 year ago

Thanks for the help! That works well for me.

From: Bowen Zhang @.> Sent: 08 November 2022 0:40 To: MicrosoftDocs/mixed-reality @.> Cc: Jinghui Hu @.>; Author @.> Subject: Re: [MicrosoftDocs/mixed-reality] Extended Eye Tracking Provider Not Working (Issue #670)

After investigation, we found that the latest version of SDK 1.0.1 missed to import some dlls when importing to Unity project, and we're fixing the issue and will release a new version of SDK soon.

A temporary solution is to use the 1.0.0 version of SDK at this time. You may notice that there will be some Unity build errors but you could ignore those and deploy the app to device.

— Reply to this email directly, view it on GitHubhttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fmixed-reality%2Fissues%2F670%23issuecomment-1306434049&data=05%7C01%7Cjh2265%40universityofcambridgecloud.onmicrosoft.com%7Cf90bc623f94045ae0cd208dac121cda4%7C49a50445bdfa4b79ade3547b4f3986e9%7C1%7C0%7C638034648168311440%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=m%2Bl7Ljo8Ui9TbOH%2Bbpr7FQ9jPwjgv9qdmreZKOalQms%3D&reserved=0, or unsubscribehttps://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FARSL6YDVKY5DTE6EBJ3J7FTWHGOO3ANCNFSM6AAAAAARQJVQ5I&data=05%7C01%7Cjh2265%40universityofcambridgecloud.onmicrosoft.com%7Cf90bc623f94045ae0cd208dac121cda4%7C49a50445bdfa4b79ade3547b4f3986e9%7C1%7C0%7C638034648168311440%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xeuYqC1AKs%2FP%2BPCruKqRfYkUAkUrPRiyHDvcQjop%2Fqo%3D&reserved=0. You are receiving this because you authored the thread.Message ID: @.***>

Bowen-MS commented 1 year ago

Thanks for providing feedback.

We have released the new EyeTracking SDK 1.0.2 with the sample app updated as well. The new 1.0.2 SDK has fixed the exception and also the build errors you may see in the Unity editor.