Dataset extractor broken on 3.0.0

bbonf commented 2 years ago

I think it has to do with the video element changes in cd26c906ae2c7227bc70d763e5d4887bdf4af725

Uncaught (in promise) TypeError: input is null
    getInputTensorDimensions index.js:150
    estimateFaces index.js:241
    step index.js:48
    verb index.js:29
    __awaiter index.js:23
    __awaiter index.js:19
    estimateFaces index.js:228
    getEyePatches facemesh.mjs:39
    getPupilFeatures index.mjs:204
    getPrediction index.mjs:237
    getCurrentPrediction index.mjs:1132
    runWebGazerSendResult webgazerExtractClient.js:347
    onload webgazerExtractClient.js:117
    onmessage webgazerExtractClient.js:112
    onLoad webgazerExtractClient.js:103
    onload webgazerExtractClient.html:1

jeffhuang commented 2 years ago

Thanks for the report. You're probably right, because 3.0.0 changed the getPupilFeatures API a little bit. I'll look into it later this week.

jeffhuang commented 2 years ago

Hmmm was this working with WebGazer 2.1.2? And are you just using the webgazer.js from the www/ folder? I'm not exactly sure what your setup is.

Generally, the video dataset was originally meant to work with a much older version of WebGazer, basically WebGazer 1.x. If it easily works with WebGazer 2.x then that's great. If you're fine with tinkering a bit, here the error is simply that estimateFaces() needs a video stream (due to the newer facemesh) and it's being passed a null one.

There might be a way to remove that need for a video stream in webgazer.js completely, and revert the parameters back to what it was in 2.x. Or there can be a way to pass that into estimateFaces().

bbonf commented 2 years ago

Thanks for looking at that so quickly. It does work with 2.1.2, which is good enough for me for now. I only ran into this issue because I wanted to test #254 with 3.0.0

shubhvachher commented 2 years ago

I can see from #253 @bbonf that dataset extractor does work on Webgazer 2.1.2 so I reverted to it using git checkout 6a43030a8f8c16784d71f3c66ab8412861971582 . Running the webgazerExtractServer python code gave me an output of

$ python .\webgazerExtractServer.py
[I 220513 03:26:10 webgazerExtractServer:395] Listening on [::]:8000
WebGazer ETRA2018 Dataset Extractor server started; please open http://localhost:8000/webgazerExtractClient.html
[I 220513 03:26:13 web:2239] 304 GET /webgazerExtractClient.html (::1) 156.25ms
[I 220513 03:26:13 web:2239] 200 GET /tensorflow.js (::1) 0.00ms
[I 220513 03:26:13 web:2239] 304 GET /webgazerExtractClient.js (::1) 31.25ms[I 220513 03:26:13 web:2239] 200 GET /webgazer.js (::1) 140.64ms
[I 220513 03:26:14 web:2239] 101 GET /websocket (::1) 0.00ms
['P_01\\1491423217564_2_-study-dot_test_instructions.webm'] P_01
P_01
Processing video: P_01/1491423217564_3_-study-dot_test.webm
    Extracting video frames (might take a few minutes)... P_01/1491423217564_3_-study-dot_test.webm
[I 220513 03:26:30 web:2239] 304 GET /P_01/1491423217564.json (::1) 0.00ms
[I 220513 03:26:38 web:2239] 206 GET /P_01/P_01.mov (::1) 7404.97ms

and then nothing... So I checked the console log on my browser and it has

webgazer.js:88772 Uncaught (in promise) TypeError: Cannot read properties of null (reading 'height')
    at getInputTensorDimensions (webgazer.js:88772:16)
    at FaceMesh.<anonymous> (webgazer.js:88863:30)
    at step (webgazer.js:88670:23)
    at Object.next (webgazer.js:88651:53)
    at webgazer.js:88645:71
    at new Promise (<anonymous>)
    at __awaiter (webgazer.js:88641:12)
    at FaceMesh.estimateFaces (webgazer.js:88850:16)
    at TFFaceMesh.getEyePatches (webgazer.js:146386:35)
    at async getPrediction (webgazer.js:148073:23)

I'm not adept at javascript :/ What can I do to fix this?

shubhvachher commented 2 years ago

The dataset extractor works on the first commit of Webgazer 2.1.2: git checkout 63eb7de29fb96a10f36fcfc18330ed08be4f8a2c instead.

Running on that though always gives a -1, -1 for webGazerX and webGazerY columns:

Lotemn102 commented 1 year ago

This is still a problem. I've used WebGazer 2.1.2 (git checkout 63eb7de29fb96a10f36fcfc18330ed08be4f8a2c), and I get the same error.

brownhci / WebGazer

Dataset extractor broken on 3.0.0 #253