Implement FaceLandmarker calibration specification

[iainbryson]

https://docs.google.com/presentation/d/1CdQD6cCWdW4oIPpE0vCRL37lVpE0oszsm8X9em4I1MA/edit#slide=id.g30a8e251af6_2_71

[madjens]

We had a small typo in the calibration pop-up screen 2.

Please update

[iainbryson]

@madjens Shouldn't that read "until they turn red"? Also, "looking at them with your eyes" seems a bit redundant, especially with the second sentence.

How about:

"Click the mouse on each of the dots on the screen until they turn red, while looking at them. Keep your head still, just move your eyes."

?

There's no need for an image, the text is rendered as text

[madjens]

No they start off being red then they turn yellow as you press them

I'm fine with text as long as there are bullet points.

[madjens]

In order for us to create a model that maps from Face to gaze position we need data about the dots and when they are pressed.

1. Event data, i.e. an event with timestamp and unique identifier every time the subject press a dot
2. Render/Layout data for each dot, i.e. where on the screen is the dot shown (x,y in pixels)

[iainbryson]

This should be deployed (both)

[madjens]

Testing the new implementation you are not able to progress to next trial/slide after the end of the calibration.

e.g. run this code and you get stuck after the calibration tests/Testcase_landmarker_calibration.py

I created one experiment https://elicit-experiment.com/studies/1171/protocols/1169

and tried to download the results using examples/learning_study/dump_results.py

However i get Got 0 datapoints for study result 4254. So it doesnt seem like any results are saved.

[iainbryson]

This should be fixed. I can't verify it in production though -- there are no more trials for the landmarker calibration. Is it generally ok to create my own studies for testing?

[madjens]

Yes go for it. Just run the test code

[iainbryson]

Done, looks like it works:

python examples/learning_study/dump_results.py --study_id 1174

...

                            datetime  phase_definition_id  trial_definition_id  component_id              entity_type                     kind           method  point_type    component_name                                              value
0   2024-11-18T06:20:27.828000+00:00                 1304                23595         54171               Instrument         RadioButtonGroup             None       State  RadioButtonGroup                          {"Id":"3","Correct":true}
1   2024-11-18T06:20:25.442000+00:00                 1304                23595         54171               Instrument         RadioButtonGroup                       Render  RadioButtonGroup  [{"Id":"1","Label":"answer 1 Lorem ipsum dolor...
2   2024-11-18T06:20:27.419000+00:00                 1304                23595         54171               Instrument         RadioButtonGroup  Mouse/Left/Down      Change  RadioButtonGroup                                                  3
3   2024-11-18T06:20:30.226000+00:00                 1304                23595         54172               Instrument         RadioButtonGroup             None       State  RadioButtonGroup                         {"Id":"5","Correct":false}
4   2024-11-18T06:20:25.451000+00:00                 1304                23595         54172               Instrument         RadioButtonGroup                       Render  RadioButtonGroup  [{"Id":"1","Label":"answer 1","Preselected":fa...
..                               ...                  ...                  ...           ...                      ...                      ...              ...         ...               ...                                                ...
65  2024-11-18T06:20:16.783000+00:00                 1304                23592             0  FaceLandmarkCalibration  FaceLandmarkCalibration  Mouse/Left/Down      Change               NaN  {"Id":"Pt5","x":1108,"y":597,"timeStamp":17319...
66  2024-11-18T06:20:16.967000+00:00                 1304                23592             0  FaceLandmarkCalibration  FaceLandmarkCalibration  Mouse/Left/Down      Change               NaN  {"Id":"Pt5","x":1108,"y":597,"timeStamp":17319...
67  2024-11-18T06:20:17.185000+00:00                 1304                23592             0  FaceLandmarkCalibration  FaceLandmarkCalibration  Mouse/Left/Down      Change               NaN  {"Id":"Pt5","x":1108,"y":597,"timeStamp":17319...
68  2024-11-18T06:20:17.402000+00:00                 1304                23592             0  FaceLandmarkCalibration  FaceLandmarkCalibration  Mouse/Left/Down      Change               NaN  {"Id":"Pt5","x":1108,"y":597,"timeStamp":17319...
69  2024-11-18T06:20:17.624000+00:00                 1304                23592             0  FaceLandmarkCalibration  FaceLandmarkCalibration             None  calibrated               NaN                                               None

[madjens]

yes that works perfectly thank you.

Next issue is that IncludeBlendshapes isnt working (maybe it never was? but we want to be able to only include the blendshapes we specify in this list.

trial_definition_specification = dict(trial_definition=dict(name='Landmarker calibration',
                                                            definition_data=dict(
                                                                    TrialType='Calibration',
                                                                    type='NewComponent::FaceLandmarkCalibration',
                                                                    # number of faces expected in the interface
                                                                    NumberOfFaces=1,
                                                                    Landmarks=True,  # return Landmark data
                                                                    Blendshapes=True,  # return Blendshape data
                                                                    FaceTransformation=True, # indicate if the affine transform should be performed or not
                                                                    CalibrationDuration=5, # duration of face within view measured in seconds
                                                                    StripZCoordinates=True,# IncludeBlendshapes='eyeLookInRight,eyeLookInLeft',
                                                                    **IncludeBlendshapes**='eyeLookInRight,eyeLookInLeft',
                                                                    IncludeLandmarks = '1,2,5,100,346',
                                                                    )))

[madjens]

Also the loading screen still says Loading Eye-tracker, could we get it to say Loading Face-tracker,

[iainbryson]

Eye-tracker name is changed

[iainbryson]

Regarding IncludeBlendshapes, it is mutually exclusive with IncludeLandmarks right now. If you need both at the same time we'll need to figure out how they interact.

[madjens]

I am getting both Blendshapes and Landmarks transmitted at the same time.

We want the ability to only save Blendshapes, Landmarks or both. I.e.

Landmarks =True, Blendshapes=False,

IncludeBlendshapes is not relevant in this case IncludeLandmarks will sort out which landmarks should be transmitted using the indices e.g. '1,2,5,100,346'

or Landmarks =False, Blendshapes=True,

IncludeBlendshapes will sort out which blendshapes should be transmitted, i.e. using the names of them e.g. 'eyeLookInRight,eyeLookInLeft' IncludeLandmarks is not relevant in this case

or Landmarks =True, Blendshapes=True,

IncludeBlendshapes will sort out which blendshapes should be transmitted, i.e. using the names of them e.g. 'eyeLookInRight,eyeLookInLeft' IncludeLandmarks will sort out which landmarks should be transmitted using the indices e.g. '1,2,5,100,346'

[iainbryson]

I've implemented new logic for this. In addition to the behavior above, note that if you do IncludeBlendshapes this will restrict the facelandmarks to just those needed referred to by the Blendshapes. If you also specify IncludeLandmarks, the facelandmarks will include the union of both the face landmarks and the landmarks referred to by the IncludeBlendshapes

[madjens]

The landmarks and blendshapes are distinctly different features and I think it would be best to separate the two variables. so

IncludeBlendshapes will only have an effect on Blendshapes IncludeLandmarks will only have an effect on Landmarks This leaves any specification up to the end user. So one doesn't effect the other one, to keep it clean. Is that possible?

[iainbryson]

The blendshapes index into the faceLandmark array, so if we want to reduce that array for size reasons, IncludeBlendshapes and IncludeLandmarks will need to affect the array.

The current approach does a logical "and" of these, but there are the options:

1. We could just always send all the faceLandmarks. This will be costly for storage
2. We could send a partial faceLandmarks only if "the opposite" feature is not enabled. i.e. subset faceLandmarks for the IncludeLandmarks if and only if Blendshapes is false, and subset faceLandmarks for the IncludeBlendshapes if and only if Landmarks is false
3. Attempt to reduce the faceLandmarks to the minimum for any blendshape and landmarks configuration a. If Landmarks is true but IncludeLandmarks is empty, send all the faceLandmarks b. If Blendshapes is true but IncludeBlendshapes is empty, send only first 51 faceLandmarks (all the blendshapes reference one of the first 51 faceLandmarks) c. IfLandmarksis true,IncludeLandmarksis specified and Blendshapesare both true, include the union of the faceLandmarks from IncludeLandmarksand either all the blendshape landmarks (b) or the subset specified byIncludeBlendshapes`

The current approach attempts 3. Perhaps this is worth a quick call?

On Mon, Nov 25, 2024 at 7:25 AM Jens Madsen @.***> wrote:

The landmarks and blendshapes are distinctly different features and I think it would be best to separate the two variables. so

IncludeBlendshapes will only have an effect on Blendshapes IncludeLandmarks will only have an effect on Landmarks This leaves any specification up to the end user. So one doesn't effect the other one, to keep it clean. Is that possible?

— Reply to this email directly, view it on GitHub https://github.com/elicit-experiment/frontend/issues/15#issuecomment-2498616601, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALVZX6EMFAKVE67BCUGHSD2CNMQVAVCNFSM6AAAAABQWM4P6SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJYGYYTMNRQGE . You are receiving this because you authored the thread.Message ID: @.***>

-- — Iain