Some questions

[zhaoyuhsin]

Hi！

1. What is affine calibration? Why the error will fall through it :)
2. For NVGaze-AR，“we gathered data from 10 subjects” from NVGaze paper，so what means "NaGaze-AR(2/40)" in your readme.md?
3. I found that in the NVGaze-AR dataset, the label is in the range of -1 to 1, but in the synthetic data set, the label is in the range of -90 to 90. Is it because the former is expressed in radians and the latter is expressed in degrees?
4. The input of the network is 127 127. If I want to test on the NVGaze data set, can I directly scale the image from 640 480 to 127 * 127?

I would be very appreciated if you can help me:) Thanks and regards.

[czero69]

1. Affine calibration is 'the simplest' way of calibrating the eye-tracker. NVGaze paper gives the result only after applying affine calibration. I give here raw results and calibrated results for two reasons: a) it is somehow possible to (more or less) compare it with NVGaze paper results (*) b) comparing to raw result, it says how much end-to-end NN result would be better after affine calibrating for selected test/subject.

https://imgur.com/a/7NzmRZV, affine calib (7)

2. It is a good question. In my case 2/40 means that I picked up 2 tests/subjects for testing, 40 for training. When I downloaded the NVGaze-AR dataset I found 42 different folders with separate tests/subjects. (*) In my opinion, it is impossible to compare the results to NVGaze paper directly for a few reasons. The most important reason is NVGaze paper does not specify how subjects were split for train/valid. From my experience, the result will differ strongly for differently picked up test/train datasets, mainly because 42 test/subjects are still a very small amount. Also, to be honest these datasets are very small from a subject amount perspective. In my opinion dataset of ~5-10k subjects should be built for nowadays solution (I know it is challenging). You can look MagicEyes and OpenEDS datasets, maybe they will open it one day.

3. Here is also an important issue. What I found about NVGaze Dataset is, that it is not the only difference in radians vs degrees, but also angles are encoded in a different way. For NVGaze Synthetic I found that the angle encoded direction is opposite (+ vs -) and angles are 2x smaller/larger compared for ex. to NVGaze-AR. Also, it is not clear how angles are decoded there but I concluded it is the projection to perpendicular planes. Note, that in Acomo I coded it as rotations with some order. Please see in my code how I deal with Acomo and NVGaze angles. This is how I assumed NVGaze angles are decoded:

https://imgur.com/a/7NzmRZV, NVGaze Syntetic vs NVGaze-AR (3), (4)

4. I think the training pipeline will support the scaling automatically, but you can use tools/downscale_images.py script for pre-scaling it to 127x127. It will save space on HDD and training time.

Please note that this repo is before I publish a paper and still, I have already planned a few interesting experiments.

Best, Kamil

[zhaoyuhsin]

1. Affine calibration is 'the simplest' way of calibrating the eye-tracker. NVGaze paper gives the result only after applying affine calibration. I give here raw results and calibrated results for two reasons: a) it is somehow possible to (more or less) compare it with NVGaze paper results (*) b) comparing to raw result, it says how much end-to-end NN result would be better after affine calibrating for selected test/subject.

https://imgur.com/a/7NzmRZV, affine calib (7)

1. It is a good question. In my case 2/40 means that I picked up 2 tests/subjects for testing, 40 for training. When I downloaded the NVGaze-AR dataset I found 42 different folders with separate tests/subjects. (*) In my opinion, it is impossible to compare the results to NVGaze paper directly for a few reasons. The most important reason is NVGaze paper does not specify how subjects were split for train/valid. From my experience, the result will differ strongly for differently picked up test/train datasets, mainly because 42 test/subjects are still a very small amount. Also, to be honest these datasets are very small from a subject amount perspective. In my opinion dataset of ~5-10k subjects should be built for nowadays solution (I know it is challenging). You can look MagicEyes and OpenEDS datasets, maybe they will open it one day.
2. Here is also an important issue. What I found about NVGaze Dataset is, that it is not the only difference in radians vs degrees, but also angles are encoded in a different way. For NVGaze Synthetic I found that the angle encoded direction is opposite (+ vs -) and angles are 2x smaller/larger compared for ex. to NVGaze-AR. Also, it is not clear how angles are decoded there but I concluded it is the projection to perpendicular planes. Note, that in Acomo I coded it as rotations with some order. Please see in my code how I deal with Acomo and NVGaze angles. This is how I assumed NVGaze angles are decoded:

https://imgur.com/a/7NzmRZV, NVGaze Syntetic vs NVGaze-AR (3), (4)

1. I think the training pipeline will support the scaling automatically, but you can use tools/downscale_images.py script for pre-scaling it to 127x127. It will save space on HDD and training time.

Please note that this repo is before I publish a paper and still, I have already planned a few interesting experiments.

Best, Kamil

Sorry to see it so late, thank you very much for your answer. Besides, is it convenient for me to study your paper? Best, Yuxin

[jiangyangzhou]

Hi, I meet some problem reproduce NVGaze, and I wonder how do you split train/test set, what are the 2 subjects used for test? Except for scaling the image to 127x127, do we need to do other data normalization operations?

Best, Yangzhou

[czero69]

Hello. I will write from my memory, I did it some time ago :)

As I remember, the authors of the NVGAZE paper did not deliver instructions on how to split the data to compare to their results and how they crop (did they use full image, or part of it). From my experiments I remember that splitting the data has a huge impact on results (forex. some subjects are easier). Also how do you crop influence the results; best cropping should be figured out for each dataset independently.

I should provide a train/valid data split so anyone can try their models and compare the results. I did one train/valid split to make two experiments possible:

• measure how good model generalize to unseen subject (2/40 would mean 2 unseen on the training subject, 40 subject use only on the training, total would be == 42)
• for each subject I took 80% gaze vectors for training and 20% gaze vectors for evaluation. So I can also measure how good the model is for unseen gaze vectors among known subjects. About cropping the NVGaze Dataset I just resized them. And I think that if NVGaze authors are cropping the images they possibly got better results. About the Acomoeye-14 dataset I already provide cropped and resized 127x127 images. (no op needed here, but some non-transforming augmentations are the way to go).

Possibly I did not include my nvgaze split txt and I only provided acomo-14 split txt? If so, let me know I will check it in the evening and send them to you via email. Would be nice to hear from you can you reproduce the results for Acomo.

Best, Kamil CZ

On Wed, Jul 7, 2021 at 11:01 AM jiangyangzhou @.***> wrote:

Hi, I meet some problem reproduce NVGaze, and I wonder how do you split train/test set, what are the 2 subjects used for test? Except for scaling the image to 127x127, do we need to do other data normalization operations?

Best, Yangzhou

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/czero69/acomoeye-NN/issues/1#issuecomment-875426088, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHCB24TWJSJ2ZHTKTYZWXDTWQJYFANCNFSM4UFTP27Q .

[jiangyangzhou]

Oh thank you very much, it helps a lot. I could not find the NVgaze split files, and I do find that the split methods have great impact. I tried Acomo, and get reasonable results. However, I think the split method is not reasonable. I found there are many very similar images（similar images with same gaze values） in both acomo and nvgaze dataset, so there exists severe label leak problem if you just random split the train and test set. By the way does subject-2-8 from same person like other subject-2? For NVgaze, I found it's hard to get a good result cross the subjects, and domain gap between different subjects is high. And I am not sure of how to implement affine calibration, could you provide some advices?

Best, Yangzhou

[czero69]

As I remember I split the Acomo dataset 80/20 not by randomizing images but by randomizing entire gaze points (set of images related to selected gaze point). So it is trained hardly with some gaze points, but at the same time, some gaze points are completely not seen during training (this way I could measure generalization to unseen gaze points). I have also excluded one subject from the training dataset to check how does it transfer to the unseen subject (but this test makes little sense at the end for only 14 subjects, as some subjects are much easier than the others).

And I am not sure of how to implement affine calibration, could you provide

some advices?

I think you should be able to find it directly coded, but the loop may be ugly a bit. But what I did is the formula on the second image (at the bottom): https://imgur.com/a/7NzmRZV Just the simplest calibration one can imagine.

Best, Kamil CZ

On Thu, Jul 8, 2021 at 9:16 AM jiangyangzhou @.***> wrote:

Oh thank you very much, it helps a lot. I could not find the NVgaze split files, and I do find that the split methods have great impact. I tried Acomo, and get reasonable results. However, I think the split method is not reasonable. I found there are many very similar images（similar images with same gaze values） in both acomo and nvgaze dataset, so there exists severe label leak problem if you just random split the train and test set. By the way does subject-2-8 from same person like other subject-2? For NVgaze, I found it's hard to get a good result cross the subjects, and domain gap between different subjects is high. And I am not sure of how to implement affine calibration, could you provide some advices?

Best, Yangzhou

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/czero69/acomoeye-NN/issues/1#issuecomment-876193994, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHCB2YCOMC4UYWBJTANEJDTWVGDFANCNFSM4UFTP27Q .