Cross-dataset validation

[BlueWinters]

Hi @Tobias-Fischer I implement the algorithm and archieve a similar result in rt_gene dataset (about 7.9% compare with 7.7% in the paper). But when I do the cross-dataset validation, the error is very high (e.g columbia, MPII). I think that its the process of warping eyes which affects the final results. Would you share the implement detail to us? Thank you for your good work.

[Tobias-Fischer]

Hi @BlueWinters, Thank you for reporting this. As described in the paper, we only cross-validated on the MPII dataset. Columbia's images are very different from ours and were taken in a very constrained setting, therefore it might well be that RT-GENE does not perform well on Columbia.

There are a few things to keep in mind: 1) We used the 4-model ensemble for the cross-validation. A single model did not perform as well as the 4-model ensemble. 2) This 4-model ensemble was trained on all subjects that are contained in the RT-GENE dataset. 3) A correction is required similar to what is implemented in https://github.com/Tobias-Fischer/rt_gene/blob/master/rt_gene/scripts/estimate_gaze.py#L107 and the corrected values should be used for comparison. I cannot recall the precise value we used but I can check in case you can't replicate our results.

We did not change the warping of the eyes when performing cross-dataset validation.

Please let me know whether this helps. In case it does not, I will prepare a Jupyter notebook with our results.

Best, Tobias

[BlueWinters]

Hi @Tobias-Fischer Last week, I try my best to implement the archieve the details, but the results are not satisfactory. I hope you can write a Jupyter notebook to re-produce the results(especially the process of warping eyes from the original face). And that would give us a lot of helps. Thank you very much. Best, Hanxiong.

[Tobias-Fischer]

Hi @BlueWinters, sure, I'll do. Not sure when I get around to doing it though, it might take a while.

[ahmed-alhindawi]

@BlueWinters & @Tobias-Fischer is this issue not outside the scope of this github repo? Reproducibility of the paper on a dataset that it wasn't designed to perform well on isn't the intended purpose of RT-GENE and would favour closing this issue...

[FigaroK]

Hi, @Tobias-Fischer Could you share the implement details of validation on UT Multiview? More precisely, in the 3-fold cross-validation experiment, which subjects' data were used and how they were grouped. It confused me for a while. Thank you very much.

[Tobias-Fischer]

Hi @FigaroK, We reported two results on UTMV: 1) 3-Fold cross-validation. The dataset is split into three groups, subjects 0-16, 17-33 and 34-49. We train on two of the groups and test on the remaining group in a cross-validation fashion: 1) train on group 1+2, test on group 3; 2) train 1+3, test 2; 3) train 2+3, test 1. 2) Cross-dataset validation: We train on the complete UTMV dataset (all subjects included) and apply the trained ensemble model to MPII.

[FigaroK]

@Tobias-Fischer Thanks for your reply. I have another question about your dataset. Did it provide gaze label for every eye like MPIIGaze or UTMV? I only found one gaze label on the same face.

[Tobias-Fischer]

Hi @FigaroK, there is just one gaze label. As the distances are much larger than in e.g. MPIIGaze (the subjects typically look at targets much further away), the difference between left and right eye angles is sufficiently small.

[Tobias-Fischer]

Closing here as this is not a priority.

[StephanPan]

@Tobias-Fischer hi, In which coordinate system is the gaze label collected? And if the gaze label is collected in eyetracking glass coordinate system, how to transform it into camara coordinate system?

[Tobias-Fischer]

Hi, please see page 6 of https://openaccess.thecvf.com/content_ECCV_2018/papers/Tobias_Fischer_RT-GENE_Real-Time_Eye_ECCV_2018_paper.pdf in subsection Coordinate transforms.