search

fpv-iplab / rulstm

Code for the Paper: Antonino Furnari and Giovanni Maria Farinella. What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attention. International Conference on Computer Vision, 2019.
http://iplab.dmi.unict.it/rulstm
132 stars 33 forks source link

Evaluating EK55 scores - missing CSVs #30

Closed lucas-ps closed 8 months ago

lucas-ps commented 1 year ago

Hi,

Firstly, thank you for providing this source code and detailed documentation, your model has been by far the easiest one to work with that I have tried so far! I'm trying to compare different anticipation models using several datasets based on RGB features alone, however when I try to validate the EK55 model, I get errors which say that there are missing CSV files.

Have these been renamed, ie, should I change them to:

From my understanding, these files serve different purposes. I have tried commenting out the validation function get_validation_ids and replaced it's functionality with the more generic provided (egtea) validation, which worked, however the results I achieved showed extremely low accuracy scores compared to other datasets I have tested. I'm not sure if this is due to how Epic Kitchens should be evaluated, or if the other datasets are producing over fitted models, or if Epic Kitchens just isn't a good dataset to rely on RGB features.

If possible, could you provide the CSV files mentioned in the code please?

For context, I am getting these scores using the generic validation method:

50-Salads dataset

2.0 1.75 1.5 1.25 1.0 0.75 0.5 0.25
Verb
Top-1 Accuracy 07.78 10.56 10.00 07.78 08.89 09.44 08.89 08.33
Top-5 Accuracy 29.44 29.44 29.44 29.44 29.44 29.44 29.44 29.44
Mean Top-5 Recall 83.33 83.33 83.33 83.33 83.33 83.33 83.33 83.33
Noun
Top-1 Accuracy 05.00 05.56 05.56 06.11 07.22 07.22 08.89 07.78
Top-5 Accuracy 37.22 40.00 38.33 37.78 38.89 39.44 37.22 37.78
Mean Top-5 Recall 39.59 42.77 40.35 39.95 43.04 43.39 41.74 42.28
Action
Top-1 Accuracy 17.22 16.11 16.11 15.56 19.44 20.00 18.89 22.78
Top-5 Accuracy 73.89 72.78 73.33 77.22 81.11 76.67 77.22 76.67
Mean Top-5 Recall 71.77 70.32 71.86 75.46 79.79 75.04 75.85 74.86

Mean TtA(5): VERB: 0.59 NOUN: 0.87 ACTION: 1.63

Epic Kitchens-55 dataset

2.0 1.75 1.5 1.25 1.0 0.75 0.5 0.25
Verb
Top-1 Accuracy 06.09 06.15 06.37 06.53 07.09 07.03 07.45 07.53
Top-5 Accuracy 20.14 20.18 20.25 20.33 20.16 20.18 20.22 20.33
Mean Top-5 Recall 04.25 04.25 04.43 04.45 04.40 04.19 04.31 04.35
Noun
Top-1 Accuracy 00.34 00.18 00.24 00.18 00.26 00.30 00.30 00.26
Top-5 Accuracy 01.71 01.57 01.77 01.95 02.11 01.75 01.71 01.75
Mean Top-5 Recall 01.48 01.49 01.64 01.90 01.99 01.75 01.68 01.71
Action
Top-1 Accuracy 09.84 10.38 11.03 12.11 12.89 13.52 14.24 15.10
Top-5 Accuracy 25.21 26.15 27.82 29.10 30.89 31.65 32.86 34.02
Mean Top-5 Recall 09.79 10.49 11.63 11.78 12.92 12.86 13.45 13.46

Mean TtA(5): VERB: 0.41 NOUN: 0.05 ACTION: 0.67

Thank you for any help, and apologies if I am missing something obvious!

antoninofurnari commented 1 year ago

Hi,

Not sure I entirely follow. Which command did you use for evaluation? The CSVs you are looking for are in https://github.com/fpv-iplab/rulstm/tree/master/RULSTM/data/ek100 and they are meant for evaluation on EK-100, not EK-55.

To check if validation results are correct, you can maybe see if your numbers match (or are similar to) the ones reported in our paper https://arxiv.org/abs/2005.02190

Best, Antonino