search

facebookresearch / EgoVLPv2

Code release for "EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone" [ICCV, 2023]
MIT License
87 stars 12 forks source link

Problems about evaluating on Epic Kitchen datasets #9

Closed Dongzhikang closed 8 months ago

Dongzhikang commented 8 months ago

Dear authors, I really appreciate your work. When I reproduced your work by performing zero-shot on epic kitchen dataset, I found that the model is still under training even I used epic.json under eval directory.

I noticed that in eval/epic.json, you set epoch to 0 and init_val to true compared to fine-tuning configs. But when I checked base_trainer.py line 332, it seems like the model will always be trained instead of directly testing on the test set. Could you please help me with that? Thank you so much!

ShramanPramanick commented 8 months ago

Hi @Dongzhikang, did you update load_checkpoint in configs/eval/charades.json with the path of the pre-trained/fine-tuned checkpoint? When using configs/eval/charades.json, only the evaluation on the test set is executed.

As you correctly stated, since the init_val is True, the evaluation will occur in line 330 of base_trainer.py. After running the evaluation, the training loop won't be executed as self.epochs is 0 in line 336, and the script will stop.

Please let me know if you still have difficulties reproducing the results or have any doubts. Thanks.

Dongzhikang commented 8 months ago

Thank you!