search

facebookresearch / mmf

A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)
https://mmf.sh/
Other
5.48k stars 934 forks source link

Unable to reproduce model performance using pretrained models #952

Open mingshanhee opened 3 years ago

mingshanhee commented 3 years ago

❓ Questions and Help

Hi everyone,

I have been trying to reproduce the model performance stated on the research paper but to no avail. Hence, I would like to clarify whether my command is correct.

For starters, I have downloaded the dataset from https://hatefulmemeschallenge.com/. Subsequently, I ran the following code (for Visual BERT COCO) based my understanding of the README.md:

mmf_run dataset=hateful_memes \  
        model=visual_bert \    
        config=projects/hateful_memes/configs/visual_bert/from_coco.yaml \
        checkpoint.resume_file=visual_bert.finetuned.hateful_memes.from_coco \
        checkpoint.resume_pretrained=False \
        run_type=val

Using this, I achieved an underwhelming result of 0.5925 for accuracy and 0.4806 ROC AUC score. This seems to be far below the performance reported in the paper...

P.S. I have also seen Issue 926. Does this means the trained model checkpoint has not been updated and we should train the model from scratch?

vedanuj commented 3 years ago

Can you try checkpoint.resume_zoo instead of checkpoint.resume_file in your run command?