mmf_predict change the order of the ids of test Hateful Meme dataset

[mmiakashs]

Feature

I was trying to use mmf_predict to create the submission result csv file. I think the mmf_predict command changes the test ids order. That's why I got "IDs for submission are not correct" when I tried to submit the results csv file. Here is the five first ids I got from the result generated using mmf_predict: 16395,94180,97015,5429 . However, the first five ids in the test.jsonl are: 16395,37405,94180,54321,97015. Yes, wrote another script to make the order of ids according to the test.josnl. However, I was wondering is there any way I can generate the ordered submission file using mmf_predict? or am I missing any flag? Thanks.

I am using the mmf latest version: 1.0.0rc11

I am using the following command to generate the submission file:

mmf_predict config="configs/experiments/defaults.yaml" \
model=model_name \
dataset=hateful_memes \
training.run_type=test \
checkpoint.resume=True \
checkpoint.resume_best=True \
training.num_workers=0

[apsdehal]

As shuffle is off for test dataset, it should technically return the same order as test.jsonl. We will have a look at what is happening.

[KaiPoChang]

Is there any modification ? I encountered this problem recently! Thanks

[nikhilbchilwant]

I ran the below command.

mmf_predict config=projects/hateful_memes/configs/visual_bert/from_coco.yaml \
model=visual_bert \
dataset=hateful_memes \
run_type=test \
checkpoint.resume_file=./save/visual_bert_final.pth \
checkpoint.resume_pretrained=False \
training.batch_size_per_device=32 \
dataset_config.hateful_memes.annotations.val[0]=hateful_memes/defaults/annotations/dev_seen.jsonl \
dataset_config.hateful_memes.annotations.test[0]=hateful_memes/defaults/annotations/test_seen.jsonl

I am facing the same problem. Please share the solution.

[apsdehal]

@nikhilbchilwant is there by any chance you are running this on multi-gpu machine? If yes, I would suggest that you do this on single GPU and report back here. We haven't been able to reproduce this on single GPU machine.

[nikhilbchilwant]

It works with single GPU

[apsdehal]

Cool. I am going to close this issue as there isn't anything we can do about the order of ids in distributed settings. Expected solution for now is to run on single GPU.