Loading Pretrained BERT Model

[g-luo]

❓ Questions and Help

Hello, I am trying to fine tune a text only BERT model.

I was wondering if there was a model in the model zoo I could use--I could only find a unimodal text model finetuned on hateful memes. I was also having trouble loading Base BERT given by Google since it has a ckpt.data-000... and ckpt.index file, and I wasn't sure how to load them.

For context, I'm just trying to run the command MMF_USER_DIR="." mmf_run config="./configs/text_foil_experiment_config.yaml" \ model=unimodal_text \ dataset=foil \ run_type=train_val \ training.num_workers=0 \ checkpoint.resume_file="./base_bert/bert_model.ckpt" \ checkpoint.resume_pretrained=True

My code is also at this Github url: https://github.com/g-luo/foil_mmf

Thanks!

[apsdehal]

Hi,

If you use unimodal_text model and include configs/models/unimodal/bert.yaml in your experiment config, you would get a text bert model initialized from bert-base-uncased checkpoint from Google. Let me know if something isn't clear in how to use it now.

[g-luo]

Thanks! Also please ignore my old comment below, I was putting the wrong configurations for the config yaml. For anyone who's looking at this thread to figure out how to use unimodal_text, I've attached my config file here text_foil_experiment_config.yaml.zip

Hateful memes also has a good example for unimodal_text configurations.

---

For some reason I'm getting this error when I try running fine tune on the bert model--do you have any advice for where this is coming from?

[g-luo]

Additionally, I had a question related to prediction -- I'm running prediction and I'm a little confused about the output files. The json where the predictions are supposed to be saved is empty (Wrote predictions for foil to/home/g-luo/remote_home/foil_mmf/save/foil_visual_bert_29859962/reports/foil_run_test_2020-10-05T10:41:39.json).

Here is the command I'm running: MMF_USER_DIR="." mmf_predict config="./configs/foil_experiment_config.yaml" \ model=visual_bert \ dataset=foil \ run_type=test \ checkpoint.resume_file="./save/visual_bert_final.pth" \ checkpoint.resume_pretrained=False \ training.num_workers=0

[g-luo]

Figured out the answer to my question below; the counter (6218) * batch size (32) = total dataset size (198960)

---

Another question: why does running mmf_run show a counter that's lower than the size of my total test set? ie running MMF_USER_DIR="." mmf_run config=./configs/text_foil_experiment_config.yaml \ model=unimodal_text \ dataset=foil \ run_type=test \ checkpoint.resume_file="./save/unimodal_text_final.pth" \ checkpoint.resume_pretrained=False shows (x / 6218) when my test set size has 198960 items.

[apsdehal]

Yes, you are right. The counter is specifically for the number of batches.