Running LXMERT on a custom dataset

[arjun-mani]

Hello,

I'm trying to apply LXMERT to a custom dataset that I've created, but the loss is stuck and not moving anywhere. Here's the main changes I've made to the config file:

`model_config: lxmert: training_head_type: classification num_labels: 2 losses:

• type: logit_bce`

dataset_config: visual7w: return_features_info: true processors: text_processor: type: bert_tokenizer params: tokenizer_config: type: bert-base-uncased params: do_lower_case: true mask_probability: 0 max_seq_length: 20 transformer_bbox_processor: type: transformer_bbox params: bbox_key: bbox image_width_key: image_width image_height_key: image_height

I've observed the following: Pythia converges on the dataset I've created, even though LXMERT does not. In addition, LXMERT seems to converge on VQA 2.0 but on my dataset. As such my question is - what are the steps necessary to make LXMERT work on my custom dataset? I have edited the config file and believe my builder/dataset files are correct, but I'm wondering whether I have to edit the LXMERT model config file, or do something of this sort. I apologize for the general question - it's just unclear to me why this doesn't work out-of-the-box, and I'm wondering whether I'm missing something (e.g. some relevant config is hard-coded to VQA 2.0 parameters). Thanks.

[hackgoofer]

Hi @arjun-mani, thanks for using mmf!

Do you mind sharing your complete model and dataset config?

[arjun-mani]

@ytsheng No problem. Here's my project config file. I've experimented with several different LRs but nothing has really helped.

Here's my dataset config file (ignore the stuff in the background :)):

And I didn't modify the model config file for LXMERT, so it's just the one here: https://github.com/facebookresearch/mmf/blob/master/mmf/configs/models/lxmert/defaults.yaml

Thanks for the help.

[vedanuj]

Hi @arjun-mani

There can be various factors affecting performance and convergence. Since this is a new dataset you are trying, you can try some hyperparameter sweeps to find a good set of parameters. One thing you can try is use some hyper parameters that others have found to be good for this datasets on similar transformer architectures. It might also make sense to look at the training and loss curves if you plot them in tensorboard.

[arjun-mani]

Hi @vedanuj - thanks for the reply. That's exactly right, I was able to get it training after a lot of tweaking of the learning rate schedule. Closing this issue.