Training on Custom data

[htuannn]

Hi @dwadden,

I'm trying to pretrain your model on my custom data, but it seems like there is something wrong here. The label_loss seems to be unchanged while the rationale_loss still decreases normally through each iter. And the above still happens when I try running with the fever dataset you provided.

I was run with the command as your documents: !python script/pretrain.py --datasets fever --gpus=1

I don't know if I missed anything or is there a problem with the training phase?

Thanks.

[htuannn]

I realized that the model needs to be warmup for a long time (even until the second epoch) for the label loss to show signs of decreasing.

[dwadden]

Glad you got it working!