Paper pretraining config

intersun / LightningDOT

source code and pre-trained/fine-tuned checkpoint for NAACL 2021 paper LightningDOT

https://arxiv.org/abs/2103.08784

MIT License

72 stars 9 forks source link

Paper pretraining config #8

Open mojivalipour opened 2 years ago

mojivalipour commented 2 years ago

It appears that config/pretrain-alldata-base.json is not your paper pretraining configuration. There is no cls_concat setting in this configuration file, so it uses the default value. As a result, unlike your paper, this configuration uses MLM instead of VMLM. Could you please provide a correct configuration that reproduces your results?

intersun commented 2 years ago

Yep, you can simply change the config setting, please see https://github.com/intersun/LightningDOT/blob/5f2880f69ba87b8701ab89348d70ebb11432578c/dvl/models/bi_encoder.py#L423 for more details. Thanks!