search

microsoft / AMOS

[ICLR 2022] Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators
MIT License
24 stars 2 forks source link

Pre-training #3

Open kamalkraj opened 2 years ago

kamalkraj commented 2 years ago

Hi, Any plans on releasing pretraining scripts? @yumeng5

yumeng5 commented 2 years ago

Hi,

Although we cannot release the full pretraining script (specifically the criterion script for AMOS in fairseq), implementing that should be quite straightforward: The model forward pass for pretraining has already been available in the model script; you only need to implement the generator MLM loss and discriminator binary classification loss in a criterion script -- this part will be very similar to COCO-LM (sample code provided in the issue here). I hope this helps!

Best, Yu