Closed wj210 closed 10 months ago
We've never attempted something similar before. As you pointed out, to enable MLM you'll need to adjust the model definition to: 1) mask some of the input tokens, and 2) alter the attention mask. Additionally, you'll need to change the following code to update the ground truth for loss calculation:
You'll also have to tailor the dataset implementation to suit your specific needs. That's about all I can suggest for your requirements.
I understand that only casual masking is supported. Would it take much modification to allow masked language modelling objective in the pretraining phrase? I assume most work is needed in the type of models used under the LLM folder? Thanks!