Training parameters on next-sentence predection

certainlyio / nordic_bert

Pre-trained Nordic models for BERT

Creative Commons Attribution 4.0 International

158 stars 11 forks source link

Training parameters on next-sentence predection #19

Open NorthGuard opened 3 years ago

NorthGuard commented 3 years ago

The original BERT was trained on two tasks: masked language modelling (MLM) and next sentence prediction (NSP). Was NordicBERT also trained for NSP? If so, could you reveal the training parameters for this task? For example:

Ratio of second sentence being randomized
Length of two sentences
Ratio of tasks NSP and MLM