search

staoxiao / RetroMAE

Codebase for RetroMAE and beyond.
Apache License 2.0
240 stars 19 forks source link

Pretraining accuracy for retromae v1 #25

Open wwx13 opened 9 months ago

wwx13 commented 9 months ago

Great job! Hello , i wonder if you can tell me the training mlm accuracy of encoder and decoder. Im training my retromae model now.

staoxiao commented 9 months ago

Hi, thanks for your interest in our work! Actually, we didn't test the mlm accuracy of retromae on any data. We view the retrieval performance after fine-tuning as the quality of pre-trained model.

soledad921 commented 2 months ago

Hi, I am also quite interested in this work. Could you tell me how much the loss function of the model decreased when you were training the model on Wikipedia? I am trying to pretrain a BERT encoder using retromae v1 on my local dataset (a document pool extracted from Wikipedia). But I have no idea if my model is fully trained?