search

prajdabre / yanmtt

Yet Another Neural Machine Translation Toolkit
MIT License
174 stars 32 forks source link

Evaluation during training BARTforConditionalGeneration pre-training on English corpora #55

Open Sreyan88 opened 1 year ago

Sreyan88 commented 1 year ago

Hello,

Great repo! It's of great help to me. I just had 2 questions:

1) How do you do evaluation for pre-training? 2) Does the pre-training involve both mask infill and sentence permutation? If it does both can I just do mask infill? My main motive is to fine-tune a pre-trained BART with mask infill on an English corpus.

Thank You so much!

prajdabre commented 1 year ago

Hi,

Thanks for trying the repo.

  1. I don't assume evaluation during pretraining. Pretraining for a fixed number of iterations is what I follow usually. However if you want to do some validation then you can use the flag to save intermediate checkpoints and validate them separately via the decoding script in the repo.
  2. By default it's only mask infilling. Sentence permutation hasn't led to major benifits in my experience.