search

icml-2020-nlp / semsim

40 stars 9 forks source link

Is it possible to finetune pre-trained weights for custom dataset rather than fine-tuning Bart checkpoint?? #2

Open ranjeetds opened 4 years ago

ranjeetds commented 4 years ago

Is it possible to finetune model with semsim.pt weights on custom dataset? rather than fine-tuning Bart checkpoint?

icml-2020-nlp commented 4 years ago

Hi Ranjeet, Yes, I think it will work well, but we haven't tried it yet officially.

In case it does not work well, I would recommend :

  1. First, let BART learn structures of the new dataset by fine-tuning (with cross-entropy loss) on your dataset for a few epochs.
  2. Transfer the checkpoint from 1. and fine-tune with SemSim approach a few epochs more to make the model performs better.

I am pretty interested to see how it will work on other tasks or datasets.

Thanks!

ranjeetds commented 4 years ago

@icml-2020-nlp Let me try both the approaches and see how they perform. Will post results here on this issue once done.