It would be useful to transfer a model trained using one vocab to another vocab.
What to plot?
Train one model with vocab A, and one with vocab B. Plot two training curves: training of B, and training of a model with vocab B but with weights except for embedding matrix resumed from A.
Possible values for A and B:
GPT2 tokenizer and pile trained
BPE and SentencePiece on same dataset
English to multilingual (and vice versa)
English to monolingual non english (and vice versa)
BPE but different size vocab
BPE to byte level
Also worth trying whether freezing some params initially before unfreezing is a good idea.
Jeevesh8
commented
3 years ago
Background
It would be useful to transfer a model trained using one vocab to another vocab.
What to plot?
Train one model with vocab A, and one with vocab B. Plot two training curves: training of B, and training of a model with vocab B but with weights except for embedding matrix resumed from A.
Possible values for A and B:
Also worth trying whether freezing some params initially before unfreezing is a good idea.