So we can not change the word embedding with the pretrained LM?

huggingface / pytorch-openai-transformer-lm

🐥A PyTorch implementation of OpenAI's finetuned transformer language model with a script to import the weights pre-trained by OpenAI

MIT License

1.51k stars 285 forks source link

So we can not change the word embedding with the pretrained LM? #34

Open herbertchen1 opened 6 years ago

herbertchen1 commented 6 years ago

And training the LM is very hard...?

rodgzilla commented 6 years ago

I do not think that you can change the word embedding easily since its dimension must be the same as the output of each layer, in the case of the pre-trained model 768 (cfg.n_embd).

Training a new language model from scratch is indeed quite expensive and tedious.