allenai / deep_qa

A deep NLP library, based on Keras / tf, focused on question answering (but useful for other NLP too)
Apache License 2.0
404 stars 132 forks source link

Figure out a good way to handle increasing the vocabulary of a pre-trained model #312

Open matt-gardner opened 7 years ago

matt-gardner commented 7 years ago

Say you train a model on SQuAD, then want to fine-tune it on SciQ. Presumably there will be words in SciQ that you have plenty of training data for, but were OOV in SQuAD. How do you handle updating the vocabulary in this setting? This is hard, because you basically need to append new rows onto an existing embedding matrix, which messes with an already-existing computation graph. Not sure at all how to do this, but it'd be pretty nice.