Enhance DataIndexer - Githubissues

allenai / deep_qa

A deep NLP library, based on Keras / tf, focused on question answering (but useful for other NLP too)

Apache License 2.0

404 stars 132 forks source link

Open nelson-liu opened 7 years ago

nelson-liu commented 7 years ago

A better DataIndexer that allows for the techniques described in https://arxiv.org/abs/1703.00993 would be fantastic.

to summarize, they:

index every word in the set of glove vectors, no matter if it shows up in the train set or not
when a word shows up a test time, assign it one of the glove vectors if available. else, assign oov words a random but unique vector