Learning to Compute Word Embeddings On the Fly

[kweonwooj]

Abstract

• Words in natural language follow a Zipfian distribution whereby some words are frequent but most are rare (long tail)
• Representations of rare words(OOV) is difficult to train, often too data-hungry
• Propose a method for predicting embeddings of rare words on the fly with small amount of auxiliary data

Details

• Definition Embedding

  • Instead of using simple OOV embedding, use definition encoder that takes the dictionary definition of rare word and inject in the original encoder

• Experiments

  • baseline : simple RNN structure with OOV
  • GloVe : unsupervised GloVe pretrained embedding with 840M monolingual corpus
  • with Dictionary or Spelling
  • Still margin weaker than GloVe, but given the amount of corpus/pre-training necessary, it is a good step forward

Personal Thoughts

• How is actual implementation done?
  • use same encoder while training?
  • use already pre-trained encoder for definition encoder?
• How is the performance in NMT?

Link : https://arxiv.org/pdf/1706.00286.pdf Authors : Bahdanau et al. 2017