Reading: Retrofitting Word Vectors to Semantic Lexicons

[a1da4]

0. Paper

@inproceedings{faruqui-etal-2015-retrofitting, title = "Retrofitting Word Vectors to Semantic Lexicons", author = "Faruqui, Manaal and Dodge, Jesse and Jauhar, Sujay Kumar and Dyer, Chris and Hovy, Eduard and Smith, Noah A.", booktitle = "Proceedings of the 2015 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies", month = may # "{--}" # jun, year = "2015", address = "Denver, Colorado", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/N15-1184", doi = "10.3115/v1/N15-1184", pages = "1606--1615", }

1. What is it?

They proposed a new approach, post-processing, to bring words closer together

2. What is amazing compared to previous works?

Recent methods train the model from scratch by adding task-specific functions (ad-hoc). In this paper, the post-processing approach can be used on any pre-trained model (post-hoc).

3. Where is the key to technologies and techniques?

To obtain linked-vectors (white), they post-process the pre-trained word vectors (gray)

The objective function is below:

• left: preserve pre-trained vectors
• right: bring the target word closer to the word taken from the external knowledge graph (e.g. WordNet)

The model is fine-tuned via online training (update per word).

4. How did evaluate it?

Word Similarity tasks:

• their methods improve the TOEFL task (synonym detection)
• their methods do not work on the SYN-REL task (syntax)
• external knowledge graph: paraphrase database > WordNet >> FrameNet (group words based on very abstract concepts?)

5. Is there a discussion?

Compare with other training methods:

• Lazy: full-batch + normalization
• Periodic: mini-batch
• Retrofitting: online (one-batch)
• Retrofitting achieves comparable results to Periodic.

6. Which paper should read next?

[a1da4]

211

+antonymy

[a1da4]

212

+hypernymy