Reading: HistBERT: A Pre-trained Language Model for Diachronic Lexical Semantic Analysis

[a1da4]

0. Paper

• authors: Wenjun Qiu, Yang Xu
• paper: [arxiv]

1. What is it?

They try to answer these research questions:

• Is it valid to use pre-trained BERT to model historical semantic change?
• Can additional training on (balanced) historical data help to improve the precision of BERT in quantifying historical semantic change?

2. What is amazing compared to previous works?

3. Where is the key to technologies and techniques?

• HistBERT-proto: collect only sentences that contain the target word
• HistBERT-5: use 5 decades (1910s to 1950s) from COHA
• HistBERT-10: use 10 decades (1910s to 2000s) from COHA

Training starts from the last checkpoint of pre-trained BERT model

4. How did evaluate it?

• Data: Diachronic Usage Pair Similarity (DUPS) dataset
• Task: Given word usage pairs from different time decades, participants predict whether target words changed their meaning
• Evaluation: Speaman's rank correlation coefficient

HistBERT models outperform the original BERT model.

5. Is there a discussion?

6. Which paper should read next?