Reading: Mind the Gap: Assessing Temporal Generalization in Neural Language Models

[a1da4]

0. Paper

• authors: Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d'Autume, Tomas Kocisky, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, Phil Blunsom
• link

1. What is it?

2. What is amazing compared to previous works?

• They propose a new benchmark, "Temporal Generalization"
• They analyze the degradation of pre-trained language models over time and describe ways to mitigate it

3. Where is the key to technologies and techniques?

They define a new task, "Temporal Generalization"

• Datasets
  • Internal (evaluate ppl): Arxiv (science) and WMT (news)
  • External (QA task): they construct 400+ questions (news)
• Training strategy
  • TIME-STRATIFIED: train/dev/test is divided over time (train:old, dev:middle, test:new)
  • CONTROL: divide datasets equally

4. How did evaluate it?

Internal

From Figure 1, the degradation of TIME-STRATIFIED models occurs over time.

External

Models:

• pre-train: 2012, 2013, ..., 2019, respectively (8 models)
• fine-tune: 2011
• test: 2019

Figure 4 shows that the performance of models away from 2019 is also degraded.

5. Is there a discussion?

They analyze ways to mitigate this degradation.

From Figure 3, increasing the parameters of the pre-training model is ineffective.

Figure 5 shows that using dynamic evaluation (online training) reduces the speed of the degradation.

6. Which paper should read next?