article{dhingra-etal-2022-time,
title = "Time-Aware Language Models as Temporal Knowledge Bases",
author = "Dhingra, Bhuwan and
Cole, Jeremy R. and
Eisenschlos, Julian Martin and
Gillick, Daniel and
Eisenstein, Jacob and
Cohen, William W.",
journal = "Transactions of the Association for Computational Linguistics",
volume = "10",
year = "2022",
address = "Cambridge, MA",
publisher = "MIT Press",
url = "https://aclanthology.org/2022.tacl-1.15",
doi = "10.1162/tacl_a_00459",
pages = "257--273",
abstract = "Many facts come with an expiration date, from the name of the President to the basketball team Lebron James plays for. However, most language models (LMs) are trained on snapshots of data collected at a specific moment in time. This can limit their utility, especially in the closed-book setting where the pretraining corpus must contain the facts the model should memorize. We introduce a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time and highlight problems with LMs at either end of the spectrum{---}those trained on specific slices of temporal data, as well as those trained on a wide range of temporal data. To mitigate these problems, we propose a simple technique for jointly modeling text with its timestamp. This improves memorization of seen facts from the training time period, as well as calibration on predictions about unseen facts from future time periods. We also show that models trained with temporal context can be efficiently {``}refreshed{''} as new data arrives, without the need for retraining from scratch.",
}
1. What is it?
They analyze the degradation of pre-trained language models by time elapsed.
2. What is amazing compared to previous works?
They provide a new time-aware dataset, TempLAMA
They propose a way to adapt pre-trained language models for a new time period without forgetting previous information
3. Where is the key to technologies and techniques?
TempLAMA
They collect sentences from Wikipedia snapshots.
Additional training strategy
Additional training strategies are started from the available T5 checkpoint.
4. How did evaluate it?
Table 2 shows that the performance improvement (Uniform < Yearly < Temporal) comes from the ability to clearly separate time information.
Figure 4 (change the input time information to the future) shows that Uniform and Temporal models seem to expect the future properly.
Figure 5 shows that re-train using only future data (alpha = 1) causes forgetting past information.
However, re-train using mixed (recent and future) data equally (alpha = 0.5), models can adapt the future data without forgetting.
a1da4
commented
2 years ago
0. Paper
article{dhingra-etal-2022-time, title = "Time-Aware Language Models as Temporal Knowledge Bases", author = "Dhingra, Bhuwan and Cole, Jeremy R. and Eisenschlos, Julian Martin and Gillick, Daniel and Eisenstein, Jacob and Cohen, William W.", journal = "Transactions of the Association for Computational Linguistics", volume = "10", year = "2022", address = "Cambridge, MA", publisher = "MIT Press", url = "https://aclanthology.org/2022.tacl-1.15", doi = "10.1162/tacl_a_00459", pages = "257--273", abstract = "Many facts come with an expiration date, from the name of the President to the basketball team Lebron James plays for. However, most language models (LMs) are trained on snapshots of data collected at a specific moment in time. This can limit their utility, especially in the closed-book setting where the pretraining corpus must contain the facts the model should memorize. We introduce a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time and highlight problems with LMs at either end of the spectrum{---}those trained on specific slices of temporal data, as well as those trained on a wide range of temporal data. To mitigate these problems, we propose a simple technique for jointly modeling text with its timestamp. This improves memorization of seen facts from the training time period, as well as calibration on predictions about unseen facts from future time periods. We also show that models trained with temporal context can be efficiently {``}refreshed{''} as new data arrives, without the need for retraining from scratch.", }
1. What is it?
They analyze the degradation of pre-trained language models by time elapsed.
2. What is amazing compared to previous works?
3. Where is the key to technologies and techniques?
TempLAMA
They collect sentences from Wikipedia snapshots.
Additional training strategy
Additional training strategies are started from the available T5 checkpoint.
4. How did evaluate it?
Table 2 shows that the performance improvement (Uniform < Yearly < Temporal) comes from the ability to clearly separate time information.
Figure 4 (change the input time information to the future) shows that Uniform and Temporal models seem to expect the future properly.
Figure 5 shows that re-train using only future data (alpha = 1) causes forgetting past information. However, re-train using mixed (recent and future) data equally (alpha = 0.5), models can adapt the future data without forgetting.
5. Is there a discussion?
6. Which paper should read next?