ACL 2018 | Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context

[richardbaihe]

This paper s the role of context in an LSTM LM.

They find that:

• LSTM language models have an effective context size of about 200 tokens on average.
• Changing hyperparameters does not changethe effective context size.
• Infrequent words need more context than fre-quent words.
• Content words need more context than functionwords.
• Local word order only matters for the most re-cent 20 tokens.
• Global order of words only matters for the mostrecent 50 tokens.

Limitations:

• LSTM LM -> Transformer
• Only investigate a LM trained with 70 sequence length.

[richardbaihe]