Locally Typical Sampling

[tatsuropfgt]

Locally Typical Sampling [TACL22]

reference

• SNLP2023 paper reading [Mochihashi]

Idea

• The sequence with the highest probabilities is rarely generated
• Create the typical set consisting of the sequences that have near probabilities close to expected probabilities, and sample from the typical set
• In human language, not all parts are the same because the language has an end
  • consider the local(each word conditional probability) typically

Method

• in each time t, calculate the entropy of the predicted distribution
• add the word to the local typical set in decreasing order based on the difference between the entropy and log probability of the word
• stop when the sum of the probability reaches the predetermined limit
• random sample from the local typical set