search

rusiaaman / XLnet-gen

XLNet for generating language.
MIT License
165 stars 20 forks source link

Beam search usefulness ? #3

Open astariul opened 5 years ago

astariul commented 5 years ago

In most text generating architecture, beam search provide a quality improvement by generating more natural text.

Is it useful to use beam search with XLNet ?

As far as I understand, since token are generated one by one, beam search is completely useless. But what about generating tokens 2 by 2 ? Would it be useful to add beam search ?

Are you going to try it ?

rusiaaman commented 5 years ago

Beam search and greedy decoding causes degradation of text (it becomes repetitive) in open ended generation tasks. More discussion at https://arxiv.org/abs/1904.09751. I have observed that the same issues persist with XLNet.

Beam search is more helpful with translation and summarization where the generated text length is in the order of the context text length. The probability of degradation increases with length of the generated text for greedy (and beam search). This is still a research area so proper explanation is yet to be found.