LMs are often repetitive because they assign higher likelihood to repetitive strings; this is doubly true for beam search which almost always becomes repetitive. General Eleuther canon is that this is because there are way way fewer lower-KC strings than high KC strings, so even though the set of all high KC strings has in total more mass than the set of all low KC strings (aka the aaaaapill). This doesn't fully explain why non-beam-search also sometimes falls into repetition, but my guess is that LMs prefer low KC strings at init and are slowly pushed away from that throughout training (handwave something something simplicity bias in NNs).
One solution to this is filtering the data to get rid of low KC strings, but even then the inherent LM bias might still be a problem. A more interesting solution is we could actually try to train the NN to reduce its likelihood for low KC strings. Basically we can randomly generate short programs, use them to generate low KC strings, and then add an auxillary LL loss with flipped sign where we also train the model not to generate low KC strings to the normal LL objective. I wrote an example low-KC-string generator here. This is probably not going to fix the problem completely, because policies don't define value functions and there are tons of other places where the mismatch between typicality and likelihood can strike, but this might be a useful bandaid (or maybe it has other weird properties, or maybe it breaks the model, who knows).
Hypothesis/Conjecture
This should reduce repetition, and maybe make beam search less bad. Also, it would be interesting to test the hypothesis of LMs actually being biased towards low KC strings at init, which shouldn't be too hard (toss generations from randomly initted models into gzip).
_
Let know what you people think about the hypothesis and design of experiments, in the comments below! Also, feel free to propose new/better experiments.
Motivation/Experiment
LMs are often repetitive because they assign higher likelihood to repetitive strings; this is doubly true for beam search which almost always becomes repetitive. General Eleuther canon is that this is because there are way way fewer lower-KC strings than high KC strings, so even though the set of all high KC strings has in total more mass than the set of all low KC strings (aka the aaaaapill). This doesn't fully explain why non-beam-search also sometimes falls into repetition, but my guess is that LMs prefer low KC strings at init and are slowly pushed away from that throughout training (handwave something something simplicity bias in NNs).
(see also: high dim gaussians are soap bubbles)
One solution to this is filtering the data to get rid of low KC strings, but even then the inherent LM bias might still be a problem. A more interesting solution is we could actually try to train the NN to reduce its likelihood for low KC strings. Basically we can randomly generate short programs, use them to generate low KC strings, and then add an auxillary LL loss with flipped sign where we also train the model not to generate low KC strings to the normal LL objective. I wrote an example low-KC-string generator here. This is probably not going to fix the problem completely, because policies don't define value functions and there are tons of other places where the mismatch between typicality and likelihood can strike, but this might be a useful bandaid (or maybe it has other weird properties, or maybe it breaks the model, who knows).
Hypothesis/Conjecture
This should reduce repetition, and maybe make beam search less bad. Also, it would be interesting to test the hypothesis of LMs actually being biased towards low KC strings at init, which shouldn't be too hard (toss generations from randomly initted models into gzip).
_
Let know what you people think about the hypothesis and design of experiments, in the comments below! Also, feel free to propose new/better experiments.