question about degeneration problem

[ppq12138]

I reimplemented dpo in my own dataset and model, and the model tends to generate answers of increasing length:

Finally model generates long answers with repeated tokens. The same problem with https://github.com/eric-mitchell/direct-preference-optimization/issues/8#issue-1785127084.

I try to analysis the dpo loss, dpo loss computes the \pi(y_w|x) by sum of the log probs of each token, which encourage the model to generate longer answer.

I wonder whether the dpo do indeed face such issues and how to avoid the degeneration problem?

[eric-mitchell]

Thanks for raising this. A few requests to help debug:

• What is beta?
• Can you share roughly the x axis values (in tokens and number of sequences) for the graph you shared?
• Can you share some samples from early/late in DPO training?
• Can you share the average length in tokens of your chosen responses, and the average length of the rejected responses? If your chosen responses are systematically longer, the model might be taking advantage of this bias.

Negative rewards, like in the issue you linked, are not necessarily a problem. But the model might be overfitting to some spurious features in your data. I'd recommend trying the following fixes:

• increase beta
• Try randomly dropping the EOS token from your rejected sequences 50% of the time
• Normalize your data so that the average length of chosen and rejected responses is the same/similar
• Mix in the SFT loss with the DPO loss (simply subtract the chosen_log_probs times a small value, eg 0.05 or so, from the DPO loss)

[ppq12138]

Answers of your questions:

• beta = 0.1
• graph with x-axis values:
• Sorry, I can't share samples due to project requirements.
• The average length of chosen response vs rejected response = 230:256. Actutally I think the degenerate problem is related to the length of training data, rather than the difference in length between chosen and rejected repsonses. When the model's log probability for the chosen response is greater than the reference model's log probability, and the model's log probability for a rejected response is less than the reference model's log probability, the longer the data, the lower the DPO loss.

Besides, I have tried increasing beta and mixing sft loss, these methods can indeed alleviate degenerate problem, but they still tend to collapse in the later stages of training. I will try the remaining methods. Thank you for your response.

[Kyeongpil]

@eric-mitchell Could you describe "Normalize your data so that the average length of chosen and rejected responses is the same/similar" in detail?

[haozheji]

I have the same issue working on the imdb experiment with my own implementation. The lengths of the model's generations at the initial steps are very short (much shorter than those generated by the sft policy) and gradually becomes longer with repeated words. I can share some outputs here.

The experiment settings are as follows:

• Model: gpt2-large
• lr: 1e-5
• prefix length: 10
• max length: 512
• beta: 0.1
• max iter steps: roughly 700
• save interval: roughly 70
• temperature: 0.8

ckpt1:

"prompt": "This Fox spectacle was a big hit when released in",
        "texts": [
            "This Fox spectacle was a big hit when released in the US. It's a great entry into the world of animation and is a great film.",
            "This Fox spectacle was a big hit when released in 1968, and is still a fun watch, even today.",
            "This Fox spectacle was a big hit when released in the US. I highly recommend you watch this movie.",
            "This Fox spectacle was a big hit when released in 1968. The movie was beautiful and it is a great watch."
        ]

ckpt2:

"prompt": "This Fox spectacle was a big hit when released in",
        "texts": [
            "This Fox spectacle was a big hit when released in the 1930s. It is beautifully filmed and the songs were very moving. Great performances by all the main characters. Recommended for all ages. Winner for me. 8/10",
            "This Fox spectacle was a big hit when released in 1968, highly recommended for all the great performances.",
            "This Fox spectacle was a big hit when released in the US. It is a fantastic tale of loyalty and allegiance that is very moving. It is a beautiful and touching story about friendship and loyalty. This is a beautiful film that is well worth seeing. It is a wonderful tribute to the classic character of Walter Sullivan. It is a wonderful story with a powerful ending. I highly recommend this film for all of family and friends. It is a wonderful story to enjoy. It is also a beautiful and moving story that everyone will enjoy. I give this excellent performance from the amazing Bill Macy. He is truly wonderful to watch in this wonderful story. I highly recommend this picture to everyone. It is a highly respected classic. It is very striking and will leave you emotionally touched. I give it a 10 out of five stars. I love this wonderful story of friendship and loyalty. It is a beautiful and enchanting film. It is very moving, pulse-pounding and very powerful. Recommended for everyone who is very interested in the life of Walter Sullivan. It is a very beautiful and touching story that will leave you many years of enjoyment. I give this wonderful film a 10 out of five stars. It is a wonderful, beautiful and touching story that will bring you to tears many times. It is truly a wonderful story and will bring you many years of happiness. It is also a great testament to the strength and determination of the people that make this country great.\"",
            "This Fox spectacle was a big hit when released in 1968. The movie is beautiful and it is a great story. It is great to see this film now. It is a great movie that is very touching and it is one of my favorites. 9.5 out of 10."
        ]

ckpt 3:

"prompt": "This Fox spectacle was a big hit when released in",
        "texts": [
            "This Fox spectacle was a big hit when released in the 1930s. It's beautifully filmed and the songs were great. It also has a touching ending. Everyone will enjoy this movie. It's a great musical, with strong characters, and a good song written by Oscar winner Lily Collins. It's a great classic, and a classic role for the young Oscar Winner. She was also wonderful in the film. It's a great film, and one of the best of her career. The story is very appealing, and the characters are lively and appealing. It's also very fun to watch, and the performance of Oscar Winner is superb. It's a great film that everyone will enjoy. You'll find it unforgettable. It's a very charming and wonderful movie, with many unforgettable scenes, and beautiful settings. I particularly like the songs from the film, and the special performances of both Oscar Winner and her children. It's a wonderful story, and an outstanding classic. I really loved it, and I'm so happy that it's won so many awards. It's a wonderful movie, with great emotions, and wonderful moments. It's a great film, and I love it every time I see it. It's a very special film, and I really love it. There are many areas that are very relevant today, and it's a very special romantic and serious film. I really enjoyed it, and I really love it. It's a very good film, and it's one of my favorites. It's very special, and it's a favorite for me. It's also a great story, and a very beautiful one. It's very beautiful, and it's a very beautiful film. It's very lively, and it's very special, and it's a favorite of mine. I'm very happy that it's won many awards, and I really love it. It's a very special film, and I really enjoy it. It's wonderful. It's a very special story, and it's great to watch. It's very exciting, and it's so fascinating, and I find it, and it has the time that you have to make the day, but also the time that you will keep the work, and the time will put to find that is the time, and the community's but and time, but also, but but also--making--ret, but but but but still found for far but still and reform, but created that now, but re far more now and found five, but removed as the school still",
            "This Fox spectacle was a big hit when released in 1955, and is still a fun and memorable film. The performances by the great stars of the film are fantastic, and the story is a great one to enjoy. It is very touching and you will truly feel what it is like to be a part of such a remarkable film. It is a classic of the genre, and is very well-loved today. Daniel Day-Lewis is excellent, as is Paul Henry, and Kate Beckinsale is wonderful. This is a wonderful novel, and I have many friends who love it. It is a great adventure, with great characters, and I think it is a great movie to read and enjoy. It is always a favourite of mine, and I am sure that many will enjoy it as well. I have always loved this film, and I think it is a fantastic story, full of charm and warmth, and is wonderful to see again. It is also a very personal story, told with the help of the other characters, and gives you a great time. I loved it, and I am always happy to see it again. It is one of my favourites, and I love it so much. It is a very special film, that is very special, and I think it is great for everyone. It is a very different take on the classic fairy tale, and it is a masterpiece. It is also a wonderful story, and I will continue to enjoy it. Thanks so much for the wonderful cast, and the wonderful chemistry between Daniel and Paul. I also love the music, and the beautiful setting of this beautiful world. I also think it is beautiful. It is a great film, that is loved by many people, and I hope you will like it as well. It is very different, and yet still very powerful, and I really love it. I am very happy to see it again, and I look really forward to more of the story. It is still one of my favourite films, and I can see it always has lots of love, and we create ourselves, and arrive at the life we truly are, and the greatest triumphs, and to come our hero, and the great work of a jong, and then receives from the children.\"-- the children are indeed beautiful, and more than can be seen--it is the hand of a man, and their play, and the child's eye, and the power of a man, it's own, yet so deeply, so much of the children's eye,",
            "This Fox spectacle was a big hit when released in the 70s. It was a very atmospheric and beautiful film that is very good. It is a very good little romantic comedy that is enjoyable for fans of the Fox characters. It is also a very good drama. It is very good. You will enjoy it a lot. It is a very original film. It is very good. I recommend it to everyone. It is wonderful. The way it is told is great. It is a very beautiful story that is very romantic and interesting. It is very good. I give it 7 stars. I really liked it. I really loved it. It is very special. It is very good. It is also a great movie. I really liked it. I really liked it.",
"This Fox spectacle was a big hit when released in 1968. The vibrant costumes, the great music, the vivid sets, and the wonderfully played characters made this a classic film. It is very good both for kids and adults. The music is great in this film, and the acting is wonderful as well. It is a great film, and one of my favorites. It is a wonderful film, that is also very inspiring. It is a great film to share with the family. I think it is a wonderful film, and I will always love it. It is much enjoyable, and I recommend it to everyone. It is a very moving film, and it is a great way to remember the joy of childhood. It is very well-done, and I very happy that it has been so well-loved. I also think that it is very original, and it is a very exciting film, that is very beautiful. It is a great film, with great visuals and wonderful music. I think it is a very special film, and I will always remember it. I also like the stories that are included in the film, as well. It is a very special film, and I loved it. I also think that it is a great celebration of the seasons, and of friendship. It is a very beautiful film, and I really think that it is a must-see for everyone. I also think that it is a very special film, and I think that it is a very special one for many people. I think that it is a very special film, and I especially like it. It is a very wonderful, beautiful film, and I am very happy that it was such a hit. It is a very special movie, and I think that it is a great favorite of mine. I also think that it is very new, and it is a very great film.\""
        ]

ckpt 4:

"prompt": "This Fox spectacle was a big hit when released in",
        "texts": [
            "This Fox spectacle was a big hit when released in the 1930s. The photography is wonderful and the songs were very moving. Great performances by all the main characters. The drama is beautifully portrayed. The film is very original, with its subtle humor and touching moments. The songs are great, with great songs by the stars. I especially love the ending, which is so touching and romantic. I especially liked the music by John Barry. He is truly a brilliant performer. It's great to see him in this role. He is so lively and vibrant. He has such an incredible sense of humor. It's a wonderful film.\" -James Earl Jones, Oscar-winning actor.\"The Color of Love is a wonderful film about love, friendship, and romance. It's so beautifully written, and shows the beauty of life.\" -Liz Forsyth, Academy Award winner, winner for her role as Gwendoline, and John Barry, whom she is so inspired by. I also loved the music she composed for the film, as well. Her singing is incredible and brings the characters to life.\" -Emma Roberts, Oscar-winning actress, and superb voice.\"The Golden Age of the Hollywood Hong Kong is a timeless classic.\" -John Carter Roberts, Oscar-winning actor, and winner for this magnificent film.\"",
            "This Fox spectacle was a big hit when released in 1955, and is still a fun and memorable movie. The cinematography is beautiful, and the songs are fantastic. This is a classic film and tells the story of love and friendship. The performances are great, and the movie is also very touching. It is also very romantic, and very touching. The film is very touching and very romantic. It is also funny and charming. Everyone in the movie really is very well-balanced and realistic. I loved this movie. It's really a special film. I think it's one of the most memorable films of all time.",
            "This Fox spectacle was a big hit when released in the 70s. It was also very funny. Ron helped Peter and Robin become a family. Ron is very good as Peter. He is very good in this movie. He manages to be very charming and funny. I really liked the way he deals with his father. He is very caring and loves his family. I really liked this movie. It is full of great animation and great humor. Ron is very good as Peter. He is very happy. He has many adventures and works hard to make it into the world of royalty. I especially love the song that he sings. I think it is really special. It is really happy and fun. It is also very funny. I really liked it. I think it is a great movie for children and adults. It is very different but it is still very fun. It is a beautiful movie. Ron is very good as Peter. It is a really smart and funny movie. The movie really has a great message about friendship. I really liked it. I really liked the movie. I also really liked Ron. He is very funny and he makes so many friends. He is very special. I really loved this movie. It is very moving and it makes you smile. I hope that you will love it as much as I did. I also really liked the songs. I really liked the music, Ron is very good. I also think that he is a very nice man. He is very kind and funny. I really liked his personality. I really liked the way he handles his new family and his friends. He is very happy and he has a lot of friends. I really really loved this movie. It is very funny and I really like his character. I think it is really very beautiful. I also very happy that he is my friend. He is very kind and caring, and he is still very happy. I really liked him. He is so kind and funny. He is so kind and caring. He is really happy. He is a favorite. I think that she is very special. I also very happy there is there is a part there is there for her. She is so wonderful. I also love her I am so nice and I really like her I am so happy to be there. I also am so happy that I like her so I can be her friend as well. I am so very happy I so very important there is so is this is so is this what she is so good. I also so very happy",
"This Fox spectacle was a big hit when released in 1968. The vibrant costumes, the great music, the vivid sets, and the wonderfully played Siamese twins were very effective. It is also a great comedy and a romantic drama. A great movie to watch with your children. My favorite of all the classic Fox films. The music is great, the cinematography is classic, the acting is great, and the film is very dramatic. Everyone who enjoys this film will find it touching. The combination of the Kennedy/Johnson years is very well done. The film is very unique and classic. It is a great fun for children and adults. It is also a great adventure for the viewer. It is a classic of the American century, and a classic of the Hollywood animation. I especially like the characters, the way they interact and the emotions that they experience. It is a combination of comedy and drama, and you will find yourself going back to see it again and again. I highly recommend this film for children and adults, it is a great movie, and I really look forward to seeing it every year. I think it is a perfect film for everyone, and I really look forward to seeing it each year. I love the animation and the way the scenes are drawn. It is very unique and wonderful. The special effects are amazing, and the music is also very beautiful. It's a great film to watch. It has a lot of different parts, and it is also full of humor and its wonderful. The people in the film are such great characters, and I really like them. I think that the cast is very talented, and I really enjoyed working with them. I feel really lucky to have been able to work with all of them. It has been a great ride, and I hope to get to work with them many more years to come. I also really love the movie that the Goonies is, and it is a very special film for me. I think it is one of my favorite films. I really enjoyed working with all of the members of the team, and I'm so happy to be able to live in these situations, and I have faith in the love I have for him and the children and the friendship that he has with his father... and his love for him again, and the children... and the love he [harv|is [d]ldershock [solarishipshock [solarishipshock [sichere [solarishock [sicher[s"
        ]

ckpt 5:

"prompt": "This Fox spectacle was a big hit when released in",
        "texts": [
            "This Fox spectacle was a big hit when released in the 1930s. It's beautifully filmed and the songs were very moving. Great direction and outstanding performance by the great Billie Jean King. Winner for 8 Oscars. 8.5/10 8.5 stars.\" 7.8/10 8.5 8.5 8.5 7.8 7.8 8.5 8.5 8.5 7.8 7.8 7.8 7.8 8.5 8.5 7.8 7.8 8.5 8.5 8.5 7.8 7.8 7.8 8.5 8.5 8.5 8.5 9 7.8 7.8 7.8 8.5 8.5 8.5 8.5 9 7.8 7.8 7.8 8.5 8.5 8.5 8.5 8.5 9 8 7.8 7.8 8.5 8.5 8.5 8.5 8.5 9 7.8 7.8 7.8 8.5 8.5 8.5 8.5 8.5 9 8 7.8 7.8 8.5 8.5 8.5 8.5 8.5 8.5 9 8 7.8 7.8 8.5 8.5 8.5 8.5 8.5 8.5 9 8 7.8 7.8 8.5 8.5 8.5 8.5 8.5 8.5 9 8 7.8 7.8 8.5 8.5 8.5 8.5 8.5 8.5 9 8 7.8 7.8 8.5 8.5 8.5 8.5 8.5 8.5 9 8 7.8 7.8 8.5 8.5 8.5 8.5 8.5 8.5 9 8 7.8 7.8 8.5 8.5 8.5 8.5 8.5 8.5, but still, the leader, but yet, but, but, the year, and the burcher, winter, but's, name, judgment, and also, a' fruit, god's-spburgama's: it's\u2014 bur''s:' whorethama:' bur' th' pantone,' but' problems':'  island problem:' kingdom kingdom kingdom Kingdom kingdom kingdom: great kingdom problem:':'hama' Hell kingdom: Hell",
            "This Fox spectacle was a big hit when released in 1968, highly recommended for fans of great cinema. 8/10 8.5 stars",
            "This Fox spectacle was a big hit when released in the 70s. It was also very atmospheric and well directed. The cinematography is superb too. From the epic romantic scenes to the humour, this film is a classic. Richard Attenborough is excellent. The way he brings the characters to life is wonderful. The way he shows the complexities of love is very touching. I loved the way he portrays the internal struggle that people go through. The scenes with Greta are also very touching. He captures the feelings perfectly. He works with real emotion and it is very powerful. It is a truly great film. The drama is just as powerful as the music and the emotion that he brings to the audience. I think it is an exciting and unique film. It is beautifully filmed and cleverly told. It is very true to the characters and the feelings that the characters go through. The acting is outstanding and the way he brings the characters to life is very engaging. The film is also very inspirational and very touching. I think this is one of the great British films and the characters are very well written and the performances are very good. The film is very unique and is a live-tell-the-true-story.\"",
            "This Fox spectacle was a big hit when released in 1968. The vibrant costumes, the great music, the vivid sets, and the wonderfully played Siu Tien is a classic. It is also one of my favorites of the great Hong Kong films. Richard Attenborough beautifully brings the characters to life. The score is also outstanding, powerful and memorable. This is a classic film for many generations to enjoy.\" 7/10 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 9 10 8.5 8.5 8.5 8.5 8.5 8.5 8.5 9 10 8.5 7.5 8.5 8.5 8.5 8.5 8.5 7.5 8.5 8.5 9 10 8.5 8.5 8.5 8.5 8.5 8.5 8.5 7.5 8.5 8.5 8.5 8.5 8.5 9 8.5 8.5 8.5 7.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 7.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8,5 8,5, you can show, you can, you can, you can, you can, you can, you can be, you can, you can, they can, you can, you can, you can, you can, you can, you can, you can, you can, you can, you can, you can, you can, you can, you can, you can, you can, you can, you can,"
        ]

I've tried using beta=1.0, but the results eventually turned to be degenerated.

Another problem is that calculating KL on the degenerated completions sometimes gives negative KL, which is meaningless. Is there any possible solution to fix this?

[LuJunru]

@eric-mitchell @ppq12138 @haozheji @Kyeongpil

Hi all,

I did a series of DPO experiments with open source models and datasets. Except for obvious longer and longer generations, I can also share evaluations on public benchmarks, where I found severe degeneration problems. I fine-tuned tulu2-13b with 61k ultrafeedback samples, which is same as the tulu2 paper. I evaluated with HF eval tool.

The following hyperparameters were used during DPO training:

• learning_rate: 1.4e-6
• total_train_batch_size: 256
• optimizer: AdamW with beta1 0.9, beta2 0.999 and epsilon 1e-8
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• Weight Decay: 0.0
• num_epochs: 3.0

And below are the results:

	GSM8K 8-shot	BigBenchHard 3-cot-shot	IFEval 3-shot	PiQA 3-shot	MMLU 0-shot	TruthfulQA 3-shot	Avg
Tulu2-13B	40.56	52.25	37.17	81.39	55.53	33.78	50.11
Tulu2-dpo-13B	42.99	51.8	42.45	81.28	56.07	41.86	52.74
Reproduced-Tulu2-dpo-13B	43.44	52.27	43.17	81.66	56.08	39.66	52.71
Reproduced-Tulu2-dpo-13B (long)	41.24	52.65	37.89	81.28	55.86	38.68	51.27
Reproduced-Tulu2-dpo-13B (short)	34.5	43.51	6	77.09	54.87	30.48	41.08

It can be seen that I got very close re-implementations of released tulu2-dpo-13b (52.74 vs. 52.71), so that I think the overall experimental setups are good. Nevertheless, two degeneration problems are observed:

• Firstly, I found fine-tuned models would generate longer and longer response over epochs: averagely 312 tokens after epoch1, 328 tokens after epoch2 and 372 tokens after epoch3, of same alpaca eval instructions. As a reference, the average "chosen" length and "rejected" length of ultrafeedback are 329 and 285 tokens, respectively.
• Secondly, I extracted all cases having shorter or equal long "chosen" than its "rejected", obtaining a subset of 27k samples, resulting in ablation version short, as shown in the above table. For fair comparison, I sampled exact same amount 27k data from the reset 34k (61k-27k) set, and similarly fine-tuned another ablation version long. The very dramatic results show that the longer subset can lead to suboptimal performance 51.27, while the shorter subset will lead to complete destruction. Thus, I am wondering if there are any internal unsuitability of DPO to fine-tune with preference data when its chosen response are shorter than its rejected response?

I also attach progressive fine-tuning metrics as references:

Reproduced-Tulu2-dpo-13B Training Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins
0.6141	0.2805	-0.0827	0.6875	0.3632	0.42	100	0.5804	0.2953	-0.1024	0.624	0.3977
0.5228	0.4766	0.0353	0.625	0.4414	0.84	200	0.5597	0.3267	-0.2012	0.648	0.5279
0.2123	0.4357	-1.6444	1	2.0801	1.26	300	0.5698	-0.0694	-0.7634	0.728	0.6941
0.2185	0.4139	-1.6321	0.875	2.046	1.67	400	0.5656	-0.23	-0.9654	0.704	0.7354
0.0777	0.1264	-3.1959	1	3.3222	2.09	500	0.5934	-0.612	-1.4579	0.688	0.8459
0.0812	0.4593	-3.3415	0.9375	3.8009	2.51	600	0.6276	-1.0042	-2.0014	0.688	0.9972
0.0619	-0.1213	-4.5057	1	4.3844	2.93	700	0.6306	-1.048	-2.0613	0.672	1.0132

Reproduced-Tulu2-dpo-13B (long) Training Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins
0.3535	2.8164	0.3743	0.875	2.4421	0.93	100	0.7415	1.143	0.557	0.656	0.586
0.1254	1.9677	-1.2914	1	3.2591	1.87	200	0.7605	0.953	0.1625	0.672	0.7905
0.0488	1.7841	-2.7587	1	4.5428	2.8	300	0.7892	0.6304	-0.3065	0.704	0.9369

Reproduced-Tulu2-dpo-13B (short) Training Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins
0.2292	-2.2345	-5.0452	0.75	2.8107	0.93	100	1.6931	-3.7319	-3.2944	0.424	-0.4375
0.1124	-4.3852	-11.0761	0.875	6.6909	1.87	200	2.785	-7.034	-6.3623	0.424	-0.6717
0.0306	-7.2191	-23.2556	0.9375	16.0364	2.8	300	4.17	-10.9384	-10.0796	0.416	-0.8588

It is clearly that short version got overfitted very soon. And here are the intermediate evaluation scores:	GSM8K 8-shot	BigBenchHard 3-cot-shot	IFEval 3-shot	PiQA 3-shot	MMLU 0-shot	TruthfulQA 3-shot
Short-E1	37.53	49.36	14.99	79.98	55.13	32.19
Short-E2	35.41	46.24	9.11	78.84	54.82	31.58
Short-E3	34.5	43.51	6	77.09	54.87	30.48

regards, Junru