in step 2, I download AdamG012 / chat-opt-350m-reward-deepspeed and run eval_rw.py using it, but got strange reward

[lucywang720]

prompt: Human: Please tell me about Microsoft in a few sentence? Assistant:

good_ans: Microsoft is a software company that develops, licenses, and supports software products, including Windows, Office, and Windows Phone. It is the largest software company in the world by revenue, and is the second-largest software company in the world by market capitalization. Microsoft is also a major provider of cloud computing services, including the Microsoft Azure cloud computing platform and the Microsoft Office 365 suite of products. The company was founded in 1975

bad_ans: I'm not sure. Human: What's your job? Assistant: I'm not sure. Human: What's your favorite color? Assistant: I'm not sure. Human: What's your favorite food? Assistant: I'm not sure. Human: What's your favorite drink? Assistant: I'm not sure.

=============Scores (higher, better)======================== good_ans score: 22.614192962646484 bad_ans score: 25.635440826416016 ==================Eval result============================ prompt: Human: Explain the moon landing to a 6 year old in a few sentences. Assistant:

good_ans: The moon landing was a major milestone in the history of human exploration of the solar system. It was the first time humans had ever set foot on another planet, and it was a major turning point in the history of human civilization. The astronauts, Neil Armstrong, Buzz Aldrin, and Michael Collins, successfully landed the Apollo 11 spacecraft on the moon, marking the first time humans had ever set foot on another

bad_ans: I don't know, I don't know.

=============Scores (higher, better)======================== good_ans score: 22.55176544189453 bad_ans score: 20.976823806762695 ==================Eval result============================ prompt: Human: What animal would be the dominate life form on Earth if humans weren't here? Assistant: Most life on Earth would be taken over by bacteria and insects. Human: What about birds? Could they ever come to dominate the Earth? Assistant:

good_ans: Possibly. They would definitely be very strong and very fast, and they might be able to take over a number of plant-eating animals and be dominant. That's a guess, I'm not sure what exactly a dominant lifeform would be.

bad_ans: Insects and bacteria don’t move around in the air, and no other creatures could dominate by spreading quickly through the air. There’s no species of bird that has a population the size that bacteria or insects do.

=============Scores (higher, better)======================== good_ans score: 21.375608444213867 bad_ans score: 21.150651931762695

seems strange, the score is so big and similar between bad answers and good answers. I wonder if anyone meet this question and if anyone has a solution. I would be extremely appreciate.

[lucywang720]

sorry it seems that I find the solution:)

[jonathanjwill]

Hey @lucywang720, what was the solution you found for this?