request for more details

[Shen-Chenhui]

Hi, could you please kindly provide more details regarding experimental settings? Specifically for SummEval,

• Which model do you use, "gpt-3.5-turbo" or "text-davinci-003" or something else?
• Do you set the temperature to a specific value, or do you use a specific top_p value?
• Does the response from ChatGPT includes explanations, and do you still have these data? Thank you and I look forward to your response.

[krystalan]

Hi, Sorry for the really late reply.

The OpenAI ChatGPT did not release the official API when we did the experiments. Thus, there might be gaps when you reproduce the results using the official API.

Currently, I recommend setting the temperature to zero in official APIs and using the gpt-3.5-turbo model. Empirically, I find that when setting the temperature to zero, the gpt-3.5-turbo model will directly produce the final scores without any explanations. If you want to collect explanations, try to raise the temperature.

Please feel free to drop me emails for any other questions.