search

composable-models / llm_multiagent_debate

ICML 2024: Improving Factuality and Reasoning in Language Models through Multiagent Debate
304 stars 42 forks source link

Repeat times of experiment results #9

Open FutureForMe opened 10 months ago

FutureForMe commented 10 months ago

hi, your paper is really a very interesting piece of work!

I saw that you stated in Appendix A.2 of your paper that you did not use all the data in the dataset, but sampled a part of the data (for example, one hundred items). I would like to ask whether the final result in the paper is to repeat the data sampling multiple times and complete the experiment, or to repeat the experiment multiple times on the same sampling data?

yilundu commented 10 months ago

Hi, it should be repeating the experiment multiple times on the same sampling data.

FutureForMe commented 10 months ago

Thank you for your reply. And what is the hyperparameter setting for the number of experiments in your paper?

yilundu commented 10 months ago

Hi, sorry what do you mean by hyperparameter setting? We selected 100 problems for each problem setting

FutureForMe commented 10 months ago

Sorry. I mean how many times experiments did you run on each of the same sampled data? like 5 times?

yilundu commented 10 months ago

Ahh I see -- all experiments are run on the same sampled data -- we report the standard error obtained from the 100 problems (standard deviation divided by sqrt(N))