Closed sunyuhan19981208 closed 1 year ago
@sunyuhan19981208 Hello, thanks for reporting the evaluation inconsistency! I'm not sure why the performance has such a large difference, and I have checked the results per task and have observed that:
logical_deduction_x_objects
in your table are much higher than those obtained from my side on FLAN-T5-Large.hyperbaton
is also really high in your reported scores.web_of_lies
is much lower than the ones reported in our paper.Not sure if these gap are caused by different evaluation methods. In our evaluation script, we just try to generate the result and evaluate the performance using exact match
(for all models), without further actions (e.g. removing redundant explanation).
One thing which may have a difference is that we split the BBH benchmark into two sets to study the effect of the number of examples to the performance on BBH in Figure 3. That is to say, our BBH benchmark contains 150 examples for each task to evaluate (and the original one has 250 examples), and we evaluate all models on these 150 examples to have a fair comparison.
I do not think that it will cause such a big difference. And I will update the reproduce procedure soon for your reference.
@sunyuhan19981208 Please follow the reproduce_bbh.py
to obtain the zero-shot results of FLAN-T5-Large reported in our paper. Note that we follow the FLAN paper to add the task description before each task input, which shares a slightly different prompt with your shared evaluation script. And also, I'm not sure if the shared repo use fuzzy matching to match the ground-truth and the prediction?
@sunyuhan19981208 Please follow the
reproduce_bbh.py
to obtain the zero-shot results of FLAN-T5-Large reported in our paper. Note that we follow the FLAN paper to add the task description before each task input, which shares a slightly different prompt with your shared evaluation script. And also, I'm not sure if the shared repo use fuzzy matching to match the ground-truth and the prediction?
Thanks a lot, and below is the evaluation code in the instruct-eval:
def evaluate(model: EvalModel, data: BBHData, ntrain: int) -> dict:
data_train = BBHData(samples=data.samples[:ntrain])
data_test = BBHData(samples=data.samples[ntrain:])
is_correct = []
for i in range(len(data_test.samples)):
# get prompt and make sure it fits
k = int(ntrain)
prompt_end = data_test.samples[i].as_prompt(include_answer=False)
train_prompt = gen_prompt(data_train, k)
prompt = train_prompt + prompt_end
while not model.check_valid_length(prompt) and k > 0:
k -= 1
train_prompt = gen_prompt(data_train, k)
prompt = train_prompt + prompt_end
label = data_test.samples[i].target
pred = model.run(prompt)
is_correct.append(pred.strip().startswith(label))
if i == 0:
print(dict(prompt=prompt, label=label, pred=pred))
return dict(score=sum(is_correct) / len(is_correct))
@SivilTaram I think the main cause is they use startswith
and you use ==
@sunyuhan19981208 Okay that makes sense. In fact I do not think startswith
is better than ==
when the mode gives wrong explanation - for example, although they predict the correct option, but they do not give the correct content. For me, if it is a fair comparison, it should reflect the ranking / trend. Since our paper uses the same exact match metric for all models, I think it should be okay to use exact match.
@SivilTaram Thanks, I will turn to use exact match in the new commit to reproduce your experiment. And also reproduce using startswith
, I will tell you the result of startswith
after I finish the experiment.
Closed the issue since the problem is solved. Feel free to reopen this issue if you want to report new results.
@sunyuhan19981208 How about the performance? Have you tried to use startswith
to evaluate lorahub?
Description
I am currently attempting to reproduce the experiment results outlined in your paper using the Flan-t5-large model on the BBH task. I've encountered a deviation in the results and I'm seeking assistance in addressing this issue.
Brief summary of my situation
Experiment Details
I utilized the script available at this GitHub repository to conduct the evaluation.
Discrepancy
My obtained score for Flan-t5-large on the BBH task was 31.82, whereas your paper reports a score of 27 for the same model-task combination.
Closure of Cot
To ensure consistency, I made sure to set ntrain=0 as instructed.
Request
I kindly ask for your guidance in reproducing the experiment results as presented in your paper. It would be immensely helpful if you could provide some insights, suggestions, or clarifications to assist me in achieving consistent results.
I greatly appreciate your time and consideration in helping me address this matter. Your expertise will go a long way in resolving this discrepancy and advancing my understanding of the experiment.
Thank you in advance for your assistance!
Log of Evaluation