search

RuochenZhao / Verify-and-Edit

A framework for editing the CoTs for better factuality
40 stars 4 forks source link

There seems to be a serious bug in the file HotpotAdv/consistency.py that leads to incorrect calculation results #2

Closed aminvisible closed 8 months ago

aminvisible commented 9 months ago

In def consistency (line 55) of HotpotAdv/consistency.py lines 70, 71, 72:

probs = [a[1] for a in answer_prob_lists[final_aggregated_answer]]
best_i = np.argmax(probs)
final_aggregated_rationale = rationales[best_i]

This seems to have incorrectly calculated best_i and final_aggregated_rationale, because of a new list of argmax indices instead of rationales.

RuochenZhao commented 9 months ago

Hi, sorry! It seems that this problem was fixed later on and I didn't update it. Could you see if the current update solves the problem?

aminvisible commented 9 months ago

Hi, sorry! It seems that this problem was fixed later on and I didn't update it. Could you see if the current update solves the problem?

I think this problem has been fixed now, but are other files up-to-date now?

RuochenZhao commented 8 months ago

Yes it was the only issue that wasn't updated. I'll be closing the issue now.