sycny
commented
1 month ago I am afraid that I cannot determine the problem here based on your log. Could you clarify whether you're using one of our supported models or a different model?
Maybe you can refer to the closed issue for a possible solution.
wangjing1012
hello~When I run the code, this happens. Do you know why? ++++++++++++++++++++++++++++++++++50-th case++++++++++++++++++++++++++++++++++++++ +++++++++++++++++0-th question+++++++++++++++++++++ Questions: Where is the headquarters location of the employer of Leonte Răutu located? Ground Truth: {'Leonte Răutu is employed by University of Bucharest', 'University of Bucharest has its headquarters located in the city of Washington, D.C.'} Request URL: https://en.wikipedia.org/w/api.php?action=query&prop=info&titles=Leonte Răutu&inprop=protection|talkid|watched|watchers|visitingwatchers|notificationtimestamp|subjectid|url|readable|preload|displaytitle !!!!49-case-0-question:Fail to extract Raw:0-->Edited Ans:'Washington, D.C./washington dc', Our Ans:'Leonte Răutu is not employed by/leonte rutu is not employed by' Pruned:0-->Edited Ans:'Washington, D.C./washington dc', Our Ans:'None of the given facts mention the employer of/none of the given facts mention the employer of' +++++++++++++++++1-th question+++++++++++++++++++++ Questions: In which city is the employer of Leonte Răutu based? Ground Truth: {'Leonte Răutu is employed by University of Bucharest', 'University of Bucharest has its headquarters located in the city of Washington, D.C.'} Request URL: https://en.wikipedia.org/w/api.php?action=query&prop=info&titles=Leonte Răutu&inprop=protection|talkid|watched|watchers|visitingwatchers|notificationtimestamp|subjectid|url|readable|preload|displaytitle !!!!49-case-1-question:Fail to extract Raw:0-->Edited Ans:'Washington, D.C./washington dc', Our Ans:'Bucharest., , Given fact:/bucharest given fact' Pruned:0-->Edited Ans:'Washington, D.C./washington dc', Our Ans:'Bucharest., , Given fact:/bucharest given fact' +++++++++++++++++2-th question+++++++++++++++++++++ Questions: What is the location of the headquarters of the employer of Leonte Răutu? Ground Truth: {'Leonte Răutu is employed by University of Bucharest', 'University of Bucharest has its headquarters located in the city of Washington, D.C.'} Request URL: https://en.wikipedia.org/w/api.php?action=query&prop=info&titles=Leonte Răutu&inprop=protection|talkid|watched|watchers|visitingwatchers|notificationtimestamp|subjectid|url|readable|preload|displaytitle !!!!49-case-2-question:Fail to extract Raw:0-->Edited Ans:'Washington, D.C./washington dc', Our Ans:'Leonte Răutu's employer/leonte rutus employer' Pruned:0-->Edited Ans:'Washington, D.C./washington dc', Our Ans:'Leonte Răutu's employer/leonte rutus employer' ++++++++++++++++++++++++++++++++++End++++++++++++++++++++++++++++++++++++++ Finished on 50-th case raw_exact_match_acc: 0.0000 raw_par_match_acc: 0.0000 prun_exact_match_acc: 0.0000 prun_par_match_acc: 0.0000 raw_ans_acc: 0.0600 prun_ans_acc: 0.0600