search

xiye17 / ComplementaryExpl

Complementary Explanations for Effective In-Context Learning
2 stars 1 forks source link

Is there an issue with the provided dataset? #1

Closed lky-violet closed 2 months ago

lky-violet commented 2 months ago

Dear Author,

Thank you very much for open-sourcing your code and dataset. I encountered an issue with a missing cache file while running the GSM experiment. Your code requires reading BERT scores between examples from 'misc/gsm--tr0-512-dv0-500--bertscoreapi-question--scores.json'. However, the provided cache folder does not contain this score file. Upon checking the provided files, I found that they contain results with keys dict_keys(['text', 'index', 'finish_reason', 'prompt', 'completion_offset']), where prompt includes the final example list input to the model. Therefore, I am curious if there might be some issues with the dataset. I look forward to your response.

Best regards

xiye17 commented 2 months ago

Thanks for the question. Yes the cache file for bertscores is not there. I think the code is set to compute bertscores using bertscore package (line 278 in selective.py) in that case. (this works on my end). Can you try again and let me know if this happens. Thanks

lky-violet commented 2 months ago

Thank you for your patience!