search

THUDM / LongBench

[ACL 2024] LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
MIT License
633 stars 45 forks source link

KeyError: 'retrieved' #32

Closed liujingcs closed 9 months ago

liujingcs commented 11 months ago

Thanks for great work! I've encountered an issue while running the pred.py script for long context compression evaluation based on retrieval. The script is throwing a KeyError: 'retrieved,' which is preventing me from making progress. Upon investigation, it appears that the 'LongBench.py' script may not have the necessary code to handle data with the 'retrieved' key.

bys0318 commented 11 months ago

Thanks for your attention! Make sure you have followed the steps in retrieval, since the 'retrieved' key is only added after you run the retrieval.

liujingcs commented 11 months ago

Thank you for your prompt response. Upon conducting a more thorough examination, it seems that there is a missing data extraction process for 'retrieved' within the LongBench.py file. This issue appears to be particularly related to the _generate_examples function.

bys0318 commented 11 months ago

We have added a customized LongBench.py under folder retrieval/. Please check if this resolves your issue.