search

danieldeutsch / sacrerouge

SacreROUGE is a library dedicated to the use and development of text generation evaluation metrics with an emphasis on summarization.
Apache License 2.0
140 stars 14 forks source link

qaeval could not been imported #119

Closed yfqiu98 closed 3 years ago

yfqiu98 commented 3 years ago

Hello I just try to use the QAEval in sacrerouge, however I found the following issue. Seems the loading of QAEval() is broken and I did not see the output as in the demo colab https://colab.research.google.com/drive/1IZUZCUQAdCfXAMDDyeF67QFHB72y2svI?usp=sharing:

"Some weights of BartForConditionalGeneration were not initialized from the model checkpoint at facebook/bart-large and are newly initialized: ['final_logits_bias'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference."

Version information: !pip install sacrerouge==0.2.2 !pip install qaeval==0.0.9

I follow the instruction in the colab demo to do !sacrerouge setup-metric qa-eval, but it did not work for downloading the answering model file (Although I got the same output as in the colab demo, there is no 'vocab.txt'). Thus I manually download it from:https://drive.google.com/file/d/1q2Z3FPP9AYNz0RJKHMlaweNhmLQoyPA8/view?usp=sharing.

Type "help", "copyright", "credits" or "license" for more information. import json from sacrerouge.metrics import QAEval qaeval = QAEval() cores = qaeval.score("asd", ["sadfsdf"]) Traceback (most recent call last): File "", line 1, in File "/X/miniconda3/envs/sacrerouge/lib/python3.6/site-packages/sacrerouge/metrics/metric.py", line 77, in score return super().score(summary, references, kwargs) File "/X/miniconda3/envs/sacrerouge/lib/python3.6/site-packages/sacrerouge/metrics/metric.py", line 48, in score return self.score_all([summary], args, kwargs)[0] File "/X/miniconda3/envs/sacrerouge/lib/python3.6/site-packages/sacrerouge/metrics/metric.py", line 83, in score_all return super().score_all(summaries, references_list, kwargs) File "/X/miniconda3/envs/sacrerouge/lib/python3.6/site-packages/sacrerouge/metrics/metric.py", line 56, in score_all metrics_lists = self.score_multi_all(summaries_list, args, kwargs) File "/X/miniconda3/envs/sacrerouge/lib/python3.6/site-packages/sacrerouge/metrics/qaeval.py", line 40, in score_multi_all f'Package "qaeval" could not be imported. Please install "qaeval" before running the metric. ' Exception: Package "qaeval" could not be imported. Please install "qaeval" before running the metric. The minimum required version is 0.0.8

Other attempts:

metric = QAEval(cuda_device=-1) Traceback (most recent call last): File "", line 1, in TypeError: init() got an unexpected keyword argument 'cuda_device' print(QAEval()) <sacrerouge.metrics.qaeval.QAEval object at 0x7f2c224e5048>

Thanks in advance for any help.

yfqiu98 commented 3 years ago

Update:

solved by adding sys.path.append('/X/.sacrerouge')

danieldeutsch commented 3 years ago

I'm glad that you solved it, but I don't actually understand how adding that directory to your path fixed it because there is no Python code which is loaded from there.

If you don't add that to the path, but just try running import qaeval, what happens? That import failed in SacreROUGE, which caused the error that you see. If import qaeval fails, then it will show an exception for why it couldn't be imported.