search

danieldeutsch / sacrerouge

SacreROUGE is a library dedicated to the use and development of text generation evaluation metrics with an emphasis on summarization.
Apache License 2.0
134 stars 11 forks source link

Old URL for the question generation model in QA eval #134

Open martiansideofthemoon opened 2 years ago

martiansideofthemoon commented 2 years ago

Hi @danieldeutsch, Thank you for building this very useful library.

I wanted to point out a small bug that I found while using the QAEval metric. I think the URL for the question generation model here links to the older model, which is incompatible with the newer versions of HuggingFace. I fixed it by using the Google Drive link on the danieldeutsch/qaeval repository.

kalpesh@Kalpeshs-MacBook-Pro:Downloads$ wget https://cogcomp.seas.upenn.edu/models/qaeval-experiments/model.tar.gz
kalpesh@Kalpeshs-MacBook-Pro:Downloads$ tar -xvzf model.tar.gz 
x config.json
x weights.th
x vocabulary/
x vocabulary/non_padded_namespaces.txt
kalpesh@Kalpeshs-MacBook-Pro:Downloads$ cat config.json | grep "bart-large"
        "model_name": "bart-large"
        "model_name": "bart-large"

The bart-large namespace is incompatible the transformers 3.0.2 version, which needs facebook/bart-large.

danieldeutsch commented 2 years ago

Thanks, this is helpful, and to be honest I don't remember why one version works and the other does not! I will work on trying to fix this.

For what it's worth, I have Dockerized implementations of QAEval and many other metrics in this repository if you have access to Docker. I find it far easier to run metrics with that code base instead of this one.

sn-masoud commented 1 year ago

Hi @danieldeutsch, Thanks for the library! I am adding may question to this issue, since it may be relevant. I get this error 

Error loading state dict for _QuestionGenerationModel 
Missing keys: ['bart.final_logits_bias', 'bart.lm_head.weight']

Any suggestions to resolve the issue?

Thanks