search

allenai / PRIMER

The official code for PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Apache License 2.0
153 stars 32 forks source link

Trying to get results from the paper #18

Open smirok opened 2 years ago

smirok commented 2 years ago

Hi, thanks a lot for sharing your work.

I want to try to get locally your zero-shot results for the DUC2004 dataset, but so far I get the wrong results (about 22, 3 and 15 points for R-1, R-2 and R-L respectively)

Tell me, please, is there a pretrained Longformer in your PRIMERA model published on HuggingFace or not?

If so, how can I repeat the above experiment locally with HF version of model?

LEYADEV commented 2 years ago

Hi! We had trouble reproducing the results from the paper with the pretrained PRIMERA from HF. Do you plan to publish your model?

Wendy-Xiao commented 2 years ago

Hi there,

Thanks for your interests in our paper. Our Longformer version model has already been released along with the code, and you can find the link in README.md under usage of PRIMERA section, for your convenience, I copied it here.

And there should not be such a big gap between HF version and the original code, so I would say there must be something wrong with the setting you used. You can double check the length limit of the generated summaries, which is 128 for DUC2004, as indicated in the paper. All the settings can be found in ./run_bash/test_primer.sh. Make sure you use stemmer for ROUGE score (which is the general setting, but may need to be set manually for some package of ROUGE).

Feel free to send email to me if you have any further questions.