search

megagonlabs / cocosum

:coconut: Code & Data for Comparative Opinion Summarization via Collaborative Decoding (Iso et al; Findings of ACL 2022)
https://aclanthology.org/2022.findings-acl.261.pdf
Apache License 2.0
21 stars 2 forks source link

Cannot reproduce few-shot results (Table 2) && Out of memory while creating train_comm_pair.jsonl #3

Open belizgunel opened 1 year ago

belizgunel commented 1 year ago

Thank you so much for open-sourcing your implementation and for your great work! I have two questions, and your help would be much appreciated!

1) I followed the decoding using pre-trained models and evaluation as described in your repo, and I was not able to reproduce your few-shot results in Table 2. There is up to ~5 points difference in BERTScore for both contrastive and common summaries and significant difference in ROUGE score up to ~4 points. Would there be any other hyperparameters I should fix? As a comparison, self-supervised results are much closer to what's reported on your paper. 2) My process keeps getting killed while trying to create train_comm_pair.jsonl at line https://github.com/megagonlabs/cocosum/blob/2a941323f07eb6152f6057710af47296990e70ef/prep.py#L154, and my machine has 192B RAM. I was wondering if you could share the file or help me figure out this/optimize that line?

Thank you so much.

lovodkin93 commented 1 year ago

Hey, I also keep getting a similar error:

numpy.core._exceptions.MemoryError: Unable to allocate 84.6 GiB for an array with shape (11355303385,) and data type
 int64

Would appreciate any help. Thanks!

isomap commented 1 year ago

Hi @lovodkin93 , I modified the prep script to run with less memory resources (see https://github.com/megagonlabs/cocosum/blob/main/prep.py#L153-L162) I hope now you can run the prep script with no memory error.

Thanks!

isomap commented 1 year ago

For the few-shot result reproduction thing, I've already fixed the readme file to show the correct hyperparameter setting.