search

lipiji / TranSummar

Transformer for abstractive summarization on cnn/daily-mail and gigawords
MIT License
140 stars 20 forks source link

How to get rouge score for the saved models? #2

Closed soheilesm closed 5 years ago

soheilesm commented 5 years ago

Shall I run tuning.sh to get the rouge scores? I do so, but I get error on perl /home/pijili/tools/ROUGE-1.5.5/ROUGE-1.5.5.pl -n 4 -w 1.2 -m -2 4 -u -c 95 -r 1000 -f A -p 0.5 -t 0 myROUGE_Config.xml C

As the original GitHub repo that contains ROUGE-1.5 is not active anymore - >https://github.com/andersjo/pyrouge/tree/master/tools/ROUGE-1.5.5, can you put a wrapper for this so that we can calculate the rouge scores by just running the tunning.sh file?

Just to make sure I am doing things the right way: So I trained the model with options["is_debugging"] = False options["is_predicting"] = False options["model_selection"] = True in the main.py, and got some models saved in the model folder.

Then for creating the validation summaries and running decoder, I need to set options["is_debugging"] = True options["is_predicting"] = True options["model_selection"] = False in the main.py and it will run for the saved models. However, doing that I only get 13 files generated as the summary and ground_truth pairs, shouldn't it be more?

Could you also please mention what combination of parameters (i.e. coverage, copy) you ran and for how many epochs and you got your reported results?

xcfcode commented 5 years ago

options["is_debugging"] is always false, I think it is only for debugging, not for real experiment, at training time, options["is_predicting"] = False, options["model_selection"] = Flase, at tuning time, options["is_predicting"] = True, options["model_selection"] = True, options["model_selection"] = True it means that only use valid set to choose the model.

lipiji commented 5 years ago

Shall I run tuning.sh to get the rouge scores? I do so, but I get error on perl /home/pijili/tools/ROUGE-1.5.5/ROUGE-1.5.5.pl -n 4 -w 1.2 -m -2 4 -u -c 95 -r 1000 -f A -p 0.5 -t 0 myROUGE_Config.xml C

As the original GitHub repo that contains ROUGE-1.5 is not active anymore - >https://github.com/andersjo/pyrouge/tree/master/tools/ROUGE-1.5.5, can you put a wrapper for this so that we can calculate the rouge scores by just running the tunning.sh file?

Thanks. I will integrate the pyrouge package later.

Just to make sure I am doing things the right way: So I trained the model with options["is_debugging"] = False options["is_predicting"] = False options["model_selection"] = True in the main.py, and got some models saved in the model folder.

Then for creating the validation summaries and running decoder, I need to set options["is_debugging"] = True options["is_predicting"] = True options["model_selection"] = False in the main.py and it will run for the saved models. However, doing that I only get 13 files generated as the summary and ground_truth pairs, shouldn't it be more?

when options["is_debugging"] is true: only use several (16) samples to conduct the training.

Training: options["is_debugging"] = False options["is_predicting"] = False options["model_selection"] = False

Validation on a subset: options["is_debugging"] = False options["is_predicting"] = True options["model_selection"] = True

Evaluation on test set: options["is_debugging"] = False options["is_predicting"] = True options["model_selection"] = False

Could you also please mention what combination of parameters (i.e. coverage, copy) you ran and for how many epochs and you got your reported results?

coverage=true, copy=true Number of epochs: https://github.com/lipiji/TranSummar/blob/master/README.md