Result

[Pwoer-zy]

Hello, I trained according to your requirements and training steps.First of all, I trained the SumMe dataset in the way of full training:batch_size=20,n_epochs=200,but I run evaluate_exp.sh,the result obtained is 39.92%.Then I continue to run inference.py, and the result is 36.01%. Do you know the cause of this phenomenon? I hope to get your answer. thank you!

[dlxdlxdlx]

I obtained 45.5% on SumMe, if you know the cause, please tell me (:

[Pwoer-zy]

I obtained 45.5% on SumMe, if you know the cause, please tell me (:

Is it the data from your own training model?Could you tell me? The result of my own training model is very low，thank you very much!

[e-apostolidis]

Hi Pwoer-zy,

first of all, thanks for your interest in our work.

I assume that something goes wrong with the training process. This can be related, for example, with the used data-splits or the applied evaluation criterion (e.g. the maximum F-score value across the different user annotations should be kept for a given test video on SumMe). We made several experiments with our model and we've never recorder such low scores on SumMe.

Furthermore, using the scripts and in the "inference" folder and the pre-trained models of PGL-SUM (on SumMe and TVSum) should result in the scores reported in our paper. So, there might be a general fault/misinterpretation in the way that you should use these pre-trained models for evaluating their summarization performance.

[e-apostolidis]

Hi dlxdlxdlx,

thanks for your interested in our work. When training and evaluating our model, did you use the data splits from our work (available here: https://github.com/e-apostolidis/PGL-SUM/tree/master/data/splits)? Moreover, did you have the same settings in terms of software components (see: https://github.com/e-apostolidis/PGL-SUM#main-dependencies)?