search

boheumd / A2Summ

The official implementation of 'Align and Attend: Multimodal Summarization with Dual Contrastive Losses' (CVPR 2023)
https://boheumd.github.io/A2Summ/
70 stars 10 forks source link

#evaluation metrics of Summe and TVSum datasets τ and ρ #11

Open Pwoer-zy opened 1 year ago

Pwoer-zy commented 1 year ago

hello, how does the evaluation metrics of Summe and TVSum datasets τ and ρ calculate? I only see the calculation method of fscore in the code. Looking forward to getting your reply, thank you very much!

boheumd commented 1 year ago

Hello. You can refer to the following code to calculate rho and tau results. https://github.com/e-apostolidis/CA-SUM/blob/main/evaluation/choose_best_model.py#L67 https://gitlab.uni-hannover.de/hussainkanafani/unsupervised-video-summarization/-/blob/master/src/evaluation/BaseEvaluator.py#L7 https://github.com/TIBHannover/MSVA/blob/master/train.py#L430

Pwoer-zy commented 1 year ago

Hello, thank you for your reply

rodosingh commented 1 year ago

Hi, thanks for this good work and such clean code.

I don't think the computation of rho and tau is consistent throughout all the links you mentioned. Can you provide your source code for this? That would be very much helpful.

Thanks in advance.

Pwoer-zy commented 8 months ago

Hello. You can refer to the following code to calculate rho and tau results. https://github.com/e-apostolidis/CA-SUM/blob/main/evaluation/choose_best_model.py#L67 https://gitlab.uni-hannover.de/hussainkanafani/unsupervised-video-summarization/-/blob/master/src/evaluation/BaseEvaluator.py#L7 https://github.com/TIBHannover/MSVA/blob/master/train.py#L430

I tried according to what you said, but the result was not good. Can you provide your source code for your code? That would be very much helpful.

Thanks in advance.

joserochh commented 7 months ago

Hello,

My question here would be. Are you using original user scores for both SumMe and TVSum? or are you averaging user summaries and taking that as user scores? Do you do it differently for SumMe and TVSum? I have not seen yet user annotations for SumMe dataset.

Thank you!