How to evaluate the transformer on the summarization task by using the tensor2tensor?

tensorflow / tensor2tensor

Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.

Apache License 2.0

15.5k stars 3.49k forks source link

How to evaluate the transformer on the summarization task by using the tensor2tensor? #1310

Open zhaoguangxiang opened 5 years ago

zhaoguangxiang commented 5 years ago

Description

e.g. CNN/dailymail dataset. I know how to train the transformer on the dataset, but I don't know how to evaluate the performance of the transformer on the task.

pig60214 commented 5 years ago

hello, Do you get the answer?

zhaoguangxiang commented 5 years ago

No, I didn' t, and I notice that there is a get_rouge.py in the utils directory

nxphi47 commented 5 years ago

Hi, have anyone got the answer yet? I also have question about this.

zhaoguangxiang commented 5 years ago

Hi, have anyone got the answer yet? I also have question about this.

not yet

RyanOngAI commented 5 years ago

Hope its not too late. Run t2t-decoder, which would generate three output files, one of which is summary generated by your model and the other its the gold-summary. Install pyrouge and feed those two output and you will get R-1, R-2 and R-L