search

snakeztc / NeuralDialog-CVAE

Tensorflow Implementation of Knowledge-Guided CVAE for dialog generation ACL 2017. It is released by Tiancheng Zhao (Tony) from Dialog Research Center, LTI, CMU
https://www.cs.cmu.edu/~tianchez/
Apache License 2.0
309 stars 85 forks source link

Problem about the multiple references test dataset #12

Closed wac81 closed 5 years ago

wac81 commented 6 years ago

Hi,

I want to use the test dataset with multiple references(which you uploaded at https://github.com/snakeztc/NeuralDialog-CVAE/blob/master/data/test_multi_ref.json.) to get the evaluation result you presented in the paper. But I'm pretty confused because the format of this one is quite different from the one with single reference, for it's in lack of some information, such as topic. Could you please tell me how to get the evaluation result with multiple references with this test dataset which seems to be incomplete?

Looking forward to your reply. ：）

snakeztc commented 5 years ago

You should run your model on the test data in the json_data folder and let your model generate multiple hypotheses. The test_mult_ref data is aligned with the test data set. Then you can have your own script to compute the score against the reference vs hypotheses.