shawnwun / NNDIAL

NNDial is an open source toolkit for building end-to-end trainable task-oriented dialogue models. It is released by Tsung-Hsien (Shawn) Wen from Cambridge Dialogue Systems Group under Apache License 2.0.
Other
348 stars 105 forks source link

Test results are different from the paper. #2

Closed robotzheng closed 7 years ago

robotzheng commented 7 years ago

LIDM_70 Venue Match Rate : 91.9% Task Success Rate : 74.3% BLEU : 0.2337 Semantic Match : 63.7%

LIDM-RL_70 Venue Match Rate : 92.6% Task Success Rate : 75.7% BLEU : 0.2459 Semantic Match : 63.4%

LIDM-RL_100 (just set I = 100) Venue Match Rate : 89.7% Task Success Rate : 70.6% BLEU : 0.2541 Semantic Match : 60.3%

not like the paper's result: LIDM Models LIDM, I = 50 66.9 0.238 LIDM, I = 70 61.0 0.246 LIDM, I = 100 63.2 0.242 LIDM Models + RL LIDM, I = 50, +RL 82.4 0.231 LIDM, I = 70, +RL 81.6 0.230 LIDM, I = 100, +RL 84.6 0.240

shawnwun, thanks.