search

salesforce / simpletod

Official repository for "SimpleTOD: A Simple Language Model for Task-Oriented Dialogue"
https://arxiv.org/abs/2005.00796
BSD 3-Clause "New" or "Revised" License
235 stars 79 forks source link

can't get the result reported in paper when using end2end training without dbsearch #7

Open fasterbuild opened 3 years ago

fasterbuild commented 3 years ago

Hi all, with default parameters, I run end2end training without desearch, train file is resources/gpt2/train.history_belief_action_sys_delex with 56778 samples in the file. The ppl in valid set stop decreasing after only 2 epoch, got final valid set ppl=2.30. The success rate of test set is around 18.5%, much lower than 70.5% reported in paper table 3. And the belief acc is around 42%, also much lower than 55% in paper table 1. I wonder if the model is trained well with default params, would you please release your hyper parameters for end2end training and traing details?

gungui98 commented 3 years ago

Hi @fasterbuild , I have the same problem as you and I don't even get the same accuracy as you did! could you share the hyper-params you are using?