search

cszmli / Rethink-RL-Sup

Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systems
9 stars 5 forks source link

can't find "goal.json" #2

Closed liyu199809 closed 3 years ago

liyu199809 commented 3 years ago

When I run python main.py --pretrain True --save_dir model_wgan/emnlp_model_agenda_pre_nosimgan_epoch9_temp0.01_action0.1_clip0.01_2 --epoch 16 --gan_type wgan --temperature 0.001 --action_lambda 0. --rl_sim no --argmax_type gumbel --clip 0.02 ,than I set "def evaluate(self, save_dialog=True)" in gan.py. But I can't find goal.json.

Than,I use data/processed_data/train_goal.json to replace it.It also doesn't work.

cszmli commented 3 years ago

You can make use of the evaluation function def evaluate(self, save_dialog=True) in two different ways:

  1. It randomly samples m goals from the dataset and measures the success rate (save_dialog=True). This is also the setup in the GDPL work.
  2. However, the result in the first setup may not be precise enough because the goal set is different for each run. So you can have a static goal set and evaluate the model performance always on the same goal set. You need to prepare a goal set first by commenting line 547 and line 548. Of course, you can also set m to a huge value to stabilize the result. The second setup is not necessary for the experiment.