search

bepoetree / MTTOD

Apache License 2.0
26 stars 8 forks source link

question about prediction #6

Open no-execution opened 2 years ago

no-execution commented 2 years ago

Hello , thanks for your work.

Recently I use the mttod model on CrossWOZ dataset and have a question about the predict code.

In predict code you have provided , you firstly get belief state generated by dialog history and then you use the dbpn from dataset as input to predict dialog act and response.

Actually when we use the dialog system, we get dbpn by searching from database by belief state generated and turn domain predicted by model instead of using the dbpn and turn domain from dataset directly.And in the code you have provided, belief state is generated before dialog act and dialog act is generated after dbpn. So I can not get dbpn before generating the dialog act.

So , If we want to build a dialog system as service , how to manage the gap between your predict procedure and real world predict procedure?

bepoetree commented 2 years ago

Hi, thanks for your interest in my code.

In the inference phase, the dbpn is generated based on generated belief state. Please refer to 775-808 lines in runner.py. The configuration for usage of true dbpn (cfg.use_true_dbpn) is set to False by default.

Actually, as you say, the turn_domain should be traced from generated belief state in the inference phase. But, the procedure is omitted because my code is for evaluation only. (If generated belief state is wrong, the dbpn would be wrong even if the true turn_domain is used.) For actual service, you can trace the turn_domain by comparing the previous and current belief state.

no-execution commented 2 years ago

Thanks for your response.

I have two questions:

  1. Though dbpn would be wrong if belief state generated is wrong, using true turn domain in the evaluate procedure is ok?

2.Maybe we can resort [bs]->[dbpn]->[dialog act]->[resp_delex] to [bs]->[dialog act]->[dbpn]->[resp_delex], to solve the turn domain problem in actual service?

By the way, I am a engineer. So I concentrate morn on the relialility of end-to-end dialog system. Do you think we could use it in some fancy applications?

bepoetree commented 2 years ago
  1. You are right.
  2. I think the dialog act should be generated after the dbpn is determined because the system act depends on the db state (so, in the inference phase, the turn_domain is determined based on belief state generated only.)
  3. In my opinion, it depends on how you model the problem you are interested in. Most of the research on end-to-end task-oriented dialog system has been conducted in very closed domains (with the pre-defined ontology). If you want to use the dialog system in the open situation, it may not work well due to the challenging problems such as out-of-slot values.