[BUG] KeyError: 'semi'

[suntea233]

Describe the bug When i use GenTUS as a normal US, and test it, the bug is "KeyError: 'semi'".

To Reproduce Steps to reproduce the behavior:

from convlab.nlu.jointBERT.multiwoz import BERTNLU
from convlab.dst.rule.multiwoz import RuleDST
from convlab.policy.rule.multiwoz import RulePolicy
from convlab.nlg.template.multiwoz import TemplateNLG
from convlab.util.analysis_tool.analyzer import Analyzer
from convlab.policy.genTUS.stepGenTUS import UserPolicy
from convlab.dialog_agent import PipelineAgent
from convlab.util.custom_util import set_seed

import random
import numpy as np
import torch

def set_seed(r_seed):
    random.seed(r_seed)
    np.random.seed(r_seed)
    torch.manual_seed(r_seed)

def test_end2end():
    # go to README.md of each model for more information
    # BERT nlu
    # sys_nlu = BERTNLU()
    sys_nlu = None
    # simple rule DST
    sys_dst = RuleDST()
    # rule policy
    # sys_policy = None
    sys_policy = RulePolicy()
    # sys_policy = MLEPolicy()
    # sys_policy = RulePolicy()
    # template NLG
    # sys_nlg = TemplateNLG(is_user=False)
    sys_nlg = None
    # assemble
    sys_agent = PipelineAgent(
        sys_nlu, sys_dst, sys_policy, sys_nlg, name='sys')

    model_checkpoint = 'convlab/policy/genTUS/unify/experiments/multiwoz21_0_1'
    usr_policy = UserPolicy(model_checkpoint, mode="semantic")
    user_agent = PipelineAgent(None, None, usr_policy, None, 'user')

    analyzer = Analyzer(user_agent=user_agent, dataset='multiwoz')

    set_seed(20200202)

    analyzer.comprehensive_analyze(
        sys_agent=sys_agent, model_name='end2end', total_dialog=100)

if __name__ == '__main__':
    test_end2end()

[zqwerty]

@hsien1993 please have a look

[hsien1993]

Hi @suntea233 As mentioned in this issue: https://github.com/ConvLab/ConvLab-3/issues/101#issue-1491300633 The RulePolicy is still using the old format in convlab-2, which means this KeyError comes from the mismatch between rule-based DST and rule-based policy. Therefore, I would recommend you try the PPOPolicy, which means you can replace these two lines

from convlab.policy.rule.multiwoz import RulePolicy
sys_policy = RulePolicy()

to

from convlab.policy.ppo import PPO
sys_policy = PPO(is_train=False, seed=0, dataset_name='multiwoz21')

Then this End2End test should be OK.

Let me know if you have further questions :)

[suntea233]

Hi, @hsien1993 I train the GenTUS, and then use PPO as the sys_policy, but the sucess rate is only 0.2, is there any mistakes? The user's responses have always been the same. The code is below:

from convlab.nlu.jointBERT.multiwoz import BERTNLU
from convlab.dst.rule.multiwoz import RuleDST
from convlab.policy.ppo import PPO
from convlab.nlg.template.multiwoz import TemplateNLG
from convlab.util.analysis_tool.analyzer import Analyzer
from convlab.policy.genTUS.stepGenTUS import UserPolicy
from convlab.dialog_agent import PipelineAgent
from convlab.util.custom_util import set_seed

import random
import numpy as np
import torch

def set_seed(r_seed):
    random.seed(r_seed)
    np.random.seed(r_seed)
    torch.manual_seed(r_seed)

def test_end2end():
    # go to README.md of each model for more information
    # BERT nlu
    # sys_nlu = BERTNLU()
    sys_nlu = None
    # simple rule DST
    sys_dst = RuleDST()
    # rule policy
    # sys_policy = None
    sys_policy = PPO(is_train=False, seed=0, dataset_name='multiwoz21')
    # sys_policy = MLEPolicy()
    # sys_policy = RulePolicy()
    # template NLG
    # sys_nlg = TemplateNLG(is_user=False)
    sys_nlg = None
    # assemble
    sys_agent = PipelineAgent(
        sys_nlu, sys_dst, sys_policy, sys_nlg, name='sys')

    model_checkpoint = r'convlab\policy\genTUS\unify\experiments\multiwoz21_0_1'
    usr_policy = UserPolicy(model_checkpoint, mode="semantic",dataset='multiwoz21')
    user_agent = PipelineAgent(None, None, usr_policy, None, 'user')

    analyzer = Analyzer(user_agent=user_agent, dataset='multiwoz')

    set_seed(20200202)

    analyzer.comprehensive_analyze(
        sys_agent=sys_agent, model_name='end2end', total_dialog=20)

if __name__ == '__main__':
    test_end2end()

The running result of the code is as follows:

sys:[]
usr:[['request', 'none', 'none', 'none']]
sys:[['inform', 'attraction', 'address', 'pool way, whitehill road, off newmarket road'], ['inform', 'attraction', 'address', 'park street'], ['inform', 'attraction', 'address', 'jesus lane'], ['inform', 'attraction', 'area', 'centre'], ['inform', 'attraction', 'choice', '10'], ['inform', 'attraction', 'choice', '10'], ['inform', 'attraction', 'entrance fee', '?'], ['inform', 'attraction', 'entrance fee', '?'], ['inform', 'attraction', 'name', 'adc theatre'], ['inform', 'attraction', 'name', 'all saints church'], ['inform', 'attraction', 'type', 'architecture'], ['inform', 'attraction', 'type', 'nightclub'], ['recommend', 'attraction', 'address', 'park street'], ['recommend', 'attraction', 'area', 'east'], ['recommend', 'attraction', 'postcode', 'cb58nt'], ['request', 'attraction', 'name', '?'], ['select', 'attraction', 'type', 'theatre'], ['greet', 'general', 'none', 'none'], ['welcome', 'general', 'none', 'none'], ['inform', 'hospital', 'address', 'Hills Rd, Cambridge'], ['request', 'hospital', 'department', '?'], ['inform', 'hotel', 'address', '124 tenison road'], ['inform', 'hotel', 'address', '154 chesterton road'], ['inform', 'hotel', 'area', 'east'], ['inform', 'hotel', 'choice', '10'], ['inform', 'hotel', 'name', 'alexander bed and breakfast'], ['inform', 'hotel', 'parking', 'no'], ['inform', 'hotel', 'price range', 'moderate'], ['inform', 'hotel', 'price range', 'moderate'], ['inform', 'hotel', 'stars', '4'], ['inform', 'hotel', 'stars', '4'], ['offerbook', 'hotel', 'name', 'a and b guest house'], ['recommend', 'hotel', 'address', '124 tenison road'], ['recommend', 'hotel', 'area', 'east'], ['recommend', 'hotel', 'internet', 'yes'], ['recommend', 'hotel', 'price range', 'moderate'], ['recommend', 'hotel', 'stars', '4'], ['recommend', 'hotel', 'type', 'guesthouse'], ['request', 'hotel', 'area', '?'], ['request', 'hotel', 'book day', '?'], ['request', 'hotel', 'book people', '?'], ['request', 'hotel', 'internet', '?'], ['request', 'hotel', 'name', '?'], ['request', 'hotel', 'parking', '?'], ['request', 'hotel', 'price range', '?'], ['request', 'hotel', 'stars', '?'], ['select', 'hotel', 'area', 'east'], ['select', 'hotel', 'area', 'north'], ['select', 'hotel', 'none', 'none'], ['select', 'hotel', 'type', 'guesthouse'], ['inform', 'police', 'address', 'Parkside, Cambridge'], ['inform', 'police', 'phone', '01223358966'], ['inform', 'police', 'name', 'Parkside Police Station'], ['inform', 'restaurant', 'address', 'Regent Street City Centre'], ['inform', 'restaurant', 'address', 'Finders Corner Newmarket Road'], ['inform', 'restaurant', 'area', 'centre'], ['inform', 'restaurant', 'choice', '10'], ['inform', 'restaurant', 'choice', '10'], ['inform', 'restaurant', 'food', 'italian'], ['inform', 'restaurant', 'food', 'international'], ['inform', 'restaurant', 'food', 'chinese'], ['inform', 'restaurant', 'name', 'pizza hut city centre'], ['inform', 'restaurant', 'name', 'curry garden'], ['inform', 'restaurant', 'name', 'the good luck chinese food takeaway'], ['inform', 'restaurant', 'price range', 'cheap'], ['offerbook', 'restaurant', 'name', 'pizza hut city centre'], ['recommend', 'restaurant', 'address', 'Regent Street City Centre'], ['recommend', 'restaurant', 'area', 'centre'], ['recommend', 'restaurant', 'name', 'pizza hut city centre'], ['recommend', 'restaurant', 'phone', '01223323737'], ['recommend', 'restaurant', 'postcode', 'cb21ab'], ['recommend', 'restaurant', 'price range', 'cheap'], ['request', 'restaurant', 'area', '?'], ['request', 'restaurant', 'book day', '?'], ['request', 'restaurant', 'book people', '?'], ['request', 'restaurant', 'book time', '?'], ['request', 'restaurant', 'name', '?'], ['select', 'restaurant', 'area', 'centre'], ['select', 'restaurant', 'food', 'international'], ['select', 'restaurant', 'none', 'none'], ['select', 'restaurant', 'price range', 'cheap'], ['inform', 'taxi', 'none', 'none'], ['inform', 'taxi', 'phone', '29428952924'], ['inform', 'taxi', 'name', 'not available'], ['inform', 'train', 'arrive by', '05:51'], ['inform', 'train', 'choice', '10'], ['inform', 'train', 'day', 'monday'], ['inform', 'train', 'departure', 'cambridge'], ['inform', 'train', 'destination', 'london kings cross'], ['inform', 'train', 'duration', '51 minutes'], ['inform', 'train', 'leave at', '05:00'], ['request', 'train', 'day', '?'], ['request', 'train', 'departure', '?'], ['request', 'train', 'leave at', '?'], ['select', 'train', 'leave at', '05:00'], ['select', 'train', 'none', 'none']]
usr:[['request', 'train', 'departure', '?']]
sys:[['inform', 'attraction', 'address', 'pool way, whitehill road, off newmarket road'], ['inform', 'attraction', 'address', 'jesus lane'], ['inform', 'attraction', 'area', 'centre'], ['inform', 'attraction', 'choice', '10'], ['inform', 'attraction', 'entrance fee', '?'], ['inform', 'attraction', 'entrance fee', '?'], ['inform', 'attraction', 'name', 'adc theatre'], ['inform', 'attraction', 'name', 'all saints church'], ['inform', 'attraction', 'type', 'architecture'], ['inform', 'attraction', 'type', 'nightclub'], ['recommend', 'attraction', 'address', 'pool way, whitehill road, off newmarket road'], ['recommend', 'attraction', 'address', 'park street'], ['recommend', 'attraction', 'area', 'east'], ['recommend', 'attraction', 'postcode', 'cb58nt'], ['request', 'attraction', 'name', '?'], ['request', 'attraction', 'type', '?'], ['select', 'attraction', 'none', 'none'], ['select', 'attraction', 'type', 'swimmingpool'], ['bye', 'general', 'none', 'none'], ['greet', 'general', 'none', 'none'], ['inform', 'hospital', 'address', 'Hills Rd, Cambridge'], ['request', 'hospital', 'department', '?'], ['inform', 'hotel', 'address', '124 tenison road'], ['inform', 'hotel', 'address', '154 chesterton road'], ['inform', 'hotel', 'area', 'east'], ['inform', 'hotel', 'choice', '10'], ['inform', 'hotel', 'name', 'alexander bed and breakfast'], ['inform', 'hotel', 'parking', 'no'], ['inform', 'hotel', 'price range', 'moderate'], ['inform', 'hotel', 'price range', 'moderate'], ['inform', 'hotel', 'stars', '4'], ['inform', 'hotel', 'stars', '4'], ['offerbook', 'hotel', 'name', 'a and b guest house'], ['recommend', 'hotel', 'address', '124 tenison road'], ['recommend', 'hotel', 'area', 'east'], ['recommend', 'hotel', 'internet', 'yes'], ['recommend', 'hotel', 'name', 'a and b guest house'], ['recommend', 'hotel', 'stars', '4'], ['recommend', 'hotel', 'type', 'guesthouse'], ['request', 'hotel', 'book day', '?'], ['request', 'hotel', 'book people', '?'], ['request', 'hotel', 'book stay', '?'], ['request', 'hotel', 'internet', '?'], ['request', 'hotel', 'name', '?'], ['request', 'hotel', 'parking', '?'], ['request', 'hotel', 'stars', '?'], ['select', 'hotel', 'area', 'east'], ['select', 'hotel', 'area', 'north'], ['select', 'hotel', 'none', 'none'], ['select', 'hotel', 'price range', 'moderate'], ['select', 'hotel', 'type', 'guesthouse'], ['inform', 'police', 'address', 'Parkside, Cambridge'], ['inform', 'police', 'phone', '01223358966'], ['inform', 'restaurant', 'address', 'Regent Street City Centre'], ['inform', 'restaurant', 'address', 'Finders Corner Newmarket Road'], ['inform', 'restaurant', 'choice', '10'], ['inform', 'restaurant', 'choice', '10'], ['inform', 'restaurant', 'food', 'italian'], ['inform', 'restaurant', 'food', 'international'], ['inform', 'restaurant', 'food', 'indian'], ['inform', 'restaurant', 'name', 'pizza hut city centre'], ['inform', 'restaurant', 'name', 'the good luck chinese food takeaway'], ['inform', 'restaurant', 'postcode', 'cb21ab'], ['inform', 'restaurant', 'price range', 'cheap'], ['inform', 'restaurant', 'ref', '00000000'], ['offerbook', 'restaurant', 'name', 'pizza hut city centre'], ['recommend', 'restaurant', 'address', 'Regent Street City Centre'], ['recommend', 'restaurant', 'name', 'pizza hut city centre'], ['recommend', 'restaurant', 'phone', '01223323737'], ['recommend', 'restaurant', 'postcode', 'cb21ab'], ['request', 'restaurant', 'area', '?'], ['request', 'restaurant', 'book day', '?'], ['request', 'restaurant', 'book people', '?'], ['request', 'restaurant', 'book time', '?'], ['select', 'restaurant', 'area', 'centre'], ['select', 'restaurant', 'area', 'east'], ['select', 'restaurant', 'food', 'italian'], ['select', 'restaurant', 'food', 'indian'], ['select', 'restaurant', 'none', 'none'], ['select', 'restaurant', 'price range', 'cheap'], ['inform', 'taxi', 'none', 'none'], ['inform', 'taxi', 'phone', '85762989627'], ['request', 'taxi', 'destination', '?'], ['request', 'taxi', 'leave at', '?'], ['inform', 'train', 'arrive by', '05:51'], ['inform', 'train', 'choice', '10'], ['inform', 'train', 'departure', 'cambridge'], ['inform', 'train', 'destination', 'london kings cross'], ['inform', 'train', 'duration', '51 minutes'], ['inform', 'train', 'leave at', '05:00'], ['inform', 'train', 'leave at', '07:00'], ['inform', 'train', 'none', 'none'], ['request', 'train', 'day', '?'], ['request', 'train', 'departure', '?'], ['request', 'train', 'destination', '?'], ['request', 'train', 'leave at', '?'], ['select', 'train', 'leave at', '05:00'], ['select', 'train', 'none', 'none']]
usr:[['request', 'train', 'departure', '?']]
sys:[['inform', 'attraction', 'address', 'pool way, whitehill road, off newmarket road'], ['inform', 'attraction', 'area', 'centre'], ['inform', 'attraction', 'choice', '10'], ['inform', 'attraction', 'entrance fee', '?'], ['inform', 'attraction', 'entrance fee', '?'], ['inform', 'attraction', 'name', 'adc theatre'], ['inform', 'attraction', 'name', 'all saints church'], ['inform', 'attraction', 'name', 'ballare'], ['inform', 'attraction', 'type', 'theatre'], ['inform', 'attraction', 'type', 'architecture'], ['recommend', 'attraction', 'address', 'park street'], ['recommend', 'attraction', 'area', 'east'], ['recommend', 'attraction', 'postcode', 'cb58nt'], ['request', 'attraction', 'name', '?'], ['select', 'attraction', 'none', 'none'], ['select', 'attraction', 'type', 'swimmingpool'], ['bye', 'general', 'none', 'none'], ['greet', 'general', 'none', 'none'], ['reqmore', 'general', 'none', 'none'], ['welcome', 'general', 'none', 'none'], ['inform', 'hospital', 'address', 'Hills Rd, Cambridge'], ['request', 'hospital', 'department', '?'], ['inform', 'hotel', 'address', '124 tenison road'], ['inform', 'hotel', 'address', '154 chesterton road'], ['inform', 'hotel', 'area', 'east'], ['inform', 'hotel', 'choice', '10'], ['inform', 'hotel', 'name', 'acorn guest house'], ['inform', 'hotel', 'name', 'alexander bed and breakfast'], ['inform', 'hotel', 'parking', 'no'], ['inform', 'hotel', 'postcode', 'cb12dp'], ['inform', 'hotel', 'price range', 'moderate'], ['inform', 'hotel', 'stars', '4'], ['inform', 'hotel', 'stars', '4'], ['offerbook', 'hotel', 'name', 'a and b guest house'], ['recommend', 'hotel', 'address', '124 tenison road'], ['recommend', 'hotel', 'area', 'east'], ['recommend', 'hotel', 'internet', 'yes'], ['recommend', 'hotel', 'name', 'a and b guest house'], ['recommend', 'hotel', 'stars', '4'], ['recommend', 'hotel', 'type', 'guesthouse'], ['request', 'hotel', 'area', '?'], ['request', 'hotel', 'book day', '?'], ['request', 'hotel', 'book people', '?'], ['request', 'hotel', 'book stay', '?'], ['request', 'hotel', 'internet', '?'], ['request', 'hotel', 'name', '?'], ['request', 'hotel', 'parking', '?'], ['request', 'hotel', 'price range', '?'], ['request', 'hotel', 'stars', '?'], ['select', 'hotel', 'area', 'east'], ['select', 'hotel', 'area', 'north'], ['select', 'hotel', 'none', 'none'], ['select', 'hotel', 'price range', 'moderate'], ['select', 'hotel', 'type', 'guesthouse'], ['inform', 'police', 'address', 'Parkside, Cambridge'], ['inform', 'police', 'name', 'Parkside Police Station'], ['inform', 'police', 'phone', '01223358966'], ['book', 'restaurant', 'none', '00000000'], ['inform', 'restaurant', 'address', 'Regent Street City Centre'], ['inform', 'restaurant', 'address', 'Finders Corner Newmarket Road'], ['inform', 'restaurant', 'choice', '10'], ['inform', 'restaurant', 'choice', '10'], ['inform', 'restaurant', 'food', 'italian'], ['inform', 'restaurant', 'food', 'international'], ['inform', 'restaurant', 'food', 'indian'], ['inform', 'restaurant', 'name', 'pizza hut city centre'], ['inform', 'restaurant', 'name', 'the good luck chinese food takeaway'], ['inform', 'restaurant', 'postcode', 'cb21ab'], ['inform', 'restaurant', 'price range', 'cheap'], ['inform', 'restaurant', 'ref', '00000000'], ['recommend', 'restaurant', 'address', 'Regent Street City Centre'], ['recommend', 'restaurant', 'area', 'centre'], ['recommend', 'restaurant', 'name', 'pizza hut city centre'], ['recommend', 'restaurant', 'phone', '01223323737'], ['recommend', 'restaurant', 'postcode', 'cb21ab'], ['request', 'restaurant', 'area', '?'], ['request', 'restaurant', 'book day', '?'], ['request', 'restaurant', 'book people', '?'], ['request', 'restaurant', 'book time', '?'], ['select', 'restaurant', 'area', 'centre'], ['select', 'restaurant', 'area', 'east'], ['select', 'restaurant', 'food', 'italian'], ['select', 'restaurant', 'food', 'indian'], ['select', 'restaurant', 'none', 'none'], ['select', 'restaurant', 'price range', 'cheap'], ['inform', 'taxi', 'none', 'none'], ['inform', 'taxi', 'name', 'none'], ['request', 'taxi', 'destination', '?'], ['inform', 'train', 'arrive by', '05:51'], ['inform', 'train', 'choice', '10'], ['inform', 'train', 'departure', 'cambridge'], ['inform', 'train', 'destination', 'london kings cross'], ['inform', 'train', 'duration', '51 minutes'], ['inform', 'train', 'leave at', '05:00'], ['inform', 'train', 'leave at', '07:00'], ['inform', 'train', 'none', 'none'], ['request', 'train', 'day', '?'], ['request', 'train', 'departure', '?'], ['request', 'train', 'destination', '?'], ['request', 'train', 'leave at', '?'], ['select', 'train', 'leave at', '05:00'], ['select', 'train', 'none', 'none']]
usr:[['request', 'train', 'departure', '?']]
sys:[['inform', 'attraction', 'address', 'pool way, whitehill road, off newmarket road'], ['inform', 'attraction', 'address', 'jesus lane'], ['inform', 'attraction', 'area', 'centre'], ['inform', 'attraction', 'choice', '10'], ['inform', 'attraction', 'entrance fee', '?'], ['inform', 'attraction', 'entrance fee', '?'], ['inform', 'attraction', 'name', 'adc theatre'], ['inform', 'attraction', 'name', 'all saints church'], ['inform', 'attraction', 'name', 'ballare'], ['inform', 'attraction', 'type', 'theatre'], ['inform', 'attraction', 'type', 'nightclub'], ['recommend', 'attraction', 'address', 'pool way, whitehill road, off newmarket road'], ['recommend', 'attraction', 'address', 'park street'], ['recommend', 'attraction', 'area', 'east'], ['recommend', 'attraction', 'postcode', 'cb58nt'], ['request', 'attraction', 'name', '?'], ['select', 'attraction', 'none', 'none'], ['select', 'attraction', 'type', 'swimmingpool'], ['greet', 'general', 'none', 'none'], ['welcome', 'general', 'none', 'none'], ['inform', 'hospital', 'address', 'Hills Rd, Cambridge'], ['request', 'hospital', 'department', '?'], ['inform', 'hotel', 'address', '124 tenison road'], ['inform', 'hotel', 'address', '154 chesterton road'], ['inform', 'hotel', 'area', 'east'], ['inform', 'hotel', 'choice', '10'], ['inform', 'hotel', 'name', 'acorn guest house'], ['inform', 'hotel', 'name', 'alexander bed and breakfast'], ['inform', 'hotel', 'parking', 'no'], ['inform', 'hotel', 'postcode', 'cb12dp'], ['inform', 'hotel', 'price range', 'moderate'], ['inform', 'hotel', 'stars', '4'], ['inform', 'hotel', 'stars', '4'], ['offerbook', 'hotel', 'name', 'a and b guest house'], ['recommend', 'hotel', 'address', '124 tenison road'], ['recommend', 'hotel', 'area', 'east'], ['recommend', 'hotel', 'internet', 'yes'], ['recommend', 'hotel', 'name', 'a and b guest house'], ['recommend', 'hotel', 'stars', '4'], ['recommend', 'hotel', 'type', 'guesthouse'], ['request', 'hotel', 'area', '?'], ['request', 'hotel', 'book day', '?'], ['request', 'hotel', 'book people', '?'], ['request', 'hotel', 'book stay', '?'], ['request', 'hotel', 'internet', '?'], ['request', 'hotel', 'name', '?'], ['request', 'hotel', 'parking', '?'], ['request', 'hotel', 'price range', '?'], ['request', 'hotel', 'stars', '?'], ['select', 'hotel', 'area', 'east'], ['select', 'hotel', 'area', 'north'], ['select', 'hotel', 'none', 'none'], ['select', 'hotel', 'price range', 'moderate'], ['select', 'hotel', 'type', 'guesthouse'], ['inform', 'police', 'address', 'Parkside, Cambridge'], ['inform', 'police', 'name', 'Parkside Police Station'], ['inform', 'police', 'phone', '01223358966'], ['inform', 'restaurant', 'address', 'Regent Street City Centre'], ['inform', 'restaurant', 'address', 'Finders Corner Newmarket Road'], ['inform', 'restaurant', 'choice', '10'], ['inform', 'restaurant', 'choice', '10'], ['inform', 'restaurant', 'food', 'italian'], ['inform', 'restaurant', 'food', 'international'], ['inform', 'restaurant', 'food', 'indian'], ['inform', 'restaurant', 'name', 'pizza hut city centre'], ['inform', 'restaurant', 'name', 'the good luck chinese food takeaway'], ['inform', 'restaurant', 'postcode', 'cb21ab'], ['inform', 'restaurant', 'price range', 'cheap'], ['inform', 'restaurant', 'ref', '00000000'], ['recommend', 'restaurant', 'address', 'Regent Street City Centre'], ['recommend', 'restaurant', 'area', 'centre'], ['recommend', 'restaurant', 'name', 'pizza hut city centre'], ['recommend', 'restaurant', 'phone', '01223323737'], ['recommend', 'restaurant', 'postcode', 'cb21ab'], ['recommend', 'restaurant', 'price range', 'cheap'], ['request', 'restaurant', 'area', '?'], ['request', 'restaurant', 'book day', '?'], ['request', 'restaurant', 'book time', '?'], ['request', 'restaurant', 'name', '?'], ['select', 'restaurant', 'area', 'centre'], ['select', 'restaurant', 'area', 'east'], ['select', 'restaurant', 'food', 'italian'], ['select', 'restaurant', 'food', 'indian'], ['select', 'restaurant', 'none', 'none'], ['select', 'restaurant', 'price range', 'cheap'], ['inform', 'taxi', 'none', 'none'], ['inform', 'taxi', 'phone', '65665851511'], ['inform', 'taxi', 'name', 'not available'], ['request', 'taxi', 'destination', '?'], ['inform', 'train', 'arrive by', '05:51'], ['inform', 'train', 'choice', '10'], ['inform', 'train', 'departure', 'cambridge'], ['inform', 'train', 'destination', 'london kings cross'], ['inform', 'train', 'duration', '51 minutes'], ['inform', 'train', 'leave at', '05:00'], ['inform', 'train', 'leave at', '07:00'], ['inform', 'train', 'none', 'none'], ['request', 'train', 'day', '?'], ['request', 'train', 'departure', '?'], ['request', 'train', 'destination', '?'], ['request', 'train', 'leave at', '?'], ['select', 'train', 'leave at', '05:00'], ['select', 'train', 'none', 'none']]
usr:[['request', 'train', 'departure', '?']]

[hsien1993]

Hi @suntea233 , According to your logs, you can find out that the system just selected almost all actions in response, because without specifying the load_path, your system policy may be initialised incorrectly. It would be better to initialise the system in this way:

sys_policy = PPO(is_train=False, seed=0, dataset_name='multiwoz21', load_path="from_pretrained")

Let me know if the success rate is still too low.

[suntea233]

Hi, @hsien1993

I followed your instructions, but the results are still not satisfactory. The complete rate is 0.15, and the success rate is 0.1.

The running result of the code is as follows:

sys:[]
usr:[['request', 'none', 'none', 'none']]
sys:[['inform', 'attraction', 'choice', '10'], ['inform', 'attraction', 'name', 'abbey pool and astroturf pitch'], ['request', 'hotel', 'area', '?'], ['inform', 'police', 'address', 'Parkside, Cambridge'], ['inform', 'police', 'phone', '01223358966'], ['inform', 'police', 'name', 'Parkside Police Station'], ['request', 'restaurant', 'area', '?']]
usr:[['request', 'none', 'none', 'none']]
sys:[['inform', 'police', 'address', 'Parkside, Cambridge'], ['inform', 'police', 'postcode', 'cb11jg']]
usr:[['request', 'none', 'none', 'none']]
sys:[['reqmore', 'general', 'none', 'none']]
usr:[['request', 'attraction', 'address', '?']]
sys:[['inform', 'attraction', 'address', 'pool way, whitehill road, off newmarket road'], ['inform', 'attraction', 'area', 'east'], ['inform', 'attraction', 'name', 'abbey pool and astroturf pitch'], ['reqmore', 'general', 'none', 'none']]
usr:[['request', 'attraction', 'address', '?']]
sys:[['inform', 'attraction', 'address', 'pool way, whitehill road, off newmarket road'], ['inform', 'attraction', 'address', 'park street'], ['inform', 'attraction', 'entrance fee', '?'], ['inform', 'attraction', 'name', 'abbey pool and astroturf pitch'], ['inform', 'attraction', 'phone', '01223902088'], ['inform', 'attraction', 'postcode', 'cb58nt'], ['reqmore', 'general', 'none', 'none']]
usr:[['request', 'none', 'none', 'none']]
sys:[['request', 'hotel', 'area', '?'], ['inform', 'restaurant', 'address', 'Regent Street City Centre'], ['inform', 'restaurant', 'choice', '10'], ['request', 'restaurant', 'area', '?']]
usr:[['request', 'none', 'none', 'none']]
sys:[['request', 'hotel', 'area', '?'], ['inform', 'restaurant', 'address', 'Regent Street City Centre'], ['inform', 'restaurant', 'choice', '10'], ['inform', 'restaurant', 'choice', '10'], ['inform', 'restaurant', 'name', 'pizza hut city centre'], ['request', 'restaurant', 'area', '?']]
usr:[['request', 'none', 'none', 'none']]
sys:[['bye', 'general', 'none', 'none'], ['request', 'hotel', 'area', '?'], ['inform', 'restaurant', 'address', 'Regent Street City Centre'], ['inform', 'restaurant', 'choice', '10'], ['inform', 'restaurant', 'choice', '10'], ['inform', 'restaurant', 'name', 'pizza hut city centre'], ['request', 'restaurant', 'area', '?']]
usr:[['request', 'none', 'none', 'none']]
sys:[['bye', 'general', 'none', 'none'], ['request', 'hotel', 'area', '?'], ['inform', 'restaurant', 'address', 'Regent Street City Centre'], ['inform', 'restaurant', 'choice', '10'], ['inform', 'restaurant', 'choice', '10'], ['inform', 'restaurant', 'name', 'pizza hut city centre'], ['request', 'restaurant', 'area', '?']]
usr:[['request', 'none', 'none', 'none']]
sys:[['bye', 'general', 'none', 'none'], ['request', 'hotel', 'area', '?'], ['inform', 'restaurant', 'address', 'Regent Street City Centre'], ['inform', 'restaurant', 'choice', '10'], ['inform', 'restaurant', 'choice', '10'], ['inform', 'restaurant', 'name', 'pizza hut city centre'], ['request', 'restaurant', 'area', '?']]
usr:[['bye', 'general', 'None', 'None']]

I trained the GenTUS following README. And i ran the evaluate.py in GenTUS, the responses are normal.

[hsien1993]

Hi @suntea233 According to your logs, GenTUS cannot select the domain correctly. Did you try to GenTUS with default weight? The problem may come from not training GenTUS correctly or not loading the model properly, for example, a wrong path name.

However, I think the original issue is solved, you can raise a new issue and we can discuss there :)

[suntea233]

Thank you for such patient answering :)