search

budzianowski / multiwoz

Source code for end-to-end dialogue model from the MultiWOZ paper (Budzianowski et al. 2018, EMNLP)
MIT License
867 stars 199 forks source link

Active Intent Labeling Issues #117

Open KeremTurgutlu opened 1 year ago

KeremTurgutlu commented 1 year ago

I am trying to understand the role that active intents play in the dataset and replicate

Active Intent Accuracy (MultiWoz 2.1) 0.924 from .

But I've noticed a lot of active intent label issues.

If we look at the definition from:

active_intent : The intent corresponding to the service of the frame which is currently being fulfilled by the system. It takes the value "NONE" if none of the intents are active.

But I am seeing inconsistent labels in the dataset. Few examples:

dialog_id: PMUL0386.json

[USER] I am looking for information on a restaurant called cafe uno
{'find_restaurant'} - correct

 [USER] I am looking for information on a restaurant called cafe uno [SYSTEM] There is a cafe uno at 32 Bridge St., would you like me to make a reservation for you there? [USER] Yes please. I need a reservation for 4 people, at 15:15, and I'd like that for Saturday,please.
{'book_restaurant', 'find_taxi'} - find_taxi is incorrect as it is not mentioned in the conversation.

 [USER] I am looking for information on a restaurant called cafe uno [SYSTEM] There is a cafe uno at 32 Bridge St., would you like me to make a reservation for you there? [USER] Yes please. I need a reservation for 4 people, at 15:15, and I'd like that for Saturday,please. [SYSTEM] Here is the booking information.Booking was successful. The table will be reserved for 15 minutes. Reference number is : 9ZL6WG4I . [USER] I am also in need of a hotel, preferably a guesthouse that is in the north and has free parking and wifi.
{'find_hotel'} - correct

 [USER] I am looking for information on a restaurant called cafe uno [SYSTEM] There is a cafe uno at 32 Bridge St., would you like me to make a reservation for you there? [USER] Yes please. I need a reservation for 4 people, at 15:15, and I'd like that for Saturday,please. [SYSTEM] Here is the booking information.Booking was successful. The table will be reserved for 15 minutes. Reference number is : 9ZL6WG4I . [USER] I am also in need of a hotel, preferably a guesthouse that is in the north and has free parking and wifi. [SYSTEM] The Acorn Guesthouse is a moderately priced, 4-star accommodation with internet and parking. Would that suit you? [USER] Yes, please book me for 4 people 4 nights, starting saturday
{'book_hotel', 'book_restaurant'} - book_restaurant is incorrect as it is already fulfilled in a previous turn
 [USER] I would like to visit a college in the centre of town.
{'find_hotel', 'find_attraction'} - find_hotel is incorrect.

 [USER] I would like to visit a college in the centre of town. [SYSTEM] There are 13 colleges in the center, but I would recommend christ's college. It's at saint andrew's street, postcode cb23bu, phone number 01223334900, and the entrance fee is free! [USER] Thank you very much for your help.
set() - correct find_attraction is fulfilled.

 [USER] I would like to visit a college in the centre of town. [SYSTEM] There are 13 colleges in the center, but I would recommend christ's college. It's at saint andrew's street, postcode cb23bu, phone number 01223334900, and the entrance fee is free! [USER] Thank you very much for your help. [SYSTEM] Is there anything else I can assist you with today? [USER] Yes, can you help me locate a guesthouse that stay in that is also in the centre of town?
{'find_hotel', 'find_attraction'} - find_attraction is incorrect it is already fulfilled.

 [USER] I would like to visit a college in the centre of town. [SYSTEM] There are 13 colleges in the center, but I would recommend christ's college. It's at saint andrew's street, postcode cb23bu, phone number 01223334900, and the entrance fee is free! [USER] Thank you very much for your help. [SYSTEM] Is there anything else I can assist you with today? [USER] Yes, can you help me locate a guesthouse that stay in that is also in the centre of town? [SYSTEM] I have two, in the cheap range. Are there any other criteria you would like me to narrow it by? [USER] Yeah, I'd like someplace with free wifi, please.
{'find_hotel'} -correct.

These are just some random examples I found without any difficulty, I guess this issues is not rare.

yuleiqin commented 7 months ago

Yes, I am also very curious about the definition of the active intent. I am also wondering if we could do to extract true annotations from these active intents of the frames.