search

facebookresearch / simmc

With the aim of building next generation virtual assistants that can handle multimodal inputs and perform multimodal actions, we introduce two new datasets (both in the virtual shopping domain), the annotation schema, the core technical tasks, and the baseline models. The code for the baselines and the datasets will be opensourced.
Other
131 stars 36 forks source link

action_evaluation expected file format #16

Closed seo-95 closed 4 years ago

seo-95 commented 4 years ago

Hi, I am trying to evaluate my model results on fashion dataset by using the script mm_action_prediction/tools/action_evaluation.py, but I have not understood the file format reported in mm_action_prediction/README.md.

  1. <action_token>: <action_log__prob>, with action_token you mean the name of the action?

  2. <attribute_label>: <attribute_val>, I have a vague idea of what attribute_label is by looking at the list IGNORE_ATTRIBUTES but it is not clear what to insert as attribute_val. For each dialogue turn, I predict as multilabel a series of arguments but I have not understood what you mean with label and arguments. Can you please better explain with an example what is a label versus a value for an attribute in the fashion dataset?

  3. I have filled the dict in this way for a particular turn of a dialogue. 

{'action': 'SearchDatabase', 'action_log_prob': {'None': -2.091989517211914, 'SearchDatabase': -0.17580336332321167, 'SearchMemory': -3.525150775909424, 'SpecifyInfo': -4.88762903213501, 'AddToCart': -7.144548416137695}, 'attributes': {}}

Anyway, I have issues with the script because attributes field is empty and the script does round_datum["attributes"][key]. How to fill the output json in the case no arguments have been found for a particular dialogue turn?

satwikkottur commented 4 years ago

Hello @seo-95 ,

  1. That is correct. action_token is indeed the name of the action, for example, SearchDataset.
  2. This is a good catch, I'll push a fix to check to account for empty attributes key.
seo-95 commented 4 years ago

Thank you @satwikkottur. Can you also please provide an example of attribute label vs attribute value? Thanks

seo-95 commented 4 years ago

I debugged the action_evaluation.py and I have noticed that it was a fault of mine. The attributes field in the model output JSON can not be empty, instead, it contains several keys, each one mapping to a list, not a single value as described in the mm_action_prediction/README.md. The right format is then:

{'action': 'SearchDatabase', 'action_log_prob': {'None': -2.091989517211914, 'SearchDatabase': -0.17580336332321167, 'SearchMemory': -3.525150775909424, 'SpecifyInfo': -4.88762903213501, 'AddToCart': -7.144548416137695}, 'attributes': {'attributes': []}}

This issue can be closed after a better description of attribute_label and attribute_val in the README. Thank you.

satwikkottur commented 4 years ago

On taking a closer look, attributes contains all the keys (API attributes) required to be predicted for a given API. The part of the code you're referring to (here) is reached only when the predicted action matches with the ground truth.

I'll add the description of attribute_label and attribute_val in the README.md. Thanks!