SubTask #3 evaluation lower case issue

facebookresearch / simmc

With the aim of building next generation virtual assistants that can handle multimodal inputs and perform multimodal actions, we introduce two new datasets (both in the virtual shopping domain), the annotation schema, the core technical tasks, and the baseline models. The code for the baselines and the datasets will be opensourced.

Other

131 stars 36 forks source link

SubTask #3 evaluation lower case issue #34

Closed hxssgaa closed 4 years ago

hxssgaa commented 4 years ago

Hi,

I just found the pre-processed data in the fashion and furniture dataset used in the baseline have different casing format regarding to words like "that" and "That":

Is this really meaningful for the evaluation method to differentiate the two? Current evaluation method doesn't do the normalization between the two. Could you add lower case method before comparing the slot values? Thanks.

hxssgaa commented 4 years ago

The model can learn to cope with the casing format, no need to change to lower case.