search

facebookresearch / simmc

With the aim of building next generation virtual assistants that can handle multimodal inputs and perform multimodal actions, we introduce two new datasets (both in the virtual shopping domain), the annotation schema, the core technical tasks, and the baseline models. The code for the baselines and the datasets will be opensourced.
Other
131 stars 36 forks source link

Bug in baseline? (missing sigmoid) #21

Closed cccntu closed 4 years ago

cccntu commented 4 years ago

https://github.com/facebookresearch/simmc/blob/9308dc2dd749665c59a28043c72aace45f3d96b5/mm_action_prediction/models/action_executor.py#L215

This is a multi-label prediction, so there should be a sigmoid. (or simply change 0.5 to 0) I tried it and the Action Attribute Accuracy increased from 54.79 to 56.06. Will the code and score in readme be updated?

satwikkottur commented 4 years ago

Hello @cccntu ,

Thanks for finding this. I'll add the sigmoid, retrain the baselines for Fashion, and update the scores.