search

facebookresearch / simmc

With the aim of building next generation virtual assistants that can handle multimodal inputs and perform multimodal actions, we introduce two new datasets (both in the virtual shopping domain), the annotation schema, the core technical tasks, and the baseline models. The code for the baselines and the datasets will be opensourced.
Other
131 stars 36 forks source link

F1 score computation for empty API arguments list #18

Closed seo-95 closed 4 years ago

seo-95 commented 4 years ago

Whenever the API arguments ground truth list is empty, the mm_action_prediction/tools/action_evaluation.py script does not evaluate the goodness of the model prediction. Thus it does not penalize a model that predicts N arguments and it does not reward a model that predicts exactly 0 arguments as the ground truth. I know that the recall computation is not feasible for empty ground truth and I want to propose this 1-0 method that assigns an f1 score of 1.0 if the model arguments prediction list is empty, given that the ground truth arguments list is empty, and 0 if model arguments prediction list is not empty and the ground truth list is empty.

The final results are pretty different in terms of scores so please consider this possible change carefully. I think that the major concern could be the fact that an empty list of arguments is always present with a None or AddToCart so I don't know if this could create an unfair and biased score. Thank you.

facebook-github-bot commented 4 years ago

Hi @seo-95!

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file.

In order for us to review and merge your code, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

facebook-github-bot commented 4 years ago

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

seo-95 commented 4 years ago

I checked, empty argument list is present if and only if the API is None or AddToCart, so it has no sense to compute the score in this case. I close this pull request.

satwikkottur commented 4 years ago

Thanks for raising this issue @seo-95. Because the empty attribute list exists for None or AddToCart, we ignore F1 computations for the instances with these ground truth APIs.