Question about submission models

facebookresearch / simmc

With the aim of building next generation virtual assistants that can handle multimodal inputs and perform multimodal actions, we introduce two new datasets (both in the virtual shopping domain), the annotation schema, the core technical tasks, and the baseline models. The code for the baselines and the datasets will be opensourced.

Other

131 stars 36 forks source link

Question about submission models #41

Closed billkunghappy closed 4 years ago

billkunghappy commented 4 years ago

Hi, For the final submission to DSTC9 Track4, are we allowed to submit multiple models(for ensemble) for each subtask? Thanks.

billkunghappy commented 4 years ago

For the submission instruction and timelines., we are a little bit confused about the description. The PR/check-in contains an output directory with the model output in JSON format that can be scored with the automatic scripts provided for that sub-task. Does this means we have to fork the simmc repository and create a directory somewhere in it which contains the output files, then make a pull request? If so, where should we create the output files directory? Or we should fork the Team repo and do the thing described above?

shanemoon commented 4 years ago

Hi @billkunghappy,

1) Yes, an ensemble approach is allowed. Please note that the performance will be measured on one predictions set (e.g. ensembeled or aggregated predictions) though.

2) Either of the approaches (forking the simmc repo or the team repo) is fine - as long as the repository is visible by the final submission deadline (Oct 5).

We look forward to your submissions - please let us know if you have any other questions.