search

jasonwu0731 / ToD-BERT

Pre-Trained Models for ToD-BERT
BSD 2-Clause "Simplified" License
291 stars 55 forks source link

Can pretrained/finetuned TOD BERT be able to check when not to predict response in case of response selection task? #21

Closed mayankgoyal1993 closed 3 years ago

mayankgoyal1993 commented 3 years ago

Can pretrained/finetuned TOD BERT be able to check when not to predict response in case of response selection task?

Since it works on softmax loss like classification settings, what needs to be done to be able to make sure it does not select either of the response for some out of domain conversation happening on bot?

jasonwu0731 commented 3 years ago

Hello,

In order to know when not to respond, this is related to out-of-distribution detection and model calibration. The easiest way. could be fine-tuned TOD-BERT using binary cross entropy loss and calibrated it to the dev set probability. And find a fixed or dynamic threshold.

mayankgoyal1993 commented 3 years ago

Thanks for the response, let me try to do that.