search

huggingface / transfer-learning-conv-ai

🦄 State-of-the-Art Conversational AI with Transfer Learning
MIT License
1.73k stars 431 forks source link

num of candidates and labels #43

Closed anandhperumal closed 4 years ago

anandhperumal commented 4 years ago

What is number of candidates ? is it the number of person in the chat? And what is the use of setting -1 for all the labels except the last reply? And in the cost function the ignore index = -1, therefore the loss will not be calculated for this labels.

 instance["lm_labels"] = [-1] * len(instance["input_ids"])

And where is the distractor in this code?

rlouf commented 4 years ago

Next time, so we can help you quicker, would you mind pointing precisely where the lines are located in the code? Even better, providing a link?

I'm not familiar with this code in particular, but the "-1" labels stand for the padding tokens which is why you are skipping them for the loss.

anandhperumal commented 4 years ago

Sorry about that. build_input_from_segments method line no 66 we're doing padding when the labels flag is set to True. But in line 65 we're also creating labels only with -1 and the entire labels value is -1, I don't understand why are we doing that and it is being done num_candiates-1 times. What is the role of num of candidates here. when I debugged the code on the second last utterance we set the proper label i.e. the last utterance as the label. In this blog persona chat they mentioned they have used distractor to train the model but in the code I don't see any distractor.

anandhperumal commented 4 years ago

@thomwolf can you please comment on this. what is the reason behind training each input having kind of two sub inputs: one with input : persona + history + reply and labels :all -1 (my question is on this part, how is this helpful) and persona + history + reply and labels = reply the above scenario is for num_candidates =2 Thanks