search

facebookresearch / DPR

Dense Passage Retriever - is a set of tools and models for open domain Q&A task.
Other
1.73k stars 304 forks source link

Question about reader training #204

Closed TZeng20 closed 2 years ago

TZeng20 commented 2 years ago

Hi,

Am wondering if you could help me understand more about the input for training the reader?

  1. Are the positive and negative contexts for each question generated using the retriever before training or during training?
  2. What does a training batch look like? Is sample_batch a batch of question, passage, answer pairs and then inputs are the transformer model inputs of shape (num_questions, num_passages, seq_length)?
vlad-karpukhin commented 2 years ago

Hi @TZeng20 ,

  1. positive and negative contexts for each question are generated before the training process. We use own DPR dense index to mine positives and negatives contexts (that is the point of open domain setup). We also use gold positive passages as positives provided by datasets and some extra heuristics regarding the choice of retrieved passages for the reader training.
  2. The sample_batch contains a set of questions each accompanied with a pool of positive and hard negative passages. The inputs are what actually go into the model. create_reader_input() selects exactly one positive and a number of negative contexts from the provided pools, concatenates with the questions and tokenizes them converting to tensors.

Hope this clarifies your questions