how to properly output reranker scores?

[ali-abz]

Hi there. Using Capreolus, I was able to train a Bert-based reranker in a "pair-wise manner" and it was sound. Now I want to do the same but this time I want it to be "point-wise". I have some troubles to do it and I don't know what am I doing wrong. I believe my problem lies in how my reranker outputs its score but I am not certain. In order to create this reranker:

• I set "softmaxloss": True in trainer config section since the comments in the code says: "True to use softmax loss (over pairs) or False to use hinge loss"
• In the extractor, added a sanity check:

  def id2vec(self, qid, posid, negid=None, label=None):
  if negid is not None:
      raise ValueError(f'got negid: {negid}')

• In reranker, the output of score and test are same:

  def score(self, d):
      return self.test(d)
  
  def test(self, d):
      score = self.model(d['paragraph_ids'], d['paragraph_mask'], d['query_ids'], d["query_mask"])
      return score

• And here is how things are done in the model itself:

  def forward(self, paragraph_ids, paragraph_mask, question_ids, question_mask):
      question_embeddings = self.question_bert(question_ids, attention_mask=question_mask)[0][:,0,:]
      paragraph_embeddings = self.paragraph_bert(paragraph_ids, attention_mask=paragraph_mask)[0][:,0,:]
  
      similarity = []
      for x, y in zip(question_embeddings, paragraph_embeddings):
          sim = torch.dot(x, y).view(1, -1)
          similarity.append(sim)
      return similarity

  Please do note that .view(1, -1) part. Right now, this reranker outputs a list of (1, 768) vectors. If I omit that .view(1, -1) part, the loss function (pair_softmax_loss in reranker/common.py) will raises an error: IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1). This is the line that would cause the problem if I omit that part: https://github.com/capreolus-ir/capreolus/blob/8695a471f9d8e911ad12778a82327e3973f92af0/capreolus/reranker/common.py#L56 Keeping that view part, the training gets done with no errors but when it comes to prediction time, there will be an error: AttributeError: 'list' object has no attribute 'view' from this line: https://github.com/capreolus-ir/capreolus/blob/8695a471f9d8e911ad12778a82327e3973f92af0/capreolus/trainer/pytorch.py#L341 If I redefine score and test like this:

  def score(self, d):
      score = self.model(d['paragraph_ids'], d['paragraph_mask'], d['query_ids'], d["query_mask"])
      return score
  
  def test(self, d):
      score = self.model(d['paragraph_ids'], d['paragraph_mask'], d['query_ids'], d["query_mask"])
      score = torch.stack(score)
      return score

  everything works. Since I had to use this tricks, I have a feeling that there is something wrong with what I am doing here. So I would be grateful if you guys help me here. What is the proper way to pass scores?

[andrewyates]

Hey, this is a good question. I think most of this approach makes sense, but it's been a while since I've looked into this scenario. One thing I'm confused about is what your forward function is doing and returning. The list error comes from the fact that this is returning a list rather than a tensor (typically with the shape (batch size, 1)). Are you using this function to create a batch within the model, rather than having the trainer (and sampler) create a batch from the available training instances?

[ali-abz]

Hi Andrew, Unfortunately, at the current time I am unable to test my code again. I will close this issue for now and I will seek help later. Thanks for your comment.