2022: SIGIR: Curriculum Learning for Dense Retrieval Distillation

[rezaBarzgar]

Title: Curriculum Learning for Dense Retrieval Distillation

year: 2022

Venue: SIGIR

Main Problem

This research aims to improve the performance of dense retrieval models using an existing base re-ranker model with the distillation knowledge method and curriculum learning.

Related Works

(Working on it)

proposed method

Although there are large-scale training datasets for dense retrieval models, there are a few datasets documents that are judged by the given query. This paper uses a combination of knowledge distillation (Teacher Student model) and Curriculum Learning to mitigate the mentioned problem and improve dense retrieval models. A teacher model is an expensive re-ranking model that uses cross-encoding which makes pairs of query-document for the input of a student model. A student model is a light weighted dense retrieval model compared to the teacher model. During iterations, the student model learns to examine coarse-grained distinctions between documents and goes to recover fine-grained distinctions. It means that at first iterations, by using the curriculum learning approach, the student learns to discriminate between related documents and non-related ones of a given query. through the next iterations, the student model learns how to rank related documents. The output of this paper is the CL-DRD Framework that can be used in the training of dense retrieval models.

Major Gaps

:) (I'm afraid I couldn't find any, but working on it)

Input

Query

output

Ranked Documents

Datasets

MS MARCO-Dev TREC-DL’19 TREC-DL’20

Codebase

Github

[rezaBarzgar]

@hosseinfani I opened this issue page for a summary of the paper, but I cannot add labels to the issue. I tried my best to summarize it.

[hosseinfani]

@rezaBarzgar fixed. let me know when the summary is complete.