zhiqihuang
commented
3 years ago Hello, I have a question about the BERT encoders. In the paper, it is said that "ANCE can be used to train any dense retrieval model. For simplicity, we use a simple set up in recent research (Luan et al., 2020) with BERT Siamese/Dual Encoder (shared between q and d), dot product similarity, and negative log likelihood (NLL) loss." So actually, only one encoder is used to encode queries and documents separately. However, in the "model.py", the "BiEncoder" is as follows:
class BiEncoder(nn.Module): """ Bi-Encoder model component. Encapsulates query/question and context/passage encoders. """ def __init__(self, args): super(BiEncoder, self).__init__() self.question_model = HFBertEncoder.init_encoder(args) self.ctx_model = HFBertEncoder.init_encoder(args)There are two encoders are defined.
Kudos! You asked the exact question I have. In the paper. it keeps using "BERT-Siamese". To my understanding, siamese here means a shared encoder between query and document.
In fact, if two encoders are used, Dense Retriever doubles the parameter size comparing to model like BERT Reranker or ColBERT.
kaishxu
Hello, I have a question about the BERT encoders. In the paper, it is said that "ANCE can be used to train any dense retrieval model. For simplicity, we use a simple set up in recent research (Luan et al., 2020) with BERT Siamese/Dual Encoder (shared between q and d), dot product similarity, and negative log likelihood (NLL) loss." So actually, only one encoder is used to encode queries and documents separately. However, in the "model.py", the "BiEncoder" is as follows:
There are two encoders are defined.