search

hongwang600 / DocRed

41 stars 6 forks source link

Question about Bert. #3

Open crystal-xu opened 4 years ago

crystal-xu commented 4 years ago

Dear authors, thanks for your efforts. I am planning to use your Bert implementation as a baseline for my MSc project concerning document-level RE. In the final part of the paper, you compare the performance of the sentence-encoding model and BiLSTM. Would you like to tell me if the BiLSTM refers to the baseline model in the DocRED paper?

hongwang600 commented 4 years ago

Sorry for the late reply. Yes, the BiLSTM is the baseline model in the DocRED paper.

crystal-xu commented 4 years ago

Thanks for your reply.

Would you mind telling you if you implemented BERT and two-phase BERT separately? I am looking at the master branch but get a bit confused? Could you point me to the correct branches of these two implementations?

Should the pre-trained model for the first phase (RE existing or not) be achieved from "rel_exist_bert_cls_sep" branch, which then can be used for the training process (RE prediction) in the second phase from the "master" branch?

hongwang600 commented 4 years ago

Yes. "rel_exist_bert_cls_sep" branch is used for the first phase, and the master branch is used for RE prediction.

crystal-xu commented 4 years ago

Thanks. I also notice that in your sentence encoder implementation:

    context_output = self.rnn(sent, context_lens)
    context_output = torch.cat([context_output, sent_emb], dim=-1)
    #context_output = sent_emb

It seems that you use the concatenation of the document-level representation and sentence-level representation. As far as I'm concerned, maybe only the sent_emb should be utilized. Do I have some misunderstanding here?

Thanks

hongwang600 commented 4 years ago

I think you are right. I did this to see if the combination of global context and sentence embedding would help. You should use sent_emb if you want to use sentence level encoding.

crystal-xu commented 4 years ago

Got it! Thanks for your quick reply.