search

hanjanghoon / BERT_FP

Fine-grained Post-training for Improving Retrieval-based Dialogue Systems - NAACL 2021
96 stars 19 forks source link

Issue on reimplementation experiment #7

Open KuzmaNg opened 2 years ago

KuzmaNg commented 2 years ago

Hi, authors of Bert-FP, the SOTA in Response Selection tasks. Excited to see that the post-training strategy works so well with the sub-context-response pairs. Recently, I try to reimplement this work, but I got some confusions. It might take you some time to do a little help.

  1. In post-training (Section 4.2), it says "constructed 6M sub-context-response pairs for Douban", but it's about 2M when running the provided code (FPT/e-commerce_final.py). I can't find out what's missing.
  2. Was EDC being pre-trained for 34 epochs with only one GPU card? Params: seq_len=240, train_batch_size=50.
  3. About results without fine-tuning, BERT-FP-NF, how to convert the pre-training NSP 3-class (0,1,2) task into a 2-class (0,1) task?
  4. About fine-tuning, default epoch is 2, is that the number for the reported results? Since BERT_finetuning.py reflects that patience is used when epoch > 2.

Thank you, really appreciate.

Kirili4ik commented 1 year ago

Hi! I'm not the author, but I think I have found the answer for your question 1.

  1. I believe 6M is 2M * 3 cause there is 1 right answer, 1 random from the same dialogue and 1 completely random. My guess is that the authors multiplied real numbers by 3 for the paper. (running ubuntu dataset I also see 4M with the state 12M)

BTW, have you found out the amount of epochs needed for PT and FT? And where does your "34 epochs" come from? That would be really helpful if you could provide any additional info, thank you in advance.