how bert do text pair matching for long document

[Awyshw]

Hi, BangLiu, I use Bert to fine-tuning, extract two long document pair first 256 words, feed them bert, output is feed classifcation layer; My model batch_size is 6, epoch 2; but the result of train is accuracy = 0.5519683, global_step = 517, loss = 0.72958165, precision = 0.49228394, recall = 0.2029262; The differences with papers's result is very big. So, I want to know how do you do use bert-text-matching?

[BangLiu]

Hi, we let Bert encode the first N (N = maximum length bert can process) words of each document to get the encoding of each doc from the [CLS] position. After that, we train FFN binary classifier to get the matching result. Note that BERT is not fixed (we finetune it). We utilize Huggingface's implementation. For hyperparameters, we utilized the default parameters in run_glue.py. I am not sure whether it has changed until now, as the Transformers library of HuggingFace has updated for multiple times.