search

google-research / bert

TensorFlow code and pre-trained models for BERT
https://arxiv.org/abs/1810.04805
Apache License 2.0
38.12k stars 9.6k forks source link

how to use bert to text summary #576

Open xiongma opened 5 years ago

xiongma commented 5 years ago

@dalequark @ywkim @abhishekraok @imcaspar @dalequark I had search many website, can't find correctly way to use bert to text summary, can anyone tell me how to use bert to text summary?

xiongma commented 5 years ago

or just add transformer deocder behind bert pre train model?

Single430 commented 5 years ago

@policeme　You can try　tensor2tensor

My previous training was not very good. Do you have any good news and want to tell me．

xiongma commented 5 years ago

@Single430 I try transformer to generator summary, but when I train transformer model, before 3W times, It's very good, But after this time, the results are same when run eval, but the loss still go down, I don't know why, can you tell me ?

xiongma commented 5 years ago

I am so sorry for long time to reply you

xiongma commented 5 years ago

@Single430 when I try add pointer-generator network to transformer, but when I run, it always occurred nan, but it won't be in seq2seq+attention, do you know the reason? I try to find the solution, but I go through many website and github of pointer-generator's author, it still not find the solution, if you interested in it, I hope you can help me?

Single430 commented 5 years ago

@policeme

I am very sorry that the reply is late. I only used tensor2tensor to do Chinese abstract generation. I ran hundreds of thousands of steps. The last rouge1, 2, L got a good score, but in practice, I went to read the generated summary. The effect was very poor. I mentioned an official issue. I haven't responded yet. Finally, for your question, you can look at this Help, I've got NaNs!

wish you a happy life! super.single430

xiongma commented 5 years ago

@Single430 my wechat is 15025700935, can we discuss about this?

xiongma commented 5 years ago

@Single430 I sing this before, but it still not work for me.

Single430 commented 5 years ago

what's Wechat? it’s too late, I am going to sleep.

xiongma commented 5 years ago

sorry to disturb you. you said you had try chinese abstract generation, so I think you are chinese, sorry, if you don't mind, can you let your facebook name, I can add you.

wen75741 commented 5 years ago

Use GPT, Bert doesn't fit well for text generation. Don't forget to modify the bi-directional attention to single if you still want to try bert.