search

huggingface / blog

Public repo for HF blog posts
https://hf.co/blog
2.28k stars 708 forks source link

encoder-decoder (bert2bert) model for summarization task doesn't work in v4.18 #292

Closed AbuUbaida closed 2 years ago

AbuUbaida commented 2 years ago

This notebook (github link) is a great roadway to summarization task. However, the BERT2BERT model demonstrated by @patrickvonplaten doesn't work properly in transformers v4.18. What the model always generates is just a summary (vector of tokens) that contains only special tokens like [CLS], and [SEP] regardless of the input. In my case, the output was something like this: [CLS] [CLS] [CLS] [CLS] [CLS] ............................ [SEP]

@patrickvonplaten or anybody could say please what the necessary adjustments might be needed in that existing blog?

patrickvonplaten commented 2 years ago

Hey @AbuUbaida,

Thanks for opening this issue. Currently I sadly don't have much time to fix this, but I'd be happy to work on it together if you want.

If you'd like, you could copy the google colab, then pin Transfomers to the new 4.18 version, and then we see together how to make it work?

AbuUbaida commented 2 years ago

Thanks, @patrickvonplaten for getting back to me. I have shared a copy of the notebook with you pinned with transformers and datasets at their latest releases 4.18.0 and 2.1.0 respectively. You could give directions on what next or take steps by yourself. Thanks!

AbuUbaida commented 2 years ago

Alhamdulillah! It’s now working for me surprisingly; just run the cells of the colab notebook with my custom model what I did earlier also, but this time explicitly mentioned transformers==4.18.0 and datasets==2.1.0 in the code. Though it doesn’t make sense to me since the latest version was being installed previously by default without mentioning explicitly.

patrickvonplaten commented 2 years ago

Interesting...I don't seem to have access to the copy of your notebook :-)

AbuUbaida commented 2 years ago

@patrickvonplaten sorry for being late. Actually, I removed the file as all were going right. But, maybe last time, I also needed to restart the runtime after checking versions several times. Thanks for the lucky instruction: :-)

then pin Transformers to the new 4.18 version