search

yusanshi / news-recommendation

Implementations of some methods in news recommendation.
MIT License
241 stars 50 forks source link

The data preprocessing part takes up too much CPU and memory #17

Closed 1607722534 closed 3 years ago

1607722534 commented 3 years ago

The data preprocessing part takes up too much CPU and memory, especially the "parse news" part. I don't know whether the memory is out of bounds or for other reasons. I hope to give the preprocessed files or try to speed up with GPU. Thank you

yusanshi commented 3 years ago

Sorry but my local project has been deleted by myself so I couldn't provide them.

I think it should be related to the BERT(RoBERTa) preprocessing part. The BERT preprocessing is for Exp2 model in README. You can try removing RoBERTa related code if you don't need the Exp2 model.

More specifically, remove L111-L172, L270, L305 in data_preprocess.py.

1607722534 commented 3 years ago

Thank you for your code and reply. I wish you more papers.

------------------ 原始邮件 ------------------ 发件人: "yusanshi/NewsRecommendation" @.>; 发送时间: 2021年8月9日(星期一) 下午3:08 @.>; @.**@.>; 主题: Re: [yusanshi/NewsRecommendation] The data preprocessing part takes up too much CPU and memory (#17)

Sorry but my local project has been deleted by myself so I couldn't provide them.

I think it should be related to the BERT(RoBERTa) preprocessing part. The BERT preprocessing is for Exp2 model in README. You can try removing RoBERTa related code if you don't need the Exp2 model.

More specifically, remove L111-L172, L270, L305 in data_preprocess.py.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

yusanshi commented 3 years ago

:)