aditeyabaral / newsnow

Automated document merging and extractive summarization of news articles
4 stars 5 forks source link

Added error handling for w2v model loading #6

Closed AindriyaBarua closed 3 years ago

AindriyaBarua commented 3 years ago

Hello @aditeyabaral I was trying to build and run this project when I came across a re-occuring error: On my console: w2v_error I tried on colab too: WORD2VEC_COLAB

On investigating more on the issue, I found out that since the size of the pre-trained word2vec-google-news-300 model is around 1.5 GB, on an unstable internet, this download may fail very often, as it happened with me multiple times. So I added some error handling on that.

Previously the w2v model was loaded without a function, I have made a function for it with proper error handling, it tries to load the model, on meeting an error, it retries up to 5 times before throwing the error and terminating.

I have called the function before the user enters the Query, that is outside the main while loop, so that the model is loaded only once and can be reused every time the user wants to make new queries, hence saving us from doing memory and data consuming task multiple times.

Hope the contribution is useful, I am open to further discussion or modification.