search

dmmiller612 / bert-extractive-summarizer

Easy to use extractive text summarization with BERT
MIT License
1.39k stars 305 forks source link

Removing elements of NLP spacy object #40

Closed evanmiller29 closed 3 years ago

evanmiller29 commented 4 years ago

This library is bloody cool, good work mate. Is there a way to disable some elements of the spacy NLP object to reduce memory usage?

dmmiller612 commented 4 years ago

That's a good question. So recently, I added some capabilities that now drop coreference from the dependencies. It is optional through a class. That should remove a large bulk of the dependencies.

dmmiller612 commented 3 years ago

Closing as stale. Since this message, we also have support for a custom sentence handler before it reaches the summarization. We also dropped requirements for coreference since it expanded the spacy model size and was causing issues with dependencies. Let me know if any other issues arise.