dmmiller612 / bert-extractive-summarizer

Easy to use extractive text summarization with BERT
MIT License
1.38k stars 305 forks source link

Summarizer with large en_core_web_lg model #80

Closed zdhernandez closed 3 years ago

zdhernandez commented 3 years ago

I am installing the en_core_weblg model by running python -m spacy download en_core_web_lg. However, the documentation does not provide an explicit clear example to make sure that if this model is downloaded that it will use it as opposed to the small model which is the one that comes by default.

If I install the large model, doe that mean that the Summarizer will pickup the large and not the small model? if I do:

model = Summarizer()
result = model(body, num_sentences=3)
print(result)

Is that enough to make use of the large model en_core_web_lg ?

ZohebAbai commented 3 years ago

I am installing the en_core_weblg model by running python -m spacy download en_core_web_lg. However, the documentation does not provide an explicit clear example to make sure that if this model is downloaded that it will use it as opposed to the small model which is the one that comes by default.

If I install the large model, doe that mean that the Summarizer will pickup the large and not the small model? if I do:

model = Summarizer()
result = model(body, num_sentences=3)
print(result)

Is that enough to make use of the large model en_core_web_lg ?

According to docs, If you want to use en_core_web_lg in CoreferenceHandler, you have to follow accordingly

from summarizer.coreference_handler import CoreferenceHandler handler = CoreferenceHandler(spacy_model = 'en_core_web_lg') model = Summarizer(sentence_handler=handler) result = model(body, num_sentences=3) print(result)

I would like to mention that, while doing this my GPU RAM crashed.

zdhernandez commented 3 years ago

@ZohebAbai Thank you. I got it to work. Your example helped. I was using the wrong order of calls.