search

dmmiller612 / bert-extractive-summarizer

Easy to use extractive text summarization with BERT
MIT License
1.4k stars 308 forks source link

Input limit? #22

Closed dzimmerman-nci closed 4 years ago

dzimmerman-nci commented 4 years ago

I know BERT has an input token limit of 512. However, this summarizer does not have an input limit because you are looping through the sentences and embedding each one with BERT separately (assuming the sentences are <512 tokens). Is this correct?

dmmiller612 commented 4 years ago

Yeah, I think that is correct, but I will have to double check.

dzimmerman-nci commented 4 years ago

Thanks. Please let me know.

dmmiller612 commented 4 years ago

Sorry, it took me so long. It is currently a requirement of 512 tokens per sentence. It can handle any number of sentences (memory constrained of course), but the sentence needs to be less than 512 tokens.