shashank-gv / hugging

0 stars 0 forks source link

If text is longer than can be handled, split into blocks #18

Open gjwgit opened 2 years ago

gjwgit commented 2 years ago

Would it make sense to split long input text into blocks that are each summarised by the model.

Write a message to STDERR that this is being done.

At the least write a message to STDERR to indicate the text is truncated because it is longer than XXXXX characters.

For example, without this information the last sentence below makes no sense:

ml summarize hugging --model=bert https://www.abc.net.au/news/2022-08-16/morrison-secret-ministerial-appointments-legal-or-not-icac/101318718
Scott Morrison's secret ministerial appointments likely legal and outside federal ICAC remit, experts say.
Independent MPs say the former prime minister's actions should be investigated by a federal integrity body.
A former NSW ICAC assistant commissioner says many questions remain unanswered.
Prime Minister Anthony Albanese is expected to provide an update today after receiving briefings from the.