Summarization

[marcverhagen]

This is now very simplistic, concatenating the abstract and the text and taking the first N characters or tokens.

Experiment with something better, maybe one of the following:

• https://www.turing.com/kb/5-powerful-text-summarization-techniques-in-python
• https://pypi.org/project/pysummarization/

[marcverhagen]

This is now recast as integrating https://github.com/brandeis-llc/xdd-LLMs into the processing chain.