keras-team / keras-nlp

Modular Natural Language Processing workflows with Keras
Apache License 2.0
740 stars 218 forks source link

Add text summarization task model #582

Open chenmoneygithub opened 1 year ago

chenmoneygithub commented 1 year ago

One interesting part is how we handle long context since our models have a limit on the input length due to positional embedding.

Ideally we should ship a task model for text summarization, but not limit the input size, otherwise it's more like a toy.

jbischof commented 1 year ago

I think something happened with the title @chenmoneygithub

jbischof commented 1 year ago

@chenmoneygithub I wonder if the Perceiver arch is a good fit (https://arxiv.org/abs/2202.07765).

It is specially motivated to accomodate long sequences.

jbischof commented 1 year ago

@chenmoneygithub this dataset seems cool: https://huggingface.co/datasets/openai/summarize_from_feedback