Chunking for summarize - Githubissues

nextcloud / integration_openai

OpenAI integration in Nextcloud

GNU Affero General Public License v3.0

51 stars 11 forks source link

Chunking for summarize #150

Open julien-nc opened 1 week ago

julien-nc commented 1 week ago

Let's implement chunking in the same way it was done in LLM2 to allow summarizing texts that are longer than the model context size.

Implement chunking (maybe have the chunking logic in a service so it can later be used by other providers like "extract topics")
Make the chunk size configurable in the admin settings
Test with GPT-3.5, GPT-4 and a few small models with LocalAI (or another self hosted service)

edward-ly commented 1 day ago

I'm looking at the OpenAI API reference and I can't find any information regarding splitting an input prompt into chunks. If the prompt is longer than context_size - max_output_tokens, is it possible that the API automatically chunks the input for us?

Here's the documentation for the endpoints used by the summary provider: https://platform.openai.com/docs/api-reference/chat/create https://platform.openai.com/docs/api-reference/completions/create