Add integration scripts with Open WebUI

[TanGentleman]

There's a few different ways to approach this one, and I think it's best to start with an MVP in Python.

Milestone 1:

• Create a wrapper function to get a simple LLM response. The API call is to a chatCompletions endpoint, and the response object should conform to the openai standard.

Milestone 2:

• Create a function for obtaining a screenpipe search filter. Inputs: prompt, instructions, and a JSON schema for the output. The instructions guide the LLM to produce an object which can be easily parsed to perform a database search on your screenpipe data.

Milestone 3:

• Create a function that accepts a user prompt, makes the search filter, and uses the screenpipe python SDK to finally generate a response synthesized from the user's personal data.

MVP: Select a model from the dropdown on Open WebUI, type "what have I worked on for the past 10 minutes?" and get served a reasonable, well contextualized response.

Notes:

• In my testing, forcing JSON schemas has varied performance across models, so I'll be using a Qwen2.5-14B-4bit model served via LMStudio locally on my M1 mac.
• From testing a few dozen different types of OCR chunks, it seems like a simple pre-processing step is worth it, even at the cost of a couple seconds of latency.

A breakdown of steps:

1. User types prompt
2. If prompt has NO information to form a screenpipe filter, fail with an appropriate response to the user. Otherwise, the search filter is created at this step.
3. Perform a search on screenpipe data. If no results are obtained, fail with a fitting response. Otherwise, the response is an array of screenpipe chunks.
4. Ensure character count is trimmed below a reasonable max, and use these chunks as context for an LLM response to the user's prompt.*

• = I think the prompt can be used to extract and organize all the relevant information in the raw chunks. Then there can be one more pass to answer the user's question accurately and with references to cleanly formatted screen/audio data. This presents a small chance of new hallucinations, but I'd say it'll heavily outperform the zero-shot response to the user using raw ocr text data.

[m13v]

can you give an example of a prompt

[TanGentleman]

can you give an example of a prompt

"My grandma has been texting me about her health problems for a couple weeks now, how can I research ways to learn more or help her?"

This uses sensitive data, and it's done 100% locally!

• An appropriate screenpipe filter: OCR chunks from my messages app over the past 3 weeks
• An appropriate response: A synthesis of her symptoms, and useful next steps in the form of resources, basic health advice, or formatting a document to give to her health provider so she can get a clear diagnosis!

I see this technology eliminating a lot of the friction/cognitive load that prevents people from doing this stuff in the first place :D

[TanGentleman]

We've got full compatibility with Open WebUI Functions, with no third party dependencies! Closing this :)