[bug] Summaries are Gibberish

mediar-ai / screenpipe

rewind.ai x cursor.com = your AI assistant that has all the context. 24/7 screen & voice recording for the age of super intelligence. get your data ready or be left behind

https://screenpi.pe

MIT License

10.15k stars 602 forks source link

[bug] Summaries are Gibberish #607

Closed DmacMcgreg closed 4 days ago

DmacMcgreg commented 3 weeks ago

describe the bug When I open the app, its difficult to find any use, since the summaries it gives me are just walls of gibberish. (I've blurred some Personal information)

to reproduce

click the default mail summary button when opening the app
check the list of results
open any of the results

expected behavior The text should be formatted, correct and useful. Right now, I'm having a hard time finding any value in this setup.

system info

macos 14.6
screenpipe version: Version 0.7.6 (20241025.162038)

additional context CleanShot 2024-10-30 at 11 09 23@2x

CleanShot 2024-10-30 at 11 11 21@2x

linear[bot] commented 3 weeks ago

MED-251 [bug] Summaries are Gibberish

DmacMcgreg commented 3 weeks ago

The way I see the app having value is that if you were to focus on having a timeline feature like rewind, or ScreenMemory, you could provide key points where you were using apps, and highlight them on the timeline. The same with searches. If you search gmail, then you highlight all the points on the timeline, where you could easily scroll back to those main points on the timeline to see what gmail was doing. Right now, the focus on OCR text is providing too much noise. Also since it splits the video files arbitrarily into small chunks, its hard to actually see what the purpose of the search results are since they are split up among many sections that have a lot of duplicate data.

louis030195 commented 3 weeks ago

@DmacMcgreg thanks for the feedback!

I created a bounty here https://github.com/mediar-ai/screenpipe/issues/609

would love any more context on it, how to make it great

regarding OCR we're going to release today a new alternative data source which extracts text natively from the UI and will be unique and highly structured

DmacMcgreg commented 3 weeks ago

This is great you're opening a bounty for this one. Some ideas:

Daily stats on used apps
Daily stats on projects - might be useful for agencies or freelancers that need time tracking on projects they worked on. User could provide name and description of projects they are working on, and the tool will keep track based on what was on their screen.
Searching for keywords highlights points on the timeline. Perhaps even highlighting the words on the screen of the timeline
Highlight a length in time, and ask for a summary from the LLM, and other LLM tasks, extracting text
Export screenshot
Option to Highlight meetings on the timeline, which when clicking on a meeting you can view the transcript window with ability to summarize, etc. can also be used to build a more robust meetings dashboard where you can group them by project, by attendees.