search

mediar-ai / screenpipe

one API to get all user desktop data (local, cross platform, 24/7, screen, voice, keyboard, mouse, camera recording). sandboxed js plugin system. keyboard and mouse control
https://screenpi.pe
MIT License
10.61k stars 649 forks source link

[bounty] SSE realtime streaming of frames #811

Open louis030195 opened 2 days ago

louis030195 commented 2 days ago

similar data structure to https://github.com/mediar-ai/screenpipe/blob/e90d426e76439120371218773588fcb32195d4d5/screenpipe-server/src/video_cache.rs#L24

https://github.com/mediar-ai/screenpipe/blob/e90d426e76439120371218773588fcb32195d4d5/screenpipe-server/src/server.rs#L1360

but real time eg as each frame come up it should stream it with OCR/UI parsing through SSE

this feature would also unlock real time API in pipes like (pseudocode):

for const frame await pipe.stream() {
  openai("what is happening in {frame}?")
}

instead of current polling

every 5 min 

data = pipe.queryScreenpipe(last5min)

openai("what is happening in {data}?")

(no need to impl this TS sdk in this issue)

this would unlock this later:

659

and next step would be to have audio + transcription too

@FractalFir do you want to work on this? I know you cannot test the UI data modality which is currently only macos but maybe i can help

let me know if need more clarification

/bounty 200

linear[bot] commented 2 days ago

MED-364 [bounty] SSE realtime streaming of frames

algora-pbc[bot] commented 2 days ago

💎 $200 bounty • Screenpi.pe

Steps to solve:

  1. Start working: Comment /attempt #811 with your implementation plan
  2. Submit work: Create a pull request including /claim #811 in the PR body to claim the bounty
  3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to mediar-ai/screenpipe!

Add a bounty • Share on socials