Open louis030195 opened 4 days ago
similar data structure to https://github.com/mediar-ai/screenpipe/blob/e90d426e76439120371218773588fcb32195d4d5/screenpipe-server/src/video_cache.rs#L24
https://github.com/mediar-ai/screenpipe/blob/e90d426e76439120371218773588fcb32195d4d5/screenpipe-server/src/server.rs#L1360
but real time eg as each frame come up it should stream it with OCR/UI parsing through SSE
this feature would also unlock real time API in pipes like (pseudocode):
for const frame await pipe.stream() { openai("what is happening in {frame}?") }
instead of current polling
every 5 min data = pipe.queryScreenpipe(last5min) openai("what is happening in {data}?")
(no need to impl this TS sdk in this issue)
this would unlock this later:
and next step would be to have audio + transcription too
@FractalFir do you want to work on this? I know you cannot test the UI data modality which is currently only macos but maybe i can help
let me know if need more clarification
/bounty 200
MED-364 [bounty] SSE realtime streaming of frames
/attempt #811
/claim #811
Thank you for contributing to mediar-ai/screenpipe!
Add a bounty • Share on socials
similar data structure to https://github.com/mediar-ai/screenpipe/blob/e90d426e76439120371218773588fcb32195d4d5/screenpipe-server/src/video_cache.rs#L24
https://github.com/mediar-ai/screenpipe/blob/e90d426e76439120371218773588fcb32195d4d5/screenpipe-server/src/server.rs#L1360
but real time eg as each frame come up it should stream it with OCR/UI parsing through SSE
this feature would also unlock real time API in pipes like (pseudocode):
instead of current polling
(no need to impl this TS sdk in this issue)
this would unlock this later:
659
and next step would be to have audio + transcription too
@FractalFir do you want to work on this? I know you cannot test the UI data modality which is currently only macos but maybe i can help
let me know if need more clarification
/bounty 200