[bounty] nextjs loom replacement pipe

mediar-ai / screenpipe

rewind.ai x cursor.com = your AI assistant that has all the context. 24/7 screen & voice recording for the age of super intelligence. get your data ready or be left behind

https://screenpi.pe

MIT License

9.64k stars 567 forks source link

[bounty] nextjs loom replacement pipe #583

Open louis030195 opened 4 weeks ago

louis030195 commented 4 weeks ago

/bounty 100

definition of done:

at the end of the day i can go to screenpipe and ask for a loom of a specific time range
it should be a nextjs pipe
minimum version is taking a time range and stitching together mp4 files (video and audio) and giving me back the video with audio

bonus:

highlights, good UX like loom (somehow)

check this nextjs pipe https://github.com/mediar-ai/screenpipe/tree/main/examples/typescript/pipe-simple-nextjs

also be aware nextjs does have some issue with deno sometimes - for example openai streaming failed for me once idk why (we're thinking in switching to bun soon)

we have endpoint to merge video but might need some changes, e.g. this task might involve a little bit of rust

my use case:

PLEASE say the UX you think of and how you're going to implement this before starting the task plz

linear[bot] commented 4 weeks ago

MED-238 [bounty] nextjs loom replacement pipe

algora-pbc[bot] commented 4 weeks ago

💎 $100 bounty • Screenpi.pe

Steps to solve:

Start working: Comment /attempt #583 with your implementation plan
Submit work: Create a pull request including /claim #583 in the PR body to claim the bounty
Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to mediar-ai/screenpipe!

Add a bounty • Share on socials

Attempt	Started (GMT+0)	Solution
🟢 @tribhuwan-kumar	Oct 25, 2024, 9:43:29 PM	#619
🟢 @volfcan	Oct 28, 2024, 9:50:57 AM	WIP

tribhuwan-kumar commented 3 weeks ago

/attempt #583

Algora profile	Completed bounties	Tech	Active attempts	Options
@tribhuwan-kumar	4 mediar-ai bounties	Rust, JavaScript, Lua & more		Cancel attempt

volfcan commented 3 weeks ago

/attempt #583

Options

Cancel my attempt

tribhuwan-kumar commented 3 weeks ago

@louis030195 i'm thinking to implement something like this, there will a time & date option, user can select a time interval for loom, then it'll merge those specific time ranged videos and audio in one video and send back to the user. there will also an option for quick share, so user can directly share that video on whatsapp, x, etc. i might take some inspiration from loom

(extra feature in this pipe could be a quick summary of that time interval and summary of that video, also it'd be great if user can interact with this pipe)

volfcan commented 3 weeks ago

pretty much the same way with the @tribhuwan-kumar.

there will be a time & date component and when users select the time range for a loom video it'll create the video and send back to the user.

tribhuwan-kumar commented 3 weeks ago

pretty much the same way with the @tribhuwan-kumar.

there will be a time & date component and when users select the time range for a loom video it'll create the video and send back to the user.

the searching and merging are done, i'm currently working on ux, if you have any suggestion for ux we can collaborate!

volfcan commented 3 weeks ago

pretty much the same way with the @tribhuwan-kumar. there will be a time & date component and when users select the time range for a loom video it'll create the video and send back to the user.

the searching and merging are done, i'm currently working on ux, if you have any suggestion for ux we can collaborate!

check your twitter dms

tribhuwan-kumar commented 3 weeks ago

check your twitter dms

there's nothing

louis030195 commented 3 weeks ago

sounds good

@louis030195 i'm thinking to implement something like this, there will a time & date option, user can select a time interval for loom, then it'll merge those specific time ranged videos and audio in one video and send back to the user. there will also an option for quick share, so user can directly share that video on whatsapp, x, etc. i might take some inspiration from loom

(extra feature in this pipe could be a quick summary of that time interval and summary of that video, also it'd be great if user can interact with this pipe)

tribhuwan-kumar commented 3 weeks ago

@louis030195 is this ui is okay ? it will open a dialog of merged video and audio

tribhuwan-kumar commented 3 weeks ago

there was a 500 internal problem with merge frame route, i fixed that. but i noticed some strange behavior while merging videos, imagine this situation, i kill the screenpipe while processing some videos. after killing the screenpipe the video remains uncompleted. and these uncompleted videos are causing errors while merging large amount of videos. maybe i need to add some checks before merging those videos.

i got a corrupt video

PS C:\Users\eirae\Downloads\app\screenpipe> C:\Users\eirae\Downloads\app\screenpipe\screenpipe-app-tauri\src-tauri\ffmpeg\bin\x64\ffmpeg.exe -v error
 -i C:\\Users\\eirae\\.screenpipe\\data\\monitor_65537_2024-10-31_01-32-59.mp4 -f null -
[mov,mp4,m4a,3gp,3g2,mj2 @ 0000022CB0AF1780] moov atom not found
[in#0 @ 0000022CB0ADA540] Error opening input: Invalid data found when processing input
Error opening input file C:\\Users\\eirae\\.screenpipe\\data\\monitor_65537_2024-10-31_01-32-59.mp4.
Error opening input files: Invalid data found when processing input

louis030195 commented 3 weeks ago

@tribhuwan-kumar

nice

video should have audio too at least

i'm actually shipping an endpoint that allows you to stream frames through SSE at high speed for #343

https://github.com/mediar-ai/screenpipe/tree/timeline-ui

i think you might wait i merge this and i'm likely going to do some hacks to stream the audio too so it might solve your problem

tribhuwan-kumar commented 3 weeks ago

@tribhuwan-kumar

nice

video should have audio too at least

right now, i'm able to merge video to video and audio to audio, (merged video and merged audio are separate)
i tried to merge video to audio but the problem i'm getting is order correction, some audio have only beep sound, some have sound but how can i merge those audio with their corresponding video frame.

i'm actually shipping an endpoint that allows you to stream frames through SSE at high speed for #343

https://github.com/mediar-ai/screenpipe/tree/timeline-ui

i think you might wait i merge this and i'm likely going to do some hacks to stream the audio too so it might solve your problem

for now, i fixed those corrupt video error, by filtering out those video. it prevents corrupt videos getting in input_list.txt, this can be slow bcz it adds an extra check before merging them

tribhuwan-kumar commented 3 weeks ago

@louis030195 a working beta version:

https://github.com/user-attachments/assets/44f985bd-172d-4595-b140-e4d30fbe7f77

algora-pbc[bot] commented 3 weeks ago

💡 @tribhuwan-kumar submitted a pull request that claims the bounty. You can visit your bounty board to reward.

louis030195 commented 3 weeks ago

@louis030195 a working beta version:

a.mp4

thats cool

is there audio in it?

tribhuwan-kumar commented 2 weeks ago

@louis030195 a working beta version:

a.mp4

thats cool

is there audio in it?

no, that merged video doesn't have audio, it's just a series of video concatenated by ffmpeg, it doesn't have embedded audio. I tried to embed video with audio but I'm facing issues of order synchronizing :(

louis030195 commented 2 weeks ago

@louis030195 a working beta version:

a.mp4

thats cool is there audio in it?

no, that merged video doesn't have audio, it's just a series of video concatenated by ffmpeg, it doesn't have embedded audio. I tried to embed video with audio but I'm facing issues of order synchronizing :(

are you using database metadata?

you can fetch all the things and order by time

i guess it's possible to have 1-3 seconds of audio desync because we capture timestamp not right sync

also if you try to create a loom over too long range things that i expect to happen:

say i closed my lid (screen capture stop but mic keeps going i think) -> will have weird desync with audio

although this should be only for short times

i guess a version 10.0 of this would be to feed this to a voice LLM and fix all the things automatically and comment on the video haha

tribhuwan-kumar commented 2 weeks ago

are you using database metadata?

you can fetch all the things and order by time

i guess it's possible to have 1-3 seconds of audio desync because we capture timestamp not right sync

also if you try to create a loom over too long range things that i expect to happen:

say i closed my lid (screen capture stop but mic keeps going i think) -> will have weird desync with audio

although this should be only for short times

i guess a version 10.0 of this would be to feed this to a voice LLM and fix all the things automatically and comment on the video haha

then i've to create an another route for embedding audio in video bcz this can only concatenate the videos and audios. & thanks for the metadata it would be helpful for embedding :)