search

dank074 / Discord-video-stream

Experiment for making video streaming work for discord selfbots.
183 stars 35 forks source link

Use ffmpeg to limit the read rate instead of `setTimeout` #52

Open longnguyen2004 opened 10 months ago

longnguyen2004 commented 10 months ago

Currently, setTimeout is used in AudioStream and VideoStream to control the read rate of the media. FFmpeg has a built-in flag -re that does the same thing, and is potentially better? Need some more experimenting to see whether there's an improvement, and whether it impacts actual live streams (RTSP, SRT,...)

Some info about the flag: https://ffmpeg.org/ffmpeg.html#:~:text=as%20live%20streaming.-,%2Dre%20(input),-Read%20input%20at

dank074 commented 10 months ago

We previously used the -re flag but it resulted in desynchronized audio/video for certain hls streams:

https://github.com/dank074/Discord-video-stream/issues/31#issuecomment-1747683830

longnguyen2004 commented 10 months ago

Interesting, I'll see if the related -readrate_initial_burst can help out here. Do you still keep the hls stream?

dank074 commented 10 months ago

This is the stream I was testing with which was bugged:

https://rbmn-live.akamaized.net/hls/live/590964/BoRB-AT/master_1660.m3u8

longnguyen2004 commented 8 months ago

Did some more testing with the -re flags, and during the process I discovered a bug with the setTimeout approach.

It assumes that the entire frame will arrive at once, which is not the case. At least on my Node.js installation, the incoming buffer size maxes out at 64KB, which is definitely not enough to contain the whole video frame, which can reach up to 1MB. This results in an effective frame rate of 1/3 to 1/4 actual. Using the -re flag fixes the issue, since ffmpeg knows where the frame boundaries are and will limit the read rate accordingly. We can do the same thing in our code by moving the setTimeout to the packetizer, but that's fragile.

About the -re flag itself, it works really well for local files, but fails on HLS streams, with stuttering audio and video. Removing the -re flag makes it even worse, with frame rates jumping all over the place. We'll probably have to find another solution for HLS streams, since neither approach works well with HLS livestreams (haven't tested HLS VOD, but I assume it'll work the same as local files)

dank074 commented 8 months ago

Did some more testing with the -re flags, and during the process I discovered a bug with the setTimeout approach.

It assumes that the entire frame will arrive at once, which is not the case. At least on my Node.js installation, the incoming buffer size maxes out at 64KB, which is definitely not enough to contain the whole video frame, which can reach up to 1MB. This results in an effective frame rate of 1/3 to 1/4 actual. Using the -re flag fixes the issue, since ffmpeg knows where the frame boundaries are and will limit the read rate accordingly. We can do the same thing in our code by moving the setTimeout to the packetizer, but that's fragile.

About the -re flag itself, it works really well for local files, but fails on HLS streams, with stuttering audio and video. Removing the -re flag makes it even worse, with frame rates jumping all over the place. We'll probably have to find another solution for HLS streams, since neither approach works well with HLS livestreams (haven't tested HLS VOD, but I assume it'll work the same as local files)

Unfortunately it wasn't just hls streams that had this weird behavior. Author of issue also reported that some local mkv video files also had delayed audio when the -re flag was used https://github.com/dank074/Discord-video-stream/issues/31#issuecomment-1747236782

longnguyen2004 commented 8 months ago

About ffplay: It has an internal packet queue, which allows it to buffer a few hundred KBs of video to smooth out the playback. Not sure if we can add some kind of buffer in the library.