search

dank074 / Discord-video-stream

Experiment for making video streaming work for discord selfbots.
183 stars 35 forks source link

Remove the NALU packing hack #74

Closed longnguyen2004 closed 8 months ago

longnguyen2004 commented 8 months ago

Instead of packing the NAL units into frames before sending to the packetizer, now we send the NAL units itself, including the AUD to the packetizer, and then set the marker bit on the AUD instead. This saves an allocation every frame.

With this change, the sendFrame function no longer receives individual frames, but individual NAL units instead. We can either do nothing, or rename it into something more appropriate. New naming welcome.

dank074 commented 8 months ago

Think this will mess up the setTimeout in VideoStream since it relies on the whole video frame being written at once, unless we also check for that AUD nal type in there, but that will be messy. Need to think of a good way to fix this mhmm 🤔

Did you ever figure out why certain streams had unsynced audio/video when we removed the setTimeout? (https://github.com/dank074/Discord-video-stream/issues/52)

longnguyen2004 commented 8 months ago

VideoStream is before the packetizer, so I think it'll be fine? (the flow is ffmpeg -> VideoStream -> AnnexBNalSplitter -> VideoPacketizerAnnexB)

About the setTimeout, I haven't experimented on that yet, but from my experience, I can say that ffmpeg isn't the best when it comes to watching HLS streams (one example: no support for EXT-X-DISCONTINUITY), and something like streamlink should be used instead.

dank074 commented 8 months ago

VideoStream is before the packetizer, so I think it'll be fine? (the flow is ffmpeg -> VideoStream -> AnnexBNalSplitter -> VideoPacketizerAnnexB)

Oh that's right, so yeah it should be fine

About the setTimeout, I haven't experimented on that yet, but from my experience, I can say that ffmpeg isn't the best when it comes to watching HLS streams (one example: no support for EXT-X-DISCONTINUITY), and something like streamlink should be used instead.

I guess we will have to experiment with that. Gstreamer is also a popular alternative to ffmpeg and it even supports building your own packetizer pipeline and has built in RTP encoders. I've never used it myself though so I cannot say whether it will solve our HLS stream issues.

I guess for now the setTimeout will still work so HLS streams should be functional

longnguyen2004 commented 8 months ago

Turns out this does affect the setTimeout() like you said 😅 The actual flow is ffmpeg -> AnnexBNalSplitter -> VideoStream -> VideoPacketizerAnnexB, and removing the frame packing messes up the timing. You can revert this for now, I checked memory consumption and it's not too bad.

dank074 commented 8 months ago

Turns out this does affect the setTimeout() like you said 😅 The actual flow is ffmpeg -> AnnexBNalSplitter -> VideoStream -> VideoPacketizerAnnexB, and removing the frame packing messes up the timing. You can revert this for now, I checked memory consumption and it's not too bad.

reverted it for now