Stream freezing after x seconds based on bitrate

[DevTobias]

Hi! First of all, awesome work right there. I played around with this package and noticed, that for me the stream is freezing after x seconds based on the provided bitrate (higher birate equals to faster freeze).

I tried to resolve this issue. For me the only logial error location would be ffmpeg, because the streaming transmission is working and ffmpeg is the only ongoing process. So I tried bruteforcing some of the arguments and noticed, that the stream is not freezing anymore after removing the '-tune zerolatency' parameter for the video operation.

Can you explain that to yourself? Why is the option to optimize the encoding settings for zero-latency streaming resulting into an error? May that be the case when running inside of WSL2? I just cant get my head around it :D

[dank074]

Think you are just experiencing lower performance because of that parameter.

According to ffmpeg, this is what settings are applied with each -tune option:

--tune <string>         Tune the settings for a particular type of source
                          or situation
                              Overridden by user settings.
                              Multiple tunings are separated by commas.
                              Only one psy tuning can be used at a time.
                              - film (psy tuning):
                                --deblock -1:-1 --psy-rd <unset>:0.15
                              - animation (psy tuning):
                                --bframes {+2} --deblock 1:1
                                --psy-rd 0.4:<unset> --aq-strength 0.6
                                --ref {Double if >1 else 1}
                              - grain (psy tuning):
                                --aq-strength 0.5 --no-dct-decimate
                                --deadzone-inter 6 --deadzone-intra 6
                                --deblock -2:-2 --ipratio 1.1 
                                --pbratio 1.1 --psy-rd <unset>:0.25
                                --qcomp 0.8
                              - stillimage (psy tuning):
                                --aq-strength 1.2 --deblock -3:-3
                                --psy-rd 2.0:0.7
                              - psnr (psy tuning):
                                --aq-mode 0 --no-psy
                              - ssim (psy tuning):
                                --aq-mode 2 --no-psy
                              - fastdecode:
                                --no-cabac --no-deblock --no-weightb
                                --weightp 0
                              - zerolatency:
                                --bframes 0 --force-cfr --no-mbtree
                                --sync-lookahead 0 --sliced-threads
                                --rc-lookahead 0

Then, according to google:

sync-lookahead=0 disables threaded lookahead, which allows lower latency at the cost of reduced performance

An ideal solution would probably be to give more flexibility for the ffmpeg parameters being used. I will have to think of a good way of doing that without adding too much complexity and so that the library is still easy to use

[misyltoad]

This also happens for me, but I am not affected. Only some of my friends on the receiving end are affected by the bug.

I've tired playing around with a bunch of the ffmpeg settings but to no avail so far.

[misyltoad]

Seems to be fine for those affected after disabling the following options in Discord

[misyltoad]

I am confused -- based on how things are set up, we should be getting an I/IDR frame sent every second, yet I am definitely seeing P frames only being sent for multiple minutes sometimes...

Something is definitely off.

[misyltoad]

Unless I am misunderstanding what `-x264-params keyint=${streamOpts.fps}:min-keyint=${streamOpts.fps},` is supposed to be doing :l

[DevTobias]

I think this has something to do with discord in general. For example the stream freezes after discord was running in the background for some time. Leaving and rejoining the stream then fixes this issue. I guess discord somehow optimizes streams running in the background, which is not considered in this implementation because nobody knows what their services are doing exaclty. This might corrupt the stream for the current user. Tbh. I just accepted this weird behaviour and didn't digged deeper into it :D

Also you may have set the max bitrate to high. I think discord wont allow streams higher than 5000/6000 kbps. If a section (e.g. with a lot of particles) exceeds this limit, the stream also freezes until the bitrate drops below the limit again. Its kinda

[misyltoad]

It doesn't seem related to the bitrate, it always hangs. My bitrate was 1000/2000 and users with HW acceleration were still hanging.

Additionally, it seems like idr/i frames are not being sent for sometimes minutes at a time somehow.

[dank074]

I am confused -- based on how things are set up, we should be getting an I/IDR frame sent every second, yet I am definitely seeing P frames only being sent for multiple minutes sometimes...

Something is definitely off.

Yeah you are right we set the key interval to the fps so it should be every second.

To clarify, the video seems to hang for users who have hardware acceleration turned on after a they minimized the screen, or is does it always hang

Not sure why it's happening but discord seems to be using custom key frame interval as one of their "experiments" if you look at the websocket messages, and their interval was 5 seconds last time I looked which was a couple months ago

[misyltoad]

It always hangs.

[dank074]

It always hangs.

Is this on a regular voice channel, or a stage channel?

Stage seems to be having this issue even with regular discord clients doing the streaming 🤔

[misyltoad]

Regular

[dank074]

It doesn't seem related to the bitrate, it always hangs. My bitrate was 1000/2000 and users with HW acceleration were still hanging.

Additionally, it seems like idr/i frames are not being sent for sometimes minutes at a time somehow.

Did removing '-tune zerolatency' not work for you, like it did for @DevTobias

[bengeek2]

I am also currently experiencing this issue with bitrate set to 1000/2500 and i have removed -tune zerolatency additionally - here are the changes i am running with. index.ts -

import { Client, StageChannel } from "discord.js-selfbot-v13";
import {
    streamLivestreamVideo,
    MediaUdp,
    setStreamOpts,
    getInputMetadata,
    Streamer,
    stopFFmpegProcess
} from "@dank074/discord-video-stream";
import config from "./config.json";
import { exec } from 'child_process';

const streamer = new Streamer(new Client());

// Set stream options
setStreamOpts(config.streamOpts);

// Ready event
streamer.client.on("ready", () => {
    console.log(`--- ${streamer.client.user.tag} is ready ---`);
});

// Message event
streamer.client.on("messageCreate", async (msg) => {
    if (msg.author.bot || !config.acceptedAuthors.includes(msg.author.id) || !msg.content) return;

    if (msg.content.startsWith(`$play-live`)) {
        await handlePlayLive(msg);
    } else if (msg.content.startsWith("$play-cam")) {
        await handlePlayCam(msg);
    } else if (msg.content.startsWith("$disconnect") || msg.content.startsWith("$stop-stream")) {
        handleDisconnectOrStopStream();
    }
});

// Login
streamer.client.login(config.token);

async function handlePlayLive(msg: any) {
    const args = parseArgs(msg.content);
    if (!args) return;

    const channel = msg.author.voice.channel;
    if (!channel) return;

    console.log(`Attempting to join voice channel ${msg.guildId}/${channel.id}`);
    await streamer.joinVoice(msg.guildId, channel.id);

    if (channel instanceof StageChannel) {
        await streamer.client.user.voice.setSuppressed(false);
    }

    const streamUdpConn = await streamer.createStream();
    await playVideo(args.url, streamUdpConn);

    streamer.stopStream();
}

async function handlePlayCam(msg: any) {
    const args = parseArgs(msg.content);
    if (!args) return;

    const channel = msg.author.voice.channel;
    if (!channel) return;

    console.log(`Attempting to join voice channel ${msg.guildId}/${channel.id}`);
    const vc = await streamer.joinVoice(msg.guildId, channel.id);

    if (channel instanceof StageChannel) {
        await streamer.client.user.voice.setSuppressed(false);
    }

    streamer.signalVideo(msg.guildId, channel.id, true);
    playVideo(args.url, vc);
}

function handleDisconnectOrStopStream() {
    stopFFmpegProcess(); // Call the function to stop the ffmpeg process
    streamer.leaveVoice();
}

async function playVideo(video: string, udpConn: MediaUdp) {
    try {
        console.log("Fetching metadata for video:", video);
        await getInputMetadata(video);
    } catch (e) {
        console.error("Error fetching metadata:", e);
        return;
    }

    console.log("Started playing video:", video);

    udpConn.mediaConnection.setSpeaking(true);
    udpConn.mediaConnection.setVideoStatus(true);
    try {
        console.log("Starting video stream:", video);
        const command = await streamLivestreamVideo(video, udpConn);
        console.log("FFmpeg command:", command);
        executeFFmpegCommand(command);
    } catch (e) {
        console.error("Error playing video:", e);
    } finally {
        udpConn.mediaConnection.setSpeaking(false);
        udpConn.mediaConnection.setVideoStatus(false);
    }
}

function parseArgs(message: string): Args | undefined {
    const args = message.split(" ");
    if (args.length < 2) return;

    const url = args[1];

    return { url };
}

type Args = {
    url: string;
};

function executeFFmpegCommand(command: string) {
    console.log("Executing FFmpeg command:", command);

    exec(command, (error, stdout, stderr) => {
        if (error) {
            console.error(`Error executing FFmpeg command: ${error.message}`);
            return;
        }
        if (stderr) {
            console.error(`FFmpeg stderr: ${stderr}`);
            return;
        }
        console.log(`FFmpeg stdout: ${stdout}`);
    });
}

streamLivestreamVideo.ts

import ffmpeg from 'fluent-ffmpeg';
import prism from 'prism-media';
import { AudioStream } from './AudioStream';
import { StreamOutput } from '@dank074/fluent-ffmpeg-multistream-ts';
import { streamOpts } from '../client/StreamOpts';
import { H264NalSplitter } from './H264NalSplitter';
import { VideoStream } from './VideoStream';

let command: ffmpeg.FfmpegCommand | null = null;

export function streamLivestreamVideo(input: string, mediaUdp: any, includeAudio = true, audioDevice = 'hw:C4K,0'): Promise<string> {    return new Promise((resolve, reject) => {
        console.log("Using audio device:", audioDevice);
        const videoStream = new VideoStream(mediaUdp, streamOpts.fps);
        let videoOutput: any;

        if (streamOpts.video_codec === 'H264') {
            videoOutput = new H264NalSplitter();
        } else {
            // Assuming you have the IvfTransformer class imported or defined elsewhere
            // Otherwise, you should import it similarly to AudioStream, VideoStream, etc.
            // videoOutput = new IvfTransformer();
        }

        const opus = new prism.opus.Encoder({ channels: 2, rate: 48000, frameSize: 960 });

        try {
            command = ffmpeg(input)
                .addOption('-loglevel', '0')
                .addOption('-fflags', 'nobuffer')
                .addOption('-analyzeduration', '0')
                .on('end', () => {
                    command = null;
                    resolve("video ended");
                })
                .on("error", (err: Error, stdout: string, stderr: string) => {
                    command = null;
                    reject('cannot play video ' + err.message);
                })
                .on('stderr', console.error);

            command.output(StreamOutput(videoOutput).url, { end: false })
                .noAudio()
                .size(`${streamOpts.width}x${streamOpts.height}`)
                .fpsOutput(streamOpts.fps)
                .videoBitrate(`${streamOpts.bitrateKbps}k`)
                .format('h264')
                .outputOptions([
                    '-profile:v baseline',
                    '-pix_fmt yuv420p',
                    '-preset ultrafast',
                    `-g ${streamOpts.fps}`,
                    `-x264-params keyint=${streamOpts.fps}:min-keyint=${streamOpts.fps}`,
                    '-bsf:v h264_metadata=aud=insert'
                ]);

            videoOutput.pipe(videoStream, { end: false });

            if (includeAudio) {
                const audioStream = new AudioStream(mediaUdp);

                command.input(audioDevice)
                    .inputFormat('alsa')
                    .output(StreamOutput(opus).url, { end: false })
                    .noVideo()
                    .audioChannels(2)
                    .audioFrequency(48000)
                    .format('s16le');

                opus.pipe(audioStream, { end: false });
            }

            if (streamOpts.hardware_acceleration) {
                command.inputOption('-hwaccel', 'auto');
            }

            console.log("FFmpeg command:", command._getArguments().join(" "));

            command.run();
        } catch (e) {
            command = null;
            reject("cannot play video " + e.message);
        }
    });
}

export function getInputMetadata(input: string): Promise<any> {
    return new Promise((resolve, reject) => {
        const instance = ffmpeg(input).on('error', (err: Error, stdout: string, stderr: string) => reject(err));
        instance.ffprobe((err: Error, metadata: any) => {
            if (err) reject(err);
            instance.removeAllListeners();
            resolve(metadata);
            instance.kill('SIGINT');
        });
    });
}

export function stopFFmpegProcess() {
    if (command) {
        command.kill("SIGINT");
        command = null;
    }
}