search

dank074 / Discord-video-stream

Experiment for making video streaming work for discord selfbots.
178 stars 35 forks source link

Issues with sending H.264 post conversion to the bot, please help! #60

Closed bengeek2 closed 8 months ago

bengeek2 commented 8 months ago

Hello, i am trying to stream a raw video source on discord. to do this, I am taking the rawvideo source and encoding it to h264, then sending the output to the bot. Joining, streaming, and disconnecting all function as intended, and properly start and stop ffmpeg. if I modify the ffmpeg output to send to an .mp4, i can verify the encoding is working.

the ffmpeg command i am generating is this:

-f v4l2 -f v4l2 -pix_fmt yuyv422 -r 30 -i /dev/video0 -vcodec rawvideo -vcodec libx264 -filter:v scale=1080:720 -loglevel info -fflags nobuffer -analyzeduration 0 -preset ultrafast -tune zerolatency -f flv unix:./2.sock

the command is a bit scuffed as it calls format twice, but does work when you change output from unix:./2.sock to output.mp4

The issue I have is the bot's stream has no content. I believe the output for the ffmpeg command is incorrect, or it is integrating with H264NalSplitter properly. Can you please let me know what I am missing, or any direction to start moving in? I'm not very good with JS so I am stumped. Here is the index.ts file I am using:

import { MediaUdp, Streamer, setStreamOpts } from "@dank074/discord-video-stream";
import config from "./config.json";
import { Client, StageChannel } from "discord.js-selfbot-v13";
import ffmpeg from "fluent-ffmpeg";
import { StreamOutput } from "@dank074/fluent-ffmpeg-multistream-ts";
import { VideoStream, H264NalSplitter, streamOpts } from "@dank074/discord-video-stream";

const streamer = new Streamer(new Client());

setStreamOpts({
    bitrateKbps: config.streamOpts.bitrateKbps,
    maxBitrateKbps: config.streamOpts.maxBitrateKbps,
    video_codec: 'H264'
});

let customFfmpegCommand: ffmpeg.FfmpegCommand | null = null;

streamer.client.on("ready", () => {
    console.log(`--- ${streamer.client.user.tag} is ready ---`);
});

streamer.client.on("messageCreate", async (msg) => {
    if (msg.author.bot) return;
    if (!config.acceptedAuthors.includes(msg.author.id)) return;
    if (!msg.content) return;

    if (msg.content.startsWith(`$play-live`)) {
        const args = msg.content.split(" ");
        if (args.length < 2) return;

        const url = args[1];
        if (!url) return;

        const channel = msg.author.voice.channel;
        if (!channel) return;

        console.log(`Attempting to join voice channel ${msg.guildId}/${channel.id}`);
        await streamer.joinVoice(msg.guildId, channel.id);

        if (channel instanceof StageChannel) {
            await streamer.client.user.voice.setSuppressed(false);
        }

        const streamUdpConn = await streamer.createStream();

        try {
            await playVideo(url, streamUdpConn);
        } catch (e) {
            console.error("Error playing video:", e.message);
            // Stop the stream if an error occurs during video playback
            streamer.stopStream();
        }
    } else if (msg.content.startsWith("$disconnect")) {
        if (customFfmpegCommand) {
            customFfmpegCommand.kill("SIGINT");
            customFfmpegCommand = null;
        }
        // Leave the voice channel and stop the stream when the user requests disconnection
        streamer.leaveVoice();
        streamer.stopStream();
    }
});

streamer.client.login(config.token);

async function playVideo(video: string, udpConn: MediaUdp) {
    try {
        console.log("Starting customStreamVideo function...");
        console.log("Video URL:", video);

        const videoStream: VideoStream = new VideoStream(udpConn, streamOpts.fps);
        const videoOutput = new H264NalSplitter();

        const ffmpegCommand = ffmpeg(video)
            .addOption("-loglevel", "info")
            .addOption("-fflags", "nobuffer")
            .addOption("-analyzeduration", "0")
            .inputFormat('v4l2')
            .inputOptions(['-f v4l2', '-pix_fmt yuyv422'])
            .inputFps(streamOpts.fps)
            .videoCodec('rawvideo')
            .outputOptions(['-preset ultrafast', '-tune zerolatency'])
            .videoFilters(`scale=${streamOpts.width}:${streamOpts.height}`)
            .format('flv')
            .videoCodec('libx264')
            .output(StreamOutput(videoOutput).url, { end: false });

        console.log("FFmpeg Command:", ffmpegCommand._getArguments().join(" "));

        customFfmpegCommand = ffmpegCommand
            .on("end", () => {
                console.log("Video streaming ended.");
                customFfmpegCommand = null;
                streamer.stopStream();
            })
            .on("error", (err, stdout, stderr) => {
                console.error("Error occurred:", err);
                customFfmpegCommand = null;
                streamer.stopStream();
            })
            .on("stderr", console.error);

        customFfmpegCommand.run();

        console.log("Video streaming completed successfully");
        udpConn.mediaConnection.setSpeaking(true);
        udpConn.mediaConnection.setVideoStatus(true);

    } catch (e) {
        console.error("Error playing video:", e.message);
        throw new Error("cannot play video " + e.message);
    }
}
bengeek2 commented 8 months ago

this is the console output when i say $play-live /dev/video0

Attempting to join voice channel xxxxxxx/xxxxxxx
Starting customStreamVideo function...
Video URL: /dev/video0
FFmpeg Command: -f v4l2 -f v4l2 -pix_fmt yuyv422 -r 30 -i /dev/video0 -vcodec rawvideo -vcodec libx264 -filter:v scale=1080:720 -loglevel info -fflags nobuffer -analyzeduration 0 -preset ultrafast -tune zerolatency -f flv unix:./2.sock
Video streaming completed successfully
ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
[video4linux2,v4l2 @ 0x558cb2665900] The driver changed the time per frame from 1/30 to 117/7013
Input #0, video4linux2,v4l2, from '/dev/video0':
  Duration: N/A, start: 38315.786159, bitrate: 1988671 kb/s
  Stream #0:0: Video: rawvideo (YUY2 / 0x32595559), yuyv422, 1920x1080, 1988671 kb/s, 59.94 fps, 59.94 tbr, 1000k tbn, 1000k tbc
Multiple -c, -codec, -acodec, -vcodec, -scodec or -dcodec options specified for stream 0, only the last option '-c:v libx264' will be used.
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 0x558cb266b540] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0x558cb266b540] profile High 4:2:2, level 3.1, 4:2:2, 8-bit
[libx264 @ 0x558cb266b540] 264 - core 163 r3060 5db6aa6 - H.264/MPEG-4 AVC codec - Copyleft 2003-2021 - http://www.videolan.org/x264.html - options: cabac=0 ref=1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=4 lookahead_threads=4 sliced_threads=1 slices=4 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=0
Output #0, flv, to 'unix:./2.sock':
  Metadata:
    encoder         : Lavf58.76.100
  Stream #0:0: Video: h264 ([7][0][0][0] / 0x0007), yuv422p(tv, progressive), 1080x720, q=2-31, 30 fps, 1k tbn
    Metadata:
      encoder         : Lavc58.134.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
frame=    1 fps=0.0 q=20.0 size=       3kB time=00:00:00.00 bitrate=26608.0kbits/s speed=N/A    
frame=   22 fps=0.0 q=13.0 size=       5kB time=00:00:00.70 bitrate=  55.2kbits/s speed=1.37x    
frame=   44 fps= 43 q=13.0 size=       6kB time=00:00:01.43 bitrate=  35.8kbits/s speed=1.41x    
frame=   69 fps= 45 q=13.0 size=       8kB time=00:00:02.26 bitrate=  29.0kbits/s speed=1.49x    
frame=   92 fps= 45 q=13.0 size=      10kB time=00:00:03.03 bitrate=  26.0kbits/s speed= 1.5x    
frame=  115 fps= 45 q=13.0 size=      11kB time=00:00:03.80 bitrate=  24.3kbits/s speed= 1.5x    
frame=  132 fps= 43 q=13.0 size=      12kB time=00:00:04.36 bitrate=  23.4kbits/s speed=1.44x    
frame=  150 fps= 42 q=13.0 size=      14kB time=00:00:04.96 bitrate=  22.6kbits/s speed= 1.4x    
frame=  173 fps= 43 q=13.0 size=      15kB time=00:00:05.73 bitrate=  21.9kbits/s speed=1.41x    
frame=  190 fps= 42 q=13.0 size=      17kB time=00:00:06.30 bitrate=  21.5kbits/s speed=1.38x    
frame=  211 fps= 42 q=13.0 size=      18kB time=00:00:07.00 bitrate=  21.1kbits/s speed=1.38x    
frame=  234 fps= 42 q=13.0 size=      20kB time=00:00:07.76 bitrate=  20.7kbits/s speed=1.39x    
frame=  253 fps= 42 q=14.0 size=      23kB time=00:00:08.40 bitrate=  22.7kbits/s speed=1.38x    
frame=  274 fps= 42 q=13.0 size=      25kB time=00:00:09.10 bitrate=  22.3kbits/s speed=1.38x    
frame=  295 fps= 42 q=13.0 size=      26kB time=00:00:09.80 bitrate=  21.9kbits/s speed=1.38x    
frame=  316 fps= 42 q=13.0 size=      28kB time=00:00:10.50 bitrate=  21.6kbits/s speed=1.38x    
frame=  335 fps= 41 q=13.0 size=      29kB time=00:00:11.13 bitrate=  21.4kbits/s speed=1.37x    
frame=  358 fps= 41 q=13.0 size=      31kB time=00:00:11.90 bitrate=  21.1kbits/s speed=1.38x    
frame=  378 fps= 41 q=13.0 size=      32kB time=00:00:12.56 bitrate=  20.9kbits/s speed=1.38x
ludwig7685 commented 8 months ago

you need to put a video URL after $play-live !

look into the example of README.md!

dank074 commented 8 months ago

-f needs to be h264, not flv

Also it's missing the bitstream filter for adding AUD into the h264 byte stream because that's how the h264nalsplitter splits the video frames (known as access units in h264). Under outputOptions add -bsf:v h264_metadata=aud=insert