Audio Async

[BitcircuitEU]

Hey there,

I try to play files from my Jellyfin Mediaserver to Discord. As there is no better way to do that then directstream the media files with something like this project i have installed the example project from this repo on my jellyfin server. So Media Files are directly accessed.

Media Files are mostly .mkv Movie Files. Currently i used 1 .mkv File to test everything. I copied the details (resolution, bitrate, fps) from the .mkv file to .env so that this values are correct (later ill directly read and set them via ffprobe). Everything seems to work fine and quality is great except that the audio has a few seconds delay to the video. Any idea why that is?

I start the file with command $play-live ./movies/example.mkv

[BitcircuitEU]

When using custom-copy-codec with the mkv i only get audio, no video. Any parameters i need to modify in ffmpeg?

[dank074]

Disable readAtNativeFps which is enabled by default. It is known to cause issues in certain streams and I still don't know why. Still enabled by default though because it is more accurate than controlling the fps with setTimeout and it doesn't always break the audio/video sync

[longnguyen2004]

Can you provide the file?

[BitcircuitEU]

Disable readAtNativeFps which is enabled by default. It is known to cause issues in certain streams and I still don't know why. Still enabled by default though because it is more accurate than controlling the fps with setTimeout and it doesn't always break the audio/video sync

I have changed readAtNativeFps to false, but nothing changed. I tried converting the file with Handbrake to MP4 (Preset 1080pfast) but error with audio desync still persists. Maybe because audio gets split and send to a different channel?

When playing those files directly in jellyfin there is no audio delay, so the files are good.

Can you provide the file?

I am unable to provide the file due to copyright and file size. But i tried different files with same results. Even the demo mp4 file is out of sync ($play-live http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4)

Current config.json:

{
    "token": "X",
    "acceptedAuthors": ["X"],
    "streamOpts": {
        "width": 1920,
        "height": 1080,
        "fps": 24,
        "bitrateKbps": 4000,
        "maxBitrateKbps": 5500,
        "hardware_acceleration": false,
    "readAtNativeFps": false,
        "videoCodec": "H264"
    }
}

[longnguyen2004]

It's...alarming that even "normal" video files suffer from desync, I'll try to see what's going on

[BitcircuitEU]

Well, as i can tell from now, the issue relates to the custom ffmpeg setup the jellyfin server uses as i had used the same ffmpeg for this project.

I created a fresh vm and installed the "normal" ffmpeg via apt. that seems to have solved the desync issues. Will check it further and give feedback.

[BitcircuitEU]

Ye okay issue was the custom jellyfin ffmpeg. With normal ffmpeg everything seems to work fine

[BitcircuitEU]

Edit: Maybe its ffmpeg under Windows.

• I have installed ffmpeg with command "choco install ffmpeg-full" on Windows and have same delay results there.

Will check if delay is too when installing Linux on same server but would still be good to know why that only happens on windows

[dank074]

Is there any version differences between the ffmpeg installed on Linux vs the one installed on windows?

I use a windows machine for development, and while certain streams do break with the readAtNativeFps, the Big Buck Bunny reference video works perfectly fine

[BitcircuitEU]

For me everything works fine on Windows (either with ffmpeg version by jellyfin or pre compiled one from ffmpeg site), just the audio desync info.

On Linux everything works fine so i just use linux for now

[dank074]

Yeah when I said "break" I really meant it suffered audio desync, I should have been more clear.

On windows I don't get audio desync issues on the Bunny video

[BitcircuitEU]

Yeah when I said "break" I really meant it suffered audio desync, I should have been more clear.

On windows I don't get audio desync issues on the Bunny video

Just use Linux, then you should be able to play all files without desync. Just make sure you use normal ffmpeg and not the one from jellyfin server when having both installed

[BitcircuitEU]

Okay nvm. Audio Desync issue is also happening on linux with mkv files. Tried hw accel on and off and nativeFPS on and off

[dank074]

Okay nvm. Audio Desync issue is also happening on linux with mkv files. Tried hw accel on and off and nativeFPS on and off

Now try toggling minimizeLatency

[longnguyen2004]

I'm thinking about making the library accepts a MPEG Transport Stream, which also carries timestamp information, then demux it in the library with https://www.npmjs.com/package/ts-demuxer. That way, both audio and video will stay together as a single stream, and we also have timestamp information as a bonus. Not sure if this is viable though, and it'll definitely not support VP8/9 (does anyone even use that?)

[BitcircuitEU]

I'm thinking about making the library accepts a MPEG Transport Stream, which also carries timestamp information, then demux it in the library with https://www.npmjs.com/package/ts-demuxer. That way, both audio and video will stay together as a single stream, and we also have timestamp information as a bonus. Not sure if this is viable though, and it'll definitely not support VP8/9 (does anyone even use that?)

Well i guess the project is mostly use to stream Movies to watch them with friends (at least thats what i am using it for, with my jellyfin media). Which are likely mp4 or mkv. So i think h264 and h265 hevc would likely be enough

[BitcircuitEU]

Just checking the Code. Does this have any effect in BaseMediaConnection.ts?

/**
     * Enables sending RTCP sender reports. Helps the receiver synchronize the audio/video frames, except in some weird
     * cases which is why you can disable it
     */
    rtcpSenderReportEnabled: boolean;

[longnguyen2004]

I'm still wondering about the viability of the MPEG-TS/MP4 demuxing approach, since all standalone demuxers I found are unmaintained. So unless we want to pull in the entirely of hls.js, I don't think we should commit to this approach.

Does this have any effect in BaseMediaConnection.ts?

This helps to keep the audio and video tracks in sync over a long time period (aka prevent drifting), but it doesn't help when the tracks are offset from the start. I guess you can try manipulating the timestamps in the sender reports, but I don't know if it'll work.

[BitcircuitEU]

I'm still wondering about the viability of the MPEG-TS/MP4 demuxing approach, since all standalone demuxers I found are unmaintained. So unless we want to pull in the entirely of hls.js, I don't think we should commit to this approach.

Does this have any effect in BaseMediaConnection.ts?

This helps to keep the audio and video tracks in sync over a long time period (aka prevent drifting), but it doesn't help when the tracks are offset from the start. I guess you can try manipulating the timestamps in the sender reports, but I don't know if it'll work.

Any idea whats up with the play function from selfbot itself? https://discordjs-self-v13.netlify.app/#/docs/docs/main/class/StreamConnection?scrollTo=playVideo

[dank074]

The built in video streaming in selfbot library was initially based off this library, so it will probably suffer from the same ffmpeg-param related pitfalls as this one

[BitcircuitEU]

The built in video streaming in selfbot library was initially based off this library, so it will probably suffer from the same ffmpeg-param related pitfalls as this one

I see.. That sucks..

[BitcircuitEU]

Does the delay between video and audio happen due to ffmpeg or is it the transfer to discord itself?

[longnguyen2004]

Does the delay between video and audio happen due to ffmpeg or is it the transfer to discord itself?

It's currently unknown, we haven't been able to debug it yet.

On the subject of demuxers, these looks promising https://github.com/Yahweasel/libav.js do not provide AAC decoder (while I agree with their reasoning, I don't think leaving it out entirely is a good idea) https://github.com/mmomtchev/ffmpeg (native binding, so will require building for non-x64 platforms)

These are packages that provide the ffmpeg API for use in Node, so we can directly uses the demuxer inside Node and get timestamp information that way.

[BitcircuitEU]

Does the delay between video and audio happen due to ffmpeg or is it the transfer to discord itself?

It's currently unknown, we haven't been able to debug it yet.

On the subject of demuxers, these looks promising ~https://github.com/Yahweasel/libav.js~ do not provide AAC decoder (while I agree with their reasoning, I don't think leaving it out entirely is a good idea) https://github.com/mmomtchev/ffmpeg (native binding, so will require building for non-x64 platforms)

These are packages that provide the ffmpeg API for use in Node, so we can directly uses the demuxer inside Node and get timestamp information that way.

Can you implement this for testing? I have no idea about all this encoding/transcoding stuff. Currently working on support for local directorys with search option.

Feel free to contact me on discord if you need anyone for testing or a server that has the current script and a video file with the delay problem. DC: abrucci

[BitcircuitEU]

Does the delay between video and audio happen due to ffmpeg or is it the transfer to discord itself?

It's currently unknown, we haven't been able to debug it yet.

On the subject of demuxers, these looks promising ~https://github.com/Yahweasel/libav.js~ do not provide AAC decoder (while I agree with their reasoning, I don't think leaving it out entirely is a good idea) https://github.com/mmomtchev/ffmpeg (native binding, so will require building for non-x64 platforms)

These are packages that provide the ffmpeg API for use in Node, so we can directly uses the demuxer inside Node and get timestamp information that way.

Not sure how exactly everything works but wouldn't it be possible to play movies with vlc api and upstream them with the vlc udp proto?

[dank074]

Not sure how exactly everything works but wouldn't it be possible to play movies with vlc api and upstream them with the vlc udp proto?

VLC uses ffmpeg internally and is intended to be used for playing media, so adding it is just adding clutter. But @longnguyen2004 proposed doing something similar to what you are probably suggesting: make ffmpeg generate a MPEG-TS/MP4 stream and demux video/audio instead of getting the audio/video separately in raw frame format from ffmpeg

[longnguyen2004]

Seems promising, we get frame rate information and individual frames along with their timestamps, so we can also remove the frame packing hack that we're doing.

[BitcircuitEU]

Seems promising, we get frame rate information and individual frames along with their timestamps, so we can also remove the frame packing hack that we're doing.

Feel free to contact me if you need any media files to test :)

[dank074]

Fixed in https://github.com/dank074/Discord-video-stream/pull/108