search

ant-media / Ant-Media-Server

Ant Media Server is a live streaming engine software that provides adaptive, ultra low latency streaming by using WebRTC technology with ~0.5 seconds latency. Ant Media Server is auto-scalable and it can run on-premise or on-cloud.
https://antmedia.io
Other
4.25k stars 626 forks source link

Transcoding 4K@25Mbps video causes video/audio async - Learn the problem in Gstreamer #3512

Closed mekya closed 2 years ago

mekya commented 3 years ago

Short description

With following devices

On the Xavier device we use Gstreamer for streaming using the following script:

gst-launch-1.0 \
        -v v4l2src device=/dev/video0 ! image/jpeg, width=3840, height=2160, framerate=30/1, format=MJPG ! jpegdec ! nvvidconv ! queue \
        ! nvv4l2h264enc insert-sps-pps=true preset-level=1 maxperf-enable=1 bitrate=25000000 ! "video/x-h264,profile=main" \
        ! h264parse ! flvmux streamable=true name=mux \
        ! rtmpsink location="rtmp://XXXXXXX" \
        -v alsasrc device="hw:2,0" ! audioconvert ! queue ! "audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)48000, channels=(int)$
        ! voaacenc bitrate=256000 \
        ! mux.

The problem is that if we don’t enable adaptive streaming, then there is absolutely no problem and video + audio are sync.

If we enable adaptive streaming (2160p@25000kbps video and 256 kbps audio, 1080p@5000kbps video and 256 kbps audio, 720p@1500kbps video and 128 kbps audio, 480p@1000 kbps video and 96 kbps audio) then the video is not sync with audio. Audio arrives about 1 second later.

Consider that GPU is not overloaded, it is about 17% and CPU is around 4%.

Environment

Steps to reproduce

  1. Run 2.3.3.1 in Workstation with GPU Nvidia RTX6000 and CPU Intel Xeon Gold 6252
  2. Add adaptive bitrate for 1080p
  3. Send 4K@25Mbps stream to the Ant Media Server
  4. Watch the stream in WebRTC playback

Expected behavior

Video and audio should be in-sync

Actual behavior

Audio is coming earlier than the video

I've the credentials to connect to the workstation. Please reach out to me to provide this information.

burak-58 commented 2 years ago

After investigating the issue deeply we found that the issue is related to the encoder settings. We create a video file with the gstreamer command in the issue. We observed the same issue while publishing this video with RTMP to AMS. When we transcode the video with following ffmpeg command sync problem is fixed. ffmpeg -i input.mp4 -c:v libx264 -preset fast -profile baseline -tune zerolatency -c:a copy output.mp4