search

nrkno / sofie-casparcg-server

Sofie-specific fork of CasparCG Server: A Part of the Sofie TV Studio Automation System
https://github.com/nrkno/Sofie-TV-automation/
GNU General Public License v3.0
34 stars 7 forks source link

Audio and video is drifting out of sync over time. #2

Open JonStaleCarlsen opened 6 years ago

JonStaleCarlsen commented 6 years ago

We use the RC1 NRK build in the latest version of NRKbetas AutoCam system. AutoCam is doing automated visual radio production in some radio studios around the NRK.

We have detected a bug in CasparCG, which makes the video drop further and further behind the audio over time. The setup consist of a Decklink Quad2, and uses the seven first SDIs as inputs, and SDI 8 for output.

The seven input are all played on layers on channel one, and the selected live source is routed to channel 2-10. A radio audio mix is embedded on Decklink input one, and routed to channel 2-1. Channel 2 has the consumer for decklink 8, and also for file recording.

Everything works as expected when the server is freshly started, but after hours of uptime we notice that the a/v-sync is lost. We have been logging with log level trace, but could not find any occurrences of neither late nor dropped frames in the log.

I have made a temporary workaround in one of the studios, that is only recording while the red light is on, and not doing live streaming.

When the red light turn off, I clear both channels and plays all the decklink producers again. That is apparently keeping thing in sync, but is not a solution for the studios that are live streaming, since the output goes black and silent for a few seconds while i reload everything. The problem is the same on builds 2.1.0 7426 and 2.1.0 11896.

jesperstarkar commented 6 years ago

@JonStaleCarlsen could you please provide the config file + the commands used for file recording (or simply a log file)

JonStaleCarlsen commented 6 years ago

I made a mistake! The log was not set to trace in the instance of the server i was running, but rather for another build i was testing. I have now corrected this, and see lots of both late and dropped frames in the attached log.

The dropped frames seems to show up on all Decklink inputs, while the late frames is primarily on Declink input 7, that is not yet connected to a SDI source.

I guess it is a good idea to disable Declink input 7 until the source is connected for a start. log_and_config.zip

JonStaleCarlsen commented 6 years ago

Here are the logs from NRK Super and NRK Buskerud after disabling the non connected Decklink inputs. I see a lot of "browser-dropped-frame []" on build 2.1.0.11896 7951dff2d NRK RC1 in NRK Buskerud, that is not logged on build 2.1.0.7426 b7ab607ae NRK in NRK Super. Casparcg_logs_2018-06-27.zip

jesperstarkar commented 6 years ago

As discussed earlier, offloading the CPU might help/prevent this bug.