Diag Window Kills Performance

[ronag]

Having the diag window open significantly affects server performance.

[hreinnbeck]

It sure does. I'd say that diagnostics in general should be configurable.

[ronag]

It shouldn't be that bad. It worked well in the past. Unsure what happened. Suspect a live loop somewhere.

[jesperstarkar]

It's always killed CPU performance. I also see som 20mbps network traffic. Guess that is OSC data being sent blindly over UDP?

[hreinnbeck]

I do have an old patch laying around that made most of diagnostics and all of osc configurable (down to the number of audio channels, if someone wanted that data).

[ronag]

It's always killed CPU performance.

No it hasn't. Was kind of ok back in 2.0 days.

[jesperstarkar]

Then we don't agree.

[dimitry-ishenko]

Same reason I never have it open during live shows (running on Linux)

[dimitry-ishenko]

Wasn't there a proposal at one point to make it into a standalone app?

[ronag]

@jesperstarkar

Then we don't agree.

Look at the far right of the graphs:

2.3

2.0

2.3 is at least twice as bad. Probably something that happened during the 2.1 refactoring.

[mint-dewit]

It's always killed CPU performance

Judging from the graphs that @ronag shared it's a lot more now (not to say it didn't snoop cpu before). I was running performance tests earlier with diag open and it didn't seem to be a big issue for me. If I can find the time I'll try to run some tests as comparison.

Perhaps now is a good time to think about planing https://github.com/CasparCG/server/issues/899 to reduce future maintenance? I know it's not in the same scope as 2.3.1, but still good to bring to the table?

[mint-dewit]

I've failed to reproduce this running 4x 1080i50 with an ffmpeg producer and decklink consumer. Opening DIAG influences my CPU usage by at most ~5%. This is on an e5-2670 so hardly a very strong CPU.

[ronag]

Might be related the fact that I was looking over TeamViewer.

[seccpur]

IMO, users should have option to disable Diag feature during normal operations. Diag codes may be refactored to avoid overloading during disabled mode.

[Julusian]

Ive had a quick look at the code. For the screen_consumer, we aren't relying on vsync, but the consumer receiving frames is what drives the window->display() calls.

In diag we are locking to vsync, but window->display() is being called at most once every 20ms (there is a sleep at the end of tick()) so that should be capping it to 50ms.

I shall test it in a teamview only setup to get some numbers and see if disabling vsync and lowering the fps will help.

[Julusian]

@ronag Is this observed with a screen attached to that pc or is it when there is none?

[ronag]

I think there is none.

[Julusian]

Hmm, when I have no screen on my machine then both the screen consumer and diag windows just show as white over teamviewer. From a quick google, this is a limitation in teamviewer/windows.

When having a screen and using teamviewer, I get only a few percent penalty with diag. With a single channel playing 21 AMB on loop

[ronag]

. From a quick google, this is a limitation in teamviewer/windows.

Where did you read this? I've got a dozen servers running without a screen connected and we always debug using screen consumer. Works fine. I can give you access to one if you'd like.

[jesperstarkar]

@Julusian i did test with a screen. My drop increased scaling by the initial load. Take it to 50% before Diag and try then.

[Julusian]

https://community.teamviewer.com/t5/General-Questions/Some-applications-does-not-show-content-white-window-Windows-10/td-p/10718#

I was running at 80% cpu before I opened diag in that test. Perhaps I should try on a lower spec machine with a screen with a fresh install of windows

[gizahNL]

I've also noticed significant performance issues leading to framerate underruns when moving or resizing the diag window on linux.

[walterav1984]

On linux @gizahNL with Nvidia (intense) moving or resizing of any program window besides CasparCG diag or screenconsumer might give framedrops for instance a terminal or texteditor window. These performance impacts are not caused by the use of the diag window its own performance penalty discussed here.

It is possible to smooth out these Nvidia desktop compositing issues on linux but this will cap overall performance, which I rather save for the CasparCG itself.

[gizahNL]

On linux @gizahNL with Nvidia (intense) moving or resizing of any program window besides CasparCG diag or screenconsumer might give framedrops for instance a terminal or texteditor window. These performance impacts are not caused by the use of the diag window its own performance penalty discussed here.

It is possible to smooth out these Nvidia desktop compositing issues on linux but this will cap overall performance, which I rather save for the CasparCG itself.

ty @walterav1984

Though I must say the framedrops seem a lot more heavy with 2.2/3 than with 2.1 We're running 2.1 in production with NDI code backported and seem to be seeing a lot less framedrops. Also jitter I see on the diag window seems to be a lot smaller on 2.1 than on 2.3

[walterav1984]

We're running 2.1 in production with NDI code backported and seem to be seeing a lot less framedrops.

Good to know the performance of 2.1 is still better. Did you get CasparCG 2.1 (NRKNO?) with NDI backport on Linux not Windows, since I had difficulties applying that backport patch because some newer ffmpeg (Windows only code) and other updated code in the NRKNO version compared to original 2.1.

[gizahNL]

We're running 2.1 in production with NDI code backported and seem to be seeing a lot less framedrops.

Good to know the performance of 2.1 is still better. Did you get CasparCG 2.1 (NRKNO?) with NDI backport on Linux not Windows, since I had difficulties applying that backport patch because some newer ffmpeg (Windows only code) and other updated code in the NRKNO version compared to original 2.1.

We're running a customized version that runs under systemd with an assortment of patches/backports ;)

[walterav1984]

It seems its possible to build and test 2.1 again on a recent Linux distro, this may give insight on the (diag) performance thanks to @dimitry-ishenko : https://github.com/dimitry-ishenko-casparcg/tv-automation-casparcg-server/releases/

Allthough "nrkno" seems to investigate switching to 2.3 they are willing to accept pull requests since having a linux build is better than none: https://github.com/nrkno/tv-automation-casparcg-server/issues/48#issuecomment-749493410

@gizahNL how different is your backported branch from current Dimitry, is it still necessary to have a custom branch for NDI&scheduling or can this also be upstreamed?
https://github.com/nrkno/tv-automation-casparcg-server/pull/50

[gizahNL]

A lot of work has been put into dejittering NDI, and making it run at "perfect" framerate as we've had issues with this in our playout systems. This is mostly concentrated in the NDI bits, but also in some other modules (implementing API for increasing scheduling prio on linux comes to mind, which is a no-op otherwise). Some changes to the build-system to use system libraries & build into a .deb package on stretch & buster, SCTE marker module in ffmpeg producer & writing out to NDI, CEA708 passing from ffmpeg producer to NDI consumer.

Lots of other things, but these come to mind. If there is interest I'll discuss pushing our private repo to github, so stuff can be picked up and up streamed (everything currently lives on our private Gitlab), no guarantees about code quality though ;)

[walterav1984]

Sounds great, just post a link if you feel its okay to publish.

[gizahNL]

@walterav1984 Enjoy: https://github.com/in2ip/server ;)

I'm by far a great C++ programmer, so forgive any WTF you might find...

I doubt it would compile on Windows, we are a Linux club only, so the focus is to run this in our Debian based stack.

Feel free to ask, if questions arise.