Auto-Framing does not work reliably with 4k resolution / interacts with other filters

[hofst]

Operating System

Windows 10 1903 and higher

OBS Studio Version?

27.1

StreamFX Version

0.11.0b3

OBS Studio Log

Attached

OBS Studio Crash Log

No response

Current Behavior

At the moment, the auto-framing selects a wrong frame (or in this case the upper left corner). However, if I add another effect, it works again. Also, the green-screen seems to not work after a couple of hours and I have to remove and re-add the effect so it works again. My only explanation is that there is some shared state between the nvidia filters and the effects influence/corrupt each other, especially with higher resolution

. Here is a recording to demonstrate it (I have added a blur filter for privacy)

https://user-images.githubusercontent.com/2535762/145092376-0505ef8c-e64d-4edd-b3f1-29f606a4ebba.mp4 2021-12-07 18-23-24.txt 2021-12-07 11-39-09.txt

Expected Behavior

The filters should work reliably

Steps to Reproduce the Bug

Use a 4k webcam with 1920p OBS projects and use the auto-framing filter independently or in conjunction with the green-screen filter.

Any additional Information we need to know?

No response

[Xaymar]

Looking at your provided log file, it appears as though you are simply running out of GPU Memory and Time. A 3070 is not powerful enough to handle multiple Tensor effects at the same time, much less at 4K. This is even visible in your log files:

14:39:48.592: [StreamFX] Failed to process due to error: unknown error

[hofst]

Hmm, I don't think that's it: If you look at my screen recording, you can see that actually both effects (green screen + auto frame) work at the moment when they are both activated. But if I activate only one effect (auto frame) it does not work. If there was a memory issue, I would expect the exact opposite behavior.

[jessemillar]

Looking at your provided log file, it appears as though you are simply running out of GPU Memory and Time. A 3070 is not powerful enough to handle multiple Tensor effects at the same time, much less at 4K. This is even visible in your log files:

14:39:48.592: [StreamFX] nvidia::vfx::greenscreen::greenscreen Failed to process due to error: unknown error

Are there system requirements anywhere for how many resources these NVIDIA effects require? Or is it entirely dependent on the filter configurations in OBS (e.g. refresh frequency and smoothing)?

[Xaymar]

If there was a memory issue, I would expect the exact opposite behavior.

No, you would expect to see this kind of behavior. You're actively causing two Tensor (Machine Learning) networks to compete asynchronously for resources, which results in unexpected and unexplainable behavior like this. With a 3070, you simply do not have the Memory or Tensor capabilities to run many Tensor-based effects in parallel, or at real time with a high resolution. Unfortunately this is further cemented by me failing the repro this with a 3080 and a 3090.

Are there system requirements anywhere for how many resources these NVIDIA effects require? Or is it entirely dependent on the filter configurations in OBS (e.g. refresh frequency and smoothing)?

As the exact usage is decided by NVIDIA providing the Machine Learning models for GPUs, no. But many of them are affected by the resolution you choose to push into the filter itself, with their base VRAM usage being around 1-2 GiB.

[hofst]

I don't know how to make this any clearer: Two simultaneous tensor effects are working fine and without any problems. I think you misunderstood me there. GPU memory consumption is not even at 50% of the available 8GB and also NVIDIA Broadcast works flawlessly on my machine with any two tensor effects with 4k resolution without any problem at all.

The problem occurs iff I have only enabled one single tensor effect (auto-framing). I don't have any problems when there are two tensor effects. There only is a problem when I have enabled exactly one single effect (auto framing), it does not work as expected and chooses a completely wrong frame.

To demonstrate again, here is the wrong auto framing as the only filter:

And here is the magically fixed auto-framing when I add one additional arbitrary filter on top of it:

[Xaymar]

The problem occurs iff I have only enabled one single tensor effect (auto-framing). I don't have any problems when there are two tensor effects. There only is a problem when I have enabled exactly one single effect (auto framing), it does not work as expected and chooses a completely wrong frame.

Your bug report itself includes two log files which, for absolutely no reason as you claim, contain multiple effects. If the problem only occurs with a single effect, then it should be within your power to provide a pre-recorded video used in a scene collection which would cause this. So far, no matter what I tried, I've been unable to get it to happen with a 3080 or a 3090, both of which have significantly more memory available at any point in time.

As far as I'm concerned, since I lack the means of reproducing this on hardware that is actually capable of running multiple expensive networks at >50 FPS, you are running either into Tensor limits or Memory limits. Possibly even both.

If you want to learn a bit about how the filter makes decisions, enable the Debug checkbox. It will tell you the exact boundaries that it picks and why it picked them.

[hofst]

Sure, here is a video for reproing:

Use OBS with both Canvas resolution and output resolution = 1920x1080. Embed this 4k video as media source and enable the auto-framing filter on it. 2021-12-09 21-58-55.zip

You will see that auto-framing does not work. If you change the OBS resolutions to 4k, then it will suddenly work.

[Xaymar]

Repro successful with the provided video file.

[Xaymar]

Underlying bug:

• Expected source to set its own projection matrix.
• Async video sources don't do this all the time.

[hofst]

Thanks a lot! :-)