RenderTexture.active state ( from custom blits ) negatively effects spout output

[noisecrime]

This is a bit of a weird one and not entirely sure you'd consider it a 'responsibility' of Klak Spout to address.

I have a project in Unity (2021.2.8f1) that uses the Kinect to grab and display 'userbody' as silhouettes. Alas when sending the 'gameview' through spout to say Resolume I was getting intermittent flashes of the 'user body' rt over the main output. The culprit was use of Graphic.Blit to transfer the kinect 'userbody' data to a RenderTexture, which happens at a lower frequency that the main project, causing it to flash in the spout output and only the spout output, its not visible in the editor or standalone builds!

Alas its not quite that simple, so I've detailed the specifics below.

So the Kinect library ( Kinect V2 from asset store ) gets raw kinect data ( < 30 fps approx i think ) that it puts into a computeBuffer and then renders that into a RenderTexture via Graphic.Blit and a material. Even when I do nothing more with that RenderTexture just performing the Blit to obtain the data is enough to cause the flashing of that RenderTexture in the spout output.

On a whim I found the Blit code in the Kinect Library I'm using and wrapped it in store/restore of renderTexture.active. To my pleasant surprised this fixed the problem. Alas there are about 40 Blits in the kinect library code and wrapping all them is a bit of pain. Further more though it didn't really explain why in this case it fixed the issue.

Additional testing revealed that the store/restore simply ended up setting renderTexture.active to null after the Blit. Without that code the active rendertexture was being left as the destination of the 'userbody' blit, and hence why I think it was getting into the spout output. There was still a question of why it was getting included, but looking a the SpoutSender.cs code I noticed the use of ScreenCapture.CaptureScreenshotIntoRenderTexture. I surmised, what if Unity's native code was simply grabbing the active rendertexture for this, based on the assumption that it should be null, but as shown above in my case is now pointing to the 'userbody' renderTexture. So I added the following to SpoutSender,

if KLAK_SPOUT_RT_FIX

        RenderTexture previousActive = RenderTexture.active;
        RenderTexture.active = null;
        ScreenCapture.CaptureScreenshotIntoRenderTexture(temp);
        RenderTexture.active = previousActive;

else

        ScreenCapture.CaptureScreenshotIntoRenderTexture(temp);

endif

Sure enough adding/removing that define to my player settings, and restoring the Kinect Library code to just using the Blit worked! With the fix enabled, no more flashing, with it disabled, I'd see the 'userbody' rt flash.

At this point it would seem clear to me that the Kinect Library not cleaning up the active renderTexture is a bit of a problem, but then so too is the Uniity screenCapture apparently not clearing the active rendertexture ( though I could see that might be a benefit ). This leads to a question of who has responsibility. I could see the argument that the code doing the Blit should restore it, but that could easily lead to a high amount of overhead wrapping every call or sequence of calls to Blit. Alternatively you could argue that if you want to capture a screenshot you should be expressly setting the active RT and thus its the responsibility of that code. Honestly I'm not sure, I think I favour the latter and therefore never assuming the state of the active RT.

With regard to Klak Spout I do like the 'defensive' approach of wrapping the ScreenCapture and expressly setting the active renderTexture to null before calling it, but I'm unsure beyond the scope of my project if that has any negative effects.

Any thoughts?

Thanks

[noisecrime]

Video showing the issue

https://user-images.githubusercontent.com/648081/213987460-8f747893-8e36-4a1c-9de2-f36f88f491c3.mp4

[keijiro]

I think your fix makes sense. I'll consider adding it in the next update.

[noisecrime]

Cool.

But I found something even weirder! I actually had it in my original post above, but deleted it as it was so strange, but have just double checked again and its true.

I add the spout sender to the scene ( only one scene in project ) via instantiating a prefab ( simple gameobject containing the spoutsender component ) at start up based on a flag ( retrieved from ini file ). With no other fixes and instantiating the prefab I get the glitch above. However if I disable instantiating the prefab and just have the prefab be part of the saved scene I don't get the glitch!

I can't explain this, other than perhaps there is something in the Klak spout code that is getting messed up by virtue of spout sender being instantiated into the scene at runtime. Though quite how that relates to rendertexture.active too I have no idea.

Unless this sparks a realisation of a potential issue, i think its weird enough to ignore for now, and maybe in the coming days I can create an actual test project to evaulate the issue.

[noisecrime]

I've created a small, simple test case of the blit glitch mentioned above, but more importantly it confirms that placing the SpoutSender into a prefab and instantiating it at runtime is an aspect of the bug.

I uploaded the demonstration project to github here.

To test

• Open Unity project.
• Open the scene.
• Open a Spout viewer ( e.g. Resolume or OBS ) assign 'Sprout Sender' as the spout input name.
• Run Unity

GameObjects

• Spout Launcher will instantiate the Spout prefab if none is found in the scene.
• Blitter - will randomly perform a blit. ------ Frequency of blits is controlled by the component via slider. ------ Toggle button allows you to enable/disable wrapping the blit call with renderTexture.active.

Default is for Sprout Launcher to add the SpoutSender to the scene via instantiate. To test with SpoutSender already in the scene, just drag the Spout prefab into the scene and save.

Results SpoutSender - SCENE | Wrap Blit calls - DISABLED | Klak Spout Fix - DISABLED = No Glitch SpoutSender - SCENE | Wrap Blit calls - ENABLED | Klak Spout Fix - DISABLED = No Glitch SpoutSender - SCENE | Wrap Blit calls - DISABLED | Klak Spout Fix - ENABLED = No Glitch

SpoutSender - PREFAB | Wrap Blit calls - DISABLED | Klak Spout Fix - DISABLED = Glitch SpoutSender - PREFAB | Wrap Blit calls - ENABLED | Klak Spout Fix - DISABLED = No Glitch SpoutSender - PREFAB | Wrap Blit calls - DISABLED | Klak Spout Fix - ENABLED = No Glitch

So while there can be an issue with Graphics.Blit calls, they only exhibit in my test case when instantiating a prefab of SpoutSender. In that case the glitch can be addressed by wrapping the blit with renderTexture.active ( either directly or in Klak.Spout ). If the SpoutSender is present in the scene by default there is never a glitch!

Honestly cannot explain this weirdness, but i'm assuming its something in the way Klak.Spout is initiated or some other weird combination of factors of Unity, Spout.