[Bug] Managed Subsystems Freezes the Unity Editor

[frankvHoof93]

Describe the bug Using managed instead of unmanaged subsystems can freeze the Unity Editor when leaving and re-entering Play Mode. This is due to specific features requiring/expecting native invokes. In particular, the MonoPInvokeCallback-Attribute in ARCameraBackground (on the BeforeBackgroundRenderHandler-Method) will cause the Editor to freeze if the XRCameraSubsystem is of a managed type. This Attribute is also used in XRCpuImage in Unity ARSubsystems (on the OnAsyncConversionComplete-Method). I have not tested this exact method, but would expect a similar result.

To Reproduce Steps to reproduce the behavior:

1. Create a Managed XRCameraSubsystem that returns Textures from a Unity Camera
2. Enter Play Mode. The Textures are displayed as background by ARCameraBackground
3. Exit Play Mode.
4. Re-Enter Play Mode

Expected behavior The second run-through of Play Mode behaves the exact same as the first

Actual behavior The editor goes into an infinite loop/await before entering Play Mode, getting stuck on either:

• Application.Reload
• AssetDatabase.Refresh
• Application.UpdateScene

Possible Resolution Adding a #if !UNITY_EDITOR around the Attribute prevents the BeforeBackgroundRenderHandler-Method from being Marshalled as a Native Method. However, this would also prevent any native XRCameraSubsystem from running this method if attached to the Editor. An optimal resolution would be a MANAGED_SUBSYTEMS-define that can be set by the developer, to prevent ARFoundation-Code from treating subsystems as Native.

Relevant Versioning Information:

• Device: PC
• OS: Windows 10
• Unity 2020.1.4
• ARFoundation 4.1.1

[tdmowrer]

Thanks for the report.

This is due to specific features requiring/expecting native invokes.

The method you pointed out (BeforeBackgroundRenderHandler) has the MonoPInvokeCallback attribute so that it can be reversed p/invoked from Unity's graphics thread, which invokes a function pointer from the render loop. While this is a reverse p/invoke from native code, nothing requires that the XRCameraSubsystem be "native". Note that the implementations for ARCore and ARKit are still managed C# objects. The callbacks and virtual methods ultimately have to p/invoke into ARCore or ARKit, but ARFoundation does not know that -- it simply invokes a method on a managed type.

In other words, wrt XRCameraSubsystem, there is no concept of "native" vs "managed" subsystems as you have described them, nor is a p/invoke required.

I tried to reproduce this by making an Editor-based implementation of the XRCameraSubsystem, using the WebCamTexture as the texture provided to ARFoundation. This works as I would expect it to and does not freeze the editor.

If you believe this is a bug in ARFoundation, please log a bug with a repro case.

[frankvHoof93]

@tdmowrer Did you enter play mode twice? It runs fine the first time, but freezes if you attempt to run it twice. I have created a repro-project and filed a bug-report: http://fogbugz.unity3d.com/default.asp?1300312_2cq5rr7kv52ib8vj The full project can also be found here: https://github.com/frankvHoof93/XRCameraSubsystemRepro As stated above; the project runs fine on its first run-through, but freezes if the user enters play mode a second time.

[tdmowrer]

Did you enter play mode twice?

Yes, several times.

I have created a repro-project and filed a bug-report

Thank you; that should help us track it down.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.