search

videokit-ai / videokit

Low-code, cross-platform media SDK for Unity Engine. Register at https://videokit.ai
https://videokit.ai
Apache License 2.0
107 stars 14 forks source link

WebGL crash with Meet VideoKit 0.0.10 - MediaDeviceQuery_DiscoverDevices #17

Closed tomaszwozniak123 closed 1 year ago

tomaszwozniak123 commented 1 year ago

Hi, In WebGL build sometimes (very often) when loading scene with NatML Meet Segmentation, there is a crash. Please take a look in the Profiler screenshot - seems like MLModelData.OnEnable() drops FPS suddenly, and maybe this is a cause of crash? If not - could you suggest some solution for this? Is there a way to improve memory size for WebGL? Or maybe it is directly connected with MediaDeviceQuery_DiscoverDevices?

This error occurs: testtest.framework.js:3 Uncaught RuntimeError: memory access out of bounds at dlfree (testtest.wasm:0x9b73e) at MemoryManager::LowLevelFree(void*, unsigned long) (testtest.wasm:0x2c6f13) at UnityDefaultAllocator<LowLevelAllocator>::Deallocate(void*) (testtest.wasm:0x18a6591) at ThreadsafeLinearAllocator<false>::TryDeallocate(void*) (testtest.wasm:0x699528) at MemoryManager::Deallocate(void*, MemLabelId, char const*, int) (testtest.wasm:0x4a2737) at operator delete(void*) (testtest.wasm:0x9125e) at NDMediaDeviceGetName (testtest.wasm:0xf193bd) at NatDevice_Name_m7BE4A206300A690F90B92697C3E540277F177766 (testtest.wasm:0x7438fc) at AudioDevice__ctor_mE8A1C592C648F889936191E95E8E557A845B30E6 (testtest.wasm:0x117f137) at U3CDiscoverAudioDevicesU3Ed__20_MoveNext_m3947F1662EDDA93971217F8C7E67172E63FDE931 (testtest.wasm:0x1979979) at InterfaceFuncInvoker0<Toggle_tBF13F3EBA485E06826FD8A38F4B4C1380DF21A1F*>::Invoke(unsigned short, Il2CppClass*, Il2CppObject*) (testtest.wasm:0x918b5) at dynCall_iiii (testtest.wasm:0x1749226) at invoke_iiii (testtest.framework.js:3:525248) at List_1_AddEnumerable_mFD4E2919839A27C7CEACADF62222C63701302D02_gshared (testtest.wasm:0x1e81a36) at RuntimeInvoker_TrueVoid_t4861ACF8F4594C3437BB48B6E56783494B843915_VoidU2A_t104EAEFBD2D237A8C29618913DA9B4D99355E965(void (*)(), MethodInfo const*, void*, void**, void*) (testtest.wasm:0xdf43cc) at InvokerActionInvoker1<RtcStats_tB82CEAB4F971E9D092B96B90C0C3BFFB035D1BE8>::Invoke(void (*)(), MethodInfo const*, void*, RtcStats_tB82CEAB4F971E9D092B96B90C0C3BFFB035D1BE8) (testtest.wasm:0x9292a) at List_1_InsertRange_m729639E2C8D1AA7579AC78D259085921E7DC72F2_gshared (testtest.wasm:0xf5b303) at RuntimeInvoker_TrueVoid_t4861ACF8F4594C3437BB48B6E56783494B843915_CancellationToken_t51142D9C6D7C02D314DA34A6A7988C528992FFED_RuntimeObject(void (*)(), MethodInfo const*, void*, void**, void*) (testtest.wasm:0x1c3d9d) at InvokerActionInvoker2<int, String_t*>::Invoke(void (*)(), MethodInfo const*, void*, int, String_t*) (testtest.wasm:0xa0b5a) at List_1_AddRange_m44DED39FFEBC8F2ACF6162DB065B649E888B9C57_gshared (testtest.wasm:0xf5ba9a) at List_1_AddRange_m66D40F60D60CF3FA653432ABEBEBF703840689DB(List_1_t437B6C3879E969156A381BDC3C459CF809D39DDF*, Il2CppObject*, MethodInfo const*) (testtest.wasm:0xa11cd) at MediaDeviceQuery_DiscoverDevices_m0520738B6315FF77EFB342F12AB3CF55102140C5 (testtest.wasm:0xf450aa) at MediaDeviceQuery__ctor_mE2C86815B53C8771C6F5BD82319E56505C59FF82 (testtest.wasm:0x2c74bc) at U252EU252DU252DU252EU252EU252EU252DU252DU252DU252EU252DU252DU252DU252EU252DU252DU252DU252EU252EU252EU252EU252DU252DU252EU252DU252EU252DU252DU252EU252EU252EU252EU252EU252DU252EU252EU252EU252DU252EU252EU252EU252EU252EU252DU252DU252EU252E_U252EU252EU252DU252DU252DU252EU252DU252EU252DU252EU252DU252EU252EU252EU252DU252DU252EU252EU252EU252EU252DU252EU252EU252EU252DU252DU252EU252EU252EU252EU252DU252EU252DU252DU252DU252DU252EU252DU252DU252EU252DU252DU252EU252DU252DU252EU252E_mA165F6D2E8CE5BDA1F792073168AB4F5C78B81EB (testtest.wasm:0x682594) at NatMlMeetBackgroundRemover_SwitchCameraDeviceToTarget_mF727B58824BDA93FF90F16C8943B75A8A71D35A5 (testtest.wasm:0x186543) at HallwayViewModel_U252DU252DU252EU252EU252EU252EU252EU252DU252EU252EU252DU252DU252EU252DU252EU252DU252EU252EU252EU252DU252DU252DU252DU252EU252DU252EU252DU252DU252EU252EU252EU252DU252DU252DU252DU252EU252DU252EU252EU252DU252EU252DU252DU252DU252EU252DU252E_mBED46317D9DAB642EACBDF579B8CC8D473313E97 (testtest.wasm:0x783e11) at HallwayViewModel_U252EU252EU252EU252EU252EU252DU252DU252EU252DU252DU252DU252EU252DU252DU252DU252DU252EU252DU252DU252DU252DU252EU252EU252EU252EU252DU252EU252EU252EU252DU252DU252EU252EU252DU252EU252EU252EU252DU252DU252DU252EU252EU252DU252EU252EU252DU252E_mE7C7F9E1E9446C4ECB510F50DDA0C22E8FE59917 (testtest.wasm:0xfe70f4) at HallwayViewModel_Start_mD1BD1C85DC59E11053F9FD98C15301F462C6A8B8 (testtest.wasm:0x1f49417) at RuntimeInvoker_TrueVoid_t4861ACF8F4594C3437BB48B6E56783494B843915(void (*)(), MethodInfo const*, void*, void**, void*) (testtest.wasm:0x1c82f76) at il2cpp::vm::Runtime::InvokeWithThrow(MethodInfo const*, void*, void**) (testtest.wasm:0x40d8b2) at dynCall_iiii (testtest.wasm:0x1749226) at invoke_iiii (testtest.framework.js:3:525248) at il2cpp::vm::Runtime::Invoke(MethodInfo const*, void*, void**, Il2CppException**) (testtest.wasm:0xf56fe) at il2cpp_runtime_invoke (testtest.wasm:0xaef264) at scripting_method_invoke(ScriptingMethodPtr, ScriptingObjectPtr, ScriptingArguments&, ScriptingExceptionPtr*, bool) (testtest.wasm:0x3c948a) at ScriptingInvocation::Invoke(ScriptingExceptionPtr*, bool) (testtest.wasm:0x1c30a1) at ScriptingInvocation::Invoke(ScriptingExceptionPtr*) (testtest.wasm:0xf60f9) at MonoBehaviour::InvokeMethodOrCoroutineChecked(ScriptingMethodPtr, ScriptingObjectPtr) (testtest.wasm:0x7473aa) at MonoBehaviour::Start() (testtest.wasm:0x36b6c3) at MonoBehaviour::DelayedStartCall(Object*, void*) (testtest.wasm:0x1e2128e) at DelayedCallManager::Update(int) (testtest.wasm:0x2059dc9) at InitPlayerLoopCallbacks()::EarlyUpdateScriptRunDelayedStartupFrameRegistrator::Forward() (testtest.wasm:0x1f0da91) at ExecutePlayerLoop(NativePlayerLoopSystem*) (testtest.wasm:0x103f5a0) at ExecutePlayerLoop(NativePlayerLoopSystem*) (testtest.wasm:0x103f614) at MainLoop() (testtest.wasm:0xf50fda) at dynCall_v (testtest.wasm:0x1748a79) at browserIterationFunc (testtest.framework.js:3:257137) at callUserCallback (testtest.framework.js:3:210100) at Object.runIter (testtest.framework.js:3:211360) at Browser_mainLoop_runner (testtest.framework.js:3:209635)

image

tomaszwozniak123 commented 1 year ago

I have found a second error, which sometimes shows at the same moment as the one above: testgoo.framework.js:3 Uncaught TypeError: Cannot read properties of null (reading 'remove') at methodCaller_void_$$ (eval at new_ (testgoo.framework.js:3:197163), <anonymous>:4:20) at __emval_call_void_method (testgoo.framework.js:3:196181) at NatML::Devices::CameraDevice::StopRunning() (testgoo.wasm:0x1e13499) at NDMediaDeviceStopRunning (testgoo.wasm:0x49cc2b) at CameraDevice_StopRunning_mD3E4E4E031EDFD4EDCFEF584F12C41C0C3131E82 (testgoo.wasm:0x8009e6) at VideoKitCameraManager_StopRunning_m3EC6267DD56C025F8E956F7659086DA23D04CBD3 (testgoo.wasm:0x1578f4) at VideoKitCameraManager_OnDestroy_m6818EA551700A0A76729AE0AD07E8812281A81F5 (testgoo.wasm:0x1846ca7) at RuntimeInvoker_TrueVoid_t4861ACF8F4594C3437BB48B6E56783494B843915(void (*)(), MethodInfo const*, void*, void**, void*) (testgoo.wasm:0x1c91be2) at il2cpp::vm::Runtime::InvokeWithThrow(MethodInfo const*, void*, void**) (testgoo.wasm:0x40d902) at dynCall_iiii (testgoo.wasm:0x1754f21) at invoke_iiii (testgoo.framework.js:3:525248) at il2cpp::vm::Runtime::Invoke(MethodInfo const*, void*, void**, Il2CppException**) (testgoo.wasm:0xf6224) at il2cpp_runtime_invoke (testgoo.wasm:0xaf0dde) at scripting_method_invoke(ScriptingMethodPtr, ScriptingObjectPtr, ScriptingArguments&, ScriptingExceptionPtr*, bool) (testgoo.wasm:0x3c97f1) at ScriptingInvocation::Invoke(ScriptingExceptionPtr*, bool) (testgoo.wasm:0x1c79ca) at ScriptingInvocation::Invoke(ScriptingExceptionPtr*) (testgoo.wasm:0xf6ac5) at SerializableManagedRef::CallMethod(Object&, ScriptingMethodPtr) (testgoo.wasm:0xdcbc77) at MonoBehaviour::CallMethodInactive(ScriptingMethodPtr) (testgoo.wasm:0x18a236) at MonoBehaviour::WillDestroyComponent() (testgoo.wasm:0x1e2a8d2) at PreDestroyRecursive(GameObject&, unsigned long*) (testgoo.wasm:0xe63c24) at DestroyObjectHighLevel(Object*, bool) (testgoo.wasm:0x126c34) at UnloadGameScene() (testgoo.wasm:0xc0a75a) at LoadSceneOperation::PlayerLoadSceneFromThread() (testgoo.wasm:0xf6061d) at LoadSceneOperation::IntegrateMainThread() (testgoo.wasm:0x1e879eb) at PreloadManager::UpdatePreloadingSingleStep(PreloadManager::UpdatePreloadingFlags, int) (testgoo.wasm:0xe62d5b) at PreloadManager::UpdatePreloading() (testgoo.wasm:0xe62b7c) at InitPlayerLoopCallbacks()::EarlyUpdateUpdatePreloadingRegistrator::Forward() (testgoo.wasm:0x1f168bc) at ExecutePlayerLoop(NativePlayerLoopSystem*) (testgoo.wasm:0x1048343) at ExecutePlayerLoop(NativePlayerLoopSystem*) (testgoo.wasm:0x10483b7) at MainLoop() (testgoo.wasm:0xf586a6) at dynCall_v (testgoo.wasm:0x1754774) at browserIterationFunc (testgoo.framework.js:3:257137) at callUserCallback (testgoo.framework.js:3:210100) at Object.runIter (testgoo.framework.js:3:211360) at Browser_mainLoop_runner (testgoo.framework.js:3:209635)

It points to this part of code in VideoKitCameraManager:

public void StopRunning () { cameraDevice?.StopRunning(); output?.Dispose(); output = null; }

And in addition I see this warning:

image

olokobayusuf commented 1 year ago

In WebGL build sometimes (very often) when loading scene with NatML Meet Segmentation, there is a crash.

The call stack seems to indicate something about getting the name of the camera device. What browser and Unity versions are you on?

have found a second error, which sometimes shows at the same moment as the one above:

This is related to the first, and happens because the camera preview is being stopped, but due to the first error, the camera is an invalid state.

tomaszwozniak123 commented 1 year ago

Hi Browser version: Chrome Version 110.0.5481.105 (Official Build) (64-bit) Unity: 2021.3.12f1

tomaszwozniak123 commented 1 year ago

Additionally, here are my methods used with MediaDeviceQuery, maybe it helps in investigation:

        public static class VideoKitCameraManagerCustom
        {
            public static CameraDevice GetNextCameraDevice(this VideoKitCameraManager videoKitCameraManager, CameraDevice currentCamera)
            {
                MediaDeviceQuery.ConfigureAudioSession = false;
                var query = new MediaDeviceQuery(MediaDeviceCriteria.CameraDevice);
                var devices = query.OrderBy(device => device is CameraDevice camera).ToList();
                var currentDeviceIndex = devices.IndexOf(currentCamera);

            if (currentDeviceIndex < devices.Count - 1)
            {
                currentDeviceIndex++;
            }
            else
            {
                currentDeviceIndex = 0;
            }
            var nextDevice = devices[currentDeviceIndex];
            return (CameraDevice)nextDevice;
        }

        public static CameraDevice GetNextTargetCameraDevice(this VideoKitCameraManager videoKitCameraManager, int index)
        {
            MediaDeviceQuery.ConfigureAudioSession = false;
            var query = new MediaDeviceQuery(MediaDeviceCriteria.CameraDevice);
            var devices = query.OrderBy(device => device is CameraDevice camera).ToList();
            var nextDevice = devices[Mathf.Max(0, index)];
            if (nextDevice != null)
            {
                return (CameraDevice)nextDevice;
            }
            else
            {
                return (CameraDevice)devices[0];
            }
        }

        public static int GetCurrentCameraDeviceIndex(this VideoKitCameraManager videoKitCameraManager, CameraDevice currentCamera)
        {
            MediaDeviceQuery.ConfigureAudioSession = false;
            var query = new MediaDeviceQuery(MediaDeviceCriteria.CameraDevice);
            var devices = query.OrderBy(device => device is CameraDevice camera).ToList();
            var currentDeviceIndex = devices.IndexOf(currentCamera);
            return currentDeviceIndex;
        }
    }
olokobayusuf commented 1 year ago

Hey @tomaszwozniak123 , can you try VideoKit 0.0.11?

tomaszwozniak123 commented 1 year ago

Hey @olokobayusuf , I used VideoKit 0.0.11 and get this error instead previous ones (it happens only in browser-Unity Editor works fine). It happens when trying switching scenes:

testgoo.loader.js:1 exception thrown: TypeError: Cannot read properties of null (reading 'remove'),TypeError: Cannot read properties of null (reading 'remove')
    at methodCaller_void_$$ (eval at new_ (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.framework.js:3:197163), <anonymous>:4:20)
    at __emval_call_void_method (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.framework.js:3:196181)
    at NatML::Devices::CameraDevice::StopRunning() (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[116334]:0x25f2b51)
    at CameraDevice_StopRunning_mD3E4E4E031EDFD4EDCFEF584F12C41C0C3131E82 (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[30092]:0xa53001)
    at VideoKitCameraManager_StopRunning_m3EC6267DD56C025F8E956F7659086DA23D04CBD3 (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[5809]:0x1bbebe)
    at VideoKitCameraManager_OnDestroy_m6818EA551700A0A76729AE0AD07E8812281A81F5 (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[79002]:0x1d97e4f)
    at RuntimeInvoker_TrueVoid_t4861ACF8F4594C3437BB48B6E56783494B843915(void (*)(), MethodInfo const*, void*, void**, void*) (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[106031]:0x23a4543)
    at il2cpp::vm::Runtime::InvokeWithThrow(MethodInfo const*, void*, void**) (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[16615]:0x52ee7f)
    at dynCall_iiii (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[73323]:0x1c3cf3a)
    at invoke_iiii (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.framework.js:3:526839)
    at il2cpp::vm::Runtime::Invoke(MethodInfo const*, void*, void**, Il2CppException**) (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[3695]:0x13f753)
    at il2cpp_runtime_invoke (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[34461]:0xd99a8f)
    at scripting_method_invoke(ScriptingMethodPtr, ScriptingObjectPtr, ScriptingArguments&, ScriptingExceptionPtr*, bool) (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[16173]:0x4e91ca)
    at ScriptingInvocation::Invoke(ScriptingExceptionPtr*, bool) (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[1685]:0xe2e5d)
    at ScriptingInvocation::InvokeChecked(ScriptingExceptionPtr*) (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[44139]:0x113695f)
    at SerializableManagedRef::CallMethod(Object&, ScriptingMethodPtr) (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[6732]:0x1f760f)
    at MonoBehaviour::WillDestroyComponent() (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[116853]:0x260e64e)
    at PreDestroyRecursive(GameObject&, unsigned long*) (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[46575]:0x11f2abf)
    at DestroyObjectHighLevel_Internal(Object*, bool) (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[29184]:0x9f9a4e)
    at UnloadGameScene() (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[38571]:0xefa508)
    at LoadSceneOperation::PlayerLoadSceneFromThread() (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[48992]:0x12ed739)
    at LoadSceneOperation::IntegrateMainThread() (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[118849]:0x26899d6)
    at PreloadManager::UpdatePreloadingSingleStep(PreloadManager::UpdatePreloadingFlags, int) (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[46527]:0x11f0b7d)
    at PreloadManager::UpdatePreloading() (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[46526]:0x11f090c)
    at InitPlayerLoopCallbacks()::EarlyUpdateUpdatePreloadingRegistrator::Forward() (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[123766]:0x27bce43)
    at ExecutePlayerLoop(NativePlayerLoopSystem*) (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[54795]:0x143cde2)
    at ExecutePlayerLoop(NativePlayerLoopSystem*) (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[54795]:0x143ce53)
    at MainLoop() (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[48815]:0x12e476a)
    at dynCall_v (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.wasm:wasm-function[73262]:0x1c3c735)
    at browserIterationFunc (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.framework.js:3:257137)
    at callUserCallback (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.framework.js:3:210100)
    at Object.runIter (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.framework.js:3:211360)
    at Browser_mainLoop_runner (https://v6p9d9t4.ssl.hwcdn.net/html/7446927/Build/testgoo.framework.js:3:209635)
tomaszwozniak123 commented 1 year ago

Hey @olokobayusuf Update for the error above: When I added these two condition checks to StopRunning() method in the CameraDevice class, it stopped crashing (haven't tested much, so please take it as a hint only):

 public void StopRunning () {
            if (running==false) { return; }
            if (device!=null)
            {
                device.StopRunning().CheckStatus();
            }

            lifecycleHelper?.Dispose();
            lifecycleHelper = default;
            if (previewHandle != default)
                previewHandle.Free();
            previewHandle = default;
        }
olokobayusuf commented 1 year ago

Hey @tomaszwozniak123 sorry for the late follow up on this. Your fix is correct, and the fix is landing in 0.0.12.

olokobayusuf commented 1 year ago

Fixed in VideoKit 0.0.12.

tomaszwozniak123 commented 1 year ago

Works great, thanks! :)