search

CesiumGS / cesium-unity

Bringing the 3D geospatial ecosystem to Unity
https://cesium.com/platform/cesium-for-unity/
Apache License 2.0
358 stars 83 forks source link

Crash on Quest 2 / Quest 3 after around 20 minutes of playtime #408

Open electrum-bowie opened 9 months ago

electrum-bowie commented 9 months ago

Cesium for Unity 1.7.1.

I should also add that it only happens in a Scene with heavier script processing (multiple runtime behaviour scripts running at the same time), What do you suggest i do? Maybe lowering the Max Cached Bytes property? is Memory Even the issue??

16 people using a quest vr device (android) experienced this exact crash on their own device atleast once: Crash Reason Abort Reason: terminating (libCesiumForUnityNative-Runtime.so)

Thread 25382

00 pc 000000000008ada0 /apex/com.android.runtime/lib64/bionic/libc.so (abort+168) (BreakpadBuildId: 06550e2bcf68e781af8072edf3aa70cf0)

01 pc 00000000018c4648 libil2cpp.so (libil2cpp.so (0x18c4648)) (BreakpadBuildId: 05ff99529c6fff0900000000000000000)

02 pc 00000000018c4798 libil2cpp.so (libil2cpp.so (0x18c4798)) (BreakpadBuildId: 05ff99529c6fff0900000000000000000)

03 pc 00000000018c4700 libil2cpp.so (libil2cpp.so (0x18c4700)) (BreakpadBuildId: 05ff99529c6fff0900000000000000000)

04 pc 00000000018c4684 libil2cpp.so (std::terminate()+36) (BreakpadBuildId: 05ff99529c6fff0900000000000000000)

05 pc 00000000018c48a8 libil2cpp.so (libil2cpp.so (0x18c48a8)) (BreakpadBuildId: 05ff99529c6fff0900000000000000000)

06 pc 0000000000a65178 /data/app/~~ImIKlHoLSZzYNYi4ko35hQ==/com.bowie.earthquest-wruUPKBVmawpC_Ev8jOzRQ==/lib/arm64/libCesiumForUnityNative-Runtime.so (std::__terminate(void ()())+12)

07 pc 0000000000a659b8 /data/app/~~ImIKlHoLSZzYNYi4ko35hQ==/com.bowie.earthquest-wruUPKBVmawpC_Ev8jOzRQ==/lib/arm64/libCesiumForUnityNative-Runtime.so (__cxxabiv1::call_terminate(bool, _Unwind_Exception)+36)

08 pc 0000000000a6595c /data/app/~~ImIKlHoLSZzYNYi4ko35hQ==/com.bowie.earthquest-wruUPKBVmawpC_Ev8jOzRQ==/lib/arm64/libCesiumForUnityNative-Runtime.so (cxxabiv1::scan_eh_tab(cxxabiv1::(anonymous namespace)::scan_results&, _Unwind_Action, bool, _Unwind_Exception, _Unwind_Context)+1508)

09 pc 0000000000a65240 /data/app/~~ImIKlHoLSZzYNYi4ko35hQ==/com.bowie.earthquest-wruUPKBVmawpC_Ev8jOzRQ==/lib/arm64/libCesiumForUnityNative-Runtime.so (__gxx_personality_v0+104)

10 pc 00000000018dbb38 libil2cpp.so (libil2cpp.so (0x18dbb38)) (BreakpadBuildId: 05ff99529c6fff0900000000000000000)

11 pc 00000000018c3bf0 libil2cpp.so (__cxa_throw+116) (BreakpadBuildId: 05ff99529c6fff0900000000000000000)

12 pc 0000000001842e0c libil2cpp.so (libil2cpp.so (0x1842e0c)) (BreakpadBuildId: 05ff99529c6fff0900000000000000000)

13 pc 00000000017fb94c libil2cpp.so (libil2cpp.so (0x17fb94c)) (BreakpadBuildId: 05ff99529c6fff0900000000000000000)

14 pc 00000000017fbd70 libil2cpp.so (libil2cpp.so (0x17fbd70)) (BreakpadBuildId: 05ff99529c6fff0900000000000000000)

15 pc 00000000019bb66c libil2cpp.so (libil2cpp.so (0x19bb66c)) (BreakpadBuildId: 05ff99529c6fff0900000000000000000)

16 pc 00000000015d1f54 libil2cpp.so (libil2cpp.so (0x15d1f54)) (BreakpadBuildId: 05ff99529c6fff0900000000000000000)

17 pc 000000000044cc98 /data/app/~~ImIKlHoLSZzYNYi4ko35hQ==/com.bowie.earthquest-wruUPKBVmawpC_Ev8jOzRQ==/lib/arm64/libCesiumForUnityNative-Runtime.so (CesiumForUnityNative::Cesium3DTilesetImpl::Update(DotNet::CesiumForUnity::Cesium3DTileset const&)+356)

18 pc 0000000000425a24 /data/app/~~ImIKlHoLSZzYNYi4ko35hQ==/com.bowie.earthquest-wruUPKBVmawpC_Ev8jOzRQ==/lib/arm64/libCesiumForUnityNative-Runtime.so (DotNet_CesiumForUnity_Cesium3DTileset_Update+60)

19 pc 0000000001a29974 libil2cpp.so (libil2cpp.so (0x1a29974)) (BreakpadBuildId: 05ff99529c6fff0900000000000000000)

20 pc 0000000001859468 libil2cpp.so (libil2cpp.so (0x1859468)) (BreakpadBuildId: 05ff99529c6fff0900000000000000000)

21 pc 00000000018593b4 libil2cpp.so (libil2cpp.so (0x18593b4)) (BreakpadBuildId: 05ff99529c6fff0900000000000000000)

22 pc 000000000063ef90 libunity.so (libunity.so (0x63ef90)) (BreakpadBuildId: 467edd95c85b31b500000000000000000)

23 pc 000000000064e8cc libunity.so (libunity.so (0x64e8cc)) (BreakpadBuildId: 467edd95c85b31b500000000000000000)

24 pc 000000000065ca50 libunity.so (libunity.so (0x65ca50)) (BreakpadBuildId: 467edd95c85b31b500000000000000000)

25 pc 0000000000484308 libunity.so (libunity.so (0x484308)) (BreakpadBuildId: 467edd95c85b31b500000000000000000)

26 pc 00000000005491d0 libunity.so (libunity.so (0x5491d0)) (BreakpadBuildId: 467edd95c85b31b500000000000000000)

27 pc 0000000000549210 libunity.so (libunity.so (0x549210)) (BreakpadBuildId: 467edd95c85b31b500000000000000000)

28 pc 000000000054949c libunity.so (libunity.so (0x54949c)) (BreakpadBuildId: 467edd95c85b31b500000000000000000)

29 pc 00000000006c0ef8 libunity.so (libunity.so (0x6c0ef8)) (BreakpadBuildId: 467edd95c85b31b500000000000000000)

30 pc 00000000006dd318 libunity.so (libunity.so (0x6dd318)) (BreakpadBuildId: 467edd95c85b31b500000000000000000)

31 pc 0000000000004e0c /data/app/~~ImIKlHoLSZzYNYi4ko35hQ==/com.bowie.earthquest-wruUPKBVmawpC_Ev8jOzRQ==/oat/arm64/base.odex (art_jni_trampoline+92)

32 pc 0000000000008634 /data/app/~~ImIKlHoLSZzYNYi4ko35hQ==/com.bowie.earthquest-wruUPKBVmawpC_Ev8jOzRQ==/oat/arm64/base.odex (com.unity3d.player.L.handleMessage+692)

33 pc 00000000000c84e0 /system/framework/arm64/boot-framework.oat (android.os.Handler.dispatchMessage+144) (BreakpadBuildId: b4cde2d562dfd48c80445adfd718ab400)

34 pc 00000000000c9700 /system/framework/arm64/boot-framework.oat (android.os.Looper.loopOnce+1424) (BreakpadBuildId: b4cde2d562dfd48c80445adfd718ab400)

35 pc 00000000000c90d4 /system/framework/arm64/boot-framework.oat (android.os.Looper.loop+516) (BreakpadBuildId: b4cde2d562dfd48c80445adfd718ab400)

36 pc 00000000000094c4 /data/app/~~ImIKlHoLSZzYNYi4ko35hQ==/com.bowie.earthquest-wruUPKBVmawpC_Ev8jOzRQ==/oat/arm64/base.odex (com.unity3d.player.O.run+244)

37 pc 0000000000218964 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+548) (BreakpadBuildId: c3d95a7d77dcec225dce96800a7a95940)

38 pc 000000000028495c /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread, unsigned int, unsigned int, art::JValue, char const)+188) (BreakpadBuildId: c3d95a7d77dcec225dce96800a7a95940)

39 pc 0000000000628a7c /apex/com.android.art/lib64/libart.so (art::JValue art::InvokeVirtualOrInterfaceWithJValuesart::ArtMethod(art::ScopedObjectAccessAlreadyRunnable const&, _jobject, art::ArtMethod, jvalue const)+460) (BreakpadBuildId: c3d95a7d77dcec225dce96800a7a95940)

40 pc 00000000006786fc /apex/com.android.art/lib64/libart.so (art::thread::CreateCallback(void*)+1164) (BreakpadBuildId: c3d95a7d77dcec225dce96800a7a95940)

41 pc 00000000000dfdb0 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+132) (BreakpadBuildId: 06550e2bcf68e781af8072edf3aa70cf0)

42 pc 000000000008c3dc /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BreakpadBuildId: 06550e2bcf68e781af8072edf3aa70cf0)

electrum-bowie commented 9 months ago

Update: i am using Google's P3DT, i have 336870912 allocated bytes in the faulty scene, i optimized some scripts and updated some packages and the crash seems to occur more rarely than before, i also found that i had Incremental Garbage Collector TURNED OFF. which i turned on right after. i will update on this tommorow or the day after.

csciguy8 commented 9 months ago

Very likely related to https://github.com/CesiumGS/cesium-native/issues/739

kring commented 9 months ago

It's hard to say what's going on here. The issue @csciguy8 linked could indeed be the problem, though in that case you should see signs the device ran out of memory. It seems we're in CesiumForUnityNative::Cesium3DTilesetImpl::Update and then "something" goes wrong and there's a crash, but the call stack in between is not very useful so it's very hard to tell what happened. Even the 356 listed doesn't make much sense as a line number, so I'm assuming that's a byte offset in the built code and I don't have an easy way to resolve that to specific code. Can you try running a debug build on the device and see if it provides more useful information?

electrum-bowie commented 7 months ago

It's hard to say what's going on here. The issue @csciguy8 linked could indeed be the problem, though in that case you should see signs the device ran out of memory.

The device does not run out of memory during or before the crash, not even close actually, in fact the device's system software shows a notification that makes me sure the crash was because the headset was out of available memory, if that was the case. (which I previously ran into months ago)

electrum-bowie commented 7 months ago

Can you try running a debug build on the device and see if it provides more useful information?

Will do that when I get the time.

electrum-bowie commented 7 months ago

Best possible workaround for now 😁

https://github.com/CesiumGS/cesium-unity/issues/447

csciguy8 commented 7 months ago

One actionable thing we could do is put some exception handling (try catch) in Cesium3DTilesetImpl::Update, link.

It would give us a chance to catch any known types of exceptions and log. This might help distinguish out of memory problems vs other problems.

meta-meta commented 2 months ago

Are there any workarounds you can recommend? In Unity on the Quest, it's inevitable that spending enough time in the app will produce a crash due to running out of memory. We've attempted to hook into Unity's Application.lowMemory event and destroy/recreate the CesiumGeoreference and Cesium3DTileset. That seems to buy some time but in practice, Unity's event happens long before a crash would have happened so it becomes a UX nuisance and the app will still crash.

kring commented 1 month ago

@meta-meta the OP seemed to think the crash described in this issue was not related to running out of memory. In which case you're presumably running into a different problem. The first thing to try if you're running out of memory is to set the Maximum Cached Bytes property on the tileset to zero. If that doesn't help, please post to the community forum with as much detail as you can about your situation. In particular, what tilesets are you using?

meta-meta commented 1 month ago

Oh whoops, I missed that message. At any rate, between updating Unity to latest LTS and making efforts to reduce the amount of memory our app uses, we may have eliminated the crashes.

electrum-bowie commented 1 month ago

No longer originates from lack of memory, the app no longer runs out of memory, and isn't even close to doing so..

Stack trace:

`Thread 4043\

00 pc 000000000008b0a0 /apex/com.android.runtime/lib64/bionic/libc.so (abort+168) (BreakpadBuildId: 52a17d79c7474226ba96a3be5f0e981c0)\

01 pc 00000000022f4ed8 libil2cpp.so (libil2cpp.so (0x22f4ed8)) (BreakpadBuildId: 0cf42e90989c1d5200000000000000000)\

02 pc 00000000022f5028 libil2cpp.so (libil2cpp.so (0x22f5028)) (BreakpadBuildId: 0cf42e90989c1d5200000000000000000)\

03 pc 00000000022f4f90 libil2cpp.so (libil2cpp.so (0x22f4f90)) (BreakpadBuildId: 0cf42e90989c1d5200000000000000000)\

04 pc 00000000022f4f14 libil2cpp.so (std::terminate()+36) (BreakpadBuildId: 0cf42e90989c1d5200000000000000000)\

05 pc 00000000022f5138 libil2cpp.so (libil2cpp.so (0x22f5138)) (BreakpadBuildId: 0cf42e90989c1d5200000000000000000)\

06 pc 0000000000a62cc4 /data/app/~~ZP_082MwIodNHODKs5sHUA==/com.bowie.earthquest-zXNvxKN0uEIqq_lA6BzX9g==/lib/arm64/libCesiumForUnityNative-Runtime.so (std::__terminate(void (*)())+12)\

07 pc 0000000000a63504 /data/app/~~ZP_082MwIodNHODKs5sHUA==/com.bowie.earthquest-zXNvxKN0uEIqq_lA6BzX9g==/lib/arm64/libCesiumForUnityNative-Runtime.so (__cxxabiv1::call_terminate(bool, _Unwind_Exception*)+36)\

08 pc 0000000000a634a8 /data/app/~~ZP_082MwIodNHODKs5sHUA==/com.bowie.earthquest-zXNvxKN0uEIqq_lA6BzX9g==/lib/arm64/libCesiumForUnityNative-Runtime.so (cxxabiv1::scan_eh_tab(cxxabiv1::(anonymous namespace)::scan_results&, _Unwind_Action, bool, _Unwind_Exception, _Unwind_Context)+1508)\

09 pc 0000000000a62d8c /data/app/~~ZP_082MwIodNHODKs5sHUA==/com.bowie.earthquest-zXNvxKN0uEIqq_lA6BzX9g==/lib/arm64/libCesiumForUnityNative-Runtime.so (__gxx_personality_v0+104)\

10 pc 000000000230c3c8 libil2cpp.so () (BreakpadBuildId: 0cf42e90989c1d5200000000000000000)\

11 pc 00000000022f4480 libil2cpp.so (__cxa_throw+116) (BreakpadBuildId: 0cf42e90989c1d5200000000000000000)\

12 pc 0000000002273424 libil2cpp.so (<__cxxabiv1::scan_eh_tab>) (BreakpadBuildId: 0cf42e90989c1d5200000000000000000)\

13 pc 00000000022734b8 libil2cpp.so (<__gxx_personality_v0>) (BreakpadBuildId: 0cf42e90989c1d5200000000000000000)\

14 pc 00000000022734a8 libil2cpp.so (libil2cpp.so (0x230c3c8)) (BreakpadBuildId: 0cf42e90989c1d5200000000000000000)\

15 pc 000000000222b92c libil2cpp.so (libil2cpp.so (0x22f4480)) (BreakpadBuildId: 0cf42e90989c1d5200000000000000000)\

16 pc 00000000024d9098 libil2cpp.so (libil2cpp.so (0x2273424)) (BreakpadBuildId: 0cf42e90989c1d5200000000000000000)\

17 pc 0000000001efbfb0 libil2cpp.so (libil2cpp.so (0x22734b8)) (BreakpadBuildId: 0cf42e90989c1d5200000000000000000)\

18 pc 0000000000448504 /data/app/~~ZP_082MwIodNHODKs5sHUA==/com.bowie.earthquest-zXNvxKN0uEIqq_lA6BzX9g==/lib/arm64/libCesiumForUnityNative-Runtime.so (CesiumForUnityNative::Cesium3DTilesetImpl::Update(DotNet::CesiumForUnity::Cesium3DTileset const&)+356)\

19 pc 0000000000420fec /data/app/~~ZP_082MwIodNHODKs5sHUA==/com.bowie.earthquest-zXNvxKN0uEIqq_lA6BzX9g==/lib/arm64/libCesiumForUnityNative-Runtime.so (DotNet_CesiumForUnity_Cesium3DTileset_Update+60)\

20 pc 000000000254bcd4 libil2cpp.so (libil2cpp.so (0x22734a8)) (BreakpadBuildId: 0cf42e90989c1d5200000000000000000)\

21 pc 0000000002289b3c libil2cpp.so (libil2cpp.so (0x222b92c)) (BreakpadBuildId: 0cf42e90989c1d5200000000000000000)\

22 pc 0000000002289a88 libil2cpp.so (libil2cpp.so (0x24d9098)) (BreakpadBuildId: 0cf42e90989c1d5200000000000000000)\

23 pc 0000000000659370 libunity.so (libil2cpp.so (0x1efbfb0)) (BreakpadBuildId: d469992fbbf8fdff00000000000000000)\

24 pc 0000000000668cb8 libunity.so () (BreakpadBuildId: d469992fbbf8fdff00000000000000000)\

25 pc 0000000000677090 libunity.so () (BreakpadBuildId: d469992fbbf8fdff00000000000000000)\

26 pc 00000000004966c8 libunity.so (libil2cpp.so (0x254bcd4)) (BreakpadBuildId: d469992fbbf8fdff00000000000000000)\

27 pc 000000000055dcd4 libunity.so (libil2cpp.so (0x2289b3c)) (BreakpadBuildId: d469992fbbf8fdff00000000000000000)\

28 pc 000000000055dd14 libunity.so (libil2cpp.so (0x2289a88)) (BreakpadBuildId: d469992fbbf8fdff00000000000000000)\

29 pc 000000000055dfa0 libunity.so (libunity.so (0x659370)) (BreakpadBuildId: d469992fbbf8fdff00000000000000000)\

30 pc 00000000006db2d0 libunity.so (libunity.so (0x668cb8)) (BreakpadBuildId: d469992fbbf8fdff00000000000000000)\

31 pc 00000000006f779c libunity.so (libunity.so (0x677090)) (BreakpadBuildId: d469992fbbf8fdff00000000000000000)\

32 pc 00000000020049ec /memfd:jit-cache (deleted) (art_jni_trampoline+92)\

33 pc 000000000201009c /memfd:jit-cache (deleted) (com.unity3d.player.L.handleMessage+1452)\

34 pc 00000000000c84c0 /system/framework/arm64/boot-framework.oat (android.os.Handler.dispatchMessage+144) (BreakpadBuildId: 67dd2692f9e23c670701afb1d5f3b8ba0)\

35 pc 00000000000c96e0 /system/framework/arm64/boot-framework.oat (android.os.Looper.loopOnce+1424) (BreakpadBuildId: 67dd2692f9e23c670701afb1d5f3b8ba0)\

36 pc 00000000000c90b4 /system/framework/arm64/boot-framework.oat (android.os.Looper.loop+516) (BreakpadBuildId: 67dd2692f9e23c670701afb1d5f3b8ba0)\

37 pc 0000000000211608 /apex/com.android.art/lib64/libart.so (nterp_helper+152) (BreakpadBuildId: 3c78912defdf4b7a049c6097c6798e120)\

38 pc 000000000000a898 /data/app/~~ZP_082MwIodNHODKs5sHUA==/com.bowie.earthquest-zXNvxKN0uEIqq_lA6BzX9g==/base.apk (com.unity3d.player.O.run+48)\

39 pc 0000000000218964 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+548) (BreakpadBuildId: 3c78912defdf4b7a049c6097c6798e120)\

40 pc 000000000028495c /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread, unsigned int, unsigned int, art::JValue, char const)+188) (BreakpadBuildId: 3c78912defdf4b7a049c6097c6798e120)\

41 pc 0000000000628a7c /apex/com.android.art/lib64/libart.so (art::JValue art::InvokeVirtualOrInterfaceWithJValues<art::ArtMethod>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject, art::ArtMethod, jvalue const)+460) (BreakpadBuildId: 3c78912defdf4b7a049c6097c6798e120)\

42 pc 00000000006786fc /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+1164) (BreakpadBuildId: 3c78912defdf4b7a049c6097c6798e120)\

43 pc 00000000000e0150 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+132) (BreakpadBuildId: 52a17d79c7474226ba96a3be5f0e981c0)\

44 pc 000000000008c6dc /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BreakpadBuildId: 52a17d79c7474226ba96a3be5f0e981c0)

2024Meta`

IMAGE WITH MORE INSIGHT

electrum-bowie commented 1 month ago

v1.12.0 Cesium for Unity

electrum-bowie commented 1 month ago

Unity 2023.1.0f1

kring commented 1 month ago

Thanks for the call stack, @electrum-bowie, but I don't think it adds any new insights for us on this one. Cesium3DTilesetImpl::Update is apparently throwing an exception (or perhaps causing an access violation?), but we don't know why, or even on what line. Some questions to help narrow it down a bit:

  1. Can you reproduce this consistently? Or does it only happen randomly on end-user devices?
  2. Can you reproduce it with the Cesium for Unity Samples project, or only in your own project? (I realize if the answer to the above is no, then this is likely to be no as well)
  3. Can you reproduce it on any other devices, or only on Quest 2/3?
electrum-bowie commented 1 month ago

I believe it happens randomly, maybe cesium receives a rare 3D tile with broken parameters or weird properties and it doesn't know how to handle it ? Just troubleshooting..

Q2 and Q3, yes.

Based on an older test, yes, it should happen on the sample scenes too, it just happens very rarely.

electrum-bowie commented 1 month ago

But it seemingly never occurs in play sessions under like, say, 20 minutes, so it's really weird.

And again, not to be confused with the Out of Memory crash, that is not happening here.