Calinou
commented
4 years ago I'm afraid a backtrace without debugging symbols isn't of much use, especially if the user who got the crash isn't around to reply to our questions.
Closed JezSonic closed 3 years ago
Calinou
commented
4 years ago I'm afraid a backtrace without debugging symbols isn't of much use, especially if the user who got the crash isn't around to reply to our questions.
mrimvo
commented
4 years ago I get quite a lot of remote crash reports (Crashlyics) from my android game, that look exactly like this (crash in libgodot_android.so), and to a lesser extend crashes in other .so binaries too. Please, how can I attach debug symbols to these reports? I would like to help you guys fix these problems.
In the crashlytics docu they say I need to execute
./gradlew crashlyticsUploadSymbolsRelease
but in a godot project I have no idea where/how to execute that?
Please, could you guide us in a general direction?
This problem is rather urgend, because crashes like this produce 1-star ratings in google play for games that would have great ratings otherwise. Quite a big bummer!
Most crash reports I get look like this:
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> com.mvolution.cuarentena1 <<<
backtrace:
#00 pc 00000000006fe9bc /data/app/com.mvolution.cuarentena1-xJ_EtKnP9LEu6rrNMiJllg==/lib/arm/libgodot_android.so
#01 pc 000000000127327c /data/app/com.mvolution.cuarentena1-xJ_EtKnP9LEu6rrNMiJllg==/lib/arm/libgodot_android.so
#02 pc 0000000000076278 /data/app/com.mvolution.cuarentena1-xJ_EtKnP9LEu6rrNMiJllg==/lib/arm/libgodot_android.so
#03 pc 0000000000009137 /data/app/com.mvolution.cuarentena1-xJ_EtKnP9LEu6rrNMiJllg==/oat/arm/base.odex (offset 0x9000)Some other .so libraries report crashes too, but not as many.
My users complain about the app crashing on startup - others about a black screen, sound is starting but nothing more.
I have Crashlytics up and running and would like to upload debugging symbols to the Crashlytics backend to provide you guys with meaningful backtraces from various devices out there in the wild. I'm kind of lost on how to do that. Any ideas?
Calinou
commented
4 years ago cc @m4gr3d
jeremyz
commented
4 years ago @mrimvo you might have to recompile godot with modified ./platform/android/java/lib/build.gradle and ./platform/android/java/app/build.gradle ? My game suffers the same, I want it to be fixed ASAP. How did you setup crashlytics, is it just a matter of registering your project ?
mrimvo
commented
4 years ago Yes, it's quite easy to set up, but as the crash reports are not meaningful, this is a bit useless. Recompiling might help I didn't try that. Another thing that just crossed my mind: maybe it's possible to add godot's logs (where all the push_error messages end up) to a crash report. Because I think some of these errors might be rooted id GDScript. The engine shouldn't crash, but at least we can do something about it if we had the logs.
Edit: I did a quick research and I think the idea to attach the godot logs might not be possible with Crashlytics:
the CrashlyticsListener doesn't support being called from when an NDK/JNI exception happens
Another option might by to use Crashlytics.log as soon as there are messages added to godot's terminal output. However i found no way how to get notified about terminal output in godot.
akien-mga
commented
4 years ago To have a full debug stacktrace when it crashes, you should build custom export templates with target=debug instead of target=release_debug (what "Debug" templates use by default).
They will be significantly bigger but will include full debug symbols so that the crash in libgodot_android.so can give us a detailed stacktrace.
Also to be sure, please verify that you're are building the export templates properly from the same commit and local modifications as the editor. If you're using the "Custom Build" option, you should also reinstall the custom compiled android_source.zip in place of your pre-existing res://android/build/ which might have been installed with the official 3.2.1-stable (which is thus not compatible with your custom changes).
Edit: I might have misinterpreted that you have custom changes made to the template. If you're using the stock Android template from the Godot 3.2.1 templates package, then you just need to rebuild a custom debug templates from the 3.2.1-stable git tag, with:
scons p=android tools=no target=debug android_arch=arm64v8
scons p=android tools=no target=debug android_arch=armv7
cd platform/android/java
./gradlew generateGodotTemplatesAnd use the generated bin/android_debug.apk as debug template here:
And export with "Debug" on of course so that this debug template is used.
mrimvo
commented
4 years ago Thank you for your answer :)
I'm using a custom android build template. I'm happy to push a debug build into the beta channel of my app to help godot get some helpful stack traces.
I'm a bit confused about where to add target=debug to build a debug template from my custom template. Could you elaborate on that?
Just to be clear about my context:
akien-mga
commented
4 years ago See the last part of my previous comment, you'd need to build the Godot export template with debug symbols (see Compiling for Android).
Then it's a bit tricky as the new workflow with custom build templates doesn't allow to easily use one's own android_source.zip to set things up (until #36728 is fixed).
Since you're using 3.2.1-stable as is, the best might be that once you have built the export template using the above instructions with target=debug, you can copy bin/godot-lib.debug.aar in place of your installed res://android/build/libs/debug/godot-lib.debug.aar. I haven't tested but that should add the necessary debug symbols while still allowing you to use the template source installed from 3.2.1-stable.
mrimvo
commented
4 years ago I tried to follow that path, but I'm sad to say I need to give up on that. The NDK Download is just too big for my limited data volume to handle.
akien-mga
commented
4 years ago I'll make a debug build myself and upload it here, that should be much easier to test.
akien-mga
commented
4 years ago Here's a target=debug build of 3.2.1-stable for Android (arm64v8 and armv7): https://send.firefox.com/download/ce4ac3341d7a2601/#XRIXZIyiWUh2oZkBpYSqAg (available for 7 days)
How to use it:
android_debug.apk as "Custom Template/Debug" in your Android preset, and export a Debug buildres://android, put the downloaded android_source.zip in your Godot templates/3.2.1.stable folder, use "Install Android Build Template..." from the editor and redo your local changes, orres://android, unzip android_source.zip somewhat and copy its android_source/libs/debug/godot-lib.debug.aar to res://android/build/libs/debug/ to replace the existing one. (Recommended as you don't need to mess with your installed Godot templates and redo your Android customization, but I'm only 95% confident that doing this is sufficient - should be good enough though :))mrimvo
commented
4 years ago This is awesome! I'm going to upload the debug version today! :+1:
If you don't use "Custom Template/Use Custom Build", then select the android_debug.apk as "Custom Template/Debug" in your Android preset, and export a Debug build
This worked for me to build and run the game with your template apk.
If you use "Custom Template/Use Custom Build", either:
I tried both ways, and even did not re-apply my changes, so I had the pure version as Custom Template set. It produces errors while building a debug version of the game:
I then removed the references to these resources from the AndroidManifest.xml and got another error while building:
Also, I noticed the godot-lib.debug.aar is
jeremyz
commented
4 years ago size is smaller because it contains arm64v8 and armv7, no x86 and x86_64. @mrimvo maybe run ./gradlew clean from within ./android/build ?
jeremyz
commented
4 years ago I can build and run using _customtemplate/debug and the given android_debug.apk or using _custom_template/use_custombuild and replacing godot-lib.debug.aar with the one from the given android_source.zip
to enable crashlytics I edit build.graddle :
classpath 'com.google.gms:google-services:4.3.3'classpath 'com.google.firebase:firebase-crashlytics-gradle:2.1.0'apply plugin: 'com.google.firebase.crashlytics'implementation 'com.google.firebase:firebase-crashlytics:17.0.0'but I must comment out implementation libraries.supportCoreUtils
because of a Manifest merger failed between com.android.support:support-compat:28.0.0 from AndroidManifest.xml and androidx.core:core:1.0.0 androidx.core.app.CoreComponentFactory.
I'm not sure if this will work, it seams like.
How can I publish one of these debug apk, google's policy does not allow debuggable applications ?
mrimvo
commented
4 years ago I just got notified by google that my app got suspended. Oh man that hurts. Their reasoning being it got covid-19 references. Well it does but I think their reaction is a bit harsh, provided it was nothing more than a cute little game.
So I'm out of this. About publishing the godot-lib.debug.aar I think it might be possible to mix and have the exported APK be a release version that uses the debug version of godot-lib. Another (better) possiblity would be to use godot-lib.release.aar and to upload the debug information to Crashlytics backend.
About enabling Crashlytics: yes, that's exactly the way I did it too, including to remove the support-compat. Also you need to download and add google-services.json from Crashlytics backend and add it to android/build/ directory.
jeremyz
commented
4 years ago Hoooo sorry, that's harsh ! I already tried to publish a release apk built with the android_debug.apk, it has be detected and refused. I'll complete crashlytics setup and upload the debug symbols to crashlytics. Yes, I forgot to mention google-services.json ...
jeremyz
commented
4 years ago That was hard ! Crashlytics does not work for me ... *.so libs are stripped in the apk, but I finally learned about ndk-stack tool. give him a dir where unstripped libraries are and you're good. So has I thought, problems comes from JSON serialisation ...
GDScript code is : fout.store_line(JSON.print(dict))
bt is :
godot-3.2.1-stable/core/object.cpp:941:33 godot-3.2.1-stable/core/variant.cpp:1592:28 godot-3.2.1-stable/core/variant.cpp:1408:9 godot-3.2.1-stable/core/io/json.cpp:117:33 godot-3.2.1-stable/core/io/json.cpp:111:10 godot-3.2.1-stable/core/io/json.cpp:111:10 godot-3.2.1-stable/core/io/json.cpp:111:10 godot-3.2.1-stable/core/io/json.cpp:123:9 godot-3.2.1-stable/core/bind/core_bind.cpp:3210:9 godot-3.2.1-stable/./core/method_bind.gen.inc:2505:17 godot-3.2.1-stable/core/object.cpp:921:17 ` full_bt
akien-mga
commented
4 years ago Here's the start of @jeremyz's backtrace in the 3.2.1-stable codebase:
https://github.com/godotengine/godot/blob/3.2.1-stable/core/object.cpp#L941
https://github.com/godotengine/godot/blob/3.2.1-stable/core/variant.cpp#L1592
https://github.com/godotengine/godot/blob/3.2.1-stable/core/variant.cpp#L1408
https://github.com/godotengine/godot/blob/3.2.1-stable/core/io/json.cpp#L117
https://github.com/godotengine/godot/blob/3.2.1-stable/core/io/json.cpp#L111
So it sounds like the dict you're serializing containers Objects with scripts (I assume Nodes?), but then trying to call to_string() from their script crashes (probably null pointer dereference?).
That sounds like it might be related to #38119 (CC @RandomShaper), which is fixed in 3.2.2 beta 2 and beta 3. Would you be able to test if 3.2.2 beta 3 fixes the issue for you?
RandomShaper
commented
4 years ago Yes, sounds a lot like the object, along with its script, has been freed.
Since the dangling Variant fix doesn't apply to release builds, it won't help you unless you find what's wrong in your scripts.
Running in the editor with the debugger attached, you'll get those freed objects converted to the "[Deleted Object]" string.
As a quick measure to avoid the crash and maybe gain some time for finding the bug before you get more crash reports, you can use a release_debug export template. With it, you won't get a crash neither the "[Deleted Object]" string, but "[Object:null]".
jeremyz
commented
4 years ago I've noticed an object that should not be here in the dict that I'm serializing. reading your comments I'm now almost sure that the crash happens when I save the turn state having just destroyed that object ... will look deeper into it in a few hours ...
@RandomShaper I haven't play with the debugger yet, cause it only crashed on android 10 so far ... will have a deeper look it this too. @akien-mga I'll test 3.2.2 behaviour ... (can you please triage my 2 EventGesture OS agnostic PR, so that I know where it's going ;)
mrimvo
commented
4 years ago Well done @jeremyz I'm glad you got the backtrace working. 🙂
Besides fixing the game itself, I would like to suggest that godot should not crash in these cases but instead pushes errors like push_error does.
RandomShaper
commented
4 years ago There's a plan about having the option to have those checks in release builds too and even add further protections.
I haven't had time to write the proposal yet.
jeremyz
commented
4 years ago I've tested the current situation, jasonify a dictionary with a dangling reference.
var d = {'stuff':'None'} var a = Node2D.new() d['dangling'] = a print(JSON.print(d)) a.queue_free() yield(get_tree().create_timer(1.0), "timeout") print(JSON.print(d))
with 3.2.1-stable
with 3.2 HEAD(6abe73f5b8)
@akien-mga so, as stated in #38119's updated comment, crashes are not handled in release builds yet.
avnerh1
commented
3 years ago Here's a
target=debugbuild of 3.2.1-stable for Android (arm64v8andarmv7): https://send.firefox.com/download/ce4ac3341d7a2601/#XRIXZIyiWUh2oZkBpYSqAg (available for 7 days)How to use it:
- If you don't use "Custom Template/Use Custom Build", then select the
android_debug.apkas "Custom Template/Debug" in your Android preset, and export a Debug buildIf you use "Custom Template/Use Custom Build", either:
- Delete
res://android, put the downloadedandroid_source.zipin your Godottemplates/3.2.1.stablefolder, use "Install Android Build Template..." from the editor and redo your local changes, or- Keep your
res://android, unzipandroid_source.zipsomewhat and copy itsandroid_source/libs/debug/godot-lib.debug.aartores://android/build/libs/debug/to replace the existing one. (Recommended as you don't need to mess with your installed Godot templates and redo your Android customization, but I'm only 95% confident that doing this is sufficient - should be good enough though :))
@akien-mga Do you happen to also have a debug build for 3.3.2?
akien-mga
commented
3 years ago Not readily available, but I'm not sure this old issue is still relevant today, and there's no reproduction project to check against recent versions.
If you reproduce a similar issue, I would suggest opening a new bug report with details on your specific situation, as it might differ from the original one in this issue.
Calinou
commented
3 years ago I built a debug APK of Godot 3.3.2 (target=debug, armv7 + arm64v8, unstripped): https://0x0.st/-Wff.zip (link expires in late 2021)
I'm not sure if it worked correctly as the APK is only 26 MB though. I remember having to use a different Gradle command like generateDebugTemplates, but it doesn't exist on 3.3.2 at least (and not in master either, from a quick search).
avnerh1
commented
3 years ago Not readily available, but I'm not sure this old issue is still relevant today, and there's no reproduction project to check against recent versions.
If you reproduce a similar issue, I would suggest opening a new bug report with details on your specific situation, as it might differ from the original one in this issue.
I see. I need it for general debugging of crash logs in Play console, not for any one specific case.
avnerh1
commented
3 years ago I built a debug APK of Godot 3.3.2 (
target=debug, armv7 + arm64v8, unstripped): https://0x0.st/-Wff.zip (link expires in late 2021)I'm not sure if it worked correctly as the APK is only 26 MB though. I remember having to use a different Gradle command like
generateDebugTemplates, but it doesn't exist on 3.3.2 at least (and not inmastereither, from a quick search).
Thanks, but unfortunately according to Android Studio those libs don't contain any debug symbols.
avnerh1
commented
3 years ago I managed to compile and create debug symbols for Godot 3.3.2. However, it doesn't seem like it helps a lot to understand the stack traces. Here's an example:
Miskler
suzanshakya
3.2.1.stable
ANDROID 10/Google Pixel 3 XL:
Issue description: Google Play Console found my game crash on Android 10/Google Pixel 3 Xl. Here is the backtrace. I don't know what is the source of that issue cause it didn't happen on any of my devices. Anyway i am creating an issue here.