Crashlytics flaky de-obfuscation

helloncode commented 1 year ago

Environment

Android Studio version: Android Studio Dolphin 2021.3.1
Firebase Component: Crashlytics
Component version:
- Firebase-bom version: 30.5.0
- Crashlytics Gradle plugin version: 2.9.2
- Firebase app distribution: 3.0.3

Describe the problem

Half of our crashes on Firebase Crashlytics are obfuscated.

Screenshot Samples:

![Screenshot 2022-10-05 at 16 11 13](https://user-images.githubusercontent.com/7222847/194085466-98929f82-f2ac-4ffc-9557-e49bc65fe9c3.png) ![Sample 2](https://user-images.githubusercontent.com/7222847/194082691-71d84f02-e23d-4a3e-b83c-322f8d7b6777.png)

We've been afflicted by the problem since November/December 2021.

Steps and actions made:

Updated Firebase dependencies (Firebase Bom, Crashlytics Gradle, Firebase App Distribution]
Checked mapping.txt (works correctly, we used it to de-obfuscated the crashes manually)
Checked proguard file (contains the options requested on Crashlytics doc)
Checked Google play console, all crashes are de-obfuscated
We run locally and on CI > ./gradlew app:assembleRelease --debug | grep "\[com.google.firebase.crashlytics\]"
Result:

Logs

The problem looks similar to #3392 that was closed for inactivity.

Please let me know what other information you may need. Thank you!

argzdev commented 1 year ago

Thanks for reporting, @francescobonni. Just to ask some details that might be needed which was also asked from #3392:

Could you provide the debugging output from your Gradle build?

If you are not comfortable sharing your Gradle output or more details about your build config publicly, please feel free to email Firebase support and reference the #3392 issue.

andbattaglia commented 1 year ago

@argzdev This is our debugging output. I runned the command both in ci and locally with the same result.

./gradlew app:assembleRelease --debug | grep "\[com.google.firebase.crashlytics\]"                                                            ✔  at 07:11:50  
 [com.google.firebase.crashlytics] Configuring com.google.firebase.crashlytics for project ':app'
 [com.google.firebase.crashlytics] Registering Crashlytics tasks for project ':app', variant DevDebug: [mappingFile : null] [mappingFileUploadEnabled : false] [nativeSymbolUploadEnabled : null] [unstrippedNativeLibsDir : null] [strippedNativeLibsDir : null] [symbolGeneratorExtension : com.google.firebase.crashlytics.buildtools.gradle.CrashlyticsExtension$SymbolGenerator@2ef5ca24] 
 [com.google.firebase.crashlytics] Mapping File Upload Enabled: false; id: 00000000000000000000000000000000
 [com.google.firebase.crashlytics] Crashlytics native symbol uploading enabled: false
 [com.google.firebase.crashlytics] Registering Crashlytics tasks for project ':app', variant ProdDebug: [mappingFile : null] [mappingFileUploadEnabled : null] [nativeSymbolUploadEnabled : null] [unstrippedNativeLibsDir : null] [strippedNativeLibsDir : null] [symbolGeneratorExtension : com.google.firebase.crashlytics.buildtools.gradle.CrashlyticsExtension$SymbolGenerator@5401c280] 
 [com.google.firebase.crashlytics] Instantiating ProjectVariantState for project ':app', variant: prodDebug
 [com.google.firebase.crashlytics] Mapping File Upload Enabled: false; id: 00000000000000000000000000000000
 [com.google.firebase.crashlytics] Crashlytics native symbol uploading enabled: false
 [com.google.firebase.crashlytics] Registering Crashlytics tasks for project ':app', variant DevRelease: [mappingFile : null] [mappingFileUploadEnabled : false] [nativeSymbolUploadEnabled : null] [unstrippedNativeLibsDir : null] [strippedNativeLibsDir : null] [symbolGeneratorExtension : com.google.firebase.crashlytics.buildtools.gradle.CrashlyticsExtension$SymbolGenerator@2ef5ca24] 
 [com.google.firebase.crashlytics] Mapping File Upload Enabled: false; id: 00000000000000000000000000000000
 [com.google.firebase.crashlytics] Crashlytics native symbol uploading enabled: false
 [com.google.firebase.crashlytics] Registering Crashlytics tasks for project ':app', variant ProdRelease: [mappingFile : null] [mappingFileUploadEnabled : null] [nativeSymbolUploadEnabled : null] [unstrippedNativeLibsDir : null] [strippedNativeLibsDir : null] [symbolGeneratorExtension : com.google.firebase.crashlytics.buildtools.gradle.CrashlyticsExtension$SymbolGenerator@5401c280] 
 [com.google.firebase.crashlytics] Instantiating ProjectVariantState for project ':app', variant: prodRelease
 [com.google.firebase.crashlytics] Mapping File Upload Enabled: true; id: a2f2b26a12534bd68e763a63203fa0b1]
 [com.google.firebase.crashlytics] Crashlytics native symbol uploading enabled: false
 [com.google.firebase.crashlytics] Using File for UP-TO-DATE check: app/build/generated/crashlytics/res/dev/release/values/com_google_firebase_crashlytics_mappingfileid.xml
 [com.google.firebase.crashlytics] UP-TO-DATE for injectCrashlyticsMappingFileIdDevRelease with 00000000000000000000000000000000: false
 [com.google.firebase.crashlytics] Injecting mappingFileId into file [mappingFileId: 00000000000000000000000000000000; file:  app/build/generated/crashlytics/res/dev/release/values/com_google_firebase_crashlytics_mappingfileid.xml]
 [com.google.firebase.crashlytics] Using File for UP-TO-DATE check: app/build/generated/crashlytics/res/prod/release/values/com_google_firebase_crashlytics_mappingfileid.xml
 [com.google.firebase.crashlytics] UP-TO-DATE for injectCrashlyticsMappingFileIdProdRelease with a2f2b26a12534bd68e763a63203fa0b1: false
 [com.google.firebase.crashlytics] Injecting mappingFileId into file [mappingFileId: a2f2b26a12534bd68e763a63203fa0b1; file:  app/build/generated/crashlytics/res/prod/release/values/com_google_firebase_crashlytics_mappingfileid.xml]
 [com.google.firebase.crashlytics] Looking for Google appId in: app/build/generated/res/google-services/prod/release/values/values.xml
 [com.google.firebase.crashlytics] Found Google appId: 1:566093163024:android:316a93b7ad683ae3
 [com.google.firebase.crashlytics] Uploading Mapping File [mappingFile: app/build/outputs/mapping/prodRelease/mapping.txt; mappingFileId: a2f2b26a12534bd68e763a63203fa0b1;packageName: **.******; googleAppId: *****************]
 [com.google.firebase.crashlytics] Zipping mapping file: app/build/outputs/mapping/prodRelease/mapping.txt -> app/build/crashlytics/prod/release/.crashlytics-mappings-tmp/a2f2b26a12534bd68e763a63203fa0b1.gz
 [com.google.firebase.crashlytics] PUT file: app/build/crashlytics/prod/release/.crashlytics-mappings-tmp/a2f2b26a12534bd68e763a63203fa0b1.gz to URL: https://firebasecrashlyticssymbols.googleapis.com/v1/project/-/app/**********:android:************/upload/java/****************
 [com.google.firebase.crashlytics] PUT headers:
 [com.google.firebase.crashlytics]  User-Agent = crashlytics-gradle/2.9.2
 [com.google.firebase.crashlytics]  X-CRASHLYTICS-API-CLIENT-TYPE = crashlytics-gradle
 [com.google.firebase.crashlytics]  X-CRASHLYTICS-API-CLIENT-VERSION = 2.9.2
 [com.google.firebase.crashlytics] PUT response: [reqId=null] 200
[com.google.firebase.crashlytics] Mapping file uploaded: app/build/outputs/mapping/prodRelease/mapping.txt

google-oss-bot commented 1 year ago

Hey @francescobonni. We need more information to resolve this issue but there hasn't been an update in 5 weekdays. I'm marking the issue as stale and if there are no new updates in the next 5 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

helloncode commented 1 year ago

Hey @argzdev, I shared under result the debugging result of gradle on my original post but @battagliandrea (my colleague) shared it again in a better way.

In the meanwhile we triggered the CI with Gradle cache remote node disabled and purged, nothing changed. Now, we are trying to trigger without any type of cache, to be sure that the problem isn't cache related

argzdev commented 1 year ago

Thanks for the extra details, @francescobonni & @battagliandrea. This is odd since you're able to see the de-obfuscated logs in Google Play Console. I think this might be an issue on the project mapping or incorrect config file, but I'm not sure since this requires taking a look in your project details with the mapping file. That said, since this requires sharing PII it would be better to reach out to our Firebase support and reference them this issue, they'll be able to assist you better since they have more visibility on your project details.

I'll close this issue for now. Feel free to comment back here if you feel this is a mistake or there's additional information that you think might help with the investigation. Thanks!

helloncode commented 1 year ago

@argzdev, I'd like to let you know that we've found a solution to our problem. Because this appears to be a common issue, I'd like to share our solution: The command "./gradlew app:assembleRelease —debug | grep "[com.google.firebase.crashlytics]" provided us an incorrect hint; in fact, the upload mapping for Crashlytics was called on CI for the assemble command, but the resulting APK was only used for Firebase App Distribution (where, when we forced a crash there, it was de-obfuscated).

We used the publishVariantName command (in the same pipeline) to publish on store, and the upload mapping was not called. There is an open bug [here]. (https://github.com/Triple-T/gradle-play-publisher/issues/1062), and we used the suggested workaround [here]. (https://github.com/Triple-T/gradle-play-publisher/issues/859#issuecomment-700658128)

firebase / firebase-android-sdk