Mapping file is too large error

[0xnm]

Originally posted by @epool in https://github.com/DataDog/dd-sdk-android-gradle-plugin/issues/66#issuecomment-2162234109

hey @0xnm 👋🏼 first of all thanks for all your support on this issue, but I'm still facing this issue with my mapping.txt file of 133.4 MB:

* What went wrong:
Execution failed for task ':app:uploadMappingRelease'.
> Unable to upload mapping file for `service:...`, `version:...`, `version_code:...`, `variant:`, `build_id:...` because mapping file is too large; please refer to documentation regarding the limits

with this setup

datadog_plugin_version = "1.14.0"
datadog = { id = "com.datadoghq.dd-sdk-android-gradle-plugin", version.ref = "datadog_plugin_version" }

plugins {
    alias(libs.plugins.datadog)
}

datadog {
    site = "US1"
}

it seems like it's falling in this condition

https://github.com/DataDog/dd-sdk-android-gradle-plugin/blob/fe2c9b6120902d6683eb7c7aedf99b8a0a08fa6a/dd-sdk-android-gradle-plugin/src/main/kotlin/com/datadog/gradle/plugin/internal/OkHttpUploader.kt#L183L186

could you please give me a hand with this one? 🙏🏼

[0xnm]

Hi @epool! I moved your question to a new ticket to avoid new comments on the closed ticket.

133.4 MB should be way below the file size limit our intake can handle. In order to help you, can you please give us more details:

• does this error happen all the time for that file? did you try to trigger upload again?
• how do you calculate file size? is it maybe transformed somehow after being generated?

[epool]

Hi @epool! I moved your question to a new ticket to avoid new comments on the closed ticket.

133.4 MB should be way below the file size limit our intake can handle. In order to help you, can you please give us more details:

• does this error happen all the time for that file? did you try to trigger upload again?
• how do you calculate file size? is it maybe transformed somehow after being generated?

@0xnm

• yeah it happens every time I try to upload the mapping.txt file with the Gradle task
• well, I deleted the mapping.txt file, ran the DD upload mapping file Gradle task, it generated the mapping.txt file back again and I saw the file size from the MacOS finder file explorer

[0xnm]

Thanks for bringing it. We had an issue with our intake and now it is resolved. Can you please retry the upload?

[epool]

it worked now, thank you so much for your quick support! 🙏🏼

[epool]

@0xnm QQ: what's the reason of this? why deobfuscation doesn't work on errors sent through Logs and only on the ones sent through RUM?

[0xnm]

@epool That is an old statement which is not relevant anymore - errors in Logs are deobfuscated as well now. I will open a PR to remove this statement.

[epool]

@epool That is an old statement which is not relevant anymore - errors in Logs are deobfuscated as well now. I will open a PR to remove this statement.

that's good to know! 🙌🏼 do you know why my stacktrace in the Logs are still not deobfuscated? this is how it looked before uploading the mapping.txt file

and this is how it looks after uploading the mapping.txt file

Note that the deobfuscation failure message is no longer present

but if I run retrace -verbose mapping.txt obfuscated.txt locally I get something like this

java.lang.Throwable: Exception received while running the Executor
        at q5.b$b.invokeSuspend(SourceFile:79)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.void resumeWith(java.lang.Object)(ContinuationImpl.kt:33)
        at F40.b1.g1(SourceFile:60)
        at kotlin.reflect.jvm.internal.impl.util.capitalizeDecapitalize.CapitalizeDecapitalizeKt.resumeWith(capitalizeDecapitalize.kt:16)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.void resumeWith(java.lang.Object)(ContinuationImpl.kt:46)
        at F40.X.run(SourceFile:115)
        at kotlinx.coroutines.flow.AbstractFlow.s(Flow.kt:1)
        at N40.a$c.d(SourceFile:15)
        at N40.a$c.p(SourceFile:29)
        at N40.a$c.run(SourceFile:1)
Caused by: NoInternetError(cause=java.net.SocketTimeoutException: timeout)
        at com.bitso.android.analytics.properties.VariantProperty.a(VariantProperty.kt:12)
        at com.bitso.android.analytics.properties.PlacementProperty.onFailure(PlacementProperty.kt:15)
        at retrofit2.OkHttpCall$1.void callFailure(java.lang.Throwable)(OkHttpCall.java:178)
        at retrofit2.OkHttpCall$1.void onFailure(okhttp3.Call,java.io.IOException)(OkHttpCall.java:173)
        at com.google.firebase.perf.network.InstrumentOkHttpEnqueueCallback.void onFailure(okhttp3.Call,java.io.IOException)(InstrumentOkHttpEnqueueCallback.java:63)
        at okhttp3.internal.connection.RealCall$AsyncCall.void run()(RealCall.kt:525)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:644)
        at java.lang.Thread.run(Thread.java:1012)
Caused by: java.net.SocketTimeoutException: timeout
        at okhttp3.internal.http2.Http2Stream$StreamTimeout.java.io.IOException newTimeoutException(java.io.IOException)(Http2Stream.kt:675)
        at okhttp3.internal.http2.Http2Stream$StreamTimeout.void exitAndThrowIfTimedOut()(Http2Stream.kt:684)
        at okhttp3.internal.http2.Http2Stream.okhttp3.Headers takeHeaders()(Http2Stream.kt:143)
        at okhttp3.internal.http2.Http2ExchangeCodec.okhttp3.Response$Builder readResponseHeaders(boolean)(Http2ExchangeCodec.kt:97)
        at okhttp3.internal.connection.Exchange.okhttp3.Response$Builder readResponseHeaders(boolean)(Exchange.kt:110)
        at okhttp3.internal.http.CallServerInterceptor.okhttp3.Response intercept(okhttp3.Interceptor$Chain)(CallServerInterceptor.kt:93)
        at okhttp3.internal.http.RealInterceptorChain.okhttp3.Response proceed(okhttp3.Request)(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.ConnectInterceptor.okhttp3.Response intercept(okhttp3.Interceptor$Chain)(ConnectInterceptor.kt:34)
        at okhttp3.internal.http.RealInterceptorChain.okhttp3.Response proceed(okhttp3.Request)(RealInterceptorChain.kt:109)
        at okhttp3.internal.cache.CacheInterceptor.okhttp3.Response intercept(okhttp3.Interceptor$Chain)(CacheInterceptor.kt:95)
        at okhttp3.internal.http.RealInterceptorChain.okhttp3.Response proceed(okhttp3.Request)(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.BridgeInterceptor.okhttp3.Response intercept(okhttp3.Interceptor$Chain)(BridgeInterceptor.kt:83)
        at okhttp3.internal.http.RealInterceptorChain.okhttp3.Response proceed(okhttp3.Request)(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.okhttp3.Response intercept(okhttp3.Interceptor$Chain)(RetryAndFollowUpInterceptor.kt:76)
        at okhttp3.internal.http.RealInterceptorChain.okhttp3.Response proceed(okhttp3.Request)(RealInterceptorChain.kt:109)
        at OU.d.k(SourceFile:1)
        at OU.d.intercept(SourceFile:95)
        at com.datadog.android.log.internal.net.LogsRequestFactory.intercept(LogsRequestFactory.kt:139)
        at okhttp3.internal.http.RealInterceptorChain.okhttp3.Response proceed(okhttp3.Request)(RealInterceptorChain.kt:109)
        at com.bitso.android.api.client.OkHttpClientFactory.intercept(OkHttpClientFactory.kt:10)
        at okhttp3.internal.http.RealInterceptorChain.okhttp3.Response proceed(okhttp3.Request)(RealInterceptorChain.kt:109)
        at j5.f.intercept(SourceFile:10)
        at okhttp3.internal.http.RealInterceptorChain.okhttp3.Response proceed(okhttp3.Request)(RealInterceptorChain.kt:109)
        at j5.g.intercept(SourceFile:10)
        at okhttp3.internal.http.RealInterceptorChain.okhttp3.Response proceed(okhttp3.Request)(RealInterceptorChain.kt:109)
        at com.bitso.android.api.client.Environment.intercept(Environment.kt:86)
        at okhttp3.internal.http.RealInterceptorChain.okhttp3.Response proceed(okhttp3.Request)(RealInterceptorChain.kt:109)
        at com.bitso.android.api.client.BitsoApiService.intercept(BitsoApiService.kt:79)
        at okhttp3.internal.http.RealInterceptorChain.okhttp3.Response proceed(okhttp3.Request)(RealInterceptorChain.kt:109)
        at com.bitso.android.api.client.RetrofitClientFactory.intercept(RetrofitClientFactory.kt:45)
        at okhttp3.internal.http.RealInterceptorChain.okhttp3.Response proceed(okhttp3.Request)(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.RealCall.okhttp3.Response getResponseWithInterceptorChain$okhttp()(RealCall.kt:201)
        at okhttp3.internal.connection.RealCall$AsyncCall.void run()(RealCall.kt:517)
        ... 3 more

you can notice that the following lines are not being deobfuscated properly

Caused by: NoInternetError(cause=java.net.SocketTimeoutException: timeout)
        at com.bitso.android.analytics.properties.VariantProperty.a(VariantProperty.kt:12)
        at com.bitso.android.analytics.properties.PlacementProperty.onFailure(PlacementProperty.kt:15)

@0xnm do you know why this could be happening? Thank you in advance! 🙏🏼

[0xnm]

Deobfuscation process relies on the build_id property which is attached to both mapping file uploaded (you can see it shown on the source map management page) and recorded in the error event. If mapping file with build_id provided in the error event doesn't exist, then we cannot deobfuscate the stacktrace.

Can you please check that build_id attribute in your error event matches the one you see at the source map management page? This property is unique for each build and if upload happened after some changes to the code which was used to generate APK file, this build ID can be different from the one which was bundled with APK. To make them match, please upload mapping file in the same Gradle execution as generating APK which you upload to Play Store. Or in the next Gradle invocation without any changes to source/resource files (or other changes which may be introduced by 3rd party plugins during the build process).

[epool]

@0xnm where can I get the build_id from a Log? we're not using RUM only Logs, I exported my log as json but I couldn't find any build_id property 🤔

[0xnm]

build_id will be included if you are using Datadog SDK version 2.8.0 or above and Datadog Gradle Plugin version 1.13.0 or above (we recommend to use the latest possible versions though).

Can you please check that Datadog SDK version you are using is 2.8.0 or above?

[epool]

@0xnm we're using the following setup

datadog_version = "2.9.0"
datadog-logs = { module = "com.datadoghq:dd-sdk-android-logs", version.ref = "datadog_version" }

datadog_plugin_version = "1.14.0"
datadog = { id = "com.datadoghq.dd-sdk-android-gradle-plugin", version.ref = "datadog_plugin_version" }

datadog_plugin_version = "1.14.0" was just updated yesterday, but the prod build used datadog_plugin_version = "1.12.0"

so is it necessary to have at least datadog_plugin_version = "1.13.0" in order to have build_id in the logs properties? is this being added at build time or how the plugin version would affect? 👀

[epool]

@0xnm I just double checked with these latests versions

datadog_version = "2.9.0"
datadog_plugin_version = "1.14.0"

and the logs still don't have the build_id property.

[epool]

@0xnm another not that related question.

The ./gradlew uploadMappingRelease task must be executed explicitly always or would it triggered after a release build or similar under the hood?

[0xnm]

Overall our deobfuscation process has 2 mechanisms (which are completely transparent and with proper setup shouldn't cause any issues):

• Legacy one: it is using values triplet service,version name,variant to match between mapping file and error. It was causing some false matches (for example, because different builds may have the same version name) and was error-prone to setup issues, so it was replaced by the new one below.
• New one: it is using build_id value which is unique for each build.

Legacy one is used if Datadog SDK version is below 2.8.0 (with any Datadog Gradle Plugin version) or if Datadog SDK version is above 2.8.0, but Datadog Gradle Plugin is below 1.13.0.

If both Datadog SDK version is above 2.8.0 and Datadog Gradle Plugin version is above 1.13.0, then new mechanism is used (Gradle plugin generates unique build ID during build time and packs it with application assets, so it can be read later by SDK and attached to the events sent to Datadog).

You've mentioned:

but the prod build used datadog_plugin_version = "1.12.0"

It means that the legacy mechanism was used, so it means triplet service,version name,variant should match between event and mapping file uploaded. If it is not the case (due to the setup difference on SDK side vs Gradle plugin side by some reason; the most common cause is specifying some custom variant value in SDK configuration, but not setting it in the Gradle plugin configuration), deobfuscation is not possible.

The ./gradlew uploadMappingRelease task must be executed explicitly always or would it triggered after a release build or similar under the hood?

By default it is not automatically finalizes build task to avoid unnecessary uploads, but we have a note how to make it in our documentation.

[epool]

everything is much clearer now, thank you so much for your detailed explanation and help @0xnm! 🙏🏼 🙇🏼‍♂️

we just launched a new app version with this latest datadog versions hoping to finally have the errors deobfuscated 🤞🏼

[epool]

Hey @0xnm! 👋🏼 I just confirmed that the deobfuscation is working as expected now, but it's only displayed in the Pretty tab. In the Raw tab, it is still obfuscated. Is this expected?

Pretty	Raw

I'm aiming to query my logs with some Kotlin packages from our emergency logs. Is there a way to do so? Is there a pipeline or something similar to populate the deobfuscated stacktrace in the logs and later query them?

Thank you so much in advance! 🙏🏼

[0xnm]

Hello @epool!

but it's only displayed in the Pretty tab. In the Raw tab, it is still obfuscated. Is this expected?

yes, it is expected behaviour currently

I'm aiming to query my logs with some Kotlin packages from our emergency logs. Is there a way to do so? Is there a pipeline or something similar to populate the deobfuscated stacktrace in the logs and later query them?

I'm not quite clear what do you mean, can you please elaborate?

[epool]

@0xnm sure! I meant, I'm trying to query my emergency logs(crashes) by package, something like service:... source: android @error.stacktrace:*some.package.here* from the logs query bar in the dashboard.

[0xnm]

Stacktraces are stored as obfuscated right now on the backend side and deobfuscation is applied only on the frontend side, when stacktrace is shown. So you can search only by the obfuscated value for now, querying by deobfuscated stacktrace value is not something we support right now.

Although if it is an important use case for you, you can open a feature request by contacting our support here or by contacting your CSM (Customer Success Manager).

[epool]

Thank you so much, @0xnm! 🙏🏼 We have just contacted our CSM 🙇🏼‍♂️