[BUG] apktool cannot recompile obfuscated apks with shrinkResources = true after updating Gradle

[jcrabanal]

Information

1. Apktool Version: 2.9.3 (latest)
2. Operating System: Windows
3. APK From?: My own compiled APK
4. Java Version: Several versions, not relevant I guess

I'm getting errors while decompiling -> recompiling my own apk. The issue started happening after updating Android Studio and the gradle plugin.

It only happens when I compile my app with obfuscation enabled and "shrinkResources true" in my build.gradle file. If I compile an apk with "shrinkResources false", apktool is able to decompile/recompile the apk.

Stacktrace/Logcat

I'm getting a few of those errors:

temp_a711ab5f-4bc1-4ff0-84f4-f5198dc96737\temp\res\values\styles.xml:1552: error: expected enum but got (raw string)

Here is the referenced part of styles.xml file of the apk that breaks apktool:

    <style name="Base.Widget.MaterialComponents.TextInputLayout" parent="@style/Widget.Design.TextInputLayout">
        <item name="android:textColorHint">@color/mtrl_indicator_text_color</item>
        <item name="boxBackgroundColor">@null</item>
        <item name="boxBackgroundMode">2</item>
        <item name="boxCollapsedPaddingTop">0.0dip</item>
        <item name="boxStrokeColor">@color/mtrl_outlined_stroke_color</item>
        <item name="boxStrokeErrorColor">@color/mtrl_error</item>
        <item name="boxStrokeWidth">@dimen/mtrl_textinput_box_stroke_width_default</item>
        <item name="boxStrokeWidthFocused">@dimen/mtrl_textinput_box_stroke_width_focused</item>
        <item name="counterOverflowTextAppearance">?textAppearanceCaption</item>
        <item name="counterOverflowTextColor">@color/mtrl_error</item>
        <item name="counterTextAppearance">?textAppearanceCaption</item>
        <item name="counterTextColor">@color/mtrl_indicator_text_color</item>
        <item name="endIconTint">@color/mtrl_outlined_icon_tint</item>
        <item name="enforceMaterialTheme">true</item>
        <item name="enforceTextAppearance">true</item>
        <item name="errorIconDrawable">@drawable/mtrl_ic_error</item>
        <item name="errorIconTint">@color/mtrl_error</item>
        <item name="errorTextAppearance">?textAppearanceCaption</item>
        <item name="errorTextColor">@color/mtrl_error</item>
        <item name="helperTextTextAppearance">?textAppearanceCaption</item>
        <item name="helperTextTextColor">@color/mtrl_indicator_text_color</item>
        <item name="hintTextAppearance">?textAppearanceCaption</item>
        <item name="hintTextColor">?colorPrimary</item>
        <item name="placeholderTextAppearance">?textAppearanceSubtitle1</item>
        <item name="placeholderTextColor">@color/mtrl_indicator_text_color</item>
        <item name="prefixTextAppearance">?textAppearanceSubtitle1</item>
        <item name="prefixTextColor">@color/mtrl_indicator_text_color</item>
        <item name="shapeAppearance">?shapeAppearanceSmallComponent</item>
        <item name="shapeAppearanceOverlay">@null</item>
        <item name="startIconTint">@color/mtrl_outlined_icon_tint</item>
        <item name="suffixTextAppearance">?textAppearanceSubtitle1</item>
        <item name="suffixTextColor">@color/mtrl_indicator_text_color</item>
    </style>

And here is the styles.xml in the apk that apktool recompiles OK:

    <style name="Base.Widget.MaterialComponents.TextInputLayout" parent="@style/Widget.Design.TextInputLayout">
        <item name="android:textColorHint">@color/mtrl_indicator_text_color</item>
        <item name="boxBackgroundColor">@null</item>
        <item name="boxBackgroundMode">outline</item>
        <item name="boxCollapsedPaddingTop">0.0dip</item>
        <item name="boxStrokeColor">@color/mtrl_outlined_stroke_color</item>
        <item name="boxStrokeErrorColor">@color/mtrl_error</item>
        <item name="boxStrokeWidth">@dimen/mtrl_textinput_box_stroke_width_default</item>
        <item name="boxStrokeWidthFocused">@dimen/mtrl_textinput_box_stroke_width_focused</item>
        <item name="counterOverflowTextAppearance">?textAppearanceCaption</item>
        <item name="counterOverflowTextColor">@color/mtrl_error</item>
        <item name="counterTextAppearance">?textAppearanceCaption</item>
        <item name="counterTextColor">@color/mtrl_indicator_text_color</item>
        <item name="endIconTint">@color/mtrl_outlined_icon_tint</item>
        <item name="enforceMaterialTheme">true</item>
        <item name="enforceTextAppearance">true</item>
        <item name="errorIconDrawable">@drawable/mtrl_ic_error</item>
        <item name="errorIconTint">@color/mtrl_error</item>
        <item name="errorTextAppearance">?textAppearanceCaption</item>
        <item name="errorTextColor">@color/mtrl_error</item>
        <item name="helperTextTextAppearance">?textAppearanceCaption</item>
        <item name="helperTextTextColor">@color/mtrl_indicator_text_color</item>
        <item name="hintTextAppearance">?textAppearanceCaption</item>
        <item name="hintTextColor">?colorPrimary</item>
        <item name="placeholderTextAppearance">?textAppearanceSubtitle1</item>
        <item name="placeholderTextColor">@color/mtrl_indicator_text_color</item>
        <item name="prefixTextAppearance">?textAppearanceSubtitle1</item>
        <item name="prefixTextColor">@color/mtrl_indicator_text_color</item>
        <item name="shapeAppearance">?shapeAppearanceSmallComponent</item>
        <item name="shapeAppearanceOverlay">@null</item>
        <item name="startIconTint">@color/mtrl_outlined_icon_tint</item>
        <item name="suffixTextAppearance">?textAppearanceSubtitle1</item>
        <item name="suffixTextColor">@color/mtrl_indicator_text_color</item>
    </style>

The line the stack trace is referencing to (1552) is this one:

<item name="boxBackgroundMode">2</item>

Apparently R8, the obfuscator, replaced an enum with its literal value. My humble guess is that apktool might not need to be that strict while checking this.

Questions to ask before submission

1. Have you tried apktool d, apktool b without changing anything?: Yes

[iBotPeaches]

Do you have a smaller apk sample to work from?

[jcrabanal]

The sample projects of Android Studio seem to do the trick. Here you have.

Both APKs here decompile OK with "java -jar apktool_2.9.3.jar d [file]", but only shrinkResources_false.apk can be recompiled.

ApkToolTest.zip

[jcrabanal]

Setting android.enableNewResourceShrinker.preciseShrinking=false in my gradle.properties file and enabling shrinkResources = true seems to make it work again (decompile & recompile)

https://developer.android.com/build/releases/gradle-plugin#resource-shrinking

[iBotPeaches]

I made a sample that leveraged that feature when investigating another bug - maybe some overlap with this. Especially if this is the feature that automatically strips namespace context from resources.

[dbgp]

maybe this is the same condition.

➜  apktool java -jar apktool2.9.3.jar b -o venus.apk m171
I: Using Apktool 2.9.3
I: Checking whether sources has changed...
I: Smaling smali folder into classes.dex...
I: Checking whether sources has changed...
I: Smaling smali_classes6 folder into classes6.dex...
I: Checking whether sources has changed...
I: Smaling smali_classes3 folder into classes3.dex...
I: Checking whether sources has changed...
I: Smaling smali_classes4 folder into classes4.dex...
I: Checking whether sources has changed...
I: Smaling smali_classes5 folder into classes5.dex...
I: Checking whether sources has changed...
I: Smaling smali_classes2 folder into classes2.dex...
I: Checking whether resources has changed...
I: Building resources...
W: /Users/dbgp/Downloads/apktool/m171/res/values/styles.xml:6670: error: expected enum but got (raw string) uniform.
W: /Users/dbgp/Downloads/apktool/m171/res/values/styles.xml:6684: error: expected enum but got (raw string) uniform.
W: error: failed linking references.
brut.androlib.exceptions.AndrolibException: brut.common.BrutException: could not exec (exit code = 1): [/var/folders/jm/1g3t6w6n5692dnhz5jk0gph40000gn/T/brut_util_Jar_56673902650333380682946784968269564306.tmp, link, -o, /var/folders/jm/1g3t6w6n5692dnhz5jk0gph40000gn/T/APKTOOL8336347317507460300.tmp, --package-id, 127, --min-sdk-version, 24, --target-sdk-version, 33, --rename-manifest-package, one.mixin.messenger.venus, --rename-instrumentation-target-package, one.mixin.messenger.venus, --version-code, 1070100, --version-name, 1.7.1, --no-auto-version, --no-version-vectors, --no-version-transitions, --no-resource-deduping, --allow-reserved-package-id, --no-compile-sdk-metadata, --warn-manifest-validation, -e, /var/folders/jm/1g3t6w6n5692dnhz5jk0gph40000gn/T/APKTOOL5073417883081723115.tmp, -0, arsc, -I, /Users/dbgp/Library/apktool/framework/1.apk, --manifest, /Users/dbgp/Downloads/apktool/m171/AndroidManifest.xml, /Users/dbgp/Downloads/apktool/m171/build/resources.zip]

related two lines:

<style name="Widget.MaterialComponents.MaterialCalendar.HeaderSelection" parent="@style/Widget.AppCompat.TextView">
        <item name="android:textAppearance">?textAppearanceHeadline4</item>
        <item name="android:textColor">?colorOnPrimary</item>
        <item name="android:ellipsize">end</item>
        <item name="android:layout_gravity">start|center|top</item>
        <item name="android:layout_width">fill_parent</item>
        <item name="android:layout_height">wrap_content</item>
        <item name="android:maxLines">@integer/mtrl_calendar_selection_text_lines</item>
        <item name="autoSizeMaxTextSize">34.0sp</item>
        <item name="autoSizeMinTextSize">2.0sp</item>
        <item name="autoSizeTextType">uniform</item> <!-- this is line 6670 -->
    </style>

<style name="Widget.MaterialComponents.MaterialCalendar.HeaderTitle" parent="@style/Widget.AppCompat.TextView">
        <item name="android:textAppearance">?textAppearanceOverline</item>
        <item name="android:textColor">?colorOnPrimary</item>
        <item name="android:ellipsize">end</item>
        <item name="android:maxLines">1</item>
        <item name="autoSizeMaxTextSize">10.0sp</item>
        <item name="autoSizeMinTextSize">2.0sp</item>
        <item name="autoSizeTextType">uniform</item>  <!-- this is line 6684 -->
    </style>

after decode , this item(autoSizeTextType) value is 1, the same error rised while re-building. I change its value to uniform manually according to this link, but this error still happen.

[rikumi]

Maybe apktool could detect those values by simply looking for their attribute names in the attrs.xml file and create pseudo keys for them there. For example:

<!-- styles.xml -->
<item name="someAttribute">1</item>

<!-- attrs.xml -->
<attr name="someAttribute">
    <enum name="anotherKey" value="0" />
</attr>

can be turned into:

<!-- styles.xml -->
<item name="someAttribute">apktool_generatedStubEnumKey_1</item>

<!-- attrs.xml -->
<attr name="someAttribute">
    <enum name="anotherKey" value="0" />
    <enum name="apktool_generatedStubEnumKey_1" value="1" />
</attr>

[iBotPeaches]

That is one possibility, but Apktool during processing may not have enough information at time of each AXML file disassembly to know at that exact moment if the scalar should be a enum or not.

I've been experimenting with relaxing the aapt2 build requirement to see if can survive with the scalar instead of re-making the enum.

https://github.com/iBotPeaches/platform_frameworks_base/tree/apktool-2.10.x

[HassanMirza01]

i have successfully recompiled apks which were compiled with shrinkResources = true i have added some attrs and replaced all matching lines in res/layout/*.xml files, like that

1. Added attributes to attrs.xml file: <attr name="layout_constraintEnd_toEndOf" format="reference"> <enum name="parent" value="0" /> </attr>

2. Changed app:layout_constraintEnd_toEndOf="0" to app:layout_constraintEnd_toEndOf="parent" and that built a fully working apk, tried 3 apks and all are working totally fine.

Conclusion: If we can include all attrs of such type from google's source or a generic file by default in each decompile and these values get auto detected or changed in res/layout and res/xml folders ?

[jcrabanal]

If you set android.enableNewResourceShrinker.preciseShrinking=false in your gradle.properties, then you can enable shrinkResources = true again and produce APKs that apktool can process, but gradle shows this message...

"The option setting 'android.enableNewResourceShrinker.preciseShrinking=false' is deprecated. The current default is 'true'. It will be removed in version 9.0 of the Android Gradle plugin."

We may not be able to do this anymore in the future. Any update on this issue?

[iBotPeaches]

We may not be able to do this anymore in the future. Any update on this issue?

Not at this time. Changes occurred in AOSP and I can no longer build aapt2 and its stretched past my knowledge. So I'm blocked on everything until I learn more on how to resolve that.

context: https://groups.google.com/g/android-building/c/TvKKsSLSlPY

[HassanMirza01]

We may not be able to do this anymore in the future. Any update on this issue?

Not at this time. Changes occurred in AOSP and I can no longer build aapt2 and its stretched past my knowledge. So I'm blocked on everything until I learn more on how to resolve that.

context: https://groups.google.com/g/android-building/c/TvKKsSLSlPY

i used "lunch aosp_arm64-trunk_staging-eng" and i can build aapt and aapt2 fine with latest repo sync which removed lunch menu.

[iBotPeaches]

i used "lunch aosp_arm64-trunk_staging-eng" and i can build aapt and aapt2 fine with latest repo sync which removed lunch menu.

thanks. I'll give that a shot tonight.

[HassanMirza01]

this version it compiled,

[drzraf]

How I'm patching XML in order to rebuild a specific package, I think that gives a sense of what's needed to get a generic workaround for this class of problems. (Here intended to be used with find res/layout res/values -type f -name '*.xml' -exec fix.sh {} \;)

#!/bin/bash

# Fix errors from apktools build process
# See https://github.com/iBotPeaches/Apktool/issues/3534

fixstyle() {
    f="$1" && shift
    xmlstarlet ed -L -u '/resources/style/item[@name="autoSizeTextType"][text()="1"]' -v none "$f"
    xmlstarlet ed -L -u '/resources/style/item[@name="offsetAlignmentMode"][text()="0"]' -v legacy "$f"
    xmlstarlet ed -L -u '/resources/style/item[@name="indeterminateAnimationType"][text()="1"]' -v contiguous "$f"
}

fixfile() {
    f="$1" && shift
    xmlstarlet ed -L -u  '//*[@app:autoSizeTextType="1"]/@app:autoSizeTextType' -v none "$f";
    xmlstarlet ed -L -u  '//*[@app:offsetAlignmentMode="0"]/@app:offsetAlignmentMode' -v legacy "$f";
    xmlstarlet ed -L -u  '//*[@app:indeterminateAnimationType="1"]/@app:indeterminateAnimationType' -v contiguous "$f";
    xmlstarlet ed -L -u  '//*[@app:layout_collapseMode="2"]/@app:layout_collapseMode' -v none "$f";
    xmlstarlet ed -L -u  '//*[contains(@app:showAsAction, "APKTOOL_MISSING_")]/@app:showAsAction' -v always "$f"
    sed -ri '/layout_scrollFlags.*APKTOOL_MISSING_0x/s/APKTOOL_MISSING_0x[0-9a-f]+/scroll/' "$f";
}

[[ $1 =~ res/values/styles.xml ]] && fixstyle $@ || fixfile $@

[iBotPeaches]

How I'm patching XML in order to rebuild a specific package, I think that gives a sense of what's needed to get a generic workaround for this class of problems. (Here intended to be used with find res/layout res/values -type f -name '*.xml' -exec fix.sh {} \;)

May you roughly explain what this is doing? In my knowledge we either have to replace scalars with enums or patch aapt2 to accept the scalar - I'm not following how this is working.

[drzraf]

I don't know (yet) about aapt2 so I fixed errors reported by apktool by patching the XML the dirty way: So the above is only valid for a particular case.

Here, res/values/styles.xml was patched to substitute scalar by enum, but it's not enough because scalars are also present in many other XML files all along res/* and this is where fixfile does the job of substituting scalar by enum. And in some particular case, like layout_collapseMode = 2, the scalar is not even defined in attrs.xml so one would have to (arbitrarily) replace it by one of the alternative existing enum (none) (but patching attrs.xml to insert the new enum would have been better, I just couldn't figure out how to do that automatically with xmlstarlet).

It's probably not bright overall (sorry if it confused anyone), but was able to successfully rebuild an app' with this. (I there was a binary release of the experimental apktool-2.10.x branch, I'd have otherwise tried it first)

[HassanMirza01]

exactly @drzraf answer is doing same what i did but in different way, i used finding xml's in res folder and patching generic values on all files where applicable.

in short, we can compile a list of scalars and enums for direct replacement when they are decompiled and some generic list to be included in attrs.xml also, it wont give error if there are some extra attributes @iBotPeaches

[iBotPeaches]

Lets take 1 of the problems and dig into it.

Missing Attrs

    <style name="Base.Widget.AppCompat.ActionBar.TabBar" parent="">
        <item name="divider">?actionBarDivider</item>
        <item name="dividerPadding">8.0dip</item>
        <item name="showDividers">APKTOOL_MISSING_0x7f08010a</item>
    </style>

So if we dig into that resource (0x7f08010a).

    resource 0x7f0303c0 attr/showDividers
      () (attr) type=flags size=4
        0x7f080137=0x00000000
        0x7f08005a=0x00000001
        0x7f08010a=0x00000002
        end(0x7f0800ae)=0x00000004

So we know the value was stripped out, but it corresponds to scalar 2. So if we take that attribute (showDividers) we don't seem to have 2.

    <attr name="showDividers">
        <flag name="end" value="0x00000004" />
    </attr>

I'm more curious why this occurred. ShinkResources clearly stripped it - if we take the false (shinkResources)

    <style name="Base.Widget.AppCompat.ActionBar.TabBar" parent="">
        <item name="divider">?actionBarDivider</item>
        <item name="dividerPadding">8.0dip</item>
        <item name="showDividers">middle</item>
    </style>

ShowDividers is still present with the middle attribute. So I guess I'm still trying to figure out what is happening to devise a good solution. Since in the non-shrunk option all the attr values are there.

    <attr name="showDividers">
        <flag name="none" value="0x00000000" />
        <flag name="beginning" value="0x00000001" />
        <flag name="middle" value="0x00000002" />
        <flag name="end" value="0x00000004" />
    </attr>

Some thoughts in my head.

• ShinkResources is stripping default values from the base style? It has no parent - so I guess it can't be that.
• The stripped values are unused
• Apktool is doing something dumb
• Gradle/Android/ShrinkResources is doing something dumb

[jcrabanal]

Just to clarify again, it's not shrinkResources true that causes the problem, it's shrinkResources = true and android.enableNewResourceShrinker.preciseShrinking=true on the gradle.properties file (the default value as of the latest gradle plugin is now true).

https://developer.android.com/build/releases/past-releases/agp-7-1-0-release-notes#experimental-further-app-size-reductions

You might be able to guess what is happening by checking what this flag does.

[jcrabanal]

Wait, wtf, I've tried again and the bug is not happening anymore. Some time ago I updated Android Studio and the gradle plugin to the latest version, and APKs compiled with shrinkResources = true and android.enableNewResourceShrinker.preciseShrinking=true can be now be recompiled correctly.

Can someone else please verify?

[jcrabanal]

I've tried compiling the same APK with gradle plugin 8.3.1 and with 8.5.0 (latest) and only the latest one solves the issue.

[iBotPeaches]

thanks, so perhaps precise shrinking was doing too much? If so I'd expect original application to have issues, but it seemed like it was only after a round of disassembly.

[iBotPeaches]

this version it compiled,

Thanks - I can build aapt/aapt2 now.