"Resource compilation failed" when replacing library which has "declare-styleable" resources

[andrewleech]

I'm trying to upgrade some of the existing libraries in my app/project.

Started by just adding new dependencies to build.gradle with problematic transitive deps excluded. The package-info.java method to declare the code overridden works fine there. Some of the libraries work perfectly well with just this done.

With others, I start to get

Execution failed for task ':mergeReleaseResources'.
> 1 exception was raised by workers:
  com.android.builder.internal.aapt.v2.Aapt2Exception: Android resource compilation failed
  C:\Users\anl\.gradle\caches\transforms-2\files-2.1\cf88320d6b120360e17f4b697335e513\res\values\values.xml:153:5-155:25: AAPT: error: duplicate value for resource 'attr/navigationMode' with config ''.
  C:\Users\anl\.gradle\caches\transforms-2\files-2.1\cf88320d6b120360e17f4b697335e513\res\values\values.xml:153:5-155:25: AAPT: error: resource previously defined here.

and a number more like this.

Comparing the listed generated file to the ones extracted from the apk show the difference: these attr resources are coming from the library and are inside a declare-styleable block;

    <declare-styleable name="ActionBar">
        <!-- The type of navigation to use. -->
        <attr name="navigationMode">
            <!-- Normal static title text -->
            <enum name="normal" value="0"/>
            <!-- The action bar will use a selection list for navigation. -->
            <enum name="listMode" value="1"/>
            <!-- The action bar will use a series of horizontal tabs for navigation. -->
            <enum name="tabMode" value="2"/>
        </attr>
        ...
    </declare-styleable>

I've run into the same issue in the past when I extracted resources from apk and used them directly copied into android studio project, compiling with decompiled code and a jar of code extracted from same app; these declare-styleable blocks are used by source as a reference but are basically stripped out of the binaries so don't exist in the apk. I had to manually recreate them by trial and error in my resource files to get the project to build.

In my new dexpatcher based project, I'm guessing the aapt resource merger can't resolve the apk-derived flat attr's as being the correct ones to override by the new declare-styleable internal ones so throws this error.

Is there any possibility of handling this sort of issue in dexpatcher? I'm guessing it generally doesn't normally do much in the way of editing resources after the initial unpack.

Do you know if there's any way I could add an extra gradle task hook or similar to manually strip out resources I don't need included from the original apk? I'm going to try to do this manually with apktool now.

[Lanchon]

hey! im travelling right now so i cant be much help. mia and nyc. if i read correctly, some process after merger transforms the lib resources so they change type (aapt i guess) and they clash with pre transformed resources.

tricky... merger is per type, so wont help. deleting apk resources is tricky: even if you get merger to delete some apk resources somehow, the build will fail because the deleted resources will still be listed in public.xml. to delete apk resources they have to be removed both from res/ and public.xml. and merger doesnt know of public.xml.

maybe you can get merger to mod the NEW resources so that they are deleted altogether, not generated, or generated with dummy names. but granted, this is not an ideal workaround.

in the meantime you can build through an apklib (are you already doing that?) and manually doctor it to remove resources.

to solve this, a resource removal step could be added to the build. or maybe a generic XML transformation step, such as XSLT or anything better/simpler.

[andrewleech]

Nice, enjoy the Sun!

Yeah wasn't expecting fixes, kinda just documenting what I find and fishing for suggestions ;-)

My efforts at deleting the troublesome resources did not go well, I knew about the public.xml references file and tried to clean in all necessary files, but then naturally apktool failed to rebuild due to missing references.

I hadn't yet investigated what apklib was all about... looks perfect and much easier for a number of the things I've done to my source apk with apktool previously!

Using apklib structure I should be able to manually add the declare-styleable tags to the resources in the apklib so that aapt can know how to merge them correctly.

[Lanchon]

nothing better than open discussion. but im short on time. maybe you want to message me on whatsapp where we can send audio messages. if uve got whatsapp u r welcome to send it over a private message. later!

[andrewleech]

So I've run into this issue again on my newer project, in this case it happened when I brought in a replacement google play library in as a dependency, with the same new <declare-styleable> attrs clashing with the flat <attr> resources extracted from the apk.

Seeing as I've just got the project back to working with the original apk (thanks!) I decided I didn't want to go back down the manually patched apklib route again.

So I've been playing around making a gradle task to fix the issue automatically during build.

The mergeDebugResources task is where it all goes down. As I presume you know, it basically pulls together the resources from the original apk and all the ones from dependencies, merges them into ${buildDir}/intermediates/incremental/mergeDebugResources/merged.dir/values/values.xml and then feeds that into aapt2.exe

I haven't been able to figure out a way to modify this file before it's processed however.

As a workaround, I've made a task that runs just before mergeDebugResources and grabs a copy of this file if it exists (from the previous build):

task snapshotResources {
    doFirst {
        def mergerXml = file("${buildDir}/intermediates/incremental/mergeDebugResources/merged.dir/values/values.xml")
        if (mergerXml.exists()) {
            def prev_dir = file("${buildDir}/previous")
            if (!prev_dir.exists()) {
                prev_dir.mkdirs()
            }
            copy {
                from mergerXml
                into prev_dir
            }
        }
    }
}

project.afterEvaluate{
    mergeDebugResources.dependsOn("snapshotResources")
    snapshotResources.dependsOn("processIdMappingsDebug")
    snapshotResources.dependsOn("generateDebugResources")
}

Then, on the next build, I've got a task running just after provideDecodedApp to parse the previously grabbed merged values.xml, search it for all <declare-styleable> attrs and then delete any flat attr copies of these from the apk attr.xml and publix.xml files. I also delete them from copies of the res I've pulled into local source.

project.afterEvaluate{
    provideDecodedApp.configure {
        it.doLast{ task ->

            def prev_values = file("${buildDir}/previous/values.xml")

            def local_attr = "${projectDir}/src/main/res/values/attrs.xml"
            def local_attr_xml = new XmlParser().parse(local_attr)

            def apk_attr = "${buildDir}/intermediates/dexpatcher/decoded-app/res/values/attrs.xml"
            def apk_attr_xml = new XmlParser().parse(apk_attr)

            def apk_public = "${buildDir}/intermediates/dexpatcher/decoded-app/res/values/public.xml"
            def apk_public_xml = new XmlParser().parse(apk_attr)

            def originalXml = new XmlParser().parse(prev_values)

            originalXml.'*'.findAll { node -> node.name() == 'declare-styleable' }.each {
                println "top: " + it.@name

                it.each {

                    def to_remove = local_attr_xml.'*'.findAll { node ->
                        (node.name() == 'attr' && node.@name == it.@name)
                    }
                    if (to_remove.size()) {
                        def parent = to_remove[0].parent()
                        parent.remove(to_remove[0])
                        println "removing: " + to_remove[0]
                    }

                    to_remove = apk_attr_xml.'*'.findAll { node ->
                        (node.name() == 'attr' && node.@name == it.@name)
                    }
                    if (to_remove.size()) {
                        def parent = to_remove[0].parent()
                        parent.remove(to_remove[0])
                        println "removing: " + to_remove[0]
                    }

                    to_remove = apk_public_xml.'*'.findAll { node ->
                        (node.@type == 'attr' && node.@name == it.@name)
                    }
                    if (to_remove.size()) {
                        def parent = to_remove[0].parent()
                        parent.remove(to_remove[0])
                        println "removing: " + to_remove[0]
                    }
                }
            }

            new File( local_attr ).withWriter { out ->
                def printer = new XmlNodePrinter( new PrintWriter(out) )
                printer.preserveWhitespace = true
                printer.print( local_attr_xml )
            }

            new File( apk_attr ).withWriter { out ->
                def printer = new XmlNodePrinter( new PrintWriter(out) )
                printer.preserveWhitespace = true
                printer.print( apk_attr_xml )
            }

            new File( apk_public).withWriter { out ->
                def printer = new XmlNodePrinter( new PrintWriter(out) )
                printer.preserveWhitespace = true
                printer.print( apk_public_xml )
            }
        }
    }
}

While this is working (!) it does basically take three builds the first time to get it to build. I think the modifying of files in multiple hacked up tasks breaks caching a bit too, so it's not ideal.

Do you know any way I could build a task / hook that modifies the final merged values.xml file just before it's passed to aapt2? This would be the ideal time to fix the file, such that I never need to touch the unpacked files from he original apk.

[Lanchon]

I haven't been able to figure out a way to modify this file before it's processed however.

this is not clear. you want to modify the output of mergeDebugResources before it gets to aapt2?

then why not...

mergeDebugResources.configure {
    doLast {
        //...

[Lanchon]

btw, this is how you get to the resource merger tasks from all the build variants:

https://github.com/DexPatcher/dexpatcher-gradle/blob/97194a6d2261f6f0019c985db056e012f2a877ef/src/main/groovy/lanchon/dexpatcher/gradle/plugins/AbstractPatcherPlugin.groovy#L298-L299

[Lanchon]

and VariantHelper just adapts to the old/new Android plugin differences:

https://github.com/DexPatcher/dexpatcher-gradle/blob/97194a6d2261f6f0019c985db056e012f2a877ef/src/main/groovy/lanchon/dexpatcher/gradle/VariantHelper.groovy#L75-L81

[andrewleech]

Thanks for the suggestions, however the aapt call seems to be happening inside mergeResources, before doLast is called. The final merged values.xml I'd like to be filtering is created after doFirst. With gradle:

android.applicationVariants.all { BaseVariant variant ->
    def mergeResources = VariantHelper.getMergeResources(variant)
    mergeResources.configure {
        it.doFirst { task ->
            println "**** before getMergeResources: "
            println file("${buildDir}/intermediates/incremental/mergeDebugResources/merged.dir/values/values.xml").exists()
        }
        it.doLast { task ->
            println "**** after getMergeResources: "
        }
    }
}

I'm getting the following errors

> Task :mainApkListPersistenceDebug UP-TO-DATE
> Task :generateDebugResValues UP-TO-DATE
> Task :generateDebugResources UP-TO-DATE
> Task :processIdMappingsDebug UP-TO-DATE

> Task :mergeDebugResources
**** before getMergeResources: 
false

> Task :mergeDebugResources FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':mergeDebugResources'.
> 1 exception was raised by workers:
  com.android.builder.internal.aapt.v2.Aapt2Exception: Android resource compilation failed
  C:\Users\anl\.gradle\caches\transforms-2\files-2.1\f7fb4b528732137d95883a17b38e0a6c\res\values\values.xml:6:1-22:21: AAPT: error: duplicate value for resource 'attr/layout_anchorGravity' with config ''.

  C:\Users\anl\.gradle\caches\transforms-2\files-2.1\f7fb4b528732137d95883a17b38e0a6c\res\values\values.xml:6:1-22:21: AAPT: error: resource previously defined here.

with a bunch more of the duplicates

[Lanchon]

since aapt2, resources are precompiled separately in a first aapt2 invocation producing separate binary files. this is to avoid recompiling all resources if you change only some of them. later, in a second invocation of aapt2, the apk is built based on a bunch of buildables including the precompiled resources. the second pass cannot feed on non-precompiled resources; everything must be precompiled.

if you are at a stage that invokes aapt2, it's already too late to alter text files. from that point on, values.xml will be ignored and only the precompiled version of it will be honored. so you must get to the files before aapt2. one way is to modify whatever files the merger task consumes. but you CAN NEVER modify input files in a doFirst!!!!!!! you'll break Gradle if you do that. you have to locate whatever task(s) produce the files in question and hook those task(s) with doLast(s).

btw, hasn't this been fixed by the apktool people? have you tried using the AAPT2 binary from apktool? take a look at the "invalid resources" sample to see how to enable use of the custom AAPT2.

[Lanchon]

so you used to mod the apklib manually. why can't you do that programmatically instead?

your best bet is: there are two tasks that produce the exploded apklib. one is used if apktool is used to decode an apk if provided. the other is used to extract an apklib if provided. only one or the other is used, never both. you can add a doLast to the task you are using (or even better, add the same doLast to both tasks so it is more general).

the names of the tasks are:

• TaskNames.DECODE_APK
• TaskNames.UNPACK_APK_LIBRARY

and they are defined here: https://github.com/DexPatcher/dexpatcher-gradle/blob/97194a6d2261f6f0019c985db056e012f2a877ef/src/main/groovy/lanchon/dexpatcher/gradle/plugins/AbstractDecoderPlugin.groovy#L151-L178

in either doLast you basically get the contents of apktool decode in directory: https://github.com/DexPatcher/dexpatcher-gradle/blob/97194a6d2261f6f0019c985db056e012f2a877ef/src/main/groovy/lanchon/dexpatcher/gradle/plugins/AbstractDecoderPlugin.groovy#L89-L91

which is just a lazy way to refer to the outputDir property of task TaskNames.PROVIDE_DECODED_APP.

you can mod this outputDir all you want in those doLast without messing with task caching or up-to-date detection.

will this work for you?

[andrewleech]

I was already using useAapt2BundledWithApktool = true, the problem isn't so much invalid attributes but conflicting ones. I ran into the same issue with direct decompiled/modded apps before I started with dexpatcher.

In android resource source files, there are the blocks like

    <declare-styleable name="ActionBar">
        <attr name="navigationMode" value="blah" />
    </declare-styleable>

However when they're compiled into an app, the declare-styleable xml tags are stripped out, leaving just

<attr name="navigationMode" value="blah" />

in the apps attr.xml file. So when the app is decompiled, this is all you see. You lose the declare-styleable grouping elements. For more info see https://github.com/iBotPeaches/Apktool/issues/1217

If you try to re-declare the <declare-styleable> elements, or bring in the same original library that declares them, they all look valid. But when compile/aapt does the initial merge both the <declare-styleable> blocks and the individual <attr> are put in the merge file, then when aapt tries to flatten the <declare-styleable> again it gives this conflict error because the <attr> is already there.

I wanted to avoid trying to remove the flattenned <attr> in an apklib because programatically, I don't know which ones need to be removed without inspecting which ever library / src resource has the full <declare-styleable> blocks.

I've managed to solve this issue for myself by adding a new gradle task just before mergeDebugResources, which then inspects all the mergeDebugResources.input files. These include the cached gradle paths to each of the resources being used to generate the merged file. By loading and iterating through them to find all <declare-styleable> blocks, I can then delete the appropriate <attr> lines from the gradle cached extracted res from the app itself, before they get sent into mergeDebugResources. This appears to be working ok.

task cleanupResources {
    dependsOn ":generateDebugResources"
    doLast {
        println 'cleanupResources'

        def inres = new ArrayList<File>()
        def outres = new ArrayList<File>()

        // We're running before mergeDebugResources, but the inputs are already defined correctly!
        for (File f: mergeDebugResources.inputs.getFiles()) {
//            println "mergeDebugResources file: " + f.getAbsoluteFile()
            if (f.getAbsoluteFile().toString().contains(".gradle\\caches")) {
                def manifest = new File(f.parent, '/AndroidManifest.xml')

                if (manifest.text.contains(" package=\"com.fossil.wearables.fossil\"")) {
                    println "Got app at " + f.getAbsoluteFile()
                    outres.add(new File(f.getAbsoluteFile(), "values/attrs.xml"))
                } else {
                    def values = new File(f.getAbsoluteFile(), "values/values.xml")
                    if (values.exists()) {
                        inres.add(values)
//                        println "Does exist:" + values.getAbsolutePath()
                    } else {
//                        println "Doesn't exist:" + values.getAbsolutePath()
                    }
                }
            }
        }

        for (File xml: inres) {
            // Search for declare-styleable attrs in all dependency library
            def originalXml = new XmlParser().parse(xml)
            originalXml.'*'.findAll { node -> node.name() == 'declare-styleable' }.each {
//                println "declare-styleable: " + it.@name
                it.each {
//                    println "attr: " + it.@name
                    for (File out : outres) {
                        // Delete any flat attr entries in decoded res that match inner declare-styleable attr
//                        println "strip from: " + out
                        def out_xml = new XmlParser().parse(out)
                        def to_remove = out_xml.'*'.findAll { node ->
                            (node.name() == 'attr' && node.@name == it.@name)
                        }
                        if (to_remove.size()) {
                            def parent = to_remove[0].parent()
                            parent.remove(to_remove[0])
//                            println "removing: " + to_remove[0] + " from " + out.getAbsolutePath()
                            new File(out.getAbsolutePath()).withWriter { writer ->
                                def printer = new XmlNodePrinter(new PrintWriter(writer))
                                printer.preserveWhitespace = true
                                printer.print(out_xml)
                            }
                        }
                    }
                }
            }
        }
    }
}
project.afterEvaluate{
    mergeDebugResources.dependsOn("snapshotResources")
    snapshotResources.dependsOn("processIdMappingsDebug")
    snapshotResources.dependsOn("generateDebugResources")
}

[Lanchon]

i see. the gradle timing of this issue is complex: the feeding of input files to the merger is just-in-time, but the resources you want to modify by peeking into that info were already processed and way ahead of that time.

there is a word-around involving:

• a setup task that at execute time peeks into the configured merger inputs, copy those to the preprocess task inputs, configures outputs for the preprocess task, and changes the merger inputs to consume the proprocess task outputs. this tasks depends on everything the merger depends, except this two ad-hoc added tasks.
• a preprocess task that modifies the resources the way you need them before the merger. you won't know which resources comes from the APK, but you don't need to: just kill items that are not in declare-styleables if any items of their group are.

however this is doable but tricky to do. i have to do stuff like that to patch the build for dexpatcher already. but the proprocess task has to do a lot, and in principle it wouldn't be incremental, which is bad.

you could try your hand at that (but read alternative below first). this is a task that tries to depend on everything the package application task depends on: https://github.com/DexPatcher/dexpatcher-gradle/blob/97194a6d2261f6f0019c985db056e012f2a877ef/src/main/groovy/lanchon/dexpatcher/gradle/plugins/PatchedAppPlugin.groovy#L77-L88

except the task i'm adding: https://github.com/DexPatcher/dexpatcher-gradle/blob/97194a6d2261f6f0019c985db056e012f2a877ef/src/main/groovy/lanchon/dexpatcher/gradle/plugins/PatchedAppPlugin.groovy#L82

it also lazily copies the task inputs: https://github.com/DexPatcher/dexpatcher-gradle/blob/97194a6d2261f6f0019c985db056e012f2a877ef/src/main/groovy/lanchon/dexpatcher/gradle/plugins/PatchedAppPlugin.groovy#L89-L91

the reason for all this is exactly what you want: this set of tasks want to preprocess what package application gets; the dex files in this case. in particular, they want to grab the project's dex files (the patch dex), patch the source apk with them, and feed the patched output to the package application task.

there's also a prepare task: https://github.com/DexPatcher/dexpatcher-gradle/blob/97194a6d2261f6f0019c985db056e012f2a877ef/src/main/groovy/lanchon/dexpatcher/gradle/plugins/PatchedAppPlugin.groovy#L119-L135

that in its execution phase changes the inputs to the package app task: https://github.com/DexPatcher/dexpatcher-gradle/blob/97194a6d2261f6f0019c985db056e012f2a877ef/src/main/groovy/lanchon/dexpatcher/gradle/plugins/PatchedAppPlugin.groovy#L125-L127

it can do this because it's only changing the contents of a lazy collection and not the inputs per se. so the caching is not broken. (yep, this is correct, and i've talked to the gradle team in-depth about it.)

why is there a separate prepare task, you ask, instead of handling the pack app preparation in the preprocess task (the patcher task)? because the preprocess task could be up-to-date, cached, or otherwise skipped, but the preparation has to always execute.

---

so... as i said: possible but complex. but picking into the gradle cache is of course a horrible hack we don't really want around either. so what else can you do? you could use the dexpatcher's aapt2 configuration to define a custom aapt2 executable, which would be a shell script to preprocess the inputs to aapt2, then invoke it.

yes, hacky, but cleaner than picking into gradle's cache. and it wont clash against gradle's task caching or avoidance as long as you don't modify the inputs: you have to make copies of them to a temp dir and fix those to aapt2.

the temp dirs would of course would be in build dir, no issues there. and you can help yourself to a real aapt2 binary to call from the script (the one included in the apktool, in this case) using this property: https://github.com/DexPatcher/dexpatcher-gradle/blob/97194a6d2261f6f0019c985db056e012f2a877ef/src/main/groovy/lanchon/dexpatcher/gradle/extensions/ApktoolExtension.groovy#L67

(or you can get it from my repo by creating a custom configuration, setting it up with the dependency, and resolving your custom config.)

[VaradharajanRajaram]

Hi All,

I am trying to decompile the APK file to use the resources of it. I have extracted the resources of the APK through APK tool.

Next I have uploaded the layout,values and drawable xml files into my local project. While building the project in android studio it throws error resource compilation failed as said above. So could you please tell me the steps to build it. However I have added the above script code in build.gradle file it is throwing error "Task with path 'processIdMappingsDebug' not found in project ':app'." Please help me on this.

[andrewleech]

@VaradharajanRajaram what version dexpatcher gradle plugin are you using? My project using that gradle script snippet is here: https://gitlab.com/alelec/fossil_smartwatches_alelec_android/blob/stylables/build.gradle you can reference the gradle plugin config in use there.