search

secure-software-engineering / FlowDroid

FlowDroid Static Data Flow Tracker
GNU Lesser General Public License v2.1
1.02k stars 292 forks source link

RuntimeException: File format violation, res1 was not zero #716

Closed jpstotz closed 3 months ago

jpstotz commented 3 months ago

It seems like something has changed in a common Android build chain, as since this week there are now multiple new apps in play store that can not be processed as they have in typeSpecTable.reserved a non-zero value and thus raise the error RuntimeException: File format violation, res1 was not zero https://github.com/secure-software-engineering/FlowDroid/blob/47c2872c6086d1668cf3d0b25c6ada4ccab74323/soot-infoflow-android/src/soot/jimple/infoflow/android/resources/ARSCFileParser.java#L2615

Affected are not only closed source apps but also apps like org.mozilla.firefox version 123.1.0.

Apktool 2.9.3 and Jadx-Gui don't have any problems decompiling this app. Looking at their sources I can see that both ignore typeSpecTable.flags (res0) and typeSpecTable.reserved (res1). Those values are not read at all:

May be we should also ignore those values in FlowDroid?

The code in AOSP doesn't seem to have changes, it still contains the two reserved fields which should be zero: https://android.googlesource.com/platform/frameworks/base/+/master/libs/androidfw/include/androidfw/ResourceTypes.h#1402

StevenArzt commented 3 months ago

We implemented the sanity check in the ARSCFileParser so that we notice changes to the file format. Such flags can be BSS ("Better Save Space"), i.e., reserved for for future use. Once the get assigned a meaning, we might need to implement that. In the worst case, this becomes a flag that tells us about a newer data structure or something similar, i.e., ignoring it blindly would make us miss actual data.

The problem is that we don't know what happend, because the AOSP hasn't changed. I'd be fine with the following approach: Instead of throwing an exception, we call a new method raiseFormatViolationIssue. This method checks a flag STRICT_MODE and then either raises an exception or just prints a warning. That would ensure that we can skip over such issues when needed, but would still retain support for alerting us when the ARSC file format changes and we need to do something,

Do we know anything about the build tools that created these APKs? Maybe they are buggy, maybe they "abuse" fields for storign additional stuff, maybe they have private extensions to the file format (what for?), ...

jpstotz commented 3 months ago

Steven, what do you have in mind, an implementation of raiseFormatViolationIssue and STRICT_MODE on project level (infoflow), module level(infoflow-android), the package level soot.jimple.infoflow.android.resources or just limited to ARSCFileParser?

I don't know anything about the used build systems, but such a change on multiple apps within one week is pretty unusual.

Looking at the apps itself I don't see indications for special build systems. According to the meta data they are built using androidGradlePluginVersion versions 7.4.2/8.0.1/8.2.2 and I did not found anything unusual that those app have in common. So from my perspective these apps are just plain Java/Kotlin apps.

What all apps have in common is that they are distributed as split APKs. So Google creates the final APKs in their infrastructure. I would assume that this also applies to resources, but I am not familiar with the process what is modified by Google when building the APK files from an upload AAB.

Affected apps:

StevenArzt commented 3 months ago

I would just add this method raiseFormatViolationIssue to the ARSCFileParser class to keep it simple.

I agree that such a fix at least enables us to move forward, i.e., to handle the new APKs.

Is the code for generating the split APKs part of the AOSP or is that Google prorietary code? Maybe that could give us a clue.

jpstotz commented 3 months ago

Is the code for generating the split APKs part of the AOSP or is that Google prorietary code? Maybe that could give us a clue.

That is a good question. For end users there is bundletool which can build apk files from an Android app bundle (aab). If Google uses this tool on their servers is unknown to me.