Some confusion about EasyTaintWrapper and Options.v().set_no_bodies_for_excluded(true)

[zhouyuhao1018]

Hi ! I have some confusion about EasyTaintWrapper and Options.v().set_no_bodies_for_excluded(true) . When I analyze a very small demo app with FlowDroid, Flowdroid can't give results in a tolerable time (even if I let it run all night). Then I debug for this, and I find that the analysis is running in some androidx.* libraries which are not registered in EasyTaintWrapper txt file and also not added to the excludeList in the default instance of SootConfigForAndroid whose default list only contains these 7 : java. sun. android. org.apache. org.eclipse. soot. javax.*

However, if I add "androidx.*" in the excludeList and call

Options.v().set_exclude(excludeList);
Options.v().set_no_bodies_for_excluded(true); 

the above is implemented by creating a new SootConfigForAndroid instance and override its setSootOptions() and then call setupApplication.setSootConfig(). Then, FLowDroid can end its analysis in a tolerable time. At the same time, the call graph also becomes much much smaller.

Why the set_no_bodies_for_excluded() can lead to this, and is there a relationship between Soot excludeList and FlowDroid's EasyTaintWrapper? I mean that, when FlowDroid analysis meets a statement containing an invoke from the library which is in Soot excludeList , how it handle it? And if this invoke method is registered in EasyTaintWrapper , it will use the EasyTaintWrapper rule instead, looking like it handles a special case?

I wish your guidance!

[StevenArzt]

The exclude list is passed to Soot. It will, by default, mark the respective classes as library classes instead of application classes, although they were loaded from the APK file. Soot will load the bodies of the methods in these classes when they are accessed, e.g., there is some callgraph path through them. With set_no_bodies_for_excluded(true);, you tell Soot to never load these bodies, no matter what. The respective classes and mehtods become phantoms. This prevents, e.g., the callgraph algorithm from ever looking into them.

The EasyTaintWrapper uses a list of methods together with a heuristic to replace the analysis of individual methods with pre-defined rules. A taint wrapper (applies to all of them) can be exclusive for a method, i.e., FlowDroid will not analyze that method and solely rely on the output of the taint wrapper. If a taint wrapper is not exclusive, FlowDroid will nevertheless propagate taints through the library method. The EasyTaintWrapper is only exclusive for classes for which it has at least one method on this list.

In total, your configuration excludes all ogf androidx.*, only relying on the taint wrapper, even if the taint wrapper has not summaries for the respective class.

Still, I wonder why this is not the default:

public class SootConfigForAndroid implements IInfoflowConfig {

    @Override
    public void setSootOptions(Options options, InfoflowConfiguration config) {
        // explicitly include packages for shorter runtime:
        List<String> excludeList = new LinkedList<String>();
        excludeList.add("java.*");
        excludeList.add("javax.*");

        excludeList.add("sun.*");

        // exclude classes of android.* will cause layout class cannot be
        // loaded for layout file based callback analysis.

        // 2020-07-26 (SA): added back the exclusion, because removing it breaks
        // calls to Android SDK stubs. We need a proper test case for the layout
        // file issue and then see how to deal with it.
        excludeList.add("android.*");
        excludeList.add("androidx.*");

        excludeList.add("org.apache.*");
        excludeList.add("org.eclipse.*");
        excludeList.add("soot.*");
        options.set_exclude(excludeList);
        Options.v().set_no_bodies_for_excluded(true);
    }

}

This seems to include androidx.*.

[zhouyuhao1018]

Thanks a lot! The exclude list passed to Soot tell it that these are libraries and set_no_bodies_for_excluded(true) leads to no bodies loading for these classes. And, it may also exist some invokes from libraries in the application class, e.g. String.valueof() is a very common case, the EasyTaintWrapper will handle this. If the library method is not exclusive, FlowDroid will propagate taints through it.
And before, the excludelist for Soot doesn't have androidx.* , so that set_no_bodies_for_excluded(true) doesn't effect the bodies loading for androidx.* , leading to FlowDroid running analysis in androidx libraries and cannot giving result in time. When I change the configuration and add androidx.* , although FlowDroid propagate taints through androidx library methods, it can quickly finish since there is no bodies for that library method. Am I right ?

I use the following jar because it maybe the latest one from the web soot-infoflow-android-2.9.0.jar. In SootConfigForAndroid, the excludeList in method setSootOptions() really doesn't hava androidx.* . And I download it again today, it also doesn't have androidx.* here is what is in IDEA:

public class SootConfigForAndroid implements IInfoflowConfig {

    @Override
    public void setSootOptions(Options options, InfoflowConfiguration config) {
        // explicitly include packages for shorter runtime:
        List<String> excludeList = new LinkedList<String>();
        excludeList.add("java.*");
        excludeList.add("sun.*");

        // exclude classes of android.* will cause layout class cannot be
        // loaded for layout file based callback analysis.

        // 2020-07-26 (SA): added back the exclusion, because removing it breaks
        // calls to Android SDK stubs. We need a proper test case for the layout
        // file issue and then see how to deal with it.
        excludeList.add("android.*");

        excludeList.add("org.apache.*");
        excludeList.add("org.eclipse.*");
        excludeList.add("soot.*");
        excludeList.add("javax.*");
        options.set_exclude(excludeList);
        Options.v().set_no_bodies_for_excluded(true);
    }

}

[StevenArzt]

You're almost right. If you exclude androidx.* and disable method body loading for excluded classes, FlowDroid doesn't propagate through these methods either. From the perspective of the analysis, the method is entirely gone, there is no code that could be analyzed. To avoid false negatives, we have the taint wrapper. If the taint wrapper does not handle the method, the analysis likely looses the taint there (depends on a few factors, but that's mostly it). The basic idea is that most of these classes are not relevant for the taint analysis, and for the few that are relevant, we create a summary.

FlowDroid is now on Maven Central. You can get version 1.10 from there.

In generel, the EasyTaintWrapper applies a very simplistic (hence the name) model. You can also look at the SummaryTaintWrapper from soot-infoflow-summaries, which is much more powerful, i.e., allows for much more expressive summaries.

[zhouyuhao1018]

Thanks a lot! I will have a try with SummaryTaintWrapper.

[zhouyuhao1018]

FlowDroid is now on Maven Central. You can get version 1.10 from there.

Sorry, I think the version you want to say is 2.10? I change my project to Maven project and get FlowDroid with

    <dependencies>
        <dependency>
            <groupId>de.fraunhofer.sit.sse.flowdroid</groupId>
            <artifactId>soot-infoflow</artifactId>
            <version>2.10.0</version>
        </dependency>
        <dependency>
            <groupId>de.fraunhofer.sit.sse.flowdroid</groupId>
            <artifactId>soot-infoflow-summaries</artifactId>
            <version>2.10.0</version>
        </dependency>
        <dependency>
            <groupId>de.fraunhofer.sit.sse.flowdroid</groupId>
            <artifactId>soot-infoflow-android</artifactId>
            <version>2.10.0</version>
        </dependency>
    </dependencies>

I check the SootConfigForAndroid , however, it also doesn't has " androidx.* ", why? is something wrong?

[StevenArzt]

I meant 2.10, sorry. The exclusion entry is strange, maybe that was added later. It will definitely be there in the next release, because it is in the develop branch.

[zhouyuhao1018]

OK~I will add it manually now. Thanks a lot for your patience!

[ggggkkkrr]

Hi,I have some problems when calling flowdroid to generate the call graph of the apk, my code is as follows: `public class CGGenerator {
public final static String androidPlatformPath = "C:/Users/guokerong/AppData/Local/Android/Sdk/platforms/android-20/android.jar";
public final static String appPath = "C:/enriched1.apk";
static Object ob = new Object();

private static Map<String,Boolean> visited = new HashMap<String,Boolean>();  
private static CGExporter cge = new CGExporter();

public static void main(String[] args){  
    SetupApplication app = new SetupApplication(androidPlatformPath, appPath);       
    soot.G.reset();  

    app.setCallbackFile(CGGenerator.class.getResource("AndroidCallbacks.txt").getFile());
    app.constructCallgraph(); 

    SootMethod entryPoint = app.getDummyMainMethod();
    CallGraph cg = Scene.v().getCallGraph();
    visit(cg,entryPoint);  
    cge.exportMIG("flowdroidCFG", "C:/");  
}  `

Then the following error will be reported： [main] INFO soot.jimple.infoflow.android.SetupApplication - Initializing Soot... [main] INFO soot.jimple.infoflow.android.SetupApplication - Loading dex files... Exception in thread "main" java.lang.ExceptionInInitializerError at soot.Scene.v(Scene.java:173) at soot.jimple.infoflow.android.SetupApplication.initializeSoot(SetupApplication.java:1202) at soot.jimple.infoflow.android.SetupApplication.runInfoflow(SetupApplication.java:1456) at soot.jimple.infoflow.android.SetupApplication.constructCallgraph(SetupApplication.java:1328) at flowdroidcg.CGGenerator.main(CGGenerator.java:38) Caused by: java.lang.NullPointerException at java.base/java.io.Reader.<init>(Reader.java:168) at java.base/java.io.InputStreamReader.<init>(InputStreamReader.java:76) at soot.ModuleUtil.parseJavaBasePackage(ModuleUtil.java:216) at soot.ModuleUtil.<clinit>(ModuleUtil.java:62) ... 5 more The error seems to be in app.constructCallgraph(); can you help me?I wish your guidance!Thank you very much @StevenArzt @zhouyuhao1018