search

FoelliX / CoDiDroid

Cooperative and Distributed Android App Analysis
https://FoelliX.github.io/CoDiDroid
GNU General Public License v3.0
5 stars 0 forks source link

Is there a guide/wiki on how to run CoDiDroid? #1

Closed zjbthomas closed 3 years ago

zjbthomas commented 3 years ago

Hi FoelliX,

Thanks for your help and now I can successfully run AQL-System to output static analysis results. However, I want to further use CoDiDroid but I am wondering if there is a guide/wiki on how to run it?

Specifically, I have the following questions:

  1. Questions on setting up CoDiDroid:

    • To run CoDiDroid, does it mean that I need to manually create two AQL-WebServices? Are there existing config.propertiess that can be used, so these WebServices work as expected in CoDiDroid?
    • I believe I can follow config_example.xml to configure my AQL-System, so it can be connected to the two AQL-WebServices. Is this idea correct?
    • In Add Tools wiki page of the AQL-System, it says that "DIALDroid (If you plan to use this converter, please adapt data/converter/dialdroid_config.properties)". May I know what exactly this sentence mean and what should I do to adapt DIALDroid?
    • May I know what is the usage of AQLCheckOperator?
    • Is there a place to download the jars of tools you listed in Table 1 of the ESEC/FSE 2019 paper? Especially for NOAH and PIM that are newly developed?

  2. Questions on configurating and running CoDiDroid:

    • How can I configure the sources and sinks used in different tools? Further, is there a way to change the configuration files of tools (e.g., easy taint wrapper list of FlowDroid)?
    • Is it correct that Flows IN App('A.apk') ? only finds intra-component flows in A.apk? If I want to ALSO find ICC flows in A.apk, do I need to use the following query?
MATCH [
    Flows IN App('A.apk') ?,
    IntentSources IN App('A.apk') ?,
    IntentSinks IN App('A.apk') ?
]

Thank you in advance!

FoelliX commented 3 years ago

Hi Junbin,

first of all, to get started with CoDiDroid, let me suggest to checkout the paper's artifact (https://dl.acm.org/do/10.1145/3345841/full).

Now let me try to answer your questions:

  1. Questions on setting up CoDiDroid:

    • To run CoDiDroid, does it mean that I need to manually create two AQL-WebServices? Are there existing config.propertiess that can be used, so these WebServices work as expected in CoDiDroid?
    • Answer: To use CoDiDroid with all the tools it comprises, you must setup all tools in any accessible AQL-System (e.g. an AQL-WebService). Considering the CoDiDroid study we setup one AQL-WebService to execute PIM, since PIM is a dynamic tool it requires an Android emulator or device. To efficiently use an emulator, hardware support is required which is typically not available in a virtual machine. Another AQL-WebService was setup for all other tools (DroidRA, NOAH, IC3, FlowDroid) on a Linux virtual machine with more computational power (e.g. >= 32GB memory). As front-end BREW (https://github.com/FoelliX/BREW) was used in most cases.
    • I believe I can follow config_example.xml to configure my AQL-System, so it can be connected to the two AQL-WebServices. Is this idea correct?
    • Answer: Yes, or follow this https://github.com/FoelliX/AQL-System/wiki/Configuration_Upgrades Wiki entry. Also tool support in form of a GUI is available. Just launch the AQL-System with one of the following parameters: -configwizard, -cw. Or access the Configuration-Wizard via the AQL-System's GUI (Help → Configuration Wizard).
    • In Add Tools wiki page of the AQL-System, it says that "DIALDroid (If you plan to use this converter, please adapt data/converter/dialdroid_config.properties)". May I know what exactly this sentence mean and what should I do to adapt DIALDroid?
    • Answer: First, DIALDroid is not required by CoDiDroid. To setup DIALDroid follow the developer's instructions: https://github.com/dialdroid-android/DIALDroid. DIALDroid stores its results in a database. To parse these results this database must be setup. In order to access this database the AQL-System needs the required information, which are stored in the mentioned file (https://github.com/FoelliX/AQL-System/blob/master/data/converter/dialdroid_config.properties).
    • May I know what is the usage of AQLCheckOperator?
    • Answer: We attempted to use HornDroid, to verify flows found by FlowDroid for example. The CHECK operator was used to do so automatically. More details can be found in the study (Strategy IV).
    • Is there a place to download the jars of tools you listed in Table 1 of the ESEC/FSE 2019 paper? Especially for NOAH and PIM that are newly developed?
    • Answer: Yes, the releases used are available on github: https://github.com/FoelliX/NOAH/releases & https://github.com/FoelliX/PIM/releases. There are also tool-(download-)links in the paper.

  2. Questions on configurating and running CoDiDroid:

    • How can I configure the sources and sinks used in different tools? Further, is there a way to change the configuration files of tools (e.g., easy taint wrapper list of FlowDroid)?
    • Answer: I guess your question aims at changing those via an AQL-Query? With the current release this is not possible, yet. You may could do so by attaching scripts in the configuration (see <runOnEntry>, <runOnExit>, <runOnSuccess>, <runOnFail> and <runOnAbort> https://github.com/FoelliX/AQL-System/wiki/Configuration).
    • Is it correct that Flows IN App('A.apk') ? only finds intra-component flows in A.apk? If I want to ALSO find ICC flows in A.apk, do I need to use the following query?
    • Answer: Depends on the configured tool. Basically, it can only find intra-app flows since only one app is provided. Yes, this query (with CoDiDroid setup as in the study) will allow to find ICC-related flows. However, with CoDiDroid we introduced configurable rules to BREW (https://github.com/FoelliX/BREW/wiki/Improvements) which automatically transform queries such as Flows FROM App('A.apk') TO App('B.apk') ? into queries as you mentioned it.

Hope it helps!

Cheers,
FoelliX

zjbthomas commented 3 years ago

Hi @FoelliX , thanks for the reply!

I decided to use a combinations of DroidRA, FlowDroid, HornDroid, IC3, NOAH, PIM as stated in the ESEC/FSE 19 paper. Currently we don't want to handle IAC related flows.

I still have two questions related to ICC:

  1. I double-checked that for MATCH, it requires PIM to take effect. Does it mean that without PIM (since it is dynamic and we also want to test a version that is totally static), the combination I used is not able to detect ICC flows within one app?
  2. For using PIM, the instructions provided in the paper's artifact is based on a real device. Is it possible for me to use a Android Emulator?

Thank you!

FoelliX commented 3 years ago

Hi Junbin (@zjbthomas),

  1. You can use the default AQL operator CONNECT instead of MATCH (PIM). However, it will be less accurate, since it only checks whether the action and category of an intent (IntentSink) and an intent-filter (IntentSource) match exactly. On a match it assumes that the intent can be "received" by the intent filter. PIM in contrast creates both, intent and intent filter, on an Android device to check if the intent can be "received" by the intent filter.

  2. Yes, any Android device can be used. The artifact comes as a Linux virtual machine and since an emulator could not be run efficiently inside a virtual machine, we decided to make a physical device mandatory regarding the artifact. Android emulators typically cannot be run WITH hardware support inside a virtual machine. WITHOUT hardware support such emulators are very slow.

Happy to help!

Cheers, FoelliX

zjbthomas commented 3 years ago

Hi @FoelliX thanks!

Here are two follow-up questions:

  1. I want to try different versions of FlowDroid. With the same query, I noticed that FlowDroid in folder "old" is able to generate an xml with the expected flow. Though, when I changed to the one in folder "271" (I just simply replace all old to 271), inside the folder "results" under "271", I can see a correct log file with the same flow. However, the xml file is now empty like the following. I am wondering is it because the results from 271 are not parsed to xml?
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<answer>
    <flows/>
</answer>
  1. I noticed the PIM is in the same WebService in your artifact. Does it really need to run it in another WebService even it is a dynamic tool?
FoelliX commented 3 years ago

Hi Junbin (@zjbthomas),

  1. You can switch the versions like that but use "FlowDroid" as name for the old version and "FlowDroid2" as name for the new version in the associated AQL-System/WebService's config file then.

  2. No, it can be run in any AQL-System implementation (including BREW, AQL-WebService, ...). If you are able to execute all tools on the same machine, there is no need to setup multiple AQL-System/WebServices.

Cheers, FoelliX

zjbthomas commented 3 years ago

Hi @FoelliX ,

  1. I am still not sure how CONNECT works with IC3 to detect ICC flows inside one app.

Take InterComponentCommunication/ActivityCommunication3 in DroidBench as an example. I used the following query. The IC3 I used is the one inside /tools/IC3/new/

CHECK [
        CONNECT [
            Flows IN App('/path/to/ActivityCommunication3' | 'DEOBFUSCATE') ?,
            IntentSinks IN App('/path/to/ActivityCommunication3' | 'DEOBFUSCATE') ?,
            IntentSources IN App('/path/to/ActivityCommunication3' | 'DEOBFUSCATE') ?
        ],
        CONNECT [
            Flows IN App('/path/to/ActivityCommunication3' | 'DEOBFUSCATE') USES 'HornDroid' ?,
            IntentSinks IN App('/path/to/ActivityCommunication3' | 'DEOBFUSCATE') ?,
            IntentSources IN App('/path/to/ActivityCommunication3' | 'DEOBFUSCATE') ?
        ]
]

I compared my results (attached below) with the one in RQ1_results (/BREW/data/storage/71fbbee247b9723506e83a00350f7e2bc30e94b3a86d3d6b1e9fbe9b44e5b214.xml). Inside the tag, it seems like mine missing those with statement getIntent(). The other look like the same.

InterComponentCommunication.ActivityCommunication3.zip

  1. As now FlowDroid is embedded with IccTA, I am also wondering if there is possible to enable IccTA in FlowDroid. That is to say, is it possible to feed the output of IC3 into FlowDroid in AQL-System? If it is possible, theoretically, I believe the precision should be: CONNECT < IccTA < PIM. Does this make sense?

Thank you in advanced.

FoelliX commented 3 years ago

Hi @zjbthomas,

  1. The flows you are missing were added by PIM. Thus, they will not be found when using the CONNECT operator. I added a description that explains how to make PIM find these additional IntentSources (see "PIM together with IC3" here: https://github.com/FoelliX/PIM)

    • Yes, IccTA in FlowDroid can be triggered, however, up to my knowledge it does not work well, yet (related: https://github.com/secure-software-engineering/FlowDroid/issues/219).
    • To "feed the output of IC3 into FlowDroid in AQL-System" currently scripts must be added either for running IC3/FlowDroid or as <runOnEntry> / <runOnExit> scripts (see https://github.com/FoelliX/AQL-System/wiki/Configuration). The newer AQL-System version which is still in development and not released yet will allow to make this "feeding" process more comfortable.
    • "If it is possible, theoretically, I believe the precision should be: CONNECT < IccTA < PIM. Does this make sense?"
      Yes, it does.

Cheers, FoelliX