Explore automated testing of apps

[kasnder]

This script contains an example: https://github.com/TrackerControl/platformcontrol-android-ios-analysis/blob/main/processApk.sh

[kasnder]

Do we want to root? (probably yes) What hardware do we then need? We don't want to have our progress be held back by spending a lot of time on rooting. This is something that we can definitely do.

[SebastianZimmeck]

Does an emulator work? How can work be scaled? For the time being, we are assuming dynamic analysis, i.e., actually running the app. We could also consider static analysis, i.e., analyzing app code without running it.

[n-aggarwal]

I believe using an emulator/virtual environment will probably be better for scaling. We can then use the Android SDK Exercise Monkey to simulate running the app.

Additionally, I think we would need to root the virtual device; we can do so using rootAVD. Then we can do one of the following:

• On a device rooted with magisk, you can install the Custom Certificate Authorities module (Android 11+) or the MagiskTrustUserCerts module, and then install the hashed certificate (replace mitmproxy-ca-cert.cer with the PCAPdroid certificate name) as a system certificate. This is the suggested solution if you have magisk
• On any rooted device, you can install the certificate into the system store, by mounting the system partition as rw

This would allow us to automize the process of analyzing the apps.

Reference: https://emanuele-f.github.io/PCAPdroid/tls_decryption#34-caveats-and-possible-solutions

[kasnder]

No, an emulator does not work. Apps behave differently.

[SebastianZimmeck]

@n-aggarwal, I would try different approaches and see how they work.

[wesley-tan]

I'm still trying to figure out how to connect the rooted device to mitmproxy and wireshark - it seems to work for my normal (non-rooted) android phone but I'm not sure why this is happening for the rooted phone. I believe it has something to do with this https://docs.mitmproxy.org/stable/howto-install-system-trusted-ca-android/#instructions, I'm still trying to work it out

[wesley-tan]

I've seen online that other people have similar issues with mitmproxy, and some suggested Charles as an alternative (charlesproxy.com) does anyone have any experience with this? I can try it out

[kasnder]

I don't recommend Charles.

[kasnder]

I've never seen a difference between rooted and non-rooted in terms of traffic analysis. Might be a problem with the VPN?

[n-aggarwal]

As of right now, I am thinking about writing a shell script that can automate the following functions for a list of apps:

1. Download and install the app (perhaps from local storage)
2. Run the Frida script to disable SSL pinning
3. Run the exerciser Monkey or some other automated user simulation process
4. stop the Frida Script

This is the core functionality needed regardless of what different privacy settings we want to analyze.

Another question we want to keep in mind is how do we want to separate the network data?

• Do we want just 1 large file with all the data?
• Do we want to separate the data by App?
• Do we want to separate the data by App and privacy setting?

I think the third option might be the best one because it gives us a proper understanding of what is going on in each particular file, making the analysis a lot easier.

[kasnder]

I would keep network captures as granular as possible (i.e. by app and privacy setting). Android Monkey is often used, but I might additionally capture network traffic before and after the first UI interaction, so as to make sure that we also measure data collection without consent.

[n-aggarwal]

The updated (basic) script is now as follows:

1. Download and install the app (perhaps from local storage)
2. Start the mitm-wireguard proxy
3. Run the Frida script to disable SSL pinning
4. Stop the Frida Script
5. Save the mitm capture

We can then add other commands like the Monkey Exerciser, deleting AdId, adding GPC signal later on.

[n-aggarwal]

I have been working on a script file to automate the testing process. One issue I am having is that the Wireguard MITM-Proxy intermittently gives the following output: [22:10:56.473] No current session for incoming WireGuard packet: Wait for the next session handshake or reconnect your client.

I have attached some snippets of my code below:

    # WITH ADID
    TYPE="_ADID"
    ( source $SCRIPT_PATH $TARGET_PACKAGE_NAME $TYPE)
    echo "ADID DONE!"
    sleep 2

    # WITH GPC
    TYPE="_ADID_GPC"
    echo "ADID+GPC STARTED!"
    sleep 2
    ( source $SCRIPT_PATH $TARGET_PACKAGE_NAME $TYPE)
    echo "ADID+GPC DONE!"
    sleep 2

    # DELETED ADID
    adb shell am start -n com.google.android.gms/.ads.settings.AdsSettingsActivity
    sleep 1
    adb shell input tap 763 538
    sleep 1
    adb shell input tap 890 1392
    sleep 1

    TYPE="_NO_ADID"
    echo "NO ADID STARTED!"
    sleep 2
    ( source $SCRIPT_PATH $TARGET_PACKAGE_NAME $TYPE)
    echo "NO ADID DONE!"
    sleep 2

where the $SCRIPT_PATH file contains the following code regarding the proxy:

# Start MITM-Wireguard
if [ "$TYPE" == "_ADID_GPC" ] || [ "$TYPE" == "_NO_ADID_GPC" ]; then
  mitmdump --mode wireguard -s $MITM_SCRIPT_PATH --showhost -w $MITM_PATH &
  MITM_PID=$!
  echo "MITM-Proxy started"
  sleep 2
else 
  mitmdump --mode wireguard --showhost -w $MITM_PATH &
  MITM_PID=$!
  echo "MITM-Proxy started"
  sleep 2
fi

# End MITM-Wireguard
killwait $MITM_PID

The killwait function is from the processApk.sh script file.

So, what happens is that the proxy/wireguard works just fine for say WITH ADID but doesn't collect any data for ADID+GPC because of the issue with handshake. I have noticed that the frequency of this error decreases if I increase the time between the successive proxy-captures, but I am not sure what is it in the first place.

[SebastianZimmeck]

Wait for the next session handshake or reconnect your client.

I have noticed that the frequency of this error decreases if I increase the time between the successive proxy-captures, but I am not sure what is it in the first place.

I do not know a concrete solution to the problem. Just from the error message it could be that no data is coming through because the secret key could not be exchanged to establish the secure connection (handshake).

As you say, maybe, it is a timing issue. Is it possible to write your script such that you give it more time for the handshake to occur (along the lines of setTimeout() or an asynchronous call for JS)?

[n-aggarwal]

I don't know if I can manually set the time for handshake. The proxy, the handshake, and everything else happens in one command.

  mitmdump --mode wireguard --showhost -w $MITM_PATH &

I think the main issue is recognizing when this error is occurring. I don't know a way to recognize when this is happening in my script (of course I can tell when I see the output). If I can find a way to "know" when this is happening I can write a mitmproxy function along the following lines:

do {
# start mitmproxy
# sleep for a certain amount of time
# check for handshake error, if error- close the connection
} while (!error)

[kasnder]

Alternative: socks5 mode of mitmproxy, and also set port, e.g. 8888

1. Install TrackerControl (original version, e.g. from F-Droid or GitHub)
2. Enable monitoring of system apps in the TrackerControl "Advanced Settings"
3. Enable logging to ADB from TrackerControl "Advanced Settings", which should also disable any blocking of tracking.
4. Set up SOCKS proxy in the TrackerControl settings, by pointing it to the IP address of your laptop and the selected port
5. disable checking for updates in TrackerControl
6. Optional: dump adb logcat and extract TC logs, then reset adb logcat for testing next app.

[n-aggarwal]

Enable monitoring of system apps in the TrackerControl "Advanced Settings"

Is that the "Manage System Apps" option?

[kasnder]

Yes!

[n-aggarwal]

I have uploaded several new script file to issue-68 branch.

The basic-automation-script starts mitm-proxy in SOCKS5 mode, runs a Frida script on the desired app, and then terminates the Frida script and the mitm-proxy.

The single-app-automation-script runs the basic-automation-script multiple times changing various settings such as deleting the ADID, sending the GPC signal, and/or force quitting the app. Additionally the script stores the package-name and the associated adid in a file called APP_ADID.txt. This is done because every app, a new adid is generated because the previous one is deleted while testing the previous app.

The multi-app-automation-script runs the single-app-automation-script over a list of package names given in txt format.

Finally, I also uploaded the HTTPToolKit Frida Unpinning script.

[n-aggarwal]

Unfortunately for some reason TrackerControl VPN/Proxy was not working for me. So instead I used the Socksdorid app.

[kasnder]

Am super glad you figured out a solution via SOCKS. Let's chase some apps soon.

[kasnder]

Looking at the script, it might be good to clear app data and cache between the tests. From what I read, the following command can accomplish this: adb shell pm clear packageName

[kasnder]

The command adb install-multiple might be helpful for installing spit apks (i.e. unzipped *.xapk files).

[n-aggarwal]

Is there any way I can prevent the Monkey UI Exerciser from touching status bar? In one of my test runs it brought down the status bar, and then turned off the internet! Of course this had the effect of stopping all future captures in that test run.

Currently this is my command:

adb shell monkey -p $TARGET_PACKAGE_NAME --throttle 30 --pct-syskeys 0 --pct-appswitch 20 -v 100

[SebastianZimmeck]

I am not sure. @wesley-tan, can you also look into that as well? That must be a problem that essentially every Monkey UI Exerciser implementer must have.

[n-aggarwal]

One workaround is to use App Pinning. But then this reduces the problem to finding a way to pin and unpin an app using adb, and as of right now I have not been able to find a solution to that.

[SebastianZimmeck]

We need to resolve #74 and then we are probably good to close this issue.

[wesley-tan]

Just for reference: the commands --pct-touch 100 --pct-syskeys 0 or --pct-touch 100 --pct-motion 0 --pct-trackball 0 --pct-syskeys 0 --pct-nav 0 --pct-majornav 0 --pct-appswitch 0 --pct-flip 0 --pct-anyevent 0 --pct-pinchzoom 0 --pct-permission 0 for the adb Exercise Monkey do work.

[SebastianZimmeck]

Here is also a Google Play downloader with API I used for downloading 1M APKs. So, this can be automated. (Not sure if it is still maintained.)

[SebastianZimmeck]

We decided generally against. Depending on time, @wesley-tan may look into some more apps. But the default we are not going to use the Monkey UI Exerciser.

[SebastianZimmeck]

We do not need a dedicated test set to make sure our techniques are working. There is very little room for error. Once we have the pipeline ready, we can go ahead. We can start with 50 apps, look at that, then continue at a larger scale.

[SebastianZimmeck]

@n-aggarwal will take another look at the PETS paper whether we can learn something from them.

The Solitude app does is not relevant in our context.

[n-aggarwal]

Here is some information from the PETS paper that we may be able to use:

1. They use tcpdump in addition to mitmproxy to capture all network data that mitmproxy doesn't capture.

   We route all mobile device traffic to an instance of mitmproxy [9] using iptables rules, and log all traffic with tcpdump. Using mitmproxy, the Gateway intercepts all mobile-device traffic. In combination with interception, we inject into each webpage a tripwire.js (in- spired by Reis et al. [111]), which is a piece of JavaScript code (≈100 LOC) that collects the Document Object Model (DOM) of content delivered to the browser. We develop a Python mitmproxy add-on to perform these actions and use tcpdump to collect traffic that mitmproxy does not handle.

2. It also seems to be the case that they are factory-resetting the device after every test (I have highlighted the part that says that). This seems a bit excessive:

   Handling dynamic web content. To build our cache of site content, we load all the selected webpages using a baseline browser. 2 Even with cached content, some webpages include non- deterministic and dynamic content (e.g., a timestamp in the HTML source or ads resulting from real-time bidding processes [14, 131]) that leads to different requests over time and impedes comparing the same websites across browsers at different times. To address this challenge, we heuristically chose to rebuild the cache periodi- cally (once every 50 browsers we test, which corresponds to about once every 2-3 hours). For each (non-baseline) browser test, our testbed prepares the mobile device by factory resetting it, installs necessary components, and collects identifiers (AdID and Android ID). Moreover, to ensure consistency in webpages delivered, the Gateway caches copies of tested websites and replays captured content using mitmproxy.

[SebastianZimmeck]

They use tcpdump in addition to mitmproxy to capture all network data that mitmproxy doesn't capture

As far as I am concerned, our analysis is good as is, but if you think that it would help, feel free to explore tcpdump, @n-aggarwal.

This seems a bit excessive

Yes, definitely. For our purposes it is not necessary.

[n-aggarwal]

I think it would take a lot of time and effort to make sense of the tcpdump, with no obvious advantage. It would give us access to requests other than HTTP/HTTP2 for eg. DNS, TLS, etc. but the information in those packets would not be very useful for our purposes. So I think we should leave our analysis as is, and focus on other issues.