Buy and set up rootable device for testing

[kasnder]

I'd recommend acquiring a Google Pixel 5. They run Android 11+ and are rootable. Steps to root:

1. Unlock bootloader on Google Pixel. This will delete all data.
2. Install Android 11 on Google Pixel. Download script and OS here: https://developers.google.com/android/images#redfin
3. Install Magisk: https://github.com/topjohnwu/Magisk
4. Install EdExposed: https://github.com/ElderDrivers/EdXposed
5. Install JustTrustMe: https://github.com/TrackerControl/JustTrustMe/releases/tag/v.3
6. Install mitmproxy root certificate
7. Install Magisk module to trust mitmproxy module: https://github.com/NVISOsecurity/MagiskTrustUserCerts

These steps, anyway, worked for me in the past. If they don't work, we might also try an alternative to Android 11 running JustTrustMe.

What is even more robust than this is approach is using a Google Nexus 5 running Android 6. It's possible to root this version easily, then install Xposed Framework, and JustTrustMe. The added advantage of this is that Android 6 automatically trusts all user certificates. It's much more robust, but might not run all apps. It should, however, run most apps.

[SebastianZimmeck]

I'd recommend acquiring a Google Pixel 5.

That would work. @n-aggarwal and @wesley-tan, please let me know if we need a phone. If so, it will be the easiest if we agree on one and then I order it to where you are.

What is even more robust than this is approach is using a Google Nexus 5 running Android 6. It's possible to root this version easily, then install Xposed Framework, and JustTrustMe.

@wesley-tan, do you have the Google Nexus I gave you with you? It should be rooted and may have Xposed installed. So, you could try @kasnder's setup.

[n-aggarwal]

I would need a phone to perform the testing!

We can get a refurbished Google Pixel 5 for around \$140 through eBay in the United States; Google Nexus 5 costs around \$40.

Links: Google Pixel 5 eBay Google Nexus 5 eBay

[SebastianZimmeck]

How about a Pixel 6a?

Is there a particular reason why you picked a Pixel 5 over 6, @kasnder?

Possible advantages of 6 over 5:

• There is not much of a price difference between a Pixel 5a and 6a, and we would only be one generation behind
• It seems there are more known issues with the Pixel 5a than the 6a

@n-aggarwal, can you check whether @kasnder's setup can be done on a Pixel 6a? Also, can you check whether xPrivacyLua (or its successor, if there is any) can be used on a Pixel 6?

[kasnder]

Yes, because Pixel 6 doesn't run Android 11 and the above instructions don't immediately work. In any case, Android 6 and Nexus 5 might be the better call. It's MUCH easier to work with.

[SebastianZimmeck]

In any case, Android 6 and Nexus 5 might be the better call. It's MUCH easier to work with.

@n-aggarwal, can you check the general coverage of apps for target API Android 6 in Android Studio. Also, it would be good if you could spotcheck a few top apps from different categories if they still support Android 6.

@wesley-tan, do you have the Android 7 I gave you?

[kasnder]

In any case, Android 6 and Nexus 5 might be the better call. It's MUCH easier to work with.

@n-aggarwal, can you check the general coverage of apps for target API Android 6 in Android Studio. Also, it would be good if you could spotcheck a few top apps from different categories if they still support Android 6.

@wesley-tan, do you have the Android 7 I gave you?

The advantage of a Pixel is that the testing would be a bit quicker. Nexus 5 is an old lady.

[SebastianZimmeck]

The advantage of a Pixel is that the testing would be a bit quicker. Nexus 5 is an old lady.

@n-aggarwal, if you can look up how to do @kasnder's setup on a Pixel 5a (or 6a).

[n-aggarwal]

I found this list of minimum sdk version for most downloaded apps from this site. The asterisk indicates that there are multiple versions, older for lower Android versions; I tried to search for the lowest one but was unsuccessful. The play store just says that the Android sdk requirements may differ depending on device.

Application	Minimum Android version
WhatsApp	4.1
Facebook*	11
Facebook Messenger*	9
Instagram*	9
Snapchat	5
Spotify	5
UC Browser	4.1
Facebook Lite	8
Twitter	5
SHAREit	4.1
Viber	5
Netflix	7
MX Player	5
Skype	6
Telegram	6

From the above table, it seems around 75% of apps do support Android 6 as the minimum sdk version. If that is good enough then we can proceed with the nexus 5. If, however, we want 100% coverage, Pixel 5a or 6a might be the way to go.

[n-aggarwal]

As for rooting Android 11, 12 or 13, I have found several videos on how I can root the Google Pixel phones. As Konrad above mentioned, this involves unlocking the bootloader, downloading the system image, and finally installing and using Magisk. There are some minor differences between 11 and 12, 13, but nothing significant.

The problem with Android 12, 13 is that those are not supported by the EdExposed framework (it only supports Android 8 through 11 so it wouldn’t work on Android 6 either), which would mean that we will not be able to install JustTrustMe and disable certificate pinning which might be an issue. Furthermore, the Magisk Module MagiskTrustUserCerts only supports Androud versions 7 through 10.

So, from the above, I think a Pixel phone running Android 8 through 10 might be the most suitable for us. This would include:

Phone	Lowest Android Version	Highest Android Version
Original Pixel	Android 7	Android 10
Pixel XL	Android 7	Android 10
Pixel 2	Android 8	Android 11
Pixel 2 XL	Android 8	Android 11
Pixel 3	Android 9	Android 12
Pixel 3 XL	Android 9	Android 12
Pixel 3A	Android 9	Android 12
Pixel 3A XL	Android 9	Android 12
Pixel 4	Android 10	Android 13
Pixel 4XL	Android 10	Android 13
Pixel 4A	Android 10	Android 13

I think we should stick to the lower side because these phones, if we buy pre-owned, will probably have the highest Android versions installed (I am not sure of this, but I do believe this would be the case).

So the original Pixel, or Pixel XL seems to be the ideal choice. I have not yet looked into the process to root these. I will do that by tomorrow.

[kasnder]

As for rooting Android 11, 12 or 13, I have found several videos on how I can root the Google Pixel phones. As Konrad above mentioned, this involves unlocking the bootloader, downloading the system image, and finally installing and using Magisk. There are some minor differences between 11 and 12, 13, but nothing significant.

The problem with Android 12, 13 is that those are not supported by the EdExposed framework (it only supports Android 8 through 11 so it wouldn’t work on Android 6 either), which would mean that we will not be able to install JustTrustMe and disable certificate pinning which might be an issue. Furthermore, the Magisk Module MagiskTrustUserCerts only supports Androud versions 7 through 10.

So, from the above, I think a Pixel phone running Android 8 through 10 might be the most suitable for us. This would include:

Phone Lowest Android Version Highest Android Version Original Pixel Android 7 Android 10 Pixel XL Android 7 Android 10 Pixel 2 Android 8 Android 11 Pixel 2 XL Android 8 Android 11 Pixel 3 Android 9 Android 12 Pixel 3 XL Android 9 Android 12 Pixel 3A Android 9 Android 12 Pixel 3A XL Android 9 Android 12 Pixel 4 Android 10 Android 13 Pixel 4XL Android 10 Android 13 Pixel 4A Android 10 Android 13 I think we should stick to the lower side because these phones, if we buy pre-owned, will probably have the highest Android versions installed (I am not sure of this, but I do believe this would be the case).

So the original Pixel, or Pixel XL seems to be the ideal choice. I have not yet looked into the process to root these. I will do that by tomorrow.

Rooting shouldn't be a problem since you can unlock the bootloader on all of them, and then flash Magisk.

I found out that Android on 11+, you can use https://github.com/whalehub/custom-certificate-authorities to accept certificates. I have, however, not yet tried this one.

We might also consider getting two phones, since they are pretty cheap.

[SebastianZimmeck]

In addition to what @kasnder said so far, here are some basic setup steps from my end:

1. Unlock the bootloader/rooting the device
2. Install Magisk
3. Install Android in form of LineageOS (or some other Android distribution)
4. Install Open GApps to have the Play Store and other Google apps
5. Install the additional items: certificate, mitmproxy, etc.

• @n-aggarwal, maybe, you can find a tutorial on doing that or piece together some credible resources that gives us sufficient confidence that all of that can be done on a certain Pixel phone
• I hope that we can get a Pixel phone as recent as possible to just have more app coverage and no performance issues. On the other hand, we need to make sure that we can get the setup done, which is more likely on an older phone. So, there is a balance to strike.

[n-aggarwal]

1. Rooting a Phone:

   • MaowDroid (YouTube channel) has instructions for rooting Pixel devices and some other Google devices like Nexus from Android 8 through 11.
   • Youtube Walkthrough for rooting Android 12 or 13.

2. Adding MitM certificate to the trusted/system certificates:

   • For Android 6 and below: We can do this directly without rooting the phone since apps trust user certificates.
   • For Android 7 through 10: MagiskTrustUserCerts
   • For Android 11 and above: custom-certificate-authorities

3. Disabling Certificate Pinning:

   • Android 8 through 11: Install EdXposed Framework and then install the JustTrustMe module.
   • Android 11+: Using Frida as described in this guide; the custom code needed to disable certificate pinning using Frida: Project: Universal Android SSL Pinning Bypass with Frida.

So, it seems we should be able to perform the analysis on any Android version starting from 8. However, since Konrad said he hasn’t used custom-certificate-authorities, it may be best to stick to Android 10. If however we want to get a newer phone to speed up the analysis, then we can certainly go for a Pixel 5 or 6a.

Additionally, has anyone used Frida before?

[wesley-tan]

I'd recommend acquiring a Google Pixel 5.

That would work. @n-aggarwal and @wesley-tan, please let me know if we need a phone. If so, it will be the easiest if we agree on one and then I order it to where you are.

What is even more robust than this is approach is using a Google Nexus 5 running Android 6. It's possible to root this version easily, then install Xposed Framework, and JustTrustMe.

@wesley-tan, do you have the Google Nexus I gave you with you? It should be rooted and may have Xposed installed. So, you could try @kasnder's setup.

Yep I still have it, I will try running the setup on the Nexus

[kasnder]

If we use Frida, this is a comprehensive guide: https://httptoolkit.com/blog/frida-certificate-pinning/

And here is a much more comprehensive script for disabling certificate pinning: https://raw.githubusercontent.com/httptoolkit/frida-android-unpinning/main/frida-script.js

Maybe, indeed, all we need is Frida. I just haven't tested this before.

On newer Android versions, you also need to modify Chrome, as far as I understand: https://httptoolkit.com/blog/chrome-android-certificate-transparency/

[n-aggarwal]

On newer Android versions, you also need to modify Chrome, as far as I understand: https://httptoolkit.com/blog/chrome-android-certificate-transparency/

I think this only applies if a user roots their phone; right now, I am running android 13 on my pixel and I seem to have no problem decrypting traffic using PCAPdroid.

So if we want to analyze traffic from Chrome we could simply do it using a non-routed device, or alternatively install another browser.

[wesley-tan]

My experience from working with the Google Nexus is that speed of phone is indeed a factor that should be considered, can be frustrating for us if the phone is an older model. If we really need to use Android 10, we can consider using a Pixel 4 and then downgrading it (https://www.androidcentral.com/downgrade-android-help) if we really need to? The min version for pixel 5 is Android 11.

For some reason, I really can't get mitmproxy to work the Google Nexus even though I could get mitmweb and wireguard to work on two other phones I have:

• An oppo ax7 on android 8.1
• A Google S20+ on android 12 I've installed the certificates manually, tried both the mitmproxy and the wireguard method. not sure if this is because lineageOS no longer supports the phone model on the Nexus. I will try rooting the oppo ax7 on android 8.1 and going through the respective steps

[SebastianZimmeck]

We settled on a Pixel 6a.

[n-aggarwal]

I have the pixels 6A (thanks Sebastian!) and as of right now it is on the 13.0.0 (TP1A.221005.003, Oct 2022)software version. The phone is prompting me to update to the latest one: 13.0.0 (TQ2A.230505.002, May 2023) saying that it fixes "critical bugs and improves the performance and stability of the device." I was wondering if there would be any significant difference between the two versions for rooting and setting up the Magisk Modules process. I don't think there should be since they are both still android 13, but I just want to make sure.

@kasnder do you have any specific advice on this?

[kasnder]

I have the pixels 6A (thanks Sebastian!) and as of right now it is on the 13.0.0 (TP1A.221005.003, Oct 2022)software version. The phone is prompting me to update to the latest one: 13.0.0 (TQ2A.230505.002, May 2023) saying that it fixes "critical bugs and improves the performance and stability of the device." I was wondering if there would be any significant difference between the two versions for rooting and setting up the Magisk Modules process. I don't think there should be since they are both still android 13, but I just want to make sure.

@kasnder do you have any specific advice on this?

No problem. Just need to unlock bootloader which will delete all data.

[n-aggarwal]

I have successfully rooted the phone using Magisk!

[n-aggarwal]

I was able to install the mitm certificate as a root Certificate using the MagiskTrustUserCerts module.

After the certificate was installed. I downloaded the Wireguard app in the phone and set up the tunnel. Finally, I started the mitmweb in Wireguard mode and was successfully able capture the data for Cut the Rope without applying apk-mitm!

I, however, wasn't able to download the app from playstore until I turned off the Wireguard proxy.

[n-aggarwal]

I was also able to successfully apply the Frida script to disable certificate pinning in Twitter! Unfortunately, the script did not work on Instagram. That seems to be the case because Meta uses a custom SSLPinning approach than a library. So any Meta apps wouldn't work with this script.

The HTTP Toolkit Frida guide is pretty good, but there are a few minor changes needed.

Follow the instructions until after you have copied the server to the device. (For Pixel 6A, I used the frida-server-16.0.19-android-arm64.xz). Then instead of

adb root
adb shell "chmod 755 /data/local/tmp/frida-server"
adb shell "/data/local/tmp/frida-server &"

use the commands:

adb shell
# Enable root access to the device
su
$enter_your_password
# Make the server binary executable
chmod 755 /data/local/tmp/frida-server
# Start the server on your device
/data/local/tmp/frida-server &

Then follow the instructions.

Some useful commands to keep in mind are:

# shows list of process running; helpful to identify the PID for Frida server
frida-ps -U

# to kill the android Frida server
adb shell
su
kill $PID 

[n-aggarwal]

Here is a comprehensive guide on how to collect Network data once everything is set up:

# terminal window 1
mitmweb --mode wireguard

Now connect to the proxy using the wireguard app on the phone! For the next steps the phone needs to connected to the computer and USB debugging should be enabled.

# terminal 2
adb shell
su
/data/local/tmp/frida-server &

Note before going through the terminal 2 commands make sure you don't have any Frida servers already running, and if you do kill the process using the commands:

# pre terminal 2
frida-ps -U
adb shell
su
kill $PID 

# terminal 3
frida -U -l ./frida-script.js -f $TARGET_PACKAGE_NAME

make sure you are in the right directory-- the one that contains the script file!

And that's it. This should allow you to view the Network traffic from the app!

[n-aggarwal]

I will take a random set of about 10-15 apps and check whether we can see the network data for the apps, given the setup above. If everything goes well, I will close the issue; if not, the problem will be documented here.

[n-aggarwal]

App Name	Issue (if any)
Mcdonalds	SSL Pinning?
Uber	Chrome Auth
GoodRx
Candy Crush	SSL Pinning?
Cut the Rope
Twitter
Instagram	SSL Pinning
Paypal	Chrome Auth
Spotify
Amazon
Tubi
Ticketmaster	Worked okay until it opened a WebView
Wallpaper Center
Hopper

It can be seen from the above example that the two major problems are chrome. I will try and implement the solution Konrad pointed out earlier, and the other is that the Frida Script is unable to disable SSLPinning in some apps. To improve this we can either try different scripts or an alternative method would be to try LSPosed as a substitute for Xposed and then use the JustTrustMe Module.

Here is some information about the LSPosed Module:

A Riru / Zygisk module trying to provide an ART hooking framework which delivers consistent APIs with the OG Xposed, leveraging LSPlant hooking framework.

Developers are welcome to write Xposed modules with hooks based on LSPosed Framework. A module based on LSPosed framework is fully compatible with the original Xposed Framework, and vice versa, a Xposed Framework-based module will work well with LSPosed framework too.

The thing that makes LSPosed more suitable for us compared to Xposed is that LSposed supports Android versions 8 through 13 while Xposed only supports versions 8 through 11. Thus, since we are running Android 13, LSPosed may be a better option.

[n-aggarwal]

Another thing I noticed was that a lot of the Apps were asking me to make an account first, often with no ability to bypass, and most of the network data started transmitting after that. Since right now I was doing this manually, I was able to get around it, or enter credentials, but if we automate the process, it might be difficult to overcome this.

[SebastianZimmeck]

Good points, @n-aggarwal!

LSPosed seems worth a try.

Also, good point on the accounts. Could a single sign on, say, via Google credentials help that we could automate somehow? (It is probably a longshot.)

[n-aggarwal]

I am not sure if it's feasible because we would first have to somehow then identify the "Google Sign in" Button and click it before we can enter the credentials. The problem is the button will be placed in different positions, and have different attributes depending on the app. So to do, we would have to run some kind of NLP AI that can identify the button. The machine learning itself won't be too tough since we will just be looking for some kind of a "google" button, but nevertheless it seems to be outside the scope of our project.

[n-aggarwal]

@kasnder which JustTrustMe Module should I use? The Tracker Control version, or the Fuzion24 version?

It says that the Fuzion24 version is one commit ahead.

[kasnder]

I would go for the Frida approach if you can. Otherwise, the TrackerControl version comes with a ready-to-use APK, so no need to build from source.

[n-aggarwal]

I was able to intercept chrome traffic by installing the MagiskBypassCertificateTransparencyError Module!

[n-aggarwal]

I have made a few more changes in the Pixel 6A settings, as suggested by @kasnder.

• I have turned on the "Data Saver" to stop apps from sending and receiving data in the background, thus limit the noise in our data.
• I have turned off "private DNS" to make sure there is no issue with the mitm-capture/socks proxy