IzzySoft
commented
3 years ago @christianrowlands did you find any hints? The non-root version has the same tracker included, by the way.
Open IzzySoft opened 3 years ago
IzzySoft
commented
3 years ago @christianrowlands did you find any hints? The non-root version has the same tracker included, by the way.
christianrowlands
commented
3 years ago Sorry, I missed this one somehow. I think you are right, last time it came in with the protobuf library. I will take a look and see if enabling minification fixes it.
IzzySoft
commented
3 years ago Ah, that one again – now that you mention it I remember. Especially in the context of minification. Thanks for taking care! I happily check a "test compile" for you if you want me to (can be a naked/unsigned one, I'd just run my scanner over it).
How typically Google. Same number they played when forcing people to switch from GCM to FCM (one had to explicitly disable analytics). But then, Flutter folks are no better, requiring you to explicitly specify -noanalytics…
MuntashirAkon
commented
3 years ago But then, Flutter folks are no better, requiring you to explicitly specify
-noanalytics…
This was the first thing I've noticed while trying out Flutter. I'm guessing they've kept this switch to discourage any attempts to build a tracker-free alternative (other than the legal issues that might cost them a few bucks).
IzzySoft
commented
3 years ago They might sell it using different phrasing – but essentially what counts is the result: it's not "privacy-first". Though for me, that's not the only issue I have with Flutter apps. There's another one that's literally big: try a "Hello World" app in Flutter. If you'd manage keeping the resulting APK below 1 MB while still being cross-platform, I'd be very surprised. If you don't manage getting it below 5 MB, I'd be not surprised at all… But we're getting off-topic (or rather, I do :see_no_evil:)
christianrowlands
commented
3 years ago I was able to exclude the audit_log.proto file using exclude("**/audit_log.proto"). However, I am not sure that it removes the full library that your apk scanner is picking up. It is likely that I need to exclude all the classes from that library that is getting packaged with the apk, but before I do that can you try to scan these two apks and see what you come up with? I had to add .zip to the files because it would not let me attach an APK.
This first one I enable minify. networksurvey-debug-minify-enabled-1.1.0-SNAPSHOT.apk.zip
This one I used the exclude for the audit_log.proto networksurvey-debug-no-audit-proto-1.1.0-SNAPSHOT.apk.zip
christianrowlands
commented
3 years ago And if either of those options work, I will update both Network Survey and Network Survey Rooted
IzzySoft
commented
3 years ago networksurvey-debug-minify-enabled-1.1.0-SNAPSHOT.apk: Clean \o/networksurvey-debug-no-audit-proto-1.1.0-SNAPSHOT.apk: Dirty /o\So use minify, as we initially guessed. Let me know when I shall replace which APK (or a new release was made). Thanks!
IzzySoft
commented
3 years ago PS: What other differences are there? Scanner output looks quite much different. Maybe minify also used obfuscation?
networksurvey-debug-minify-enabled-1.1.0-SNAPSHOT.apk:
Libraries detected:
-------------------
* Android Support v4 (/android/support/v4): Development Framework
* AppCompat (/androidx/appcompat): Utility
* Constraint Layout Library (/androidx/constraintlayout): Utility
* Androidx Core (/androidx/core): Utility
* Lifecycle (/androidx/lifecycle): Utility
* Media (/androidx/media): Utility
* Navigation (/androidx/navigation): Utility
* Preference (/androidx/preference): Utility
* Google Material Design (/com/google/android/material): Utility
* Eclipse Paho Android Service (/org/eclipse/paho/android/service): Utility
* SQLite JDBC Driver (/org/sqlite): Utility
No offending libs found.networksurvey-debug-no-audit-proto-1.1.0-SNAPSHOT.apk:
Libraries detected:
-------------------
* Android Support v4 (/android/support/v4): Development Framework
* Arch (/androidx/arch): Utility
* AppCompat (/androidx/appcompat): Utility
* Constraint Layout Library (/androidx/constraintlayout): Utility
* Androidx Core (/androidx/core): Utility
* Documentfile (/androidx/documentfile): UI Component
* androidx.legacy (/androidx/legacy): Utility
* Lifecycle (/androidx/lifecycle): Utility
* Loader (/androidx/loader): Utility
* Media (/androidx/media): Utility
* Navigation (/androidx/navigation): Utility
* Preference (/androidx/preference): Utility
* Print (/androidx/print): Utility
* Transition (/androidx/transition): UI Component
* Vectordrawable (/androidx/vectordrawable): UI Component
* PNGJ (/ar/com/hjg/pngj): Utility
* network-survey-messaging (/com/craxiom/messaging): Utility
* Fasterxml (/com/fasterxml): Utility
* Google Material Design (/com/google/android/material): Utility
* Cloud Audit Logs (/com/google/cloud/audit): Mobile Analytics Tracking
* Google Core Libraries for Java 6+ (/com/google/common): Utility
* Error Prone (/com/google/errorprone): Utility
* Google Gson (/com/google/gson): Utility
* J2ObjC (/com/google/j2objc): Utility
* Google Protocol Buffers (/com/google/protobuf): Utility
* ormlite (/com/j256/ormlite): Utility
* OkHttp (/com/squareup/okhttp): Utility
* gRPC-Java (/io/grpc): Utility
* PerfMark (/io/perfmark): Utility
* JavaX Annotation API (/javax/annotation): Utility
* Kotlin (/kotlin): Utility
* GeoPackage Java (/mil/nga/geopackage): Utility
* OGC API Features JSON Lib (/mil/nga/oapi): Utility
* Simple Features Java (/mil/nga/sf): Utility
* TIFF Java (/mil/nga/tiff): Utility
* OkHttp okio Framework (/okio): Utility
* Checker Framework (/org/checkerframework): Utility
* MojoHaus AnimalSniffer Maven Plugin (/org/codehaus/mojo/animal_sniffer): Utility
* Eclipse Paho Android Service (/org/eclipse/paho/android/service): Utility
* Paho Java Client (/org/eclipse/paho/client/mqttv3): Utility
* IntelliJ IDEA (/org/intellij): Utility
* Proj4J (/org/locationtech/proj4j): Utility
* SQLite JDBC Driver (/org/sqlite): Utility
* Timber (/timber/log): Utility
Offending libs:
---------------
* Cloud Audit Logs (/com/google/cloud/audit): Tracking
1 offenders.Checking the output generated by apktool d <apk_file> confirms my suspicion: the first (smaller) APK has quite a lot of obfuscated code, the second (larger) APK has none. Can you please try minify with obfuscation disabled?
PPS: Funny. Found Cloud Audit in the first APK as well, just not in Smali: unknown/google/cloud/audit. So looks like minify either keeps a log of what it removed – or that is the trace to what was obfuscated. Need to check what the docs/issues/wiki/whatever say on this.
- unknown = Files / folders that are not part of the standard AOSP build procedure. These files will be injected back into the rebuilt APK.
If you check what apktool d networksurvey-debug-minify-enabled-1.1.0-SNAPSHOT.apk places in the unknown directory: can you confirm all those classes are used by the app (and thus should be expected to be found)? That would be important to me, as I'd then need to update my scanner to take that place into consideration, too. Doing so, what's shown in addition to the above is
* network-survey-messaging (/com/craxiom/messaging): Utility
* ormlite (/com/j256/ormlite): Utility
* Paho Java Client (/org/eclipse/paho/client/mqttv3): UtilityThose are used by Network Survey, and should be detected in the first APK, right?
christianrowlands
commented
3 years ago It seems that everything in the unknown folder for the minify apk is supporting files, and not actually any code. For example, the unknown/com/craxiom/messaging directory contains all the protobuf definitions I have defined for the wireless protocol message formats. For some of the other files in unknown, they are csv, properties, etc. They all seem to be supporting files for the code and not code itself. Given that, I don't see how they could be used for tracking purposes. As to if they are all used, I can't say for sure, but I don't think so. There are several for libraries that I can't imagine are being used by any code paths I cause... that being said it is hard to know all the code paths for the libraries that are used.
Here is a version with minify enabled but obfuscation off: networksurvey-debug-minify-dontobfuscate-1.1.0-SNAPSHOT.apk.zip
I am curious if it finds the google audit tracker in there since it has minify enabled but the code is not obfuscated.
IzzySoft
commented
3 years ago It seems that everything in the unknown folder for the minify apk is supporting files, and not actually any code.
Ah, that makes sense somehow. I wasn't sure how to understand the phrasing in that documentation ("folders that are not part of the standard AOSP build procedure" or "folders that are not part of the standard AOSP build procedure") – if it's only "supporting files", the latter emphasis must be how to read it. So just to clarify: those "additional" 3 libraries are used by Network Survey or not? My guess is minify removed their code, but not the "supporting files"; for ormlite it's e.g. only the license files, for Paho some properties, and the 3rd one you already outlined.
it is hard to know all the code paths for the libraries that are used.
Dependencies of dependencies of your dependencies and all that, yes. What basically means that even the developers themselves cannot tell what code they use – which is somehow worrying.
Here is a version with minify enabled but obfuscation off:
Output including the unknown folder scanned:
Libraries detected:
-------------------
* Android Support v4 (/android/support/v4): Development Framework
* Arch (/androidx/arch): Utility
* AppCompat (/androidx/appcompat): Utility
* Constraint Layout Library (/androidx/constraintlayout): Utility
* Androidx Core (/androidx/core): Utility
* Lifecycle (/androidx/lifecycle): Utility
* Loader (/androidx/loader): Utility
* Media (/androidx/media): Utility
* Navigation (/androidx/navigation): Utility
* Preference (/androidx/preference): Utility
* Transition (/androidx/transition): UI Component
* Vectordrawable (/androidx/vectordrawable): UI Component
* network-survey-messaging (/com/craxiom/messaging): Utility
* Fasterxml (/com/fasterxml): Utility
* Google Material Design (/com/google/android/material): Utility
* Google Core Libraries for Java 6+ (/com/google/common): Utility
* Google Gson (/com/google/gson): Utility
* Google Protocol Buffers (/com/google/protobuf): Utility
* ormlite (/com/j256/ormlite): Utility
* OkHttp (/com/squareup/okhttp): Utility
* gRPC-Java (/io/grpc): Utility
* PerfMark (/io/perfmark): Utility
* Kotlin (/kotlin): Utility
* GeoPackage Java (/mil/nga/geopackage): Utility
* Simple Features Java (/mil/nga/sf): Utility
* OkHttp okio Framework (/okio): Utility
* Eclipse Paho Android Service (/org/eclipse/paho/android/service): Utility
* Paho Java Client (/org/eclipse/paho/client/mqttv3): Utility
* Proj4J (/org/locationtech/proj4j): Utility
* SQLite JDBC Driver (/org/sqlite): Utility
* Timber (/timber/log): Utility
No offending libs found.So no Cloud Audit detected (even unknown has just a single .proto file of it – which again confirms my above suspicion of minify forgetting to remove the "supporting files"; removing them would have saved another ~200k here). I've checked the files inside unknown and can confirm your analysis: no code, just supporting files like *.proto or *.csv etc. Thanks for pointing that out – so I do not need to apply that "test change" (just kept it on for our case here, it was never committed and will be rolled-back now). I've also checked smali/* and found no more obfuscating.
So if you agree, I'd say that would be the setup to use: minify on, obfuscate off. Please give me a nudge once the next release using that was published, so I cross-check ASAP and remove the AntiFeature from both apps' listings.
Thanks a lot! Your analysis really helped me to 1) some deeper understanding of the structures Apktool creates, 2) ensuring/confirming I use the proper ones for my library scanner and 3) getting some insights to the tricks Proguard plays and how to optimize them for privacy and transparency! That's what I'll recommend developers from now on: use minify and disable obfuscation :smiley:
christianrowlands
commented
3 years ago Yes, those three libraries are used by Network Survey. The messaging library contains the cellular, wifi, gnss, and bluetooth record definitions. The eclipse paho library is used for the MQTT connection, and the ormlite library is a transitive dependency needed for logging the wireless protocol records to the SQL lite database file.
As for this comment:
What basically means that even the developers themselves cannot tell what code they use – which is somehow worrying
You are 100% correct. And even worse that is how easy it is to do things without a user knowing. Once you give an app camera permissions, they can take photos at random times and ship that off over an Internet connection all without the user ever knowing. This includes both the rear facing and forward facing camera 😨 . And the amount of data that can be pulled even with just a basic app... yikes. Developing Android apps has made me remove all the unnecessary apps on my phone, it is just too easy for app developers to pull all sorts of information.
Yep, I agree, minify on, obfuscation off. I don't plan on doing a release anytime soon, so it might be a couple weeks, but I will reach out when I do. I will leave this issue open until I post the release and you can check it.
I agree, it was an interesting exercise to run through!
IzzySoft
commented
3 years ago Yes, those three libraries are used by Network Survey.
Funny then they only get reported if I also scan unknown/ – so only assets of them are found but not the actual code. Still, would I include unknown/ it might lead to a lot of "false positives" (where Proguard removed the code but not the assets), so I better not add that to the scanner.
Once you give an app camera permissions, they can take photos at random times and ship that off over an Internet connection all without the user ever knowing.
And actually even without requiring the INTERNET permission itself, utilizing intents of other apps I guess?
And the amount of data that can be pulled even with just a basic app... yikes.
I remember several years ago there were a few apps showing what details could be pulled with or without certain permissions – simply accessing all details and displaying them on-screen. I made a test then, explicitly blocking all permissions to make sure it didn't fool me. Yikes, indeed…
Unfortunately, those apps are no longer available. Might be a good idea for a new project in the privacy sector. To give you an idea, one of those apps was permission.READ_PHONE_STATE (link goes to Aptoide), a second one permission.READ_SMS (ApkTom; not sure how safe those downloads would be), both from the same author. He seems to have had several of those apps: I just ducked the latter and found mentionings of DispWifiInfo and others. Should you go for it, I happily take the app(s) into my repo!
… has made me remove all the unnecessary apps on my phone
One of my core advices: do not install apps you find interesting, but only those you cannot do without. More often than not you won't need them. I never understand folks having hundreds of "extra apps" (i.e. not pre-installed ones) on their devices.
Yep, I agree, minify on, obfuscation off. I don't plan on doing a release anytime soon, so it might be a couple weeks, but I will reach out when I do. I will leave this issue open until I post the release and you can check it.
Yes please :smiley:
I agree, it was an interesting exercise to run through!
Oh yeah! And I'm very thankful you did that with me! Couldn't have done it without you.
christianrowlands
commented
3 years ago So I did some more testing and it appears turning on minify broke some of the logging logic I am using. I tried to add -keep settings to the proguard rules file, but I could not get it to keep everything I needed. I am sure I am doing something wrong there so I will need to dig in more when I have time. For now I am going to turn off minify until I can get it working correctly. It seems minify is not quite smart enough to know what is used and what is not used.
IzzySoft
commented
3 years ago Ouch. Good you caught that before letting it free. Yes, of course: it's ready when it's ready, not before. Thanks for your efforts!
PS: would we build this at F-Droid, we'd simply remove it before the build starts. We've got special commands for that in our build recipes. Simplified: "do a checkout, remove unwanted stuff, build". Might also be a viable approach here, if all else fails…
Did this tracker sneak in without intention?
I didn't find it referenced directly from your
build.gradle, so I thought some dependency might have drawn it in and you missed it (IIRC Google's ProtoBuf recently started that, and you are using that – and again IIRC enabling minification, which you've explicitly disabled, would kick it out). An app with root powers having proprietary tracking inside is a bit scary :wink: