Exodus tracker finds GMS classes

[licaon-kter]

Last time we did this we were looking at apktool and decompiled classes, fixed in https://github.com/deltachat/deltachat-android/pull/2303

F-Droid now integrates (but not yet enforces) Exodus trackers scanning capabilities, and we scanned the current repo and we're in the middle of a clean up.

See long thread here: https://gitlab.com/fdroid/fdroidserver/-/issues/1004

In this scan DeltaChat popped up again but Element did not (!!)

$fdroid scanner --verbose --exodus com.b44t.messenger_632.apk
DEBUG: Starting new HTTPS connection (1): reports.exodus-privacy.eu.org:443
DEBUG: https://reports.exodus-privacy.eu.org:443 "GET /api/trackers HTTP/1.1" 200 None
DEBUG: 428 trackers signatures loaded
INFO: Scanning APK with dexdump for known non-free classes.
DEBUG: > /usr/bin/dexdump /tmp/tmpw_8kem8z/classes.dex
DEBUG: > /usr/bin/dexdump /tmp/tmpw_8kem8z/classes2.dex
DEBUG: Found class 'com/google/android/gms/tasks/OnFailureListener'
DEBUG: Found class 'com/google/android/gms/location/LocationResult'
DEBUG: Found class 'com/google/android/gms/location/FusedLocationProviderClient'
DEBUG: Found class 'com/google/android/gms/location/LocationRequest'
DEBUG: Found class 'com/google/android/gms/location/LocationCallback'
DEBUG: Found class 'com/google/android/gms/tasks/OnSuccessListener'
CRITICAL: Found problems in /home/fdroid/scanner/com.b44t.messenger_632.apk
WARNING: Scanner found 6 problems in /home/fdroid/scanner/com.b44t.messenger_632.apk:

Looking at deps

$gradle dependencies --configuration fatReleaseCompileClasspath
...
+--- org.maplibre.gl:android-sdk:9.4.0
|    +--- com.mapbox.mapboxsdk:mapbox-sdk-geojson:5.3.0
|    |    \--- com.google.code.gson:gson:2.8.6
|    +--- com.mapbox.mapboxsdk:mapbox-android-gestures:0.7.0
|    \--- com.mapbox.mapboxsdk:mapbox-android-accounts:0.7.0
...

First thing com.mapbox.mapboxsdk:mapbox-android-accounts is not-FOSS see maven and this issue

Also not sure why maplibre still depends on com.mapbox, but that's another issue I guess.

apktool > com.b44t.messenger_632/smali_classes2/

$ grep gms -rin 
/com/mapbox/mapboxsdk/location/engine/GoogleLocation...smali
8<-----moar of this
...

...coming from here I guess https://github.com/maplibre/maplibre-gl-native/blob/android-v9.4.0/platform/android/MapboxGLAndroidSDK/src/main/java/com/mapbox/mapboxsdk/location/engine/GoogleLocationEngineImpl.java#L13

/LE: disabled 631&632 https://gitlab.com/fdroid/fdroiddata/-/commit/bbe1be46dffe071335d24296133b9815edfe198d

[r10s]

hm, in my understanding, the classes are actually not part of the apk.

eg, if you do apktool d deltachat-fat-debug-1.31.1.apk and then grep -ril FusedLocationProviderClient . you find code that would call FusedLocationProviderClient - but that code would fails if actually executed and imu that code is FOSS. and it is not executed. the removal of referred, maybe not FOSS code is done at https://github.com/deltachat/deltachat-android/pull/2303

so, it is correct that if you scan for FusedLocationProviderClient you will find some references, but that does not mean that the class is part of the apk.

i am wondering how Exodus is actually working here.

also, i am pretty unhappy about the fast removal; this makes checking carefully and discussions very hard.

[adbenitez]

also, i am pretty unhappy about the fast removal; this makes checking carefully and discussions very hard.

yes, now I can't download the apk and analize it :-/

[licaon-kter]

I've attached the APK to the GItlab issue.

F-Droid gets flack if it's too slow to update/publish, now was considered fast? Or if it did not find a non-FOSS issue, or now if it did? :roll_eyes:

[adbenitez]

I've attached the APK to the GItlab issue.

yes, found already, sorry, read here before there ;P

[adbenitez]

@licaon-kter analyzing the apk the gms warnings are false positives from exodus

[hpk42]

We uploaded the F-droid APK in question here https://download.delta.chat/android/beta/com.b44t.messenger_632.apk as a full file, so that more people can look into it. Note that F-droid has already removed all versions of Delta Chat since September 2021 which means everyone who now tries to install Delta Chat gets a very old version.

[r10s]

@licaon-kter analyzing the apk the gms warnings are false positives from exodus

yip, i can confirm that. false positives.

[adbenitez]

pasting here comment from gitlab:

I analyzed the Delta Chat apk and it doesn't have any gms blobs, that are false positives from exodus, so the real issue is about mapbox-android-accounts as @tad said? to test:

wget https://download.delta.chat/android/beta/com.b44t.messenger_632.apk
apktool decode -o com.b44t.messenger_632  com.b44t.messenger_632.apk

# this fails, no gms is in the apk, false positive from exodus:
du -sh com.b44t.messenger_632/smali*/com/google/android/gms/

# this is present:
du -sh com.b44t.messenger_632/smali*/com/mapbox/android/accounts

besides mapbox-android-accounts, anything else that would be needed to be fixed for the app to be 100% free software?

[licaon-kter]

@adbenitez

anything else that would be needed to be fixed for the app to be 100% free software?

Isn't this on a checklist when adding a new dependency to begin with?

[adbenitez]

Isn't this on a checklist when adding a new dependency to begin with?

this is nothing super new, it was used since a lot of time without it even being noticed by f-droid folks :walking_man:

[hpk42]

@licaon-kter of course we take care to only include foss dependencies -- which is why we moved to maplibre -- but we can not always check the dependencies of dependencies but rather trust the "top" level declaration.

[licaon-kter]

We've been having the "maps in DeltaChat are problematic" discussion for >3 years already.

[adbenitez]

@hpk42 who would have expected something called mapLibre to be not libre XD

[adbenitez]

@licaon-kter I am struggling to load gitlab page (via proxy because gitlab.com blocks access to my country) to reply your comment :hourglass_flowing_sand:

[licaon-kter]

https://gitlab.com/fdroid/admin/-/issues/159 :shrug:

[r10s]

@licaon-kter to sum up:

• the mapbox-android-accounts issue should be solved by #2351
• the GMS issue was a false positive of the Exodus scanner

can you please re-enable auto-update that was disabled at https://gitlab.com/fdroid/fdroiddata/-/commit/bbe1be46dffe071335d24296133b9815edfe198d ? we likely tag a new version soon EDIT: we tagged 1.32.0 and like to avoid more delays. thanks!

while it would have been more fun if it had been timed differently and we had not annoyed new users getting outdated versions - please let me note that i appreciate the check for non-FOSS code in general, thanks a lot for that.

[licaon-kter]

So this will fix both issues like Element did, no references in smali too?/

[adbenitez]

@licaon-kter no idea if the false positive will still pop up, in theory if it is not popping up for element it shouldn't for DC, I don't have the fdroid scanner and can't download it at the moment, maybe other dev could test, but still it would be only false positives, I checked the decompiled apk and there is no gms blobs

[licaon-kter]

Build testing now...

[licaon-kter]

no joy :(

...
INFO: Scanning APK with dexdump for known non-free classes.
DEBUG: > /opt/android-sdk/build-tools/30.0.2/dexdump /tmp/tmpcj_x8tmj/classes.dex
DEBUG: > /opt/android-sdk/build-tools/30.0.2/dexdump /tmp/tmpcj_x8tmj/classes2.dex
DEBUG: Found class 'com/google/android/gms/location/LocationResult'
DEBUG: Found class 'com/google/android/gms/location/LocationRequest'
DEBUG: Found class 'com/google/android/gms/location/FusedLocationProviderClient'
DEBUG: Found class 'com/google/android/gms/tasks/OnFailureListener'
DEBUG: Found class 'com/google/android/gms/tasks/OnSuccessListener'
DEBUG: Found class 'com/google/android/gms/location/LocationCallback'
CRITICAL: Found problems in build/com.b44t.messenger/build/outputs/apk/fat/release/com.b44t.messenger-fat-release-unsigned-1.32.0.apk
ERROR: Could not build app com.b44t.messenger: Found blocklisted packages in final apk!

Deps

+--- org.maplibre.gl:android-sdk:9.5.2
|    +--- org.maplibre.gl:android-sdk-geojson:5.9.0
|    |    \--- com.google.code.gson:gson:2.8.6
|    \--- com.mapbox.mapboxsdk:mapbox-android-gestures:0.7.0

.../smali_classes2/com $ grep \/gms -rin .|cut -d ":" -f1|sort|uniq
./mapbox/mapboxsdk/location/engine/GoogleLocationEngineImpl$GoogleLastLocationEngineCallbackTransport.smali
./mapbox/mapboxsdk/location/engine/GoogleLocationEngineImpl$GoogleLocationEngineCallbackTransport.smali
./mapbox/mapboxsdk/location/engine/GoogleLocationEngineImpl.smali
./mapbox/mapboxsdk/location/engine/LocationEngineProvider.smali
./mapbox/mapboxsdk/location/engine/LocationEngineResult.smali

...as expected since that lib is build using GMS libs.

/LE: I'll start an Element build now to peek at the tree later

/LE: Regarding APK scanning, just clone fdroidserver and run fdroid scanner --verbose --exodus myapp.apk

[r10s]

thanks for checking. exodus and grep seem to find references to non-existent code; this seems expected by the current state of the tools. also, deps has mapbox-android-accounts removed, so, all in all, that looks good to me wrt the scope of this issue :)

so, to be precise: is there anything that stands in the way of 1.32.0 being added to f-droid again?

[licaon-kter]

Element deps tree

+--- org.maplibre.gl:android-sdk:9.5.2
|    +--- org.maplibre.gl:android-sdk-geojson:5.9.0
|    |    \--- com.google.code.gson:gson:2.8.6
|    \--- com.mapbox.mapboxsdk:mapbox-android-gestures:0.7.0  
+--- org.maplibre.gl:android-plugin-annotation-v9:1.0.0

apktool diassemble the APK but grep \/gms -rin smali* yields no results.

Not sure what Element does differently.

[link2xt]

I have just compiled 1.32 and unpacked resulting .apk with 7z x:

$ strings classes3.dex  | grep gms
5Lcom/google/android/gms/common/GoogleApiAvailability;
=Lcom/google/android/gms/location/FusedLocationProviderClient;
2Lcom/google/android/gms/location/LocationCallback;
1Lcom/google/android/gms/location/LocationRequest;
0Lcom/google/android/gms/location/LocationResult;
2Lcom/google/android/gms/location/LocationServices;
0Lcom/google/android/gms/tasks/OnFailureListener;
0Lcom/google/android/gms/tasks/OnSuccessListener;
0Lcom/google/android/gms/tasks/OnSuccessListener<
#Lcom/google/android/gms/tasks/Task;
3com.google.android.gms.common.GoogleApiAvailability
.com.google.android.gms.location.LocationResult
0com.google.android.gms.location.LocationServices

As for element, probably the difference is that they use proguard which removes/obfuscates unused symbols.

[link2xt]

I have now installed jadx and run it, here is the result:

$ jadx app-fat-debug-1.32.0.apk
...
$ grep -r gms app-fat-debug-1.32.0
grep: app-fat-debug-1.32.0/resources/classes.dex: binary file matches
app-fat-debug-1.32.0/resources/META-INF/CERT.SF:SHA1-Digest: ...
grep: app-fat-debug-1.32.0/resources/classes3.dex: binary file matches
grep: app-fat-debug-1.32.0/resources/lib/x86/libnative-utils.so: binary file matches
grep: app-fat-debug-1.32.0/resources/lib/x86_64/libnative-utils.so: binary file matches
grep: app-fat-debug-1.32.0/resources/lib/armeabi-v7a/libnative-utils.so: binary file matches
grep: app-fat-debug-1.32.0/resources/lib/arm64-v8a/libnative-utils.so: binary file matches
app-fat-debug-1.32.0/sources/okhttp3/internal/platform/AndroidPlatform.java:                Class<?> gmsSslParametersClass = Class.forName("com.google.android.gms.org.conscrypt.SSLParametersImpl", false, sslSocketFactory.getClass().getClassLoader());
app-fat-debug-1.32.0/sources/okhttp3/internal/platform/AndroidPlatform.java:                context = readFieldOrNull(sslSocketFactory, gmsSslParametersClass, "sslParameters");
app-fat-debug-1.32.0/sources/com/mapbox/mapboxsdk/location/engine/LocationEngineProvider.java:import com.google.android.gms.common.GoogleApiAvailability;
app-fat-debug-1.32.0/sources/com/mapbox/mapboxsdk/location/engine/LocationEngineProvider.java:    private static final String GOOGLE_API_AVAILABILITY = "com.google.android.gms.common.GoogleApiAvailability";
app-fat-debug-1.32.0/sources/com/mapbox/mapboxsdk/location/engine/LocationEngineProvider.java:    private static final String GOOGLE_LOCATION_SERVICES = "com.google.android.gms.location.LocationServices";
app-fat-debug-1.32.0/sources/com/mapbox/mapboxsdk/location/engine/LocationEngineResult.java:import com.google.android.gms.location.LocationResult;
app-fat-debug-1.32.0/sources/com/mapbox/mapboxsdk/location/engine/LocationEngineResult.java:    private static final String GOOGLE_PLAY_LOCATION_RESULT = "com.google.android.gms.location.LocationResult";
app-fat-debug-1.32.0/sources/com/mapbox/mapboxsdk/location/engine/GoogleLocationEngineImpl.java:import com.google.android.gms.location.FusedLocationProviderClient;
app-fat-debug-1.32.0/sources/com/mapbox/mapboxsdk/location/engine/GoogleLocationEngineImpl.java:import com.google.android.gms.location.LocationCallback;
app-fat-debug-1.32.0/sources/com/mapbox/mapboxsdk/location/engine/GoogleLocationEngineImpl.java:import com.google.android.gms.location.LocationRequest;
app-fat-debug-1.32.0/sources/com/mapbox/mapboxsdk/location/engine/GoogleLocationEngineImpl.java:import com.google.android.gms.location.LocationResult;
app-fat-debug-1.32.0/sources/com/mapbox/mapboxsdk/location/engine/GoogleLocationEngineImpl.java:import com.google.android.gms.location.LocationServices;
app-fat-debug-1.32.0/sources/com/mapbox/mapboxsdk/location/engine/GoogleLocationEngineImpl.java:import com.google.android.gms.tasks.OnFailureListener;
app-fat-debug-1.32.0/sources/com/mapbox/mapboxsdk/location/engine/GoogleLocationEngineImpl.java:import com.google.android.gms.tasks.OnSuccessListener;

(only import statements and some strings, no non-free code related to gms)

[link2xt]

The problem is that https://github.com/f-droid/fdroidserver/blob/master/fdroidserver/scanner.py is rather dumb, it simply greps dextool output.

I did not run it as is, but here is my adaptation:

import re
import subprocess
import logging

CODE_SIGNATURES = {
    exp: re.compile(r'.*' + exp, re.IGNORECASE) for exp in [
        r'com/google/firebase',
        r'com/google/android/gms',
        r'com/google/android/play/core',
        r'com/google/tagmanager',
        r'com/google/analytics',
        r'com/android/billing',
    ]
}

run = subprocess.run(["podman", "exec", "deltachat", "/home/deltachat/android-sdk/build-tools/30.0.2/dexdump", "build/outputs/apk/fat/debug/classes3.dex"], capture_output=True)
output = run.stdout.decode("utf-8", errors='replace')
result = set(re.findall(r'[A-Z]+((?:\w+\/)+\w+)', output))

for classname in result:
    for suspect, regexp in CODE_SIGNATURES.items():
        if regexp.match(classname):
            print("Found class '%s'" % classname)

(deltachat container is a container built with Dockerfile from the root of this repo)

It outputs:

Found class 'com/google/android/gms/tasks/OnSuccessListener'
Found class 'com/google/android/gms/location/LocationCallback'
Found class 'com/google/android/gms/location/LocationResult'
Found class 'com/google/android/gms/location/LocationRequest'
Found class 'com/google/android/gms/tasks/OnFailureListener'
Found class 'com/google/android/gms/location/FusedLocationProviderClient'

These classes are not built into binary. They are simply referenced from https://github.com/maplibre/maplibre-gl-native/blob/main/platform/android/MapboxGLAndroidSDK/src/main/java/com/mapbox/mapboxsdk/location/engine/GoogleLocationEngineImpl.java

[Hocuri]

@licaon-kter is there anything left we have to do so that DeltaChat can get back into F-Droid?

• Element deps are same as for Delta Chat
• The script finds a false positive, because https://github.com/maplibre/maplibre-gl-native/blob/main/platform/android/MapboxGLAndroidSDK/src/main/java/com/mapbox/mapboxsdk/location/engine/GoogleLocationEngineImpl.java references classes that would be non-free if they were compiled in, but they are not compiled in.
• Element doesn't trigger the false-positive, probably because they are using Proguard.
• I just played around a bit with ProGuard and build.grade, trying to exclude this file, but didn't find way after a few hours.

[link2xt]

I filed an MR with update to 1.32.0, let's track it in F-Droid repo: https://gitlab.com/fdroid/fdroiddata/-/merge_requests/11475

Closing this as resolved, open a new issue if there are findings related to new version.

[licaon-kter]

As for element, probably the difference is that they use proguard which removes/obfuscates unused symbols.

@bmarty While I saw a "fdroidMinifySomething" build step scroll by on the screen, I could not find where to disable any minify/shrink/proguard steps. Can you share an one-liner that allows me to disable such obfuscation at build time?

[link2xt]

@licaon-kter Could you review https://gitlab.com/fdroid/fdroiddata/-/merge_requests/11475 ? Disabling proguard in Element is not blocking it I think, we have gms removed in any case.

Btw @bmarty seems to be afk for the last week according to github page.