Rework get_embedded_classes

[jspricke]

• Support embedded apk files as found by apkeep.
• Drop usage of perl, sort, uniq and subprocess shell (security issue).
• Drop which() (already done by subprocess).
• Ignore invalid utf-8 encoding from dexdump.
• Support multiple dex files with the same name in the apk.
• Separate caching logic from extraction logic to allow recursive call.

[jspricke]

I took get_embedded_classes as a basis to implement this into fdroidserver: https://gitlab.com/fdroid/fdroidserver/-/merge_requests/1110

Would be great to get feedback to synchronize both implementations.

[U039b]

Thank you @jspricke! I think that the code could even simpler since it possible to run dexdump directly on an APK file. If we want to support AppBundle, we should just check if the downloaded Zip archive contains a file base.apk. If so, unzip the aab and run dexdump on base.apk, otherwise, just run dexdump on the downloaded file.

@pnu-s any thoughts about that?

[eighthave]

I think this approach of looking for nested APKs is good for a number of reasons:

• apkeep or ApkPure has a potentially custom method of bundling up downloaded App Bundles which a ZIP with APKs in it (it could be AAB, but i don't think so)
• It should also work find with AAB files
• Apps do weird things and include APKs sometimes, here's a list of examples from our APK collection:

  10249d80dd78bb2cea4bd3d403fe53e5        res/raw/wear.apk
  1065b36f38904b0acc011d9fb54abf12        assets/MoXiuLauncher_alone.apk
  10703d2017d14f5bc03e0f1cc9a05a73        assets/alipay_msp.apk
  10979b944e6168166653d8baf54afd19        assets/MoXiuLauncher_alone.apk
  109f3f544dbcfe341291b323f8923c95        res/raw/android_wear_micro_apk.apk
  10aa2f1cede33697f30fa026a2f320b8        assets/bdpwxpayplugin.apk
  10c10554d4e67419cc8126e04fb7d428        assets/DexToLoad.apk
  10de97efdd42aa517fd358d100af41c5        assets/20121018133442msp.apk
  10e11278a8b2db81973771858fe6c61f        baidu-api-sdk-android.apk
  10e11278a8b2db81973771858fe6c61f        baidu-api-sdk-android.unaligned.apk
  111cfe92d271c18c68d8c058866dc4a1        assets/alipay_plugin_20120428msp.apk
  1137c19ce1867bb7427ec2a524e4262d        assets/alipay_plugin_20120428msp.apk
  11425cf07065bf50eb3a0bbca1556f4a        assets/alipay_plugin_20120428msp.apk
  1146cb822a2c7ebcfc275bfbdbe4c991        assets/UPPayPluginEx.apk
  11537e2477f13b471a49d856ba69f460        assets/alipay_plugin_20120428msp.apk
  11b9c560f75163f4dfb5a605a48adc5a        assets/ATSystem.apk
  11e6394efc1b296a79ee528f78cc29f5        assets/alipay_msp.apk
  1226a0783d819fd203c4a55f7e8f4a83        assets/alipay_msp.apk
  125b2be84d9efbadac3d79b67e914bcd        assets/TopVideo.apk
  125b2be84d9efbadac3d79b67e914bcd        assets/kuaiwan.apk
  125b2be84d9efbadac3d79b67e914bcd        assets/msp_branch_5.0.9_kuaibo.apk
  1266216b4b9e16b11c7780e88777c3ed        assets/QihooPay.apk
  1289bb3e82999348158355b13698fd0f        assets/alipay_plugin_20120428msp.apk
  129dae31910a71658bded34c5b831c23        assets/alipay_plugin223_0309.apk
  12c241486b3438bf1c707a39663a9f0f        assets/PlayerUIApk.apk
  12ec132f349c746e0be9815a1421271a        assets/alipay_plugin231_120417.apk
  12f46e56151d9e0fb5e9c19ca9f2f5a2        assets/MoXiuLauncher_alone.apk
  130de81b8d016152e499ce01704f3ffa        assets/bangcleplugin/collector.apk
  130de81b8d016152e499ce01704f3ffa        assets/bangcleplugin/container.apk
  13355dbb0729ebbfd0b179282ef609f9        assets/MoXiuLauncher_alone.apk
  133c5d4094c9602a21b813381973db38        assets/UPPayPluginEx.apk
  133c5d4094c9602a21b813381973db38        assets/alipay_plugin_20120428msp.apk
  135dbd3ff8faaf100551045622c70aab        assets/UPPayPluginEx.apk
  135dbd3ff8faaf100551045622c70aab        assets/alipay_msp.apk
  13607d9d35104143f6b3c2655664a72d        assets/AnyShare-Lenovo-Phone-KUAISHOU-4020706.apk
  13607d9d35104143f6b3c2655664a72d        assets/alipay_mobile_sp_20130818.apk
  13625dbc6eb28fef44ed0e0392922656        assets/UPPayPluginEx.apk
  13789c9c5972f8a858e6e1ad609c566f        assets/alipay_msp.apk
  13c0be4f51003dd264350a61bba6da5d        assets/bdpwxpayplugin.apk
  13c1e5b2c2b5df9ebad4194a51703e69        assets/MoXiuLauncher_alone.apk
  13d719496ce3cb928f63bf4a8a3f3e9f        assets/alipay_msp.apk
  13d93a4944ccbe4692a3ec8e79e5e167        assets/jackson.apk
  13eaf5cd04ed178694422dcc849e5c6a        assets/system_service_1.18.apk
  13f4a0fdc6241eaae8b55d69e7709c95        assets/module/com.autonavi.libs.apk
  13ff94018f93103e3857485306eefdc3        bin/RootTools.apk

As for letting dexdump find the classes.dex itself, it is hard to say whether that's a better approach or not IMHO. @U039b do you know more about that? I suppose in the future, if Google changes something about how the Java code is bundled, they would also update dexdump, but my gut feeling is that they are also about as likely to do something weird in dexdump that breaks our usage. Do you think its a mistake to go with the current approach?

[eighthave]

I just had one other thought, but perhaps this could be left til later: the recursive search for ZIP files could actually check the first 4 bytes of each file to see if it is a ZIP rather than looking only at the extension. Maybe this is more a malware trick, so isn't so important for privacy scanning of legitimate apps.

[pnu-s]

@U039b I approved this PR but I'll wait for your approval before merging it!

[U039b]

Looks good to me :) In a near future I will propose an even better, faster and pure python way to grab embedded classes names. I am currently working on this exact same feature on DexTerity which is still private for the moment.

[eighthave]

To detect things hidden in nested ZIPs, we've added robust recursion to this work: https://gitlab.com/fdroid/fdroidserver/-/merge_requests/1123