Referencing another SDK failed

[highwater]

Now I feel worried for submitting issue 4th time in a row, but here we go.

So I'm currently still trying APK com.path and turns out it uses Amazon Maps SDK, and that SDK of course does not get build into classes.jar. I don't know why Path uses that SDK, though (just for information: I'm still new in this analysis stuff). So here's the logs:

....
Transforming /home/vip/droidel_test/com.path/
Parsing layout
Warning: couldn't find manifest-declared event handler method goToPreviousQuestion as a method on Lcom/example/android/apis/view/CustomView1
Warning: couldn't find manifest-declared event handler method completeSurvey as a method on Lcom/example/android/apis/view/CustomView1
Warning: couldn't find manifest-declared event handler method goToNextQuestion as a method on Lcom/example/android/apis/view/CustomView1
Parsing layout took 69.434
Generating stubs
[error] (run-main-0) java.lang.RuntimeException: Couldn't find class name corresponding to any of List(Lcom/amazon/geo/maps/MapView) in class hierarchy
java.lang.RuntimeException: Couldn't find class name corresponding to any of List(Lcom/amazon/geo/maps/MapView) in class hierarchy
    at scala.sys.package$.error(package.scala:27)
    at edu.colorado.droidel.codegen.AndroidLayoutStubGenerator.edu$colorado$droidel$codegen$AndroidLayoutStubGenerator$$getTypeForAndroidClassName(AndroidLayoutStubGenerator.scala:113)
    at edu.colorado.droidel.codegen.AndroidLayoutStubGenerator$$anonfun$getFieldsAndAllocsForLayoutElems$1$1.apply(AndroidLayoutStubGenerator.scala:141)
    at edu.colorado.droidel.codegen.AndroidLayoutStubGenerator$$anonfun$getFieldsAndAllocsForLayoutElems$1$1.apply(AndroidLayoutStubGenerator.scala:140)
    at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:111)
    at scala.collection.immutable.List.foldLeft(List.scala:84)
    at edu.colorado.droidel.codegen.AndroidLayoutStubGenerator.getFieldsAndAllocsForLayoutElems$1(AndroidLayoutStubGenerator.scala:140)
    at edu.colorado.droidel.codegen.AndroidLayoutStubGenerator.generateWalaStubs(AndroidLayoutStubGenerator.scala:156)
    at edu.colorado.droidel.codegen.AndroidLayoutStubGenerator.generateStubs(AndroidLayoutStubGenerator.scala:73)
    at edu.colorado.droidel.driver.AndroidAppTransformer.generateStubs(AndroidAppTransformer.scala:541)
    at edu.colorado.droidel.driver.AndroidAppTransformer.transformApp(AndroidAppTransformer.scala:567)
    at edu.colorado.droidel.driver.Main$.main(Main.scala:68)
    at edu.colorado.droidel.driver.Main.main(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
[trace] Stack trace suppressed: run last compile:runMain for the full output.
java.lang.RuntimeException: Nonzero exit code: 1
    at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:runMain for the full output.
[error] (compile:runMain) Nonzero exit code: 1
[error] Total time: 78 s, completed Sep 25, 2014 8:20:59 PM

What's the solution to this, do I have to build the Amazon SDK myself and include it to bin/classes or else?

Thank you very much.

[sblackshear]

Hi Coraline, No worries, every issue you submit helps me improve Droidel! I appreciate your feedback a lot.

Yes, you need to download the JAR for the missing SDK (looks like you can get the Amazon one here) and copy it into the libs directory of the decompiled APK. If you re-run Droidel after doing this, it should hopefully take care of the issue. Please let me know if not.

• Sam

[highwater]

Yeah I think of that solution too. So, it seems it's going to be difficult to automate detecting many apps if this missing library keeps occurring.

[highwater]

It's still not successful with Path.

....
Parsing layout took 58.415
Generating stubs
generatedstubs/GeneratedAndroidLayoutStubs.java:234: error: com.mixpanel.android.surveys.SurveyChoiceView is not public in com.mixpanel.android.surveys; cannot be accessed from outside package
  private static com.mixpanel.android.surveys.SurveyChoiceView com_mixpanel_android_multiple_choice_answer_text;
                                             ^
generatedstubs/GeneratedAndroidLayoutStubs.java:577: error: com.mixpanel.android.surveys.FadeOnPressButton is not public in com.mixpanel.android.surveys; cannot be accessed from outside package
  private static com.mixpanel.android.surveys.FadeOnPressButton com_mixpanel_android_button_exit;
                                             ^

[sblackshear]

To address your earlier question: yes, the missing libraries (among other problems) make it hard to automate running Droidel on a lot of apps without babysitting. However, missing libraries typically occur because either because the decompiler Droidel uses (dex2jar) fails to decompile some of the classes used by the app (I suspect that's the case here), or because the Android project has not been set up in the directory structure that Droidel understands. The first problem is not Droidel's fault, and the second one could be improved by writing some code that tries to automatically reorganize arbitrary project layouts into the format that the Droidel expects.

[sblackshear]

To address the second question, I think I see what the problem is and have pushed a fix. If it doesn't solve the problem, would you mind sending me the APK that's causing the issue?

[highwater]

New error after trying the new fix. I've attached the app below which I downloaded from http://apps.evozi.com/apk-downloader/

...
Parsing layout took 63.554
Generating stubs
[error] (run-main-0) scala.MatchError: include focus_off.xml (of class edu.colorado.droidel.parser.LayoutInclude)
scala.MatchError: include focus_off.xml (of class edu.colorado.droidel.parser.LayoutInclude)
    at edu.colorado.droidel.codegen.AndroidLayoutStubGenerator$$anonfun$1$$anonfun$apply$1.apply(AndroidLayoutStubGenerator.scala:45)
    at edu.colorado.droidel.codegen.AndroidLayoutStubGenerator$$anonfun$1$$anonfun$apply$1.apply(AndroidLayoutStubGenerator.scala:45)
    at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:144)
    at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:144)
    at scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:153)
    at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306)
    at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:306)
    at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:144)
    at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:105)
    at edu.colorado.droidel.codegen.AndroidLayoutStubGenerator$$anonfun$1.apply(AndroidLayoutStubGenerator.scala:45)
    at edu.colorado.droidel.codegen.AndroidLayoutStubGenerator$$anonfun$1.apply(AndroidLayoutStubGenerator.scala:44)
    at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:144)
    at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:144)
    at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224)
    at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
    at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
    at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:144)
    at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:105)
    at edu.colorado.droidel.codegen.AndroidLayoutStubGenerator.generateStubs(AndroidLayoutStubGenerator.scala:44)
    at edu.colorado.droidel.driver.AndroidAppTransformer.generateStubs(AndroidAppTransformer.scala:503)
    at edu.colorado.droidel.driver.AndroidAppTransformer.transformApp(AndroidAppTransformer.scala:529)
    at edu.colorado.droidel.driver.Main$.main(Main.scala:68)
    at edu.colorado.droidel.driver.Main.main(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
[trace] Stack trace suppressed: run last compile:runMain for the full output.
java.lang.RuntimeException: Nonzero exit code: 1
    at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:runMain for the full output.
[error] (compile:runMain) Nonzero exit code: 1
[error] Total time: 100 s, completed Sep 26, 2014 7:48:34 PM

https://github.com/chighwater/mirror/blob/master/com.path.apk https://github.com/chighwater/mirror/blob/master/amazonmaps-1.0.2.jar

[highwater]

[NullPointerException from JPhantom]

I think this is JPhantom bug but I report to you. On class jphantom.hier.ClassHierarchies.java, method ClassHierarchy fromJar(JarFile file) around this statement:

Type clazz = Type.getObjectType(reader.getClassName());                    
Type superclass = Type.getObjectType(reader.getSuperName());

When reader is of the type Object, the superName will be null, thus it throws NPE. However when I tried to patch it myself by continuing loop when it finds superName == null, it still throws RuntimeException.

I tested with this apk https://github.com/chighwater/mirror/blob/master/me.organize.android.apk

[sblackshear]

Yep, looks like a JPhantom issue. Fortunately, George (JPhantom author) is very helpful and has fixed every problem that I reported to him. If you file an issue on the JPhantom page and send him the JAR that's causing the problem, I'm sure he'll be willing to take a look at it.

In the meantime, you can always try running Droidel with the --no-jphantom flag, but you do risk missing a lot of code in your analysis by doing this.

[highwater]

Forwarding to you >> https://github.com/gbalats/jphantom/issues/4#issuecomment-57509365

[highwater]

Wait, it seems the android.jar from Sable and from Grepcore are different. The jar from Grepcode does not contain java library. Do you know about this?

[sblackshear]

Hmm, I haven't actually used the Android JARs from Sable in quite some time because they do not have any of the more recent Android versions. I did not know that the Sable JARs contain the Android reimplementation of the core Java libraries/that including the core Java libraries in the input JAR causes problems for JPhantom. Perhaps I should delete my recommendation that folks use the Sable JARs for this reason. I'll update the README. Thanks for tracking this down.

[sblackshear]

Closing after updating the README.

[highwater]

What about this? https://github.com/cuplv/droidel/issues/4#issuecomment-56960267

[sblackshear]

I no longer get the MatchError with the latest version of Droidel. The com.path.apk benchmark still fails for me, but with a different error. It's a much nastier problem with inner class enum's that's hard to handle correctly with the information WALA gives. I'll open a separate issue for that problem.