search

soot-oss / SootUp

A new version of Soot with a completely overhauled architecture
https://soot-oss.github.io/SootUp/
GNU Lesser General Public License v2.1
565 stars 73 forks source link

How to analysis an android apk ? #528

Closed niorgai closed 1 year ago

niorgai commented 1 year ago

ApkAnalysisInputLocation is a private class, is it available now ? How to analysis an android apk ?

kadirayk commented 1 year ago

Please have a look at the example here: https://github.com/soot-oss/SootUp/blob/develop/sootup.java.bytecode/src/test/java/sootup/java/bytecode/inputlocation/PathBasedAnalysisInputLocationTest.java#L271

niorgai commented 1 year ago

@kadirayk thx, but it still not work.

code like:

PathBasedAnalysisInputLocation pathBasedNamespace =
        new PathBasedAnalysisInputLocation(apkPath, null);

final JavaProject project =
        JavaProject.builder(new JavaLanguage(8)).addInputLocation(pathBasedNamespace).build();
final JavaView view = project.createFullView();

ViewTypeHierarchy typeHierarchy = new ViewTypeHierarchy(view);

CallGraphAlgorithm cha =
        new ClassHierarchyAnalysisAlgorithm(view, typeHierarchy);

cha.initialize();

it is a simple demo apk, but i get this error:

Exception in thread "main" sootup.core.frontend.ResolveException: Could not find java.lang.Object in View. ./file-does-not-exist No position info
    at sootup.core.views.View.lambda$getClassOrThrow$0(View.java:96)
    at java.util.Optional.orElseThrow(Optional.java:290)
    at sootup.core.views.View.getClassOrThrow(View.java:96)
    at sootup.core.typehierarchy.ViewTypeHierarchy.sootClassFor(ViewTypeHierarchy.java:347)
    at sootup.core.typehierarchy.ViewTypeHierarchy.superClassOf(ViewTypeHierarchy.java:239)
    at sootup.core.typehierarchy.TypeHierarchy.superClassesOf(TypeHierarchy.java:173)
    at sootup.callgraph.AbstractCallGraphAlgorithm.findMethodInHierarchy(AbstractCallGraphAlgorithm.java:167)
    at sootup.callgraph.ClassHierarchyAnalysisAlgorithm.lambda$resolveCall$1(ClassHierarchyAnalysisAlgorithm.java:92)
    at java.util.Optional.orElseGet(Optional.java:267)
    at sootup.callgraph.ClassHierarchyAnalysisAlgorithm.resolveCall(ClassHierarchyAnalysisAlgorithm.java:92)
    at sootup.callgraph.AbstractCallGraphAlgorithm.lambda$resolveAllCallsFromSourceMethod$2(AbstractCallGraphAlgorithm.java:146)
    at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:269)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
    at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1580)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
    at sootup.callgraph.AbstractCallGraphAlgorithm.processWorkList(AbstractCallGraphAlgorithm.java:110)
    at sootup.callgraph.AbstractCallGraphAlgorithm.constructCompleteCallGraph(AbstractCallGraphAlgorithm.java:80)
    at sootup.callgraph.ClassHierarchyAnalysisAlgorithm.initialize(ClassHierarchyAnalysisAlgorithm.java:60)
    at SootUpMain.main(SootUpMain.java:49)

How to fix it please?

And another question , how can i add Spark dependency ? i can't import Spark from soot.

niorgai commented 1 year ago

by the way. code like testApk can work :

final Optional<? extends AbstractClassSource<JavaSootClass>> clazzOpt =
                pathBasedNamespace.getClassSource(mainClass, view);
final Collection<? extends AbstractClassSource<?>> classSources = pathBasedNamespace.getClassSources(view);

But i want to get a call graph of apk, so i need CallGraphAlgorithm. I also try to add an entryMethod.

CallGraphAlgorithm cha =
        new ClassHierarchyAnalysisAlgorithm(view, typeHierarchy);

final ClassType mainClass =
        JavaIdentifierFactory.getInstance().getClassType("com.example.rvdemo.MainActivity");

ClassType classTypeA = project.getIdentifierFactory().getClassType("android.os.Bundle");

MethodSignature entryMethodSignature =
        JavaIdentifierFactory.getInstance()
                .getMethodSignature(
                        mainClass,
                        JavaIdentifierFactory.getInstance()
                                .getMethodSubSignature(
                                        "onCreate", VoidType.getInstance(), Collections.singletonList(classTypeA)));

CallGraph cg = cha.initialize(Collections.singletonList(entryMethodSignature));

since Android MainActivity contains method onCreate

class MainActivity : AppCompatActivity() {

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
    }
}

but i also get error like:

Exception in thread "main" sootup.core.frontend.ResolveException: Could not find android.view.ViewGroup in View. ./file-does-not-exist No position info
    at sootup.core.views.View.lambda$getClassOrThrow$0(View.java:96)
    at java.util.Optional.orElseThrow(Optional.java:290)
    at sootup.core.views.View.getClassOrThrow(View.java:96)
    at sootup.core.typehierarchy.ViewTypeHierarchy.sootClassFor(ViewTypeHierarchy.java:347)
    at sootup.core.typehierarchy.ViewTypeHierarchy.superClassOf(ViewTypeHierarchy.java:239)
    at sootup.core.typehierarchy.TypeHierarchy.superClassesOf(TypeHierarchy.java:173)
    at sootup.callgraph.AbstractCallGraphAlgorithm.findMethodInHierarchy(AbstractCallGraphAlgorithm.java:167)
    at sootup.callgraph.ClassHierarchyAnalysisAlgorithm.lambda$resolveCall$1(ClassHierarchyAnalysisAlgorithm.java:92)
    at java.util.Optional.orElseGet(Optional.java:267)
    at sootup.callgraph.ClassHierarchyAnalysisAlgorithm.resolveCall(ClassHierarchyAnalysisAlgorithm.java:92)
    at sootup.callgraph.AbstractCallGraphAlgorithm.lambda$resolveAllCallsFromSourceMethod$2(AbstractCallGraphAlgorithm.java:146)
    at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:269)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
    at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1580)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
    at sootup.callgraph.AbstractCallGraphAlgorithm.processWorkList(AbstractCallGraphAlgorithm.java:110)
    at sootup.callgraph.AbstractCallGraphAlgorithm.constructCompleteCallGraph(AbstractCallGraphAlgorithm.java:80)
    at sootup.callgraph.ClassHierarchyAnalysisAlgorithm.initialize(ClassHierarchyAnalysisAlgorithm.java:66)
    at SootUpMain.main(SootUpMain.java:62)

Please help me~

kadirayk commented 1 year ago

for java.lang.Object you need to add the rt.jar and for Android classes you need to add android.jar from Android sdk. We currently only have CHA and RTA for creating call graphs. For example:

JavaProject project =
                JavaProject.builder(new JavaLanguage(8))
                        .addInputLocation(pathBasedNamespace) // apk 
                        .addInputLocation(
                                new JavaClassPathAnalysisInputLocation(
                                        System.getProperty("java.home") + "/lib/rt.jar"))
                        .addInputLocation(
                                new JavaClassPathAnalysisInputLocation(".../Android/sdk/platforms/android-30/android.jar") // change this
                        )
                        .build();
niorgai commented 1 year ago

After add rt.jar and android.jar, get another exception:

Exception in thread "main" sootup.core.frontend.ResolveException: Could not find androidx.window.extensions.layout.WindowLayoutComponent in View. ./file-does-not-exist No position info
    at sootup.core.views.View.lambda$getClassOrThrow$0(View.java:96)
    at java.util.Optional.orElseThrow(Optional.java:290)
    at sootup.core.views.View.getClassOrThrow(View.java:96)
    at sootup.core.typehierarchy.ViewTypeHierarchy.sootClassFor(ViewTypeHierarchy.java:347)
    at sootup.core.typehierarchy.ViewTypeHierarchy.superClassOf(ViewTypeHierarchy.java:239)
    at sootup.core.typehierarchy.TypeHierarchy.isSubtype(TypeHierarchy.java:146)
    at sootup.core.typehierarchy.MethodDispatchResolver.canDispatch(MethodDispatchResolver.java:132)
    at sootup.core.typehierarchy.MethodDispatchResolver.lambda$resolveAbstractDispatch$3(MethodDispatchResolver.java:58)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
    at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
    at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
    at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1580)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
    at sootup.core.typehierarchy.MethodDispatchResolver.resolveAbstractDispatch(MethodDispatchResolver.java:61)
    at sootup.callgraph.ClassHierarchyAnalysisAlgorithm.resolveCall(ClassHierarchyAnalysisAlgorithm.java:101)
    at sootup.callgraph.AbstractCallGraphAlgorithm.lambda$resolveAllCallsFromSourceMethod$2(AbstractCallGraphAlgorithm.java:146)
    at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:269)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
    at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1580)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
    at sootup.callgraph.AbstractCallGraphAlgorithm.processWorkList(AbstractCallGraphAlgorithm.java:110)
    at sootup.callgraph.AbstractCallGraphAlgorithm.constructCompleteCallGraph(AbstractCallGraphAlgorithm.java:80)
    at sootup.callgraph.ClassHierarchyAnalysisAlgorithm.initialize(ClassHierarchyAnalysisAlgorithm.java:60)
    at SootUpMain.main(SootUpMain.java:70)

But i didn't import androidx.window to my project manually and i don't know how to get the jar file.

Is it necessary to add jar of all third-part library to input location to work?

kadirayk commented 1 year ago

Yes, otherwise the call graph will be incomplete. However, we could skip the class and produce a warning in such cases. Could you share the apk?

niorgai commented 1 year ago

Of course, it is a demo apk of Android Studio project. you can download from github.

Is there any api i can use that skip such missing classes and output an incomplete call graph?

kadirayk commented 1 year ago

Could you try it with the new changes to the develop branch?

niorgai commented 1 year ago

@kadirayk develop branch can work, thanks!

niorgai commented 1 year ago

Since I have the source code of apk, i also try with JavaSourcePathAnalysisInputLocation

    Path apkPath = Paths.get("/AndroidStudioProjects/RVDemo/app");

    AnalysisInputLocation<JavaSootClass> pathBasedNamespace =
        new JavaSourcePathAnalysisInputLocation(apkPath.toString());

and i get this error:


    at com.ibm.wala.ipa.cha.ClassHierarchy.<init>(ClassHierarchy.java:291)
    at com.ibm.wala.ipa.cha.ClassHierarchy.<init>(ClassHierarchy.java:203)
    at com.ibm.wala.ipa.cha.ClassHierarchyFactory.make(ClassHierarchyFactory.java:85)
    at com.ibm.wala.ipa.cha.ClassHierarchyFactory.makeWithRoot(ClassHierarchyFactory.java:95)
    at sootup.java.sourcecode.frontend.WalaJavaClassProvider.buildClassHierachy(WalaJavaClassProvider.java:217)
    at sootup.java.sourcecode.frontend.WalaJavaClassProvider.iterateWalaClasses(WalaJavaClassProvider.java:268)
    at sootup.java.sourcecode.frontend.WalaJavaClassProvider.getClassSources(WalaJavaClassProvider.java:230)
    at sootup.java.sourcecode.inputlocation.JavaSourcePathAnalysisInputLocation.getClassSources(JavaSourcePathAnalysisInputLocation.java:153)
    at sootup.java.core.views.JavaView.lambda$resolveAll$3(JavaView.java:163)
    at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:269)
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
    at sootup.java.core.views.JavaView.resolveAll(JavaView.java:164)
    at sootup.java.core.views.JavaView.getClasses(JavaView.java:104)
    at sootup.java.core.JavaProject.createFullView(JavaProject.java:78)
    at sootup.tests.SimpleSootClient.main(SimpleSootClient.java:85)
Caused by: java.lang.NullPointerException
    at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.createClassDeclaration(JDTJava2CAstTranslator.java:510)
    at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visitTypeDecl(JDTJava2CAstTranslator.java:435)
    at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.visit(JDTJava2CAstTranslator.java:3778)
    at com.ibm.wala.cast.java.translator.jdt.JDTJava2CAstTranslator.translateToCAst(JDTJava2CAstTranslator.java:274)
    at com.ibm.wala.cast.java.translator.jdt.ecj.ECJSourceModuleTranslator$ECJAstToIR.acceptAST(ECJSourceModuleTranslator.java:158)
    at org.eclipse.jdt.core.dom.CompilationUnitResolver.resolve(CompilationUnitResolver.java:1068)
    at org.eclipse.jdt.core.dom.CompilationUnitResolver.resolve(CompilationUnitResolver.java:659)
    at org.eclipse.jdt.core.dom.ASTParser.createASTs(ASTParser.java:1003)
    at com.ibm.wala.cast.java.translator.jdt.ecj.ECJSourceModuleTranslator.loadAllSources(ECJSourceModuleTranslator.java:258)
    at com.ibm.wala.cast.java.loader.JavaSourceLoaderImpl.loadAllSources(JavaSourceLoaderImpl.java:599)
    at com.ibm.wala.classLoader.ClassLoaderImpl.init(ClassLoaderImpl.java:506)
    at com.ibm.wala.cast.java.loader.JavaSourceLoaderImpl.init(JavaSourceLoaderImpl.java:608)
    at com.ibm.wala.cast.java.translator.jdt.ecj.ECJClassLoaderFactory.makeNewClassLoader(ECJClassLoaderFactory.java:31)
    at com.ibm.wala.classLoader.ClassLoaderFactoryImpl.getLoader(ClassLoaderFactoryImpl.java:61)
    at com.ibm.wala.ipa.cha.ClassHierarchy.<init>(ClassHierarchy.java:270)
    ... 20 more
Exception in thread "main" java.lang.NullPointerException
    at sootup.java.sourcecode.frontend.WalaJavaClassProvider.iterateWalaClasses(WalaJavaClassProvider.java:273)
    at sootup.java.sourcecode.frontend.WalaJavaClassProvider.getClassSources(WalaJavaClassProvider.java:230)
    at sootup.java.sourcecode.inputlocation.JavaSourcePathAnalysisInputLocation.getClassSources(JavaSourcePathAnalysisInputLocation.java:153)
    at sootup.java.core.views.JavaView.lambda$resolveAll$3(JavaView.java:163)
    at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:269)
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
    at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
    at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
    at sootup.java.core.views.JavaView.resolveAll(JavaView.java:164)
    at sootup.java.core.views.JavaView.getClasses(JavaView.java:104)
    at sootup.java.core.JavaProject.createFullView(JavaProject.java:78)
    at sootup.tests.SimpleSootClient.main(SimpleSootClient.java:85)

code is 

this.classHierarchy = ClassHierarchyFactory.make(scope, factory);

I also try makeWithRoot or makeWithPhantom, all failed. Is it a wala error?

kadirayk commented 1 year ago

yes, I'd suggest using the bytecode front end for now.