search

Instagram / ig-json-parser

Fast JSON parser for java projects
https://instagram-engineering.com/fast-auto-generated-streaming-json-parsing-for-android-ab8e7be21033
MIT License
1.32k stars 124 forks source link

The Guava Dependency adds 15k methods, can you reduce this? #31

Closed aneemtalukder closed 8 years ago

aneemtalukder commented 8 years ago

Currently this parser has a dependency on Google's guava library that adds 15082 methods to any project that uses it. Given that the limit for a dex file is only 65k, is there any way you guys can reduce this? If not, can you at least give better proguard documentation? The current documentation for proguard does not seem to work, and neither does the proguard config file in the demo application.

kangzhang commented 8 years ago

@aneemtalukder The annotation processor's dependency shouldn't be added the app. The ig-json-parser have two jars, one for the annotation processor and another one for actual runtime. The app should only need the runtime jar (which has only two classes). Are you including the annotation processor jar in your app?

aneemtalukder commented 8 years ago

Oh interesting, I am using gradle and simply adding the parser to my dependencies:

compile 'com.instagram:ig-json-parser-processor:0.0.6'

aneemtalukder commented 8 years ago

@kangzhang @ttung would you happen to know of a way using gradle that I can prevent the downloading of guava?

kangzhang commented 8 years ago

yeah, looks like our current gradle setup pulled the dependency of the annotation processor into the app. I'll send a fix to split the runtime and the processor in our demo app.

in the meanwhile, could you try compile the lib as prebuilt jars and integrate into your project using apt? https://bitbucket.org/hvisser/android-apt

kangzhang commented 8 years ago

I started working on this and got the demo app to use android-apt https://github.com/Instagram/ig-json-parser/pull/32

We also need to add a new runtime artifact to the maven repo, and update the readme file with instructions to use android-apt.

aneemtalukder commented 8 years ago

great, thanks! i'll switch over to apt later on this week, hopefully there will be a runtime artifact down the line

aneemtalukder commented 8 years ago

@kangzhang I think I'm close, but in this way, where do you specify the generate files directory? (after deleting, i can no longer regenerate)

kangzhang commented 8 years ago

@aneemtalukder how's your build.gradle looks like? if you are using apt, shouldn't need to specify the generate source dir. (apt takes care of it automatically)

aneemtalukder commented 8 years ago

@kangzhang my issues are with compiling, in this form:

Error:(7, 51) error: package com.model.json.responses does not exist

(this package is the module that contains all my json models for the instagram parser)

this is my build.gradle:

apply plugin: 'com.android.library'
apply plugin: 'com.neenbedankt.android-apt'

android {
    compileSdkVersion 23
    buildToolsVersion "23.0.2"
    sourceCompatibility = 1.7
    targetCompatibility = 1.7

    sourceSets {
        main {
            manifest {
                srcFile "AndroidManifest.xml"
            }
            java {
                srcDir 'src/main/java'
            }
        }
    }
}

dependencies {
    compile files('libs/common.jar')
    apt files('libs/processor.jar')
}
kangzhang commented 8 years ago

the gradle file looks alright, the error message sounds like the model lib module doesn't point to the correct source code directory. (are all java source in src/main/java folder?).

could you put a minimal repro on github so that we can take a look?

aneemtalukder commented 8 years ago

I have created a minimal project that compiles with the jars, but crashes on launch, please have a look: https://github.com/aneemtalukder/AptTest

I believe the jars are creating a dex issue, not exactly sure though. This is the log:

11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: FATAL EXCEPTION: main 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: Process: com.demo.apttest, PID: 4584 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: java.lang.RuntimeException: Unable to instantiate activity ComponentInfo{com.demo.apttest/com.demo.apttest.MainActivity}: java.lang.ClassNotFoundException: Didn't find class "com.demo.apttest.MainActivity" on path: DexPathList[[zip file "/data/app/com.demo.apttest-1/base.apk"],nativeLibraryDirectories=[/vendor/lib, /system/lib]] 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2546) 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2758) 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at android.app.ActivityThread.access$900(ActivityThread.java:177) 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1448) 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at android.os.Handler.dispatchMessage(Handler.java:102) 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at android.os.Looper.loop(Looper.java:145) 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at android.app.ActivityThread.main(ActivityThread.java:5942) 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at java.lang.reflect.Method.invoke(Native Method) 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at java.lang.reflect.Method.invoke(Method.java:372) 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1399) 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1194) 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: Caused by: java.lang.ClassNotFoundException: Didn't find class "com.demo.apttest.MainActivity" on path: DexPathList[[zip file "/data/app/com.demo.apttest-1/base.apk"],nativeLibraryDirectories=[/vendor/lib, /system/lib]] 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:56) 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at java.lang.ClassLoader.loadClass(ClassLoader.java:511) 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at java.lang.ClassLoader.loadClass(ClassLoader.java:469) 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at android.app.Instrumentation.newActivity(Instrumentation.java:1079) 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2536) 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2758)  11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at android.app.ActivityThread.access$900(ActivityThread.java:177)  11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1448)  11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at android.os.Handler.dispatchMessage(Handler.java:102)  11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at android.os.Looper.loop(Looper.java:145)  11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at android.app.ActivityThread.main(ActivityThread.java:5942)  11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at java.lang.reflect.Method.invoke(Native Method)  11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at java.lang.reflect.Method.invoke(Method.java:372)  11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1399)  11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1194)  11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: Suppressed: java.lang.ClassNotFoundException: com.demo.apttest.MainActivity 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at java.lang.Class.classForName(Native Method) 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at java.lang.BootClassLoader.findClass(ClassLoader.java:781) 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at java.lang.BootClassLoader.loadClass(ClassLoader.java:841) 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: at java.lang.ClassLoader.loadClass(ClassLoader.java:504) 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: ... 13 more 11-13 10:54:52.936 4584-4584/? E/AndroidRuntime: Caused by: java.lang.NoClassDefFoundError: Class not found using the boot class loader; no stack available

kangzhang commented 8 years ago

yeah, it looks like you were using the wrong jar files. (We should use the jar in build folder, generated by ./gradlew build )

I created a patch at https://github.com/aneemtalukder/AptTest/pull/1 , which should fix the issue.

aneemtalukder commented 8 years ago

ahh great, thank you! @kangzhang as expected, the sample project AptTest compiles - however this was without any annotated classes in the models module.

I have added a class ExampleResponse into the project AptTest, and now does not compile with the following failure (I have pushed these changes to master at https://github.com/aneemtalukder/AptTest):

Warning:(9, 8) ERROR: Unable to generate injector for @JsonType.

java.lang.annotation.IncompleteAnnotationException: com.instagram.common.json.annotation.JsonType missing element generateSerializer at sun.reflect.annotation.AnnotationInvocationHandler.invoke(AnnotationInvocationHandler.java:80) at com.sun.proxy.$Proxy123.generateSerializer(Unknown Source) at com.instagram.common.json.annotation.processor.JsonAnnotationProcessor.processClassAnnotation(JsonAnnotationProcessor.java:194) at com.instagram.common.json.annotation.processor.JsonAnnotationProcessor.gatherClassAnnotations(JsonAnnotationProcessor.java:140) at com.instagram.common.json.annotation.processor.JsonAnnotationProcessor.process(JsonAnnotationProcessor.java:104) at com.sun.tools.javac.processing.JavacProcessingEnvironment.callProcessor(JavacProcessingEnvironment.java:793) at com.sun.tools.javac.processing.JavacProcessingEnvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:722) at com.sun.tools.javac.processing.JavacProcessingEnvironment.access$1700(JavacProcessingEnvironment.java:97) at com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1029) at com.sun.tools.javac.processing.JavacProcessingEnvironment.doProcessing(JavacProcessingEnvironment.java:1163) at com.sun.tools.javac.main.JavaCompiler.processAnnotations(JavaCompiler.java:1108) at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:824) at com.sun.tools.javac.main.Main.compile(Main.java:439) at com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:132) at org.gradle.api.internal.tasks.compile.JdkJavaCompiler.execute(JdkJavaCompiler.java:45) at org.gradle.api.internal.tasks.compile.JdkJavaCompiler.execute(JdkJavaCompiler.java:33) at org.gradle.api.internal.tasks.compile.NormalizingJavaCompiler.delegateAndHandleErrors(NormalizingJavaCompiler.java:101) at org.gradle.api.internal.tasks.compile.NormalizingJavaCompiler.execute(NormalizingJavaCompiler.java:50) at org.gradle.api.internal.tasks.compile.NormalizingJavaCompiler.execute(NormalizingJavaCompiler.java:36) at org.gradle.api.internal.tasks.compile.CleaningJavaCompilerSupport.execute(CleaningJavaCompilerSupport.java:34) at org.gradle.api.internal.tasks.compile.CleaningJavaCompilerSupport.execute(CleaningJavaCompilerSupport.java:25) at org.gradle.api.tasks.compile.JavaCompile.performCompilation(JavaCompile.java:157) at org.gradle.api.tasks.compile.JavaCompile.compile(JavaCompile.java:137) at org.gradle.api.tasks.compile.JavaCompile.compile(JavaCompile.java:91) at sun.reflect.GeneratedMethodAccessor886.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:75) at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$IncrementalTaskAction.doExecute(AnnotationProcessingTaskFactory.java:243) at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$StandardTaskAction.execute(AnnotationProcessingTaskFactory.java:219) at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$IncrementalTaskAction.execute(AnnotationProcessingTaskFactory.java:230) at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$StandardTaskAction.execute(AnnotationProcessingTaskFactory.java:208) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:80) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:61) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:46) at org.gradle.api.internal.tasks.execution.PostExecutionAnalysisTaskExecuter.execute(PostExecutionAnalysisTaskExecuter.java:35) at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:64) at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:58) at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:42) at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:52) at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:53) at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43) at org.gradle.api.internal.AbstractTask.executeWithoutThrowingTaskFailure(AbstractTask.java:310) at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.executeTask(AbstractTaskPlanExecutor.java:79) at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.processTask(AbstractTaskPlanExecutor.java:63) at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.run(AbstractTaskPlanExecutor.java:51) at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor.process(DefaultTaskPlanExecutor.java:23) at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter.execute(DefaultTaskGraphExecuter.java:88) at org.gradle.execution.SelectedTaskExecutionAction.execute(SelectedTaskExecutionAction.java:37) at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:62) at org.gradle.execution.DefaultBuildExecuter.access$200(DefaultBuildExecuter.java:23) at org.gradle.execution.DefaultBuildExecuter$2.proceed(DefaultBuildExecuter.java:68) at org.gradle.execution.DryRunBuildExecutionAction.execute(DryRunBuildExecutionAction.java:32) at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:62) at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:55) at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:149) at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:106) at org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:86) at org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.run(InProcessBuildActionExecuter.java:90) at org.gradle.tooling.internal.provider.runner.BuildModelActionRunner.run(BuildModelActionRunner.java:54) at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35) at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:41) at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:28) at org.gradle.launcher.daemon.server.exec.ExecuteBuild.doBuild(ExecuteBuild.java:49) at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36) at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120) at org.gradle.launcher.daemon.server.exec.WatchForDisconnection.execute(WatchForDisconnection.java:37) at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120) at org.gradle.launcher.daemon.server.exec.ResetDeprecationLogger.execute(ResetDeprecationLogger.java:26) at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120) at org.gradle.launcher.daemon.server.exec.RequestStopIfSingleUsedDaemon.execute(RequestStopIfSingleUsedDaemon.java:34) at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120) at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:74) at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:72) at org.gradle.util.Swapper.swap(Swapper.java:38) at org.gradle.launcher.daemon.server.exec.ForwardClientInput.execute(ForwardClientInput.java:72) at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120) at org.gradle.launcher.daemon.server.health.DaemonHealthTracker.execute(DaemonHealthTracker.java:47) at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120) at org.gradle.launcher.daemon.server.exec.LogToClient.doBuild(LogToClient.java:66) at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36) at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120) at org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment.doBuild(EstablishBuildEnvironment.java:71) at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36) at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120) at org.gradle.launcher.daemon.server.health.HintGCAfterBuild.execute(HintGCAfterBuild.java:41) at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120) at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy$1.run(StartBuildOrRespondWithBusy.java:50) at org.gradle.launcher.daemon.server.DaemonStateCoordinator$1.run(DaemonStateCoordinator.java:246) at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54) at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)

aneemtalukder commented 8 years ago

I was able to bring the changes to my main project, and indeed, 15k methods have been cut. thanks for the support @kangzhang !