generation time seems much slower

[ghost]

Hi, I'm using the doclet from an ant script. I noticed that my build time seems to have increased drastically: 2adb450 - 23 seconds 831e8f7 - 1 minute 24 seconds

I experimented with the number of packages I included in my doclet, and the time seemed to go up slightly exponentially.

My ant script looks like this:

        <javadoc encoding="8859_1"
                 destdir="${dist.base}/api-docs"
                 useexternalfile="true"
                 failonerror="true">
            <sourcepath>
                <pathelement path="${src.dir}" />
            </sourcepath>
            <classpath>
                <path refid="class.path"/>
            </classpath>
            <doclet name="com.carma.swagger.doclet.ServiceDoclet"
                    path="${dependencies.dir}/swagger-doclet.jar">
                <param name="-docBasePath" value="${swagger-docs-url}"/>
                <param name="-apiBasePath" value="${swagger-api-url}" />
                <param name="-apiVersion" value="${official.build.number}"/>
                <param name="-modelFieldsRequiredByDefault" value="true" />
                <param name="-disableCopySwaggerUi" value="true" />
            </doclet>
        </javadoc>

The reason this is a problem for me is that I want it to be quick for developers to make javadoc changes, build, and see what the resultant documentation looks like.

I see the same degradation on tag v1.1.0.

Just wondering if others are seeing this?

[conorroche]

am i correct in thinking you looked at v1.5.0 and noticed it was a lot slower than v1.0.4.2 then drilled into it further and saw the degradation seems to have occurred with the fixes for issue 71? We have quite a lot of classes that are documented and looking at the 1.1.0 version thats built a little quicker than the previous (though that might just mean the doclet runs faster under java 8 for our project) I will look to do some profiling next week to find the bottlenecks. I would be curious to know if there is much of a difference for your project if you run the doclet with java 8 vs java 7

[ghost]

Hi @conorroche, thanks for your quick response... actually I had checked out revision 2adb450 as it was the latest on master at the time and had some fixes I needed. As a result of your recent v1.1.0 I upgraded and so I just simply trawled through the revisions building and trying each one until I found the place where the generation time suddenly increased. I wasn't really looking at tags specifically.

Results of swagger json generation time testing 831e8f7: Java 7: 1 minute 29 seconds Java 8: 1 minute 23 seconds Not much different. In both I get plenty of javadoc warnings, though interestingly the Java 7 run seems to give warnings about more files.

It would be great if you could do some profiling. The difference in performance is not noticeable if I include only a few packages. It is apparent if I include many, or all packages, which results in about 60 services, i.e. 60 top-level items you see when you open Swagger UI.

Thanks.

p.s. Oh I see why I get more warnings in Java 7: Java 8 gives up after 100 warnings.

[conorroche]

Hi @dwalden I have just committed some performance improvements for some bottlenecks that I identified through profiling, can you let me know how this impacts on your generation time?

[ghost]

Hi @conorroche, thanks for looking at this. Sadly there isn't any significant change in generation time: v1.1.0: 1 minute 22 seconds 528153beacec4b2d253d5e2b07e2a58755fd12c5: 1 minute 25 seconds

I can try doing some profiling myself.

[conorroche]

@dwalden i've made a commit to support adding a 30 sec sleep before and after the doclet processes, this is to give enough time to connect a profiler to it and record, this is enabled by adding the doclet option -profileMode Our APIs are made up of 33 resources that references about 216 domain entities and generation is quick e.g. 8 secs. If you could profile that would help as its something specific with your code base.

if you havent used a profiler then this is some info:

one you can try is YourKit java profiler https://www.yourkit.com which has a 15 day eval.

You need to add the -agentpath jvm option to the javadoc https://www.yourkit.com/docs/java/help/agent.jsp

i used the below maven config to run the doclet via mvn and start the profiler agent that i could then attach to:

<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-javadoc-plugin</artifactId> <version>2.10.3</version> <executions> <execution> <phase>package</phase> <goals> <goal>jar</goal> </goals> <configuration> <additionalparam>-Xdoclint:none</additionalparam> </configuration> </execution> <execution> <id>generate-service-docs</id> <phase>process-resources</phase> <configuration> <doclet>com.carma.swagger.doclet.ServiceDoclet</doclet> <docletArtifact> <groupId>com.carma</groupId> <artifactId>swagger-doclet</artifactId> <version>1.1.1-SNAPSHOT</version> </docletArtifact> <reportOutputDirectory>${project.build.directory}/swagger</reportOutputDirectory> <useStandardDocletOptions>false</useStandardDocletOptions> <additionalparam>-profileMode -sortResourcesByPriority -disableCopySwaggerUi -apiVersion 1 -docBasePath ${swagger.doc.base.path} -apiBasePath ${swagger.api.base.path} -authOperationScopes rtr -apiAuthorizationsFile ${project.build.directory}/swagger/includes/apiauth.json -apiInfoFile ${project.build.directory}/swagger/includes/apiinfo.json -extraApiDeclarations ${project.build.directory}/swagger/includes/oauth.json -variablesPropertiesFile ${project.build.directory}/swagger/includes/vars.properties</additionalparam> <additionalJOptions> <additionalJOption>-J-agentpath:/Applications/YourKit_Java_Profiler_2015_build_15068.app/Contents/Resources/bin/mac/libyjpagent.jnilib</additionalJOption> </additionalJOptions> </configuration> <goals> <goal>javadoc</goal> </goals> </execution> </executions> </plugin>

When it is running i see the javadoc process show in the local applications and double clicking javadoc opens it, you can then click start cpu profiling. For reference here is what shows for me:

[conorroche]

@dwalden can you also try to restrict the classes/packages that are processed to only the ones that contain the jaxrs resources and associated objects

e.g. -includeResourcePrefixes com.mycompany.api.services,com.mycompany.api.domain

ive just done a commit to make class/package name exclusion happen first before bothering to inspect any tags/annotations on the classes so that will make this a little faster than it was

if this does have a significant performance improvement for you it probably means that there are a large number of classes being passed in to the doclet that are not relevant but it would be good to be able to not have the number of irrelevant classes impact peformance

[ghost]

I've had a quick look with this (very cool) profiler. Results so far...

The addAll() method invocations take roughly one minute, which appears to account completely for the performance degradation.

When profiling my "pre-degradation" revision (2adb450) the addAll() method invocation time is negligible.

Of possible interest, the Collection is a HashSet, which means it's calling hashCode() and equals() on its elements.

I have not yet tried filtering on packages.

[conorroche]

@dwalden interesting i wonder if some of your classes have odd/slow implementations of hashcode and or equals or if its the volume of classes, anyways i've commited a change that moves the merging of normal model classes and the special type classes to the class parser so means it will be done once per class not once per method, also ive changed it to use an arraylist instead of a hashset

[ghost]

Hi @conorroche, Time: 1 minute 7 seconds

Profiling results:

[conorroche]

@dwalden i've committed a performance improvement for the latest bottleneck u showed

[ghost]

Hi @conorroche, Time: 28 seconds :smiley:

Thank you for fixing this.

[conorroche]

@dwalden could you send me on a profiling screenshot for the latest i would like to check if there is any other obvious performance improvement i could make

[ghost]

@conorroche here you go.

Of note, the timings I have been giving in this issue also include some dependent tasks. With your fixes, the total time is reduced to 28 seconds. Of that, the actual swagger-doclet time is 6 seconds. Very impressive.