search

googlegenomics / dataflow-java

Google Cloud Dataflow pipelines such as Identity-By-State as well as useful utility classes.
Apache License 2.0
36 stars 31 forks source link

Running mvn package takes too long #98

Closed Careyjmac closed 9 years ago

Careyjmac commented 9 years ago

Building this package using the new system of running mvn package instead of mvn compile and mvn bundle:bundle takes around an hour on my machine, which is too long for a development environment. For now, I have reverted my local pom.xml file to the previous style (reverting the changes made here https://github.com/googlegenomics/dataflow-java/pull/96/files#diff-2 ) but we should have a fix for this using the mvn package command in the future.

pgrosu commented 9 years ago

@Careyjmac Are you sure? I just performed it, and mine takes less than 30 seconds. Are you performing this via Eclipse? Below are my results:

$ date
Tue Jun 16 14:05:33 EDT 2015
$ git clone https://github.com/googlegenomics/dataflow-java
Initialized empty Git repository in /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/.git/
remote: Counting objects: 5321, done.
remote: Total 5321 (delta 0), reused 0 (delta 0), pack-reused 5321
Receiving objects: 100% (5321/5321), 121.14 MiB | 25.65 MiB/s, done.
Resolving deltas: 100% (1721/1721), done.
$ cd dataflow-java/
$ cp ../../../client_secrets.json .
$ time( mvn package )
[INFO] Scanning for projects...
[INFO]
[INFO] Using the builder org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder with a thread count of 1
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Google Genomics and Dataflow v1beta2-0.12-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ google-genomics-dataflow ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.2:compile (default-compile) @ google-genomics-dataflow ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 49 source files to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/classes
[INFO] /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/coders/DelegatingAtomicCoder.java: Some input files use or override a deprecated API.
[INFO] /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/coders/DelegatingAtomicCoder.java: Recompile with -Xlint:deprecation for details.
[INFO] /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/pipelines/CalculateCoverage.java: /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/pipelines/CalculateCoverage.java uses unchecked or unsafe operations.
[INFO] /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/pipelines/CalculateCoverage.java: Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ google-genomics-dataflow ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.2:testCompile (default-testCompile) @ google-genomics-dataflow ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 16 source files to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/test-classes
[INFO] /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/src/test/java/com/google/cloud/genomics/dataflow/pipelines/CalculateCoverageTest.java: Some input files use unchecked or unsafe operations.
[INFO] /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/src/test/java/com/google/cloud/genomics/dataflow/pipelines/CalculateCoverageTest.java: Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ google-genomics-dataflow ---
[INFO] Surefire report directory: /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running com.google.cloud.dataflow.coders.GenericJsonCoderTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.243 sec
Running com.google.cloud.genomics.dataflow.utils.PairGeneratorTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.073 sec
Running com.google.cloud.genomics.dataflow.utils.AnnotationUtilsTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.03 sec
Running com.google.cloud.genomics.dataflow.utils.VariantUtilsTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec
Running com.google.cloud.genomics.dataflow.readers.bam.ShardingPolicyTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec
Running com.google.cloud.genomics.dataflow.readers.bam.BAMShardTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec
Running com.google.cloud.genomics.dataflow.readers.bam.ReadConverterTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.089 sec
Running com.google.cloud.genomics.dataflow.pipelines.CalculateCoverageTest
Jun 16, 2015 2:06:10 PM com.google.cloud.dataflow.sdk.Pipeline applyInternal
WARNING: Transform Create2 does not have a stable unique name.  In the future, this will prevent reloading streaming pipelines
Jun 16, 2015 2:06:10 PM com.google.cloud.dataflow.sdk.Pipeline applyInternal
WARNING: Transform AsIterable2 does not have a stable unique name.  In the future, this will prevent reloading streaming pipelines
Jun 16, 2015 2:06:10 PM com.google.cloud.dataflow.sdk.Pipeline applyInternal
WARNING: Transform Create3 does not have a stable unique name.  In the future, this will prevent reloading streaming pipelines
Jun 16, 2015 2:06:10 PM com.google.cloud.dataflow.sdk.coders.CoderRegistry getDefaultCoderFactory
INFO: No Coder registered for class com.google.cloud.genomics.dataflow.model.PosRgsMq$MappingQuality
Jun 16, 2015 2:06:10 PM com.google.cloud.dataflow.sdk.coders.CoderRegistry getDefaultCoderFactory
INFO: No Coder registered for class com.google.cloud.genomics.dataflow.model.PosRgsMq$MappingQuality
Jun 16, 2015 2:06:10 PM com.google.cloud.dataflow.sdk.Pipeline applyInternal
WARNING: Transform Create2 does not have a stable unique name.  In the future, this will prevent reloading streaming pipelines
Jun 16, 2015 2:06:10 PM com.google.cloud.dataflow.sdk.Pipeline applyInternal
WARNING: Transform AsIterable2 does not have a stable unique name.  In the future, this will prevent reloading streaming pipelines
Jun 16, 2015 2:06:10 PM com.google.cloud.dataflow.sdk.Pipeline applyInternal
WARNING: Transform Create3 does not have a stable unique name.  In the future, this will prevent reloading streaming pipelines
Jun 16, 2015 2:06:10 PM com.google.cloud.dataflow.sdk.coders.CoderRegistry getDefaultCoderFactory
INFO: No Coder registered for class com.google.cloud.genomics.dataflow.model.PosRgsMq$MappingQuality
Jun 16, 2015 2:06:10 PM com.google.cloud.dataflow.sdk.coders.CoderRegistry getDefaultCoderFactory
INFO: No Coder registered for class com.google.cloud.genomics.dataflow.model.PosRgsMq$MappingQuality
Jun 16, 2015 2:06:10 PM com.google.cloud.dataflow.sdk.Pipeline applyInternal
WARNING: Transform Create2 does not have a stable unique name.  In the future, this will prevent reloading streaming pipelines
Jun 16, 2015 2:06:10 PM com.google.cloud.dataflow.sdk.Pipeline applyInternal
WARNING: Transform AsIterable2 does not have a stable unique name.  In the future, this will prevent reloading streaming pipelines
Jun 16, 2015 2:06:10 PM com.google.cloud.dataflow.sdk.Pipeline applyInternal
WARNING: Transform Create3 does not have a stable unique name.  In the future, this will prevent reloading streaming pipelines
Jun 16, 2015 2:06:10 PM com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner run
INFO: Executing pipeline using the DirectPipelineRunner.
Jun 16, 2015 2:06:11 PM com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner run
INFO: Pipeline execution complete.
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.457 sec
Running com.google.cloud.genomics.dataflow.functions.CallSimilarityCalculatorTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec
Running com.google.cloud.genomics.dataflow.functions.JoinNonVariantSegmentsWithVariantsTest
Jun 16, 2015 2:06:11 PM com.google.cloud.genomics.dataflow.utils.DataflowWorkarounds registerGenomicsCoders
INFO: Registering coders for genomics classes
Jun 16, 2015 2:06:11 PM org.reflections.Reflections scan
INFO: Reflections took 70 ms to scan 10 urls, producing 0 keys and 0 values
Jun 16, 2015 2:06:11 PM com.google.cloud.dataflow.sdk.Pipeline applyInternal
WARNING: Transform Create2 does not have a stable unique name.  In the future, this will prevent reloading streaming pipelines
Jun 16, 2015 2:06:11 PM com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner run
INFO: Executing pipeline using the DirectPipelineRunner.
Jun 16, 2015 2:06:11 PM com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner run
INFO: Pipeline execution complete.
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.218 sec
Running com.google.cloud.genomics.dataflow.functions.AlleleSimilarityCalculatorTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec
Running com.google.cloud.genomics.dataflow.functions.ExtractSimilarCallsetsTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec

Results :

Tests run: 32, Failures: 0, Errors: 0, Skipped: 0

[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ google-genomics-dataflow ---
[INFO] Building jar: /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/google-genomics-dataflow-v1beta2-0.12-SNAPSHOT.jar
[INFO]
[INFO] --- maven-shade-plugin:2.3:shade (default) @ google-genomics-dataflow ---
[INFO] Including com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:0.4.150414 in the shaded jar.
[INFO] Including com.google.apis:google-api-services-dataflow:jar:v1beta3-rev9-1.19.1 in the shaded jar.
[INFO] Including com.google.cloud.dataflow:google-cloud-dataflow-java-proto-library-all:jar:0.3.150406 in the shaded jar.
[INFO] Including com.google.apis:google-api-services-bigquery:jar:v2-rev187-1.19.1 in the shaded jar.
[INFO] Including com.google.apis:google-api-services-compute:jar:v1-rev46-1.19.1 in the shaded jar.
[INFO] Including com.google.apis:google-api-services-pubsub:jar:v1beta1-rev18-1.19.1 in the shaded jar.
[INFO] Including com.google.apis:google-api-services-storage:jar:v1-rev25-1.19.1 in the shaded jar.
[INFO] Including com.google.apis:google-api-services-datastore-protobuf:jar:v1beta2-rev1-2.1.2 in the shaded jar.
[INFO] Including com.google.http-client:google-http-client-protobuf:jar:1.15.0-rc in the shaded jar.
[INFO] Including com.google.http-client:google-http-client-jackson:jar:1.15.0-rc in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.4.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.4.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-databind:jar:2.4.2 in the shaded jar.
[INFO] Including org.slf4j:slf4j-api:jar:1.7.7 in the shaded jar.
[INFO] Including org.slf4j:slf4j-jdk14:jar:1.7.7 in the shaded jar.
[INFO] Including org.apache.avro:avro:jar:1.7.7 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-core-asl:jar:1.9.13 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 in the shaded jar.
[INFO] Including com.thoughtworks.paranamer:paranamer:jar:2.3 in the shaded jar.
[INFO] Including org.apache.commons:commons-compress:jar:1.9 in the shaded jar.
[INFO] Including joda-time:joda-time:jar:2.4 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-server:jar:9.2.6.v20141205 in the shaded jar.
[INFO] Including javax.servlet:javax.servlet-api:jar:3.1.0 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-http:jar:9.2.6.v20141205 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-io:jar:9.2.6.v20141205 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-jmx:jar:9.2.6.v20141205 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-util:jar:9.2.6.v20141205 in the shaded jar.
[INFO] Including com.google.api-client:google-api-client:jar:1.19.0 in the shaded jar.
[INFO] Including com.google.oauth-client:google-oauth-client:jar:1.19.0 in the shaded jar.
[INFO] Including com.google.code.findbugs:jsr305:jar:1.3.9 in the shaded jar.
[INFO] Including com.google.oauth-client:google-oauth-client-jetty:jar:1.19.0 in the shaded jar.
[INFO] Including com.google.oauth-client:google-oauth-client-java6:jar:1.19.0 in the shaded jar.
[INFO] Including com.google.http-client:google-http-client:jar:1.19.0 in the shaded jar.
[INFO] Including org.apache.httpcomponents:httpclient:jar:4.0.1 in the shaded jar.
[INFO] Including org.apache.httpcomponents:httpcore:jar:4.0.1 in the shaded jar.
[INFO] Including commons-codec:commons-codec:jar:1.3 in the shaded jar.
[INFO] Including com.google.http-client:google-http-client-jackson2:jar:1.19.0 in the shaded jar.
[INFO] Including com.google.apis:google-api-services-genomics:jar:v1beta2-rev25-1.19.1 in the shaded jar.
[INFO] Including com.google.cloud.genomics:google-genomics-utils:jar:v1beta2-0.25 in the shaded jar.
[INFO] Including com.google.api-client:google-api-client-java6:jar:1.19.0 in the shaded jar.
[INFO] Including io.grpc:grpc-all:jar:0.7.0 in the shaded jar.
[INFO] Including io.grpc:grpc-stub:jar:0.7.0 in the shaded jar.
[INFO] Including io.grpc:grpc-okhttp:jar:0.7.0 in the shaded jar.
[INFO] Including com.squareup.okhttp:okhttp:jar:2.3.0 in the shaded jar.
[INFO] Including com.squareup.okio:okio:jar:1.3.0 in the shaded jar.
[INFO] Including io.grpc:grpc-netty:jar:0.7.0 in the shaded jar.
[INFO] Including io.netty:netty-codec-http2:jar:4.1.0.Beta5 in the shaded jar.
[INFO] Including io.netty:netty-codec-http:jar:4.1.0.Beta5 in the shaded jar.
[INFO] Including io.netty:netty-codec:jar:4.1.0.Beta5 in the shaded jar.
[INFO] Including io.netty:netty-handler:jar:4.1.0.Beta5 in the shaded jar.
[INFO] Including io.netty:netty-buffer:jar:4.1.0.Beta5 in the shaded jar.
[INFO] Including io.netty:netty-common:jar:4.1.0.Beta5 in the shaded jar.
[INFO] Including io.netty:netty-transport:jar:4.1.0.Beta5 in the shaded jar.
[INFO] Including io.netty:netty-resolver:jar:4.1.0.Beta5 in the shaded jar.
[INFO] Including com.twitter:hpack:jar:0.10.1 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf:jar:0.7.0 in the shaded jar.
[INFO] Including io.grpc:grpc-core:jar:0.7.0 in the shaded jar.
[INFO] Including io.grpc:grpc-auth:jar:0.7.0 in the shaded jar.
[INFO] Including com.google.auth:google-auth-library-oauth2-http:jar:0.1.0 in the shaded jar.
[INFO] Including com.google.auth:google-auth-library-credentials:jar:0.1.0 in the shaded jar.
[INFO] Including javax:javaee-api:jar:7.0 in the shaded jar.
[INFO] Including com.sun.mail:javax.mail:jar:1.5.0 in the shaded jar.
[INFO] Including javax.activation:activation:jar:1.1 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf-nano:jar:0.7.0 in the shaded jar.
[INFO] Including com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-2 in the shaded jar.
[INFO] Including com.google.guava:guava:jar:18.0 in the shaded jar.
[INFO] Including gov.nist.math.jama:gov.nist.math.jama:jar:1.1.1 in the shaded jar.
[INFO] Including org.reflections:reflections:jar:0.9.9 in the shaded jar.
[INFO] Including org.javassist:javassist:jar:3.18.2-GA in the shaded jar.
[INFO] Including com.google.code.findbugs:annotations:jar:2.0.1 in the shaded jar.
[INFO] Including org.mortbay.jetty:jetty:jar:6.1.26 in the shaded jar.
[INFO] Including org.mortbay.jetty:servlet-api:jar:2.5-20081211 in the shaded jar.
[INFO] Including org.mortbay.jetty:jetty-util:jar:6.1.26 in the shaded jar.
[INFO] Including javax.servlet:servlet-api:jar:2.5 in the shaded jar.
[INFO] Including com.github.samtools:htsjdk:jar:1.128 in the shaded jar.
[INFO] Including org.scala-lang:scala-library:jar:2.11.4 in the shaded jar.
[INFO] Including org.apache.commons:commons-jexl:jar:2.1.1 in the shaded jar.
[INFO] Including commons-logging:commons-logging:jar:1.2 in the shaded jar.
[INFO] Including org.xerial.snappy:snappy-java:jar:1.1.1.3 in the shaded jar.
[INFO] Including com.google.cloud.genomics:gatk-tools-java:jar:1.0 in the shaded jar.
[INFO] Including com.beust:jcommander:jar:1.35 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:3.0.0-alpha-3 in the shaded jar.
[WARNING] grpc-all-0.7.0.jar, grpc-protobuf-nano-0.7.0.jar define 4 overlappping classes:
[WARNING]   - io.grpc.protobuf.nano.DeferredNanoProtoInputStream
[WARNING]   - io.grpc.protobuf.nano.NanoUtils
[WARNING]   - io.grpc.protobuf.nano.NanoUtils$1
[WARNING]   - io.grpc.protobuf.nano.Parser
[WARNING] google-genomics-utils-v1beta2-0.25.jar, protobuf-java-3.0.0-alpha-3.jar define 168 overlappping classes:
[WARNING]   - com.google.protobuf.DescriptorProtos$FileDescriptorProtoOrBuilder
[WARNING]   - com.google.protobuf.DescriptorProtos$DescriptorProto$Builder
[WARNING]   - com.google.protobuf.DescriptorProtos
[WARNING]   - com.google.protobuf.FieldMask
[WARNING]   - com.google.protobuf.DescriptorProtos$FileOptions$OptimizeMode
[WARNING]   - com.google.protobuf.Any$1
[WARNING]   - com.google.protobuf.Timestamp
[WARNING]   - com.google.protobuf.TimestampProto
[WARNING]   - com.google.protobuf.StructProto
[WARNING]   - com.google.protobuf.DescriptorProtos$OneofDescriptorProto$1
[WARNING]   - 158 more...
[WARNING] javaee-api-7.0.jar, javax.servlet-api-3.1.0.jar define 37 overlappping classes:
[WARNING]   - javax.servlet.WriteListener
[WARNING]   - javax.servlet.annotation.HttpConstraint
[WARNING]   - javax.servlet.DispatcherType
[WARNING]   - javax.servlet.descriptor.JspPropertyGroupDescriptor
[WARNING]   - javax.servlet.Registration
[WARNING]   - javax.servlet.SessionTrackingMode
[WARNING]   - javax.servlet.http.WebConnection
[WARNING]   - javax.servlet.http.HttpSessionIdListener
[WARNING]   - javax.servlet.annotation.ServletSecurity$EmptyRoleSemantic
[WARNING]   - javax.servlet.annotation.HandlesTypes
[WARNING]   - 27 more...
[WARNING] grpc-all-0.7.0.jar, grpc-auth-0.7.0.jar define 2 overlappping classes:
[WARNING]   - io.grpc.auth.ClientAuthInterceptor$1
[WARNING]   - io.grpc.auth.ClientAuthInterceptor
[WARNING] grpc-all-0.7.0.jar, grpc-protobuf-0.7.0.jar define 4 overlappping classes:
[WARNING]   - io.grpc.protobuf.ProtoUtils$2
[WARNING]   - io.grpc.protobuf.ProtoUtils
[WARNING]   - io.grpc.protobuf.ProtoUtils$1
[WARNING]   - io.grpc.protobuf.DeferredProtoInputStream
[WARNING] grpc-all-0.7.0.jar, grpc-stub-0.7.0.jar define 30 overlappping classes:
[WARNING]   - io.grpc.stub.ServerCalls$UnaryRequestMethod
[WARNING]   - io.grpc.stub.StreamObserver
[WARNING]   - io.grpc.stub.StreamRecorder
[WARNING]   - io.grpc.stub.ServerCalls$EmptyServerCallListener
[WARNING]   - io.grpc.stub.MetadataUtils$2$1
[WARNING]   - io.grpc.stub.AbstractStub$StubConfigBuilder
[WARNING]   - io.grpc.stub.MetadataUtils$1$1
[WARNING]   - io.grpc.stub.MetadataUtils$2$1$1
[WARNING]   - io.grpc.stub.Calls$CallToStreamObserverAdapter
[WARNING]   - io.grpc.stub.MetadataUtils
[WARNING]   - 20 more...
[WARNING] javaee-api-7.0.jar, javax.mail-1.5.0.jar define 135 overlappping classes:
[WARNING]   - javax.mail.internet.HeaderTokenizer
[WARNING]   - javax.mail.search.MessageIDTerm
[WARNING]   - javax.mail.Session$3
[WARNING]   - javax.mail.Message
[WARNING]   - javax.mail.internet.MimePart
[WARNING]   - javax.mail.EventQueue
[WARNING]   - javax.mail.search.DateTerm
[WARNING]   - javax.mail.Session
[WARNING]   - javax.mail.Session$6
[WARNING]   - javax.mail.search.SentDateTerm
[WARNING]   - 125 more...
[WARNING] grpc-all-0.7.0.jar, grpc-core-0.7.0.jar define 156 overlappping classes:
[WARNING]   - io.grpc.AbstractServerBuilder
[WARNING]   - io.grpc.ClientInterceptors$1
[WARNING]   - io.grpc.transport.ClientTransport$Listener
[WARNING]   - io.grpc.Marshaller
[WARNING]   - io.grpc.SharedResourceHolder$1
[WARNING]   - io.grpc.AbstractChannelBuilder$2
[WARNING]   - io.grpc.DeferredInputStream
[WARNING]   - io.grpc.Call
[WARNING]   - io.grpc.ServerCallHandler
[WARNING]   - io.grpc.ServerImpl$ServerCallImpl
[WARNING]   - 146 more...
[WARNING] grpc-all-0.7.0.jar, grpc-netty-0.7.0.jar define 64 overlappping classes:
[WARNING]   - io.grpc.transport.netty.WriteQueue
[WARNING]   - io.grpc.transport.netty.NettyClientStream$1
[WARNING]   - io.grpc.transport.netty.ProtocolNegotiators$3
[WARNING]   - io.grpc.transport.netty.NettyServerBuilder$1
[WARNING]   - io.grpc.transport.netty.NettyWritableBufferAllocator
[WARNING]   - io.grpc.transport.netty.SendResponseHeadersCommand
[WARNING]   - io.grpc.transport.netty.GrpcSslContexts
[WARNING]   - io.grpc.transport.netty.NettyServerTransport$1
[WARNING]   - io.grpc.transport.netty.Utils
[WARNING]   - io.grpc.transport.netty.NettyClientHandler
[WARNING]   - 54 more...
[WARNING] jsr305-1.3.9.jar, annotations-2.0.1.jar define 34 overlappping classes:
[WARNING]   - javax.annotation.RegEx
[WARNING]   - javax.annotation.concurrent.Immutable
[WARNING]   - javax.annotation.meta.TypeQualifierDefault
[WARNING]   - javax.annotation.meta.TypeQualifier
[WARNING]   - javax.annotation.Syntax
[WARNING]   - javax.annotation.Nonnull
[WARNING]   - javax.annotation.CheckReturnValue
[WARNING]   - javax.annotation.CheckForNull
[WARNING]   - javax.annotation.meta.TypeQualifierNickname
[WARNING]   - javax.annotation.MatchesPattern
[WARNING]   - 24 more...
[WARNING] javaee-api-7.0.jar, servlet-api-2.5.jar, servlet-api-2.5-20081211.jar, javax.servlet-api-3.1.0.jar define 42 overlappping classes:
[WARNING]   - javax.servlet.http.HttpSessionBindingEvent
[WARNING]   - javax.servlet.http.Cookie
[WARNING]   - javax.servlet.http.NoBodyResponse
[WARNING]   - javax.servlet.ServletOutputStream
[WARNING]   - javax.servlet.ServletContext
[WARNING]   - javax.servlet.http.HttpSessionListener
[WARNING]   - javax.servlet.http.HttpSessionContext
[WARNING]   - javax.servlet.FilterChain
[WARNING]   - javax.servlet.GenericServlet
[WARNING]   - javax.servlet.http.HttpServletRequestWrapper
[WARNING]   - 32 more...
[WARNING] grpc-all-0.7.0.jar, grpc-okhttp-0.7.0.jar define 37 overlappping classes:
[WARNING]   - io.grpc.transport.okhttp.OutboundFlowController$WriteStatus
[WARNING]   - io.grpc.transport.okhttp.OkHttpWritableBuffer
[WARNING]   - io.grpc.transport.okhttp.AsyncFrameWriter$8
[WARNING]   - io.grpc.transport.okhttp.AsyncFrameWriter$3
[WARNING]   - io.grpc.transport.okhttp.AsyncFrameWriter$11
[WARNING]   - io.grpc.transport.okhttp.AsyncFrameWriter$1
[WARNING]   - io.grpc.transport.okhttp.OutboundFlowController$OutboundFlowState$Frame
[WARNING]   - io.grpc.transport.okhttp.AsyncFrameWriter
[WARNING]   - io.grpc.transport.okhttp.AsyncFrameWriter$6
[WARNING]   - io.grpc.transport.okhttp.AsyncFrameWriter$10
[WARNING]   - 27 more...
[WARNING] maven-shade-plugin has detected that some .class files
[WARNING] are present in two or more JARs. When this happens, only
[WARNING] one single version of the class is copied in the uberjar.
[WARNING] Usually this is not harmful and you can skeep these
[WARNING] warnings, otherwise try to manually exclude artifacts
[WARNING] based on mvn dependency:tree -Ddetail=true and the above
[WARNING] output
[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin
[INFO] Attaching shaded artifact.
[INFO]
[INFO] --- maven-dependency-plugin:2.8:copy-dependencies (copy-dependencies) @ google-genomics-dataflow ---
[INFO] Copying annotations-2.0.1.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/annotations-2.0.1.jar
[INFO] Copying grpc-netty-0.7.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/grpc-netty-0.7.0.jar
[INFO] Copying javassist-3.18.2-GA.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/javassist-3.18.2-GA.jar
[INFO] Copying google-api-services-compute-v1-rev46-1.19.1.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/google-api-services-compute-v1-rev46-1.19.1.jar
[INFO] Copying google-api-services-bigquery-v2-rev187-1.19.1.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/google-api-services-bigquery-v2-rev187-1.19.1.jar
[INFO] Copying google-http-client-protobuf-1.15.0-rc.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/google-http-client-protobuf-1.15.0-rc.jar
[INFO] Copying commons-compress-1.9.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/commons-compress-1.9.jar
[INFO] Copying jetty-server-9.2.6.v20141205.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/jetty-server-9.2.6.v20141205.jar
[INFO] Copying google-http-client-jackson2-1.19.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/google-http-client-jackson2-1.19.0.jar
[INFO] Copying google-http-client-jackson-1.15.0-rc.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/google-http-client-jackson-1.15.0-rc.jar
[INFO] Copying netty-resolver-4.1.0.Beta5.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/netty-resolver-4.1.0.Beta5.jar
[INFO] Copying netty-codec-4.1.0.Beta5.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/netty-codec-4.1.0.Beta5.jar
[INFO] Copying google-oauth-client-1.19.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/google-oauth-client-1.19.0.jar
[INFO] Copying grpc-protobuf-nano-0.7.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/grpc-protobuf-nano-0.7.0.jar
[INFO] Copying commons-jexl-2.1.1.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/commons-jexl-2.1.1.jar
[INFO] Copying jcommander-1.35.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/jcommander-1.35.jar
[INFO] Copying scala-library-2.11.4.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/scala-library-2.11.4.jar
[INFO] Copying jetty-util-6.1.26.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/jetty-util-6.1.26.jar
[INFO] Copying google-http-client-1.19.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/google-http-client-1.19.0.jar
[INFO] Copying netty-common-4.1.0.Beta5.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/netty-common-4.1.0.Beta5.jar
[INFO] Copying google-api-services-dataflow-v1beta3-rev9-1.19.1.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/google-api-services-dataflow-v1beta3-rev9-1.19.1.jar
[INFO] Copying jetty-http-9.2.6.v20141205.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/jetty-http-9.2.6.v20141205.jar
[INFO] Copying javax.mail-1.5.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/javax.mail-1.5.0.jar
[INFO] Copying protobuf-javanano-3.0.0-alpha-2.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/protobuf-javanano-3.0.0-alpha-2.jar
[INFO] Copying hamcrest-core-1.3.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/hamcrest-core-1.3.jar
[INFO] Copying jackson-mapper-asl-1.9.13.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/jackson-mapper-asl-1.9.13.jar
[INFO] Copying google-api-client-java6-1.19.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/google-api-client-java6-1.19.0.jar
[INFO] Copying google-auth-library-oauth2-http-0.1.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/google-auth-library-oauth2-http-0.1.0.jar
[INFO] Copying grpc-all-0.7.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/grpc-all-0.7.0.jar
[INFO] Copying google-oauth-client-jetty-1.19.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/google-oauth-client-jetty-1.19.0.jar
[INFO] Copying commons-logging-1.2.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/commons-logging-1.2.jar
[INFO] Copying grpc-stub-0.7.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/grpc-stub-0.7.0.jar
[INFO] Copying gatk-tools-java-1.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/gatk-tools-java-1.0.jar
[INFO] Copying joda-time-2.4.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/joda-time-2.4.jar
[INFO] Copying jetty-jmx-9.2.6.v20141205.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/jetty-jmx-9.2.6.v20141205.jar
[INFO] Copying snappy-java-1.1.1.3.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/snappy-java-1.1.1.3.jar
[INFO] Copying paranamer-2.3.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/paranamer-2.3.jar
[INFO] Copying google-cloud-dataflow-java-proto-library-all-0.3.150406.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/google-cloud-dataflow-java-proto-library-all-0.3.150406.jar
[INFO] Copying jackson-core-asl-1.9.13.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/jackson-core-asl-1.9.13.jar
[INFO] Copying netty-transport-4.1.0.Beta5.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/netty-transport-4.1.0.Beta5.jar
[INFO] Copying google-oauth-client-java6-1.19.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/google-oauth-client-java6-1.19.0.jar
[INFO] Copying httpclient-4.0.1.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/httpclient-4.0.1.jar
[INFO] Copying google-api-services-pubsub-v1beta1-rev18-1.19.1.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/google-api-services-pubsub-v1beta1-rev18-1.19.1.jar
[INFO] Copying jackson-databind-2.4.2.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/jackson-databind-2.4.2.jar
[INFO] Copying hamcrest-all-1.3.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/hamcrest-all-1.3.jar
[INFO] Copying jsr305-1.3.9.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/jsr305-1.3.9.jar
[INFO] Copying okhttp-2.3.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/okhttp-2.3.0.jar
[INFO] Copying protobuf-java-3.0.0-alpha-3.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/protobuf-java-3.0.0-alpha-3.jar
[INFO] Copying jetty-util-9.2.6.v20141205.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/jetty-util-9.2.6.v20141205.jar
[INFO] Copying grpc-okhttp-0.7.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/grpc-okhttp-0.7.0.jar
[INFO] Copying commons-codec-1.3.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/commons-codec-1.3.jar
[INFO] Copying javaee-api-7.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/javaee-api-7.0.jar
[INFO] Copying netty-codec-http2-4.1.0.Beta5.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/netty-codec-http2-4.1.0.Beta5.jar
[INFO] Copying junit-4.11.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/junit-4.11.jar
[INFO] Copying activation-1.1.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/activation-1.1.jar
[INFO] Copying gov.nist.math.jama-1.1.1.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/gov.nist.math.jama-1.1.1.jar
[INFO] Copying google-api-services-datastore-protobuf-v1beta2-rev1-2.1.2.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/google-api-services-datastore-protobuf-v1beta2-rev1-2.1.2.jar
[INFO] Copying google-api-services-storage-v1-rev25-1.19.1.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/google-api-services-storage-v1-rev25-1.19.1.jar
[INFO] Copying grpc-protobuf-0.7.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/grpc-protobuf-0.7.0.jar
[INFO] Copying okio-1.3.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/okio-1.3.0.jar
[INFO] Copying jackson-annotations-2.4.2.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/jackson-annotations-2.4.2.jar
[INFO] Copying grpc-core-0.7.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/grpc-core-0.7.0.jar
[INFO] Copying google-cloud-dataflow-java-sdk-all-0.4.150414.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/google-cloud-dataflow-java-sdk-all-0.4.150414.jar
[INFO] Copying google-genomics-utils-v1beta2-0.25.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/google-genomics-utils-v1beta2-0.25.jar
[INFO] Copying servlet-api-2.5-20081211.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/servlet-api-2.5-20081211.jar
[INFO] Copying javax.servlet-api-3.1.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/javax.servlet-api-3.1.0.jar
[INFO] Copying jackson-core-2.4.2.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/jackson-core-2.4.2.jar
[INFO] Copying hpack-0.10.1.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/hpack-0.10.1.jar
[INFO] Copying slf4j-jdk14-1.7.7.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/slf4j-jdk14-1.7.7.jar
[INFO] Copying avro-1.7.7.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/avro-1.7.7.jar
[INFO] Copying google-auth-library-credentials-0.1.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/google-auth-library-credentials-0.1.0.jar
[INFO] Copying jetty-6.1.26.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/jetty-6.1.26.jar
[INFO] Copying guava-18.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/guava-18.0.jar
[INFO] Copying httpcore-4.0.1.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/httpcore-4.0.1.jar
[INFO] Copying servlet-api-2.5.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/servlet-api-2.5.jar
[INFO] Copying netty-handler-4.1.0.Beta5.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/netty-handler-4.1.0.Beta5.jar
[INFO] Copying htsjdk-1.128.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/htsjdk-1.128.jar
[INFO] Copying grpc-auth-0.7.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/grpc-auth-0.7.0.jar
[INFO] Copying reflections-0.9.9.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/reflections-0.9.9.jar
[INFO] Copying jetty-io-9.2.6.v20141205.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/jetty-io-9.2.6.v20141205.jar
[INFO] Copying google-api-client-1.19.0.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/google-api-client-1.19.0.jar
[INFO] Copying slf4j-api-1.7.7.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/slf4j-api-1.7.7.jar
[INFO] Copying google-api-services-genomics-v1beta2-rev25-1.19.1.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/google-api-services-genomics-v1beta2-rev25-1.19.1.jar
[INFO] Copying netty-buffer-4.1.0.Beta5.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/netty-buffer-4.1.0.Beta5.jar
[INFO] Copying netty-codec-http-4.1.0.Beta5.jar to /home/pgrosu/me/gg_dataflow/test/time-test/dataflow-java/target/dependency/netty-codec-http-4.1.0.Beta5.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 16.655 s
[INFO] Finished at: 2015-06-16T14:06:18-05:00
[INFO] Final Memory: 27M/158M
[INFO] ------------------------------------------------------------------------

real    0m28.714s
user    1m23.366s
sys     0m2.783s
$

~p

Careyjmac commented 9 years ago

Here is mine: [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 41:11 min [INFO] Finished at: 2015-06-16T11:30:56-07:00 [INFO] Final Memory: 75M/606M [INFO] ------------------------------------------------------------------------

I'll be using the work around for now until @deflaux can take a look at it.

pgrosu commented 9 years ago

Hmm, I'm not sure. I just ran it locally on my Windows laptop - just in case there might be any throughput issue - and it's less than 2 minutes:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:59 min
[INFO] Finished at: 2015-06-16T15:59:24-05:00
[INFO] Final Memory: 28M/90M
[INFO] ------------------------------------------------------------------------
c:\Google Genomics\gg_dataflow\dataflow-java>

`p

deflaux commented 9 years ago

@Careyjmac can you give it a try on a different machine? You could spin up a Google Compute Engine instance and then run the following to get it set up:

sudo apt-get update
sudo apt-get install --assume-yes openjdk-7-jdk maven git
sudo update-alternatives --config java
Careyjmac commented 9 years ago

It worked in under a minute on the compute engine instance, and then I tried running those commands locally and it worked in about 10 seconds. I think it was the way I set up maven, I had it installed via my PATH variable, and this is a lot better. Thanks!

pgrosu commented 9 years ago

Glad to hear it worked, but I doubt the PATH variable setting makes a difference since I also have it setup via PATH as well and it runs fairly quickly. Try to look for processes that take up a lot of memory and disk - it could be sometimes the antivirus that can slow things down as well.

deflaux commented 9 years ago

Glad to hear it.

If it becomes slow again, try mvn --debug package to see what its spending its time on (e.g., downloading a particular file from a slow remote server).