search

qbicsoftware / variantstore-service

The Variantstore is a Java/Groovy-based service application implemented using the Micronaut framework and enables storage and access to information on genomic variants and metadata from a connected database via a RESTful API.
MIT License
4 stars 1 forks source link

Maven clean package - Error in Packaging Tests #47

Open muellerdo opened 3 years ago

muellerdo commented 3 years ago

Heyho,

when running './mvnw clean compile' everything works fine. However, when running './mvnw clean package', I obtained the following bug.

Seems like some tests are not working:

[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running life.qbic.controller.GeneControllerSpec
10:32:28.609 [main] ERROR o.t.d.EnvironmentAndSystemPropertyClientProviderStrategy - ping failed with configuration Environment variables, system properties and defaults. Resolved dockerHost=unix:///var/run/docker.sock due to org.rnorth.ducttape.TimeoutException: Timeout waiting for result with exception
org.rnorth.ducttape.TimeoutException: Timeout waiting for result with exception
    at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:54)
    at org.testcontainers.dockerclient.DockerClientProviderStrategy.ping(DockerClientProviderStrategy.java:182)
    at org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy.test(EnvironmentAndSystemPropertyClientProviderStrategy.java:41)
    at org.testcontainers.dockerclient.DockerClientProviderStrategy.lambda$getFirstValidStrategy$2(DockerClientProviderStrategy.java:118)
    at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)
    at java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:302)
    at java.util.stream.Streams$ConcatSpliterator.tryAdvance(Streams.java:731)
    at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
    at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.findAny(ReferencePipeline.java:469)
    at org.testcontainers.dockerclient.DockerClientProviderStrategy.getFirstValidStrategy(DockerClientProviderStrategy.java:154)
    at org.testcontainers.DockerClientFactory.getOrInitializeStrategy(DockerClientFactory.java:126)
    at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:147)
    at org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
    at org.testcontainers.LazyDockerClient.listImagesCmd(LazyDockerClient.java:12)
    at org.testcontainers.images.LocalImagesCache.maybeInitCache(LocalImagesCache.java:68)
    at org.testcontainers.images.LocalImagesCache.get(LocalImagesCache.java:32)
    at org.testcontainers.images.AbstractImagePullPolicy.shouldPull(AbstractImagePullPolicy.java:18)
    at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:59)
    at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:26)
    at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:20)
    at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:27)
    at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1277)
    at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:613)
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:320)
    at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:311)
    at org.testcontainers.lifecycle.Startable$start.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:119)
    at life.qbic.controller.TestContainerSpecification.<clinit>(TestContainerSpecification.groovy:19)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.lang.Class.newInstance(Class.java:442)
    at org.spockframework.runtime.BaseSpecRunner.createSpecInstance(BaseSpecRunner.java:104)
    at org.spockframework.runtime.BaseSpecRunner.run(BaseSpecRunner.java:62)
    at org.spockframework.runtime.Sputnik.run(Sputnik.java:63)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:364)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:237)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:158)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
    at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
Caused by: java.lang.RuntimeException: java.io.IOException: native connect() failed : No such file or directory
    at com.github.dockerjava.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:301)
    at com.github.dockerjava.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:271)
    at com.github.dockerjava.okhttp.OkHttpInvocationBuilder.get(OkHttpInvocationBuilder.java:239)
    at com.github.dockerjava.core.exec.PingCmdExec.execute(PingCmdExec.java:26)
    at com.github.dockerjava.core.exec.PingCmdExec.execute(PingCmdExec.java:12)
    at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
    at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
    at org.testcontainers.dockerclient.DockerClientProviderStrategy.lambda$null$4(DockerClientProviderStrategy.java:185)
    at org.rnorth.ducttape.ratelimits.RateLimiter.getWhenReady(RateLimiter.java:51)
    at org.testcontainers.dockerclient.DockerClientProviderStrategy.lambda$ping$5(DockerClientProviderStrategy.java:183)
    at org.rnorth.ducttape.unreliables.Unreliables.lambda$retryUntilSuccess$0(Unreliables.java:43)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: native connect() failed : No such file or directory
    at com.github.dockerjava.okhttp.UnixDomainSocket.connect(UnixDomainSocket.java:157)
    at com.github.dockerjava.okhttp.UnixSocketFactory$1.connect(UnixSocketFactory.java:27)
    at org.testcontainers.shaded.okhttp3.internal.platform.Platform.connectSocket(Platform.java:130)
    at org.testcontainers.shaded.okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:263)
    at org.testcontainers.shaded.okhttp3.internal.connection.RealConnection.connect(RealConnection.java:183)
    at org.testcontainers.shaded.okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
    at org.testcontainers.shaded.okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
    at org.testcontainers.shaded.okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
    at org.testcontainers.shaded.okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
    at org.testcontainers.shaded.okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
    at org.testcontainers.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at org.testcontainers.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
    at org.testcontainers.shaded.okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
    at org.testcontainers.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at org.testcontainers.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
    at org.testcontainers.shaded.okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
    at org.testcontainers.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at org.testcontainers.shaded.okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
    at org.testcontainers.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at org.testcontainers.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
    at org.testcontainers.shaded.okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
    at org.testcontainers.shaded.okhttp3.RealCall.execute(RealCall.java:81)
    at com.github.dockerjava.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:276)
    ... 14 common frames omitted
10:32:28.632 [main] ERROR o.t.d.DockerClientProviderStrategy - Could not find a valid Docker environment. Please check configuration. Attempted configurations were:
10:32:28.632 [main] ERROR o.t.d.DockerClientProviderStrategy -     EnvironmentAndSystemPropertyClientProviderStrategy: failed with exception InvalidConfigurationException (ping failed)
10:32:28.633 [main] ERROR o.t.d.DockerClientProviderStrategy -     UnixSocketClientProviderStrategy: failed with exception InvalidConfigurationException (ping failed). Root cause NoSuchFileException (/var/run/docker.sock)
10:32:28.633 [main] ERROR o.t.d.DockerClientProviderStrategy - As no valid configuration was found, execution cannot continue
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 11.219 s <<< FAILURE! - in life.qbic.controller.GeneControllerSpec
[ERROR] life.qbic.controller.GeneControllerSpec  Time elapsed: 11.184 s  <<< ERROR!
org.spockframework.util.InternalSpockError: Failed to instantiate spec 'GeneControllerSpec'
Caused by: java.lang.ExceptionInInitializerError
Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed
Caused by: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=mariadb:10.4, imagePullPolicy=DefaultPullPolicy())
Caused by: java.lang.IllegalStateException: Could not find a valid Docker environment. Please see logs and check configuration

[INFO] Running life.qbic.controller.VariantControllerSpec
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.001 s <<< FAILURE! - in life.qbic.controller.VariantControllerSpec
[ERROR] life.qbic.controller.VariantControllerSpec  Time elapsed: 0.001 s  <<< ERROR!
org.spockframework.util.InternalSpockError: Failed to instantiate spec 'VariantControllerSpec'
Caused by: java.lang.NoClassDefFoundError: Could not initialize class life.qbic.controller.TestContainerSpecification

[INFO] Running life.qbic.controller.PropertyValueSpec
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 s <<< FAILURE! - in life.qbic.controller.PropertyValueSpec
[ERROR] life.qbic.controller.PropertyValueSpec  Time elapsed: 0 s  <<< ERROR!
org.spockframework.util.InternalSpockError: Failed to instantiate spec 'PropertyValueSpec'
Caused by: java.lang.NoClassDefFoundError: Could not initialize class life.qbic.controller.TestContainerSpecification

[INFO] Running life.qbic.controller.DatabaseSpec
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.001 s <<< FAILURE! - in life.qbic.controller.DatabaseSpec
[ERROR] life.qbic.controller.DatabaseSpec  Time elapsed: 0.001 s  <<< ERROR!
org.spockframework.util.InternalSpockError: Failed to instantiate spec 'DatabaseSpec'
Caused by: java.lang.NoClassDefFoundError: Could not initialize class life.qbic.controller.TestContainerSpecification

[INFO] Running life.qbic.controller.CaseControllerSpec
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 s <<< FAILURE! - in life.qbic.controller.CaseControllerSpec
[ERROR] life.qbic.controller.CaseControllerSpec  Time elapsed: 0 s  <<< ERROR!
org.spockframework.util.InternalSpockError: Failed to instantiate spec 'CaseControllerSpec'
Caused by: java.lang.NoClassDefFoundError: Could not initialize class life.qbic.controller.TestContainerSpecification

[INFO] Running life.qbic.controller.SampleControllerSpec
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 s <<< FAILURE! - in life.qbic.controller.SampleControllerSpec
[ERROR] life.qbic.controller.SampleControllerSpec  Time elapsed: 0 s  <<< ERROR!
org.spockframework.util.InternalSpockError: Failed to instantiate spec 'SampleControllerSpec'
Caused by: java.lang.NoClassDefFoundError: Could not initialize class life.qbic.controller.TestContainerSpecification

[INFO] Running life.qbic.controller.BeaconControllerSpec
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.001 s <<< FAILURE! - in life.qbic.controller.BeaconControllerSpec
[ERROR] life.qbic.controller.BeaconControllerSpec  Time elapsed: 0 s  <<< ERROR!
org.spockframework.util.InternalSpockError: Failed to instantiate spec 'BeaconControllerSpec'
Caused by: java.lang.NoClassDefFoundError: Could not initialize class life.qbic.controller.TestContainerSpecification

[INFO] Running life.qbic.controller.SecuritySpec
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 s <<< FAILURE! - in life.qbic.controller.SecuritySpec
[ERROR] life.qbic.controller.SecuritySpec  Time elapsed: 0 s  <<< ERROR!
org.spockframework.util.InternalSpockError: Failed to instantiate spec 'SecuritySpec'
Caused by: java.lang.NoClassDefFoundError: Could not initialize class life.qbic.controller.TestContainerSpecification

[INFO] Running life.qbic.controller.SwaggerSpec
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.001 s <<< FAILURE! - in life.qbic.controller.SwaggerSpec
[ERROR] life.qbic.controller.SwaggerSpec  Time elapsed: 0.001 s  <<< ERROR!
org.spockframework.util.InternalSpockError: Failed to instantiate spec 'SwaggerSpec'
Caused by: java.lang.NoClassDefFoundError: Could not initialize class life.qbic.controller.TestContainerSpecification

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   BeaconControllerSpec » InternalSpock Failed to instantiate spec 'BeaconControl...
[ERROR]   CaseControllerSpec » InternalSpock Failed to instantiate spec 'CaseControllerS...
[ERROR]   DatabaseSpec » InternalSpock Failed to instantiate spec 'DatabaseSpec'
[ERROR]   GeneControllerSpec » InternalSpock Failed to instantiate spec 'GeneControllerS...
[ERROR]   PropertyValueSpec » InternalSpock Failed to instantiate spec 'PropertyValueSpe...
[ERROR]   SampleControllerSpec » InternalSpock Failed to instantiate spec 'SampleControl...
[ERROR]   SecuritySpec » InternalSpock Failed to instantiate spec 'SecuritySpec'
[ERROR]   SwaggerSpec » InternalSpock Failed to instantiate spec 'SwaggerSpec'
[ERROR]   VariantControllerSpec » InternalSpock Failed to instantiate spec 'VariantContr...
[INFO] 
[ERROR] Tests run: 9, Failures: 0, Errors: 9, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 49.589 s
[INFO] Finished at: 2021-04-20T10:32:28Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on project variantstore: There are test failures.
[ERROR] 
[ERROR] Please refer to /root/variantstore/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
muellerdo commented 3 years ago

Full log: log.txt

christopher-mohr commented 3 years ago

Hi @muellerdo, thanks for reporting. I guess you don't have a running docker instance? That should be the issue since it is required by Testcontainers which is used in the tests.

muellerdo commented 3 years ago

Hey @christopher-mohr,

thanks for the fast response.

A docker instance of a connected database or what do you mean with Testcontainers? No. In my current setup, the jar packaging runs in a separated container without any connected DB.

Ah, or does the testing spawn new containers?

muellerdo commented 3 years ago

Maybe, this can be interesting for this issue: https://www.testcontainers.org/supported_docker_environment/continuous_integration/dind_patterns/

christopher-mohr commented 3 years ago

Hey @christopher-mohr,

thanks for the fast response.

A docker instance of a connected database or what do you mean with Testcontainers? No. In my current setup, the jar packaging runs in a separated container without any connected DB.

Ah, or does the testing spawn new containers?

You actually need docker to use testconainers. I guess this might be helpful.

christopher-mohr commented 3 years ago

Maybe, this can be interesting for this issue: https://www.testcontainers.org/supported_docker_environment/continuous_integration/dind_patterns/

Exactly. Sorry, I didn't see it before submitting my previous answer. :)

muellerdo commented 3 years ago

Mhm, it seems that the problem requires a little bit of structure redesigning of our docker-compose.

The problem is that testcontainers spawns containers in our container. This could be fixed when running a privileged instance or mount the docker socket into the container. However, the issue is now that we run the tests in the building process of our docker image. In the building process it is not possible to give the building container the ability to spawn new containers.

As far as I understand this issue, we have now the following options:

Any other suggestions or ideas?

I personally favor the mvn clean compile in the dockerfile and running mvn test and then mvn exec:exec in a entry-script. Packing it to a jar and starting it from there, would also be an option but be redundant after compiling it already. Testing it everytime a docker container restart can absolutely make sense.

christopher-mohr commented 3 years ago

Do you think we should keep the tests as part of the docker image? Would there be cases where we would use the docker image for untested code?

christopher-mohr commented 3 years ago

I finally added a GitHub Action workflow for compiling, testing, packaging (verify) which is executed for every pushand PR.

christopher-mohr commented 3 years ago

Hi @muellerdo, since there hasn't been any progress for a while, should I just merge your PR for now and take it from there?

muellerdo commented 3 years ago

Hi @christopher-mohr,

I have to apologize for the stalling.

Hi @muellerdo, since there hasn't been any progress for a while, should I just merge your PR for now and take it from there?

This would be a good idea. Thank you!