search

testcontainers / testcontainers-java

Testcontainers is a Java library that supports JUnit tests, providing lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container.
https://testcontainers.org
MIT License
8.02k stars 1.65k forks source link

Codeship builds fail: `can not connect to Ryuk` #581

Open gavinmh opened 6 years ago

gavinmh commented 6 years ago

TestContainers works locally. Builds on Codeship fail with the following:

2018-02-10_19:22:35.027 WARN  o.t.utility.ResourceReaper - Can not connect to Ryuk at 172.17.0.1:32768
java.net.SocketException: Broken pipe (Write failed)
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:134)
    at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:136)
    at java.lang.Thread.run(Thread.java:748)
2018-02-10_19:22:35.044 WARN  o.t.utility.ResourceReaper - Can not connect to Ryuk at 172.17.0.1:32768
java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:210)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.fill(BufferedReader.java:161)
    at java.io.BufferedReader.readLine(BufferedReader.java:324)
    at java.io.BufferedReader.readLine(BufferedReader.java:389)
    at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:139)
    at java.lang.Thread.run(Thread.java:748)

I do not use volume mapping for this service, but I use volumes to build other services. My understanding of https://www.testcontainers.org/usage/inside_docker.html is that I do not need to mount the Docker socket volume.

I've set add_docker: true in my codeship-services.yml, which does the following:

Add the environment variables DOCKER_HOST, DOCKER_TLS_VERIFY and DOCKER_CERT_PATH from the host. If DOCKER_CERT_PATH is set, it will mount the certificate directory through to the container. See add_docker for an example using Docker-in-Docker.

https://documentation.codeship.com/pro/builds-and-configuration/services/

estebangarcia commented 6 years ago

I'm having the same issue with CircleCI. Using v1.6.0

kiview commented 6 years ago

Hi @estebangarcia, would it be possible for you to migrate to CircleCI 2.0? We are using CircleCI 2.0 in Testcontainers ourselves, so everything should work fine out of the box (look at our CircleCI config).

Since pre 2.0 CircleCI was using lxc directly themselves (AFAIK?), you have to jump through a lot more hoops to get Testcontainers working there.

bsideup commented 6 years ago

Hi @gavinmh, Could you please try with 1.7.0?

kiview commented 6 years ago

Since this is possibly fixed by the fix for #621 I'll close this for now. @gavinmh I'll gladly reopen the issue if the problems persists for you on 1.7.0 :slightly_smiling_face:

kszalontai commented 6 years ago

I'm having the same issue with version 1.8.1 Codeship pro, add_docker=true, ryuk.container.timeout = 120

sic2 commented 6 years ago

Yes, please. Re-open this issue since it still persists in 1.8.1

kszalontai commented 6 years ago

@kiview Could you please reopen this? ty

kiview commented 6 years ago

Reopened as requested, bug seems to be still present.

bountin commented 6 years ago

We're experiencing the same when the load on the build server (Jenkins) is (very) high.

One thing that eased the pain was upgrading docker where we used to use an early version of 2017. The docker service wasn't responsive during peak testing (eight parallel test runs) which lead me to that direction.

Still, we had to reduce the number of parallel builds because of testcontainer timeouts. I haven't tried to increase ryuk.container.timeout because IMO thirty seconds should be enough - also in a high load scenario.

thammerl commented 6 years ago

@bountin I don't think the timeout is related to the issue we are having. I turned on the testcontainers logging and this is what I got:

14:56:23.689 INFO  [main] o.t.d.DockerClientProviderStrategy - [] - Loaded org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy from ~/.testcontainers.properties, will try it first
14:56:23.741 INFO  [main] o.t.d.DockerClientProviderStrategy - [] - Will use 'netty' transport
14:56:24.425 INFO  [main] o.t.d.EnvironmentAndSystemPropertyClientProviderStrategy - [] - Found docker client settings from environment
14:56:24.455 INFO  [main] o.t.d.DockerClientProviderStrategy - [] - Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=unix:///var/run/docker.sock
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='jenkins'
    registryPassword='null'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=unix:///var/run/docker.sock,registryUsername=jenkins,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/var/lib/jenkins/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

14:56:24.458 INFO  [main] o.t.DockerClientFactory - [] - Docker host IP address is localhost
14:56:25.030 INFO  [main] o.t.DockerClientFactory - [] - Connected to docker: 
  Server Version: 18.03.1-ce
  API Version: 1.37
  Operating System: Ubuntu 16.04.5 LTS
  Total Memory: 20076 MB
14:56:26.727 WARN  [testcontainers-ryuk] o.t.u.ResourceReaper - [] - Can not connect to Ryuk at localhost:33333
java.net.SocketException: Broken pipe (Write failed)
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:134)
    at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:144)
    at java.lang.Thread.run(Thread.java:748)
14:56:26.728 WARN  [testcontainers-ryuk] o.t.u.ResourceReaper - [] - Can not connect to Ryuk at localhost:33333
java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:210)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.fill(BufferedReader.java:161)
    at java.io.BufferedReader.readLine(BufferedReader.java:324)
    at java.io.BufferedReader.readLine(BufferedReader.java:389)
    at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:147)
    at java.lang.Thread.run(Thread.java:748)

As you can see, only 1.7 seconds passed between connecting to docker and the socket exception.

thammerl commented 6 years ago

After having a closer look at the code, I came up with the following findings/questions:

  1. Why are there two stack traces in the log output above? One when trying to write to the Ryuk socket and one when trying to read. ~Obviously there were two different threads trying to initialize the ryuk container.~ Shouldn't this only be executed once? Is the potential race condition in DockerClientFactory problematic? UPDATE: There are two stack traces because the index variable isn't incremented and the outermost while(true)-loop will try to connect again. As there aren't any further stack traces, I assume that the thread gets stuck in the while-loop waiting for the "ACK" response.
  2. ~Is it reasonable to just log a warning here? The ryukScheduledLatch will never be decremented to zero and will result in an IllegalStateException once the timeout is reached. Is it possible to recover from an IOException that is thrown when sending the filters to ryuk?~ I understand now that this retry-logic is intentional to wait until the ryuk server is available.
  3. It would be interesting to get the stdout of the ryuk server. Maybe it's possible to find the cause of the connection reset there. Anyone having an idea on how to do that?
  4. The sources JAR for 1.8.3 in Maven Central obviously does not contain the actual code of the release. I noticed when navigating the code in IDEA and realising that the line numbers in ResourceReaper from the stack trace don't match. The same is true for 1.8.2. Haven't checked any other releases. The sources for the releases on GitHub are fine.

Sorry if the answers to some of my questions are obvious. I'm new to the project and only guessing how things should play together here.

thammerl commented 6 years ago

Now I had a build with numerous stack traces instead of just two. But after some attempts to connect it seems to get stuck in the while-loop waiting for the acknowledgment and after the 30 second timeout the IllegalStateException kicks in:

testcontainers.log

thammerl commented 6 years ago

I've created a pull request that should fix the issue. See #843.

vpavic commented 6 years ago

We're also occasionally experiencing the same issue with Spring Session builds on Jenkins. Here's the stacktrace from one of those builds:

java.lang.IllegalStateException: Failed to load ApplicationContext
    at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:125)
    at org.springframework.test.context.support.DefaultTestContext.getApplicationContext(DefaultTestContext.java:108)
    at org.springframework.test.context.web.ServletTestExecutionListener.setUpRequestContextIfNecessary(ServletTestExecutionListener.java:190)
    at org.springframework.test.context.web.ServletTestExecutionListener.prepareTestInstance(ServletTestExecutionListener.java:132)
    at org.springframework.test.context.TestContextManager.prepareTestInstance(TestContextManager.java:246)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.createTest(SpringJUnit4ClassRunner.java:227)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner$1.runReflectiveCall(SpringJUnit4ClassRunner.java:289)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.methodBlock(SpringJUnit4ClassRunner.java:291)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:246)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
    at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
    at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
    at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
    at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
    at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:117)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:155)
    at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:137)
    at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
    at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'indexController': Unsatisfied dependency expressed through field 'sessions'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.boot.autoconfigure.session.RedisSessionConfiguration$SpringBootRedisHttpSessionConfiguration': Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redisConnectionFactory' defined in sample.FindByUsernameTests$Config: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory]: Factory method 'redisConnectionFactory' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redisContainer' defined in sample.FindByUsernameTests$Config: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.testcontainers.containers.GenericContainer]: Factory method 'redisContainer' threw exception; nested exception is java.lang.IllegalStateException: Can not connect to Ryuk
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:598)
    at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:90)
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:376)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1376)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:575)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:498)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:829)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:865)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:548)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:785)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:418)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:337)
    at org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:139)
    at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:99)
    at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:117)
    ... 49 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.boot.autoconfigure.session.RedisSessionConfiguration$SpringBootRedisHttpSessionConfiguration': Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redisConnectionFactory' defined in sample.FindByUsernameTests$Config: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory]: Factory method 'redisConnectionFactory' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redisContainer' defined in sample.FindByUsernameTests$Config: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.testcontainers.containers.GenericContainer]: Factory method 'redisContainer' threw exception; nested exception is java.lang.IllegalStateException: Can not connect to Ryuk
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:584)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:498)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:392)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1282)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1126)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:538)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:498)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
    at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:290)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1222)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1149)
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:595)
    ... 67 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redisConnectionFactory' defined in sample.FindByUsernameTests$Config: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory]: Factory method 'redisConnectionFactory' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redisContainer' defined in sample.FindByUsernameTests$Config: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.testcontainers.containers.GenericContainer]: Factory method 'redisContainer' threw exception; nested exception is java.lang.IllegalStateException: Can not connect to Ryuk
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:624)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:455)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1282)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1126)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:538)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:498)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
    at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:290)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1222)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory$DependencyObjectProvider.getObject(DefaultListableBeanFactory.java:1775)
    at org.springframework.session.data.redis.config.annotation.web.http.RedisHttpSessionConfiguration.setRedisConnectionFactory(RedisHttpSessionConfiguration.java:181)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:710)
    at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:90)
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:376)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1376)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:575)
    ... 85 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory]: Factory method 'redisConnectionFactory' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redisContainer' defined in sample.FindByUsernameTests$Config: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.testcontainers.containers.GenericContainer]: Factory method 'redisContainer' threw exception; nested exception is java.lang.IllegalStateException: Can not connect to Ryuk
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:619)
    ... 107 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'redisContainer' defined in sample.FindByUsernameTests$Config: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.testcontainers.containers.GenericContainer]: Factory method 'redisContainer' threw exception; nested exception is java.lang.IllegalStateException: Can not connect to Ryuk
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:624)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:455)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1282)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1126)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:538)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:498)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.resolveBeanReference(ConfigurationClassEnhancer.java:394)
    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:366)
    at sample.FindByUsernameTests$Config$$EnhancerBySpringCGLIB$$5030775a.redisContainer(<generated>)
    at sample.FindByUsernameTests$Config.redisConnectionFactory(FindByUsernameTests.java:95)
    at sample.FindByUsernameTests$Config$$EnhancerBySpringCGLIB$$5030775a.CGLIB$redisConnectionFactory$1(<generated>)
    at sample.FindByUsernameTests$Config$$EnhancerBySpringCGLIB$$5030775a$$FastClassBySpringCGLIB$$e4d393ce.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:363)
    at sample.FindByUsernameTests$Config$$EnhancerBySpringCGLIB$$5030775a.redisConnectionFactory(<generated>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
    ... 108 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.testcontainers.containers.GenericContainer]: Factory method 'redisContainer' threw exception; nested exception is java.lang.IllegalStateException: Can not connect to Ryuk
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:619)
    ... 131 more
Caused by: java.lang.IllegalStateException: Can not connect to Ryuk
    at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:166)
    at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:119)
    at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:151)
    at sample.FindByUsernameTests$Config.redisContainer(FindByUsernameTests.java:87)
    at sample.FindByUsernameTests$Config$$EnhancerBySpringCGLIB$$5030775a.CGLIB$redisContainer$0(<generated>)
    at sample.FindByUsernameTests$Config$$EnhancerBySpringCGLIB$$5030775a$$FastClassBySpringCGLIB$$e4d393ce.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:363)
    at sample.FindByUsernameTests$Config$$EnhancerBySpringCGLIB$$5030775a.redisContainer(<generated>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
    ... 132 more
kiview commented 6 years ago

@vpavic Does this still happen for you, or was it resolved with #843 in 1.9.0?

vpavic commented 6 years ago

Thanks for the ping - I forgot to follow up on this.

We were still experiencing the problem with 1.9.0-rc1, and have set ryuk.container.timeout=120 property since. This appears to have fixed it, but we've also had some updated on this infrastructure side and have upgrade to 1.9.1 since.

I'll try to remove ryuk.container.timeout=120 some time this week and see how it behaves, just to try and isolate the problem.

guss77 commented 6 years ago

I'm having the same issue with 1.9.1 on Circle CI (2.0). I'm running testcontainers for a JDBC server for Jooq code generation:

[INFO] Will use 'okhttp' transport
[INFO] Found docker client settings from environment
[INFO] Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=tcp://35.231.185.114:2376
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='circleci'
    registryPassword='null'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=tcp://35.231.185.114:2376,registryUsername=circleci,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/home/circleci/.docker,sslConfig=LocalDirectorySSLConfig{dockerCertPath=/tmp/docker-certs194934439},apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

[INFO] Docker host IP address is 35.231.185.114
[INFO] Connected to docker: 
  Server Version: 17.09.0-ce
  API Version: 1.32
  Operating System: Ubuntu 16.04.3 LTS
  Total Memory: 7477 MB
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:42 min
[INFO] Finished at: 2018-10-29T07:45:45Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.jooq:jooq-codegen-maven:3.11.5:generate (default) on project model: Error running jOOQ code generation tool: Can not connect to Ryuk -> [Help 1]

I can't figure out how to set ryuk.container.timeout=120 - usind -D on the maven command line doesn't seem to do anything.

Here's the full debug log when running Maven with -X:

[DEBUG] Container not found in cache, creating new instance
[DEBUG] Testcontainers configuration overrides will be loaded from file:/home/circleci/.testcontainers.properties
[INFO] Will use 'okhttp' transport
[DEBUG] Pinging docker daemon...
[DEBUG] Cmd: org.testcontainers.dockerclient.transport.okhttp.OkHttpDockerCmdExecFactory$1@34349e19
[INFO] Found docker client settings from environment
[INFO] Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=tcp://35.190.177.84:2376
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='circleci'
    registryPassword='null'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=tcp://35.190.177.84:2376,registryUsername=circleci,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/home/circleci/.docker,sslConfig=LocalDirectorySSLConfig{dockerCertPath=/tmp/docker-certs196266953},apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

[INFO] Docker host IP address is 35.190.177.84
[DEBUG] Cmd: com.github.dockerjava.core.exec.InfoCmdExec@986b619
[DEBUG] Cmd: com.github.dockerjava.core.exec.VersionCmdExec@7df3da0b
[INFO] Connected to docker: 
  Server Version: 17.09.0-ce
  API Version: 1.32
  Operating System: Ubuntu 16.04.3 LTS
  Total Memory: 7477 MB
[DEBUG] Cmd: ListImagesCmdImpl[imageNameFilter=quay.io/testcontainers/ryuk:0.2.2,showAll=false,filters=com.github.dockerjava.core.util.FiltersBuilder@0,execution=com.github.dockerjava.core.exec.ListImagesCmdExec@3514ac7d]
[DEBUG] Looking up auth config for image: quay.io/testcontainers/ryuk:0.2.2
[DEBUG] RegistryAuthLocator has configFile: /home/circleci/.docker/config.json (exists) and commandPathPrefix: 
[DEBUG] registryName [quay.io] for dockerImageName [quay.io/testcontainers/ryuk:0.2.2]
[DEBUG] no matching Auth Configs - falling back to defaultAuthConfig [null]
[DEBUG] Effective auth config [null]
[DEBUG] PullResponseItem[stream=<null>,status=Pulling from testcontainers/ryuk,progressDetail=<null>,progress=<null>,id=0.2.2,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Pulling fs layer,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=ff3a5c916c92,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Pulling fs layer,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=c9ea39c9283a,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Pulling fs layer,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=f95d3189d433,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Downloading,progressDetail=ResponseItem.ProgressDetail[current=16384,total=308109,start=<null>],progress=[==>                                                ]  16.38kB/308.1kB,id=c9ea39c9283a,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Downloading,progressDetail=ResponseItem.ProgressDetail[current=32768,total=1776708,start=<null>],progress=[>                                                  ]  32.77kB/1.777MB,id=f95d3189d433,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Downloading,progressDetail=ResponseItem.ProgressDetail[current=32279,total=2065537,start=<null>],progress=[>                                                  ]  32.28kB/2.066MB,id=ff3a5c916c92,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Download complete,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=c9ea39c9283a,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Downloading,progressDetail=ResponseItem.ProgressDetail[current=655360,total=1776708,start=<null>],progress=[==================>                                ]  655.4kB/1.777MB,id=f95d3189d433,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Downloading,progressDetail=ResponseItem.ProgressDetail[current=588961,total=2065537,start=<null>],progress=[==============>                                    ]    589kB/2.066MB,id=ff3a5c916c92,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Verifying Checksum,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=f95d3189d433,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Download complete,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=f95d3189d433,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Verifying Checksum,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=ff3a5c916c92,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Download complete,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=ff3a5c916c92,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Extracting,progressDetail=ResponseItem.ProgressDetail[current=32768,total=2065537,start=<null>],progress=[>                                                  ]  32.77kB/2.066MB,id=ff3a5c916c92,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Extracting,progressDetail=ResponseItem.ProgressDetail[current=1605632,total=2065537,start=<null>],progress=[======================================>            ]  1.606MB/2.066MB,id=ff3a5c916c92,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Extracting,progressDetail=ResponseItem.ProgressDetail[current=2065537,total=2065537,start=<null>],progress=[==================================================>]  2.066MB/2.066MB,id=ff3a5c916c92,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Pull complete,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=ff3a5c916c92,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Extracting,progressDetail=ResponseItem.ProgressDetail[current=32768,total=308109,start=<null>],progress=[=====>                                             ]  32.77kB/308.1kB,id=c9ea39c9283a,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Extracting,progressDetail=ResponseItem.ProgressDetail[current=308109,total=308109,start=<null>],progress=[==================================================>]  308.1kB/308.1kB,id=c9ea39c9283a,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Extracting,progressDetail=ResponseItem.ProgressDetail[current=308109,total=308109,start=<null>],progress=[==================================================>]  308.1kB/308.1kB,id=c9ea39c9283a,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Pull complete,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=c9ea39c9283a,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Extracting,progressDetail=ResponseItem.ProgressDetail[current=32768,total=1776708,start=<null>],progress=[>                                                  ]  32.77kB/1.777MB,id=f95d3189d433,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Extracting,progressDetail=ResponseItem.ProgressDetail[current=1146880,total=1776708,start=<null>],progress=[================================>                  ]  1.147MB/1.777MB,id=f95d3189d433,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Extracting,progressDetail=ResponseItem.ProgressDetail[current=1776708,total=1776708,start=<null>],progress=[==================================================>]  1.777MB/1.777MB,id=f95d3189d433,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Pull complete,progressDetail=ResponseItem.ProgressDetail[current=<null>,total=<null>,start=<null>],progress=<null>,id=f95d3189d433,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Digest: sha256:3d53f437c84a355eafccfef4a0fe3270c49893bd2ac80b5ef57698c1a9102fec,progressDetail=<null>,progress=<null>,id=<null>,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] PullResponseItem[stream=<null>,status=Status: Downloaded newer image for quay.io/testcontainers/ryuk:0.2.2,progressDetail=<null>,progress=<null>,id=<null>,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] Looking up auth config for image: quay.io/testcontainers/ryuk:0.2.2
[DEBUG] RegistryAuthLocator has configFile: /home/circleci/.docker/config.json (exists) and commandPathPrefix: 
[DEBUG] registryName [quay.io] for dockerImageName [quay.io/testcontainers/ryuk:0.2.2]
[DEBUG] no matching Auth Configs - falling back to defaultAuthConfig [null]
[DEBUG] Effective auth config [null]
[DEBUG] Cmd: com.github.dockerjava.core.command.CreateContainerCmdImpl@fd5689d[name=testcontainers-ryuk-b1afc90d-70e5-418e-b741-ed3433252737,hostName=<null>,domainName=<null>,user=<null>,attachStdin=<null>,attachStdout=<null>,attachStderr=<null>,portSpecs=<null>,tty=<null>,stdinOpen=<null>,stdInOnce=<null>,env=<null>,cmd=<null>,entrypoint=<null>,image=quay.io/testcontainers/ryuk:0.2.2,volumes=com.github.dockerjava.api.model.Volumes@1d9af731,workingDir=<null>,macAddress=<null>,networkDisabled=<null>,exposedPorts=com.github.dockerjava.api.model.ExposedPorts@39afe59f,stopSignal=<null>,hostConfig=com.github.dockerjava.api.model.HostConfig@16f34376[binds=com.github.dockerjava.api.model.Binds@4272d8cf,blkioWeight=<null>,blkioWeightDevice=<null>,blkioDeviceReadBps=<null>,blkioDeviceReadIOps=<null>,blkioDeviceWriteBps=<null>,blkioDeviceWriteIOps=<null>,memorySwappiness=<null>,capAdd=<null>,capDrop=<null>,containerIDFile=<null>,cpuPeriod=<null>,cpuShares=<null>,cpuQuota=<null>,cpusetCpus=<null>,cpusetMems=<null>,devices=<null>,diskQuota=<null>,dns=<null>,dnsSearch=<null>,extraHosts=<null>,links=<null>,logConfig=<null>,lxcConf=<null>,memory=<null>,memorySwap=<null>,memoryReservation=<null>,kernelMemory=<null>,networkMode=<null>,oomKillDisable=<null>,autoRemove=true,oomScoreAdj=<null>,portBindings=<null>,privileged=<null>,publishAllPorts=true,readonlyRootfs=<null>,restartPolicy=<null>,ulimits=<null>,volumesFrom=<null>,pidMode=<null>,securityOpts=<null>,cgroupParent=<null>,volumeDriver=<null>,shmSize=<null>,pidsLimit=<null>,runtime=<null>,tmpFs=<null>],labels={org.testcontainers=true},networkingConfig=<null>,ipv4Address=<null>,ipv6Address=<null>,aliases=<null>,authConfig=<null>,execution=com.github.dockerjava.core.exec.CreateContainerCmdExec@4e96cb04]
[DEBUG] Cmd: 5905dfcfefceb40da10b551e3c8b9c51e5618001ef9d19683eb844eef70b7352,com.github.dockerjava.core.exec.StartContainerCmdExec@2b974137
[DEBUG] Cmd: 5905dfcfefceb40da10b551e3c8b9c51e5618001ef9d19683eb844eef70b7352,false,com.github.dockerjava.core.exec.InspectContainerCmdExec@76fc5687
[DEBUG] GET: OkHttpWebTarget(okHttpClient=org.testcontainers.shaded.okhttp3.OkHttpClient@53fbb2b4, baseUrl=https://35.190.177.84:2376/, path=[/containers/5905dfcfefceb40da10b551e3c8b9c51e5618001ef9d19683eb844eef70b7352/json], queryParams={})
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:48 min
[INFO] Finished at: 2018-10-29T07:51:37Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.jooq:jooq-codegen-maven:3.11.5:generate (default) on project model: Error running jOOQ code generation tool: Can not connect to Ryuk -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.jooq:jooq-codegen-maven:3.11.5:generate (default) on project model: Error running jOOQ code generation tool
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:954)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: Error running jOOQ code generation tool
    at org.jooq.codegen.maven.Plugin.execute (Plugin.java:200)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:954)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: java.lang.IllegalStateException: Can not connect to Ryuk
    at org.testcontainers.utility.ResourceReaper.start (ResourceReaper.java:148)
    at org.testcontainers.utility.ResourceReaper.start (ResourceReaper.java:65)
    at org.testcontainers.DockerClientFactory.client (DockerClientFactory.java:124)
    at org.testcontainers.containers.GenericContainer.<init> (GenericContainer.java:162)
    at org.testcontainers.containers.JdbcDatabaseContainer.<init> (JdbcDatabaseContainer.java:40)
    at org.testcontainers.containers.MariaDBContainer.<init> (MariaDBContainer.java:25)
    at org.testcontainers.containers.MariaDBContainerProvider.newInstance (MariaDBContainerProvider.java:19)
    at org.testcontainers.containers.MariaDBContainerProvider.newInstance (MariaDBContainerProvider.java:14)
    at org.testcontainers.containers.JdbcDatabaseContainerProvider.newInstance (JdbcDatabaseContainerProvider.java:49)
    at org.testcontainers.jdbc.ContainerDatabaseDriver.connect (ContainerDatabaseDriver.java:94)
    at org.jooq.codegen.GenerationTool.run (GenerationTool.java:318)
    at org.jooq.codegen.GenerationTool.generate (GenerationTool.java:220)
    at org.jooq.codegen.maven.Plugin.execute (Plugin.java:197)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:954)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
vpavic commented 6 years ago

@guss See the docs on custom properties.

As a side note for @kiview and @bsideup, it would be nice to have ryuk.container.timeout (or any other missing properties) added to that page for a complete reference.

guss77 commented 6 years ago

Sorry about the weird link, Github is sometimes too enthusiastic about referencing issues.

I've tried with setting the timeout to 120, and indeed the build takes 2 more minutes but the Ryuk connection still fails:

[DEBUG] PullResponseItem[stream=<null>,status=Status: Downloaded newer image for quay.io/testcontainers/ryuk:0.2.2,progressDetail=<null>,progress=<null>,id=<null>,from=<null>,time=<null>,errorDetail=<null>,error=<null>,aux=<null>]
[DEBUG] Looking up auth config for image: quay.io/testcontainers/ryuk:0.2.2
[DEBUG] RegistryAuthLocator has configFile: /home/circleci/.docker/config.json (exists) and commandPathPrefix: 
[DEBUG] registryName [quay.io] for dockerImageName [quay.io/testcontainers/ryuk:0.2.2]
[DEBUG] no matching Auth Configs - falling back to defaultAuthConfig [null]
[DEBUG] Effective auth config [null]
[DEBUG] Cmd: com.github.dockerjava.core.command.CreateContainerCmdImpl@6ba0ee4a[name=testcontainers-ryuk-0124f49d-4059-439a-93b6-cb8f82b9ac4e,hostName=<null>,domainName=<null>,user=<null>,attachStdin=<null>,attachStdout=<null>,attachStderr=<null>,portSpecs=<null>,tty=<null>,stdinOpen=<null>,stdInOnce=<null>,env=<null>,cmd=<null>,entrypoint=<null>,image=quay.io/testcontainers/ryuk:0.2.2,volumes=com.github.dockerjava.api.model.Volumes@27db45f,workingDir=<null>,macAddress=<null>,networkDisabled=<null>,exposedPorts=com.github.dockerjava.api.model.ExposedPorts@6ec3d8e4,stopSignal=<null>,hostConfig=com.github.dockerjava.api.model.HostConfig@1aeff8ca[binds=com.github.dockerjava.api.model.Binds@6068ebb2,blkioWeight=<null>,blkioWeightDevice=<null>,blkioDeviceReadBps=<null>,blkioDeviceReadIOps=<null>,blkioDeviceWriteBps=<null>,blkioDeviceWriteIOps=<null>,memorySwappiness=<null>,capAdd=<null>,capDrop=<null>,containerIDFile=<null>,cpuPeriod=<null>,cpuShares=<null>,cpuQuota=<null>,cpusetCpus=<null>,cpusetMems=<null>,devices=<null>,diskQuota=<null>,dns=<null>,dnsSearch=<null>,extraHosts=<null>,links=<null>,logConfig=<null>,lxcConf=<null>,memory=<null>,memorySwap=<null>,memoryReservation=<null>,kernelMemory=<null>,networkMode=<null>,oomKillDisable=<null>,autoRemove=true,oomScoreAdj=<null>,portBindings=<null>,privileged=<null>,publishAllPorts=true,readonlyRootfs=<null>,restartPolicy=<null>,ulimits=<null>,volumesFrom=<null>,pidMode=<null>,securityOpts=<null>,cgroupParent=<null>,volumeDriver=<null>,shmSize=<null>,pidsLimit=<null>,runtime=<null>,tmpFs=<null>],labels={org.testcontainers=true},networkingConfig=<null>,ipv4Address=<null>,ipv6Address=<null>,aliases=<null>,authConfig=<null>,execution=com.github.dockerjava.core.exec.CreateContainerCmdExec@5a466dd]
[DEBUG] Cmd: ee8c93190996a894ee8eb46d2aa2435b81b6c902cec44fdc831c019453cbd73a,com.github.dockerjava.core.exec.StartContainerCmdExec@4aac81ca
[DEBUG] Cmd: ee8c93190996a894ee8eb46d2aa2435b81b6c902cec44fdc831c019453cbd73a,false,com.github.dockerjava.core.exec.InspectContainerCmdExec@269c7104
[DEBUG] GET: OkHttpWebTarget(okHttpClient=org.testcontainers.shaded.okhttp3.OkHttpClient@6de84336, baseUrl=https://35.185.49.118:2376/, path=[/containers/ee8c93190996a894ee8eb46d2aa2435b81b6c902cec44fdc831c019453cbd73a/json], queryParams={})
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:23 min
[INFO] Finished at: 2018-10-30T11:38:48Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.jooq:jooq-codegen-maven:3.11.5:generate (default) on project model: Error running jOOQ code generation tool: Can not connect to Ryuk -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.jooq:jooq-codegen-maven:3.11.5:generate (default) on project model: Error running jOOQ code generation tool
...
Caused by: java.lang.IllegalStateException: Can not connect to Ryuk
    at org.testcontainers.utility.ResourceReaper.start (ResourceReaper.java:148)
    at org.testcontainers.utility.ResourceReaper.start (ResourceReaper.java:65)
    at org.testcontainers.DockerClientFactory.client (DockerClientFactory.java:124)
    at org.testcontainers.containers.GenericContainer.<init> (GenericContainer.java:162)
    at org.testcontainers.containers.JdbcDatabaseContainer.<init> (JdbcDatabaseContainer.java:40)
    at org.testcontainers.containers.MariaDBContainer.<init> (MariaDBContainer.java:25)
...
rafaelalmeida commented 5 years ago

@guss77 Did you manage to solve this? Running into the same issue now with 1.10.2.

ysb33r commented 5 years ago

I am seeing the same issue with 1.10.6 but only on Mac running Docker Desktop

prent

Similar stack trace:

    java.lang.IllegalStateException: Can not connect to Ryuk
        at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:148)
        at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:125)
        at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:142)

I am running this side-by-side with a Linux laptop (Docker engine 18.06.0-dev build e68fc7a) that has less memory than the Mac, but I am not seeing the issue there.

kiview commented 5 years ago

@ysb33r Did you already try to increase the timeout in ryuk.container.timeout? https://www.testcontainers.org/features/configuration/

ysb33r commented 5 years ago

Did you already try to increase the timeout in ryuk.container.timeout? https://www.testcontainers.org/features/configuration/

Yes I have set it to 120, but possibly a complete restart of Docker Desktop seemed to have resolved the problem.

ciri-cuervo commented 5 years ago

have tried creating the docker group? it solves permission issues

# create new group
sudo groupadd docker
# add current user to the group
sudo usermod -aG docker $USER
# log in to the group
newgrp - docker
AlexanderArgyriou commented 1 year ago

Hello, any update here ? https://stackoverflow.com/questions/76442027/cannot-connect-to-ryuk