search

testcontainers / testcontainers-java

Testcontainers is a Java library that supports JUnit tests, providing lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container.
https://testcontainers.org
MIT License
8.01k stars 1.65k forks source link

Could not connect to Ryuk - Linux, Docker 20.10.0, 1.15.0-rc2 #3572

Closed ullrichp closed 3 years ago

ullrichp commented 3 years ago

As this is my first issue i would like to say "Thank you" to all maintainers and contributors.

Apparently, out of nothing my tests started to throw:

Caused by: java.lang.IllegalStateException: Could not connect to Ryuk at localhost:49160
    at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:171)
    at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:201)
    at org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
    at org.testcontainers.LazyDockerClient.listImagesCmd(LazyDockerClient.java:12)
    at org.testcontainers.images.LocalImagesCache.maybeInitCache(LocalImagesCache.java:68)
    at org.testcontainers.images.LocalImagesCache.get(LocalImagesCache.java:32)
    at org.testcontainers.images.AbstractImagePullPolicy.shouldPull(AbstractImagePullPolicy.java:18)
    at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:65)
    at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:26)
    at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:17)
    at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:39)
    at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1280)
    ... 266 more

I am on Linux with: openjdk 11.0.8 2020-07-14 testcontainers 1.15.0-rc2 docker 20.10.0

The container seems to start:

$ docker ps -a
CONTAINER ID   IMAGE                       COMMAND   CREATED        STATUS                  PORTS                     NAMES
85335add1def   testcontainers/ryuk:0.3.0   "/app"    1 second ago   Up Less than a second   0.0.0.0:49160->8080/tcp   testcontainers-ryuk-2b08ab75-8559-48fc-8f73-8d233b45c430

and the logs do not seem too wrong:

$ docker logs 85335add1def
2020/12/09 09:47:37 Pinging Docker...
2020/12/09 09:47:37 Docker daemon is available!
2020/12/09 09:47:37 Starting on port 8080...
2020/12/09 09:47:37 Started!

Still, the above exception happens after the default timeout of 30 seconds.

Let me know if you need more information, stack traces or logs.

bsideup commented 3 years ago

Hi @ullrichp,

Thanks for your kind words! :)

Have you tried the latest release (1.15.0)? Also, does it work if you attempt to connect to Ryuk from CLI?

arhohuttunen commented 3 years ago

Experiencing the same out of the blue when executing in EC2, this was still working this morning. The stack trace has this:

Caused by:
    org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=postgres:11.8-alpine, imagePullPolicy=DefaultPullPolicy())
    at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1278)
    at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:612)
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:317)
    ... 114 more
        Caused by:
        com.github.dockerjava.api.exception.NotFoundException: Status 404: {"message":"No such image: testcontainers/ryuk:0.3.0"}

But looking at the Docker Hub that image does exists there.

bsideup commented 3 years ago

@arhohuttunen you issue seems to be different and caused by the registry returning 404 for an image that exists.

arhohuttunen commented 3 years ago

My mistake, was too hasty to draw conclusions between the two issues.

swierszczu4j commented 3 years ago

I have exactly the same issue on latest (1.15.0). What is more seems to be not-deterministic one :/ Anyway seems like they are some compatibility issues with the newest docker.

amelnikov-mylivn commented 3 years ago

Experiencing the same out of the blue when executing in EC2, this was still working this morning. The stack trace has this:

Caused by:
    org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=postgres:11.8-alpine, imagePullPolicy=DefaultPullPolicy())
    at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1278)
    at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:612)
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:317)
    ... 114 more
        Caused by:
        com.github.dockerjava.api.exception.NotFoundException: Status 404: {"message":"No such image: testcontainers/ryuk:0.3.0"}

But looking at the Docker Hub that image does exists there.

We had the same issue today. Temporary fixed by pinning docker version to 19.03.

jdelucaa commented 3 years ago

@amelnikov-mylivn what does it have to do with the docker version? have they changed anything in the latest version?

KyleAure commented 3 years ago

I am also seeing the same issue with the stack trace: 

[12/09/2020 11:09:31:957 CST] 040 ResourceReaper                 lambda$null$1                  W Can not connect to Ryuk at localhost:32769
java.net.SocketException: Protocol wrong type for socket (Write failed)
    at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
    at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:138)
    at org.testcontainers.utility.ResourceReaper$FilterRegistry.register(ResourceReaper.java:406)
    at org.testcontainers.utility.ResourceReaper.lambda$null$1(ResourceReaper.java:135)
    at org.testcontainers.utility.ResourceReaper$$Lambda$131/000000002833D020.run(Unknown Source)
    at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
    at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:119)
    at org.testcontainers.utility.ResourceReaper$$Lambda$130/00000000279F0A20.run(Unknown Source)
    at java.base/java.lang.Thread.run(Thread.java:834)
croudet commented 3 years ago

Also got com.github.dockerjava.api.exception.NotFoundException: Status 404: {"message":"No such image: testcontainers/ryuk:0.3.0"

Looks like related to docker version: Working version:

13:02:19.695 [main] DEBUG org.testcontainers.utility.TestcontainersConfiguration - Testcontainers configuration overrides will be loaded from file:/root/.testcontainers.properties
13:02:19.718 [main] WARN org.testcontainers.utility.TestcontainersConfiguration - Attempted to read Testcontainers configuration file at file:/root/.testcontainers.properties but the file was not found. Exception message: FileNotFoundException: /root/.testcontainers.properties (No such file or directory)
13:02:19.725 [main] INFO org.testcontainers.utility.ImageNameSubstitutor - Image name substitution will be performed by: DefaultImageNameSubstitutor (ConfigurationFileImageNameSubstitutor)
13:02:19.735 [main] DEBUG org.testcontainers.utility.ImageNameSubstitutor - Did not find a substitute image for rabbitmq:3.8.9 (using image substitutor: DefaultImageNameSubstitutor (ConfigurationFileImageNameSubstitutor))
13:02:19.764 [main] INFO org.testcontainers.dockerclient.DockerMachineClientProviderStrategy - docker-machine executable was not found on PATH ([/usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin, /usr/lib/jvm/java-1.8-openjdk/jre/bin, /usr/lib/jvm/java-1.8-openjdk/bin])
13:02:19.764 [main] DEBUG org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy - $XDG_RUNTIME_DIR is not set.
13:02:19.765 [main] DEBUG org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy - '/root/.docker/run' does not exist.
13:02:19.807 [main] DEBUG org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy - '/run/user/0' does not exist.
13:02:20.330 [ducttape-0] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Pinging docker daemon...
13:02:20.352 [ducttape-0] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 
13:02:20.658 [main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
13:02:20.659 [main] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Transport type: 'okhttp', Docker host: 'unix:///var/run/docker.sock'
13:02:20.659 [main] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Checking Docker OS type for local Unix socket (unix:///var/run/docker.sock)
13:02:20.668 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: bridge
13:02:20.684 [main] INFO org.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
13:02:20.686 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 
13:02:20.700 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 
13:02:20.718 [main] INFO org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 19.03.14
  API Version: 1.40
  Operating System: Ubuntu 16.04.4 LTS
  Total Memory: 31907 MB
13:02:20.718 [main] DEBUG org.testcontainers.DockerClientFactory - Ryuk is enabled
13:02:20.721 [main] DEBUG org.testcontainers.utility.ImageNameSubstitutor - Did not find a substitute image for testcontainers/ryuk:0.3.0 (using image substitutor: DefaultImageNameSubstitutor (ConfigurationFileImageNameSubstitutor))
13:02:20.724 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: ListImagesCmdImpl[imageNameFilter=testcontainers/ryuk:0.3.0,showAll=false,filters=org.testcontainers.shaded.com.github.dockerjava.core.util.FiltersBuilder@0]
13:02:20.803 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: testcontainers/ryuk:0.3.0 at registry: index.docker.io
13:02:20.804 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /root/.docker/config.json (does not exist) and commandPathPrefix: 
13:02:20.804 [main] INFO org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: testcontainers/ryuk:0.3.0, configFile: /root/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /root/.docker/config.json (No such file or directory)
13:02:20.805 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - No matching Auth Configs - falling back to defaultAuthConfig [null]
13:02:20.805 [main] DEBUG org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig - Effective auth config [null]
13:02:21.019 [docker-java-stream--1668592714] INFO org.testcontainers.DockerClientFactory - Starting to pull image
13:02:21.041 [docker-java-stream--1668592714] DEBUG com.github.dockerjava.api.command.PullImageResultCallback - ResponseItem(stream=null, status=Pulling from testcontainers/ryuk, progressDetail=null, progress=null, id=0.3.0, from=null, time=null, errorDetail=null, error=null, aux=null)

Non working version:

19:03:19.418 [main] DEBUG org.testcontainers.utility.TestcontainersConfiguration - Testcontainers configuration overrides will be loaded from file:/root/.testcontainers.properties
19:03:19.441 [main] WARN org.testcontainers.utility.TestcontainersConfiguration - Attempted to read Testcontainers configuration file at file:/root/.testcontainers.properties but the file was not found. Exception message: FileNotFoundException: /root/.testcontainers.properties (No such file or directory)
19:03:19.448 [main] INFO org.testcontainers.utility.ImageNameSubstitutor - Image name substitution will be performed by: DefaultImageNameSubstitutor (ConfigurationFileImageNameSubstitutor)
19:03:19.459 [main] DEBUG org.testcontainers.utility.ImageNameSubstitutor - Did not find a substitute image for rabbitmq:3.8.9 (using image substitutor: DefaultImageNameSubstitutor (ConfigurationFileImageNameSubstitutor))
19:03:19.489 [main] INFO org.testcontainers.dockerclient.DockerMachineClientProviderStrategy - docker-machine executable was not found on PATH ([/usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin, /usr/lib/jvm/java-1.8-openjdk/jre/bin, /usr/lib/jvm/java-1.8-openjdk/bin])
19:03:19.490 [main] DEBUG org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy - $XDG_RUNTIME_DIR is not set.
19:03:19.490 [main] DEBUG org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy - '/root/.docker/run' does not exist.
19:03:19.533 [main] DEBUG org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy - '/run/user/0' does not exist.
19:03:20.117 [ducttape-0] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Pinging docker daemon...
19:03:20.140 [ducttape-0] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 
19:03:20.458 [main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
19:03:20.458 [main] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Transport type: 'okhttp', Docker host: 'unix:///var/run/docker.sock'
19:03:20.458 [main] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Checking Docker OS type for local Unix socket (unix:///var/run/docker.sock)
19:03:20.468 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: bridge
19:03:20.486 [main] INFO org.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
19:03:20.488 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 
19:03:20.505 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: 
19:03:20.528 [main] INFO org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 20.10.0
  API Version: 1.41
  Operating System: Ubuntu 16.04.4 LTS
  Total Memory: 31907 MB
19:03:20.528 [main] DEBUG org.testcontainers.DockerClientFactory - Ryuk is enabled
19:03:20.531 [main] DEBUG org.testcontainers.utility.ImageNameSubstitutor - Did not find a substitute image for testcontainers/ryuk:0.3.0 (using image substitutor: DefaultImageNameSubstitutor (ConfigurationFileImageNameSubstitutor))
19:03:20.534 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: ListImagesCmdImpl[imageNameFilter=testcontainers/ryuk:0.3.0,showAll=false,filters=org.testcontainers.shaded.com.github.dockerjava.core.util.FiltersBuilder@0]
19:03:20.585 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: testcontainers/ryuk:0.3.0 at registry: index.docker.io
19:03:20.585 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /root/.docker/config.json (does not exist) and commandPathPrefix: 
19:03:20.586 [main] INFO org.testcontainers.utility.RegistryAuthLocator - Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: testcontainers/ryuk:0.3.0, configFile: /root/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /root/.docker/config.json (No such file or directory)
19:03:20.586 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - No matching Auth Configs - falling back to defaultAuthConfig [null]
19:03:20.587 [main] DEBUG org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig - Effective auth config [null]
19:03:20.593 [main] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: org.testcontainers.shaded.com.github.dockerjava.core.command.CreateContainerCmdImpl@4ff8d125[name=testcontainers-ryuk-478be443-d6a0-46f7-93af-512bc9f73bfa,hostName=<null>,domainName=<null>,user=<null>,attachStdin=<null>,attachStdout=<null>,attachStderr=<null>,portSpecs=<null>,tty=<null>,stdinOpen=<null>,stdInOnce=<null>,env=<null>,cmd=<null>,healthcheck=<null>,argsEscaped=<null>,entrypoint=<null>,image=testcontainers/ryuk:0.3.0,volumes=com.github.dockerjava.api.model.Volumes@19c65cdc,workingDir=<null>,macAddress=<null>,onBuild=<null>,networkDisabled=<null>,exposedPorts=com.github.dockerjava.api.model.ExposedPorts@74bf1791,stopSignal=<null>,stopTimeout=<null>,hostConfig=HostConfig(binds=[/var/run/docker.sock:/var/run/docker.sock:rw], blkioWeight=null, blkioWeightDevice=null, blkioDeviceReadBps=null, blkioDeviceWriteBps=null, blkioDeviceReadIOps=null, blkioDeviceWriteIOps=null, memorySwappiness=null, nanoCPUs=null, capAdd=null, capDrop=null, containerIDFile=null, cpuPeriod=null, cpuRealtimePeriod=null, cpuRealtimeRuntime=null, cpuShares=null, cpuQuota=null, cpusetCpus=null, cpusetMems=null, devices=null, deviceCgroupRules=null, deviceRequests=null, diskQuota=null, dns=null, dnsOptions=null, dnsSearch=null, extraHosts=null, groupAdd=null, ipcMode=null, cgroup=null, links=[], logConfig=LogConfig(type=null, config=null), lxcConf=null, memory=null, memorySwap=null, memoryReservation=null, kernelMemory=null, networkMode=null, oomKillDisable=null, init=null, autoRemove=true, oomScoreAdj=null, portBindings=null, privileged=false, publishAllPorts=true, readonlyRootfs=null, restartPolicy=null, ulimits=null, cpuCount=null, cpuPercent=null, ioMaximumIOps=null, ioMaximumBandwidth=null, volumesFrom=null, mounts=null, pidMode=null, isolation=null, securityOpts=null, storageOpt=null, cgroupParent=null, volumeDriver=null, shmSize=null, pidsLimit=null, runtime=null, tmpFs=null, utSMode=null, usernsMode=null, sysctls=null, consoleSize=null),labels={org.testcontainers=true},shell=<null>,networkingConfig=<null>,ipv4Address=<null>,ipv6Address=<null>,aliases=<null>,authConfig=<null>]
19:03:20.695 [main] DEBUG org.testcontainers.utility.ImageNameSubstitutor - Did not find a substitute image for rabbitmq:3.8.9 (using image substitutor: DefaultImageNameSubstitutor (ConfigurationFileImageNameSubstitutor))
jdelucaa commented 3 years ago

I am also seeing this com.github.dockerjava.api.exception.NotFoundException: {"message":"No such image: quay.io/testcontainers/ryuk:0.3.0"}

amelnikov-mylivn commented 3 years ago

@amelnikov-mylivn what does it have to do with the docker version? have they changed anything in the latest version?

Honestly idk, I did not dig enough into. Our CI runs on spot EC2 instances which are created for each task. Today builds started to fail with 404 No such image. We wasted most of the day, nothing helped, but when found that docker 20.10 appeared in repos yesterday, we pinned its version to an old one, and it worked.

P.S. Sorry for the off-topic.

gesellix commented 3 years ago

A manual docker pull testcontainers/ryuk:0.3.0 helps. Without having debugged the issue, yet, it seems like docker-java doesn't transparently pull the missing image on status code 404. This should be the default behaviour. I'm also running into this issue using the recent Docker for Mac 2.5.4/Docker Engine 20.10.0-rc2 and Testcontainers 1.15.0.

bsideup commented 3 years ago

Without having debugged the issue, yet, it seems like docker-java doesn't transparently pull the missing image on status code 404.

It doesn't, yes.

This should be the default behaviour.

Why? 404 is "not found" and 4xx type of an http code, non-retriable. I don't think registries should return 404 for images that exist.

Although the original issue report is talking about some other, TCP related issue, so please consider reporting 404 separately, this is some other issue.

gesellix commented 3 years ago

Filed an issue for "No such image: testcontainers/ryuk:0.3.0" (Status 404) at #3574. Sorry for the spam.

rnorth commented 3 years ago

I'm trying to reproduce this TCP/connection to Ryuk issue, but haven't managed to so far 😬

Using a fresh VM with Ubuntu 20.04 (LTS) x64 and Docker version 20.10.0, build 7287ab3

I'm testing with 1.15-0rc2, 1.15.0 and master/HEAD and so far not able to reproduce. Will continue to try.

ullrichp commented 3 years ago

Have you tried the latest release (1.15.0)?

Yes, i get the same exception.

Also, does it work if you attempt to connect to Ryuk from CLI?

I assume you mean $ docker attach -> Apparenlty, no. That seems not to be possible:

$ docker ps -a
CONTAINER ID        IMAGE                       COMMAND             CREATED                  STATUS              PORTS               NAMES
e2fa3727c9aa        testcontainers/ryuk:0.3.0   "/app"              Less than a second ago   Created                                 testcontainers-ryuk-e31ea349-ae04-4762-9c9d-c45e600d4b19
$ docker attach e2fa3727c9aa
panic: Timed out waiting for the initial connection

goroutine 1 [running]:
main.main()
        /go/src/github.com/testcontainers/moby-ryuk/main.go:104 +0xce7
rnorth commented 3 years ago

@ullrichp your last docker ps output doesn't seem to have any listening ports listed for ryuk, whereas your first does. This seems very strange.

Out of interest, which Linux distro and version are you using?

ullrichp commented 3 years ago

I played a little with my docker installation and are now back to version 19.03.14. Maybe the output changes because of that.

I am using Gentoo.

asaikali commented 3 years ago

I am having the same issue on ubuntu 20.04 with 

$ docker --version
Docker version 20.10.0, build 7287ab3

$ docker images
REPOSITORY                    TAG       IMAGE ID       CREATED         SIZE
postgres                      12        4b52913b0a3a   4 months ago    313MB
testcontainersofficial/ryuk   0.3.0     344c0229fe44   6 months ago    12MB
hello-world                   latest    bf756fb1ae65   11 months ago   13.3kB

Test containers 1.14.3 works but with 1.15.0 I get the exception saying there is no ryuk image. Stack trace below 

       Caused by:
        java.lang.IllegalStateException: Failed to load ApplicationContext
            at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:132)
            at org.springframework.test.context.support.DefaultTestContext.getApplicationContext(DefaultTestContext.java:123)
            at org.springframework.test.context.junit.jupiter.SpringExtension.getApplicationContext(SpringExtension.java:207)
            at org.springframework.test.context.junit.jupiter.SpringExtension.resolveParameter(SpringExtension.java:193)
            at org.junit.jupiter.engine.execution.ExecutableInvoker.resolveParameter(ExecutableInvoker.java:216)
            ... 65 more

            Caused by:
            org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flywayInitializer' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Invocation of init method failed; nested exception is com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Container startup failed
                at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1788)
                at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:609)
                at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:531)
                at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
                at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
                at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
                at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
                at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
                at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
                at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1161)
                at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:915)
                at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:588)
                at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:767)
                at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759)
                at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:426)
                at org.springframework.boot.SpringApplication.run(SpringApplication.java:326)
                at org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:122)
                at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:99)
                at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:124)
                ... 69 more

                Caused by:
                com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Container startup failed
                    at com.zaxxer.hikari.pool.HikariPool.throwPoolInitializationException(HikariPool.java:595)
                    at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:581)
                    at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115)
                    at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:112)
                    at org.flywaydb.core.internal.jdbc.JdbcUtils.openConnection(JdbcUtils.java:57)
                    at org.flywaydb.core.internal.jdbc.JdbcConnectionFactory.<init>(JdbcConnectionFactory.java:69)
                    at org.flywaydb.core.Flyway.execute(Flyway.java:475)
                    at org.flywaydb.core.Flyway.migrate(Flyway.java:164)
                    at com.programmingmastery.database.schema.SchemaManagerImpl.migrate(SchemaManagerImpl.java:72)
                    at com.programmingmastery.database.schema.SchemaManagerImpl.migrate(SchemaManagerImpl.java:54)
                    at com.programmingmastery.database.schema.MigrationStrategy.migrate(MigrationStrategy.java:24)
                    at org.springframework.boot.autoconfigure.flyway.FlywayMigrationInitializer.afterPropertiesSet(FlywayMigrationInitializer.java:62)
                    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1847)
                    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1784)
                    ... 87 more

                    Caused by:
                    org.testcontainers.containers.ContainerLaunchException: Container startup failed
                        at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:327)
                        at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:308)
                        at org.testcontainers.jdbc.ContainerDatabaseDriver.connect(ContainerDatabaseDriver.java:118)
                        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
                        at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:358)
                        at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206)
                        at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:477)
                        at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:560)
                        ... 99 more

                        Caused by:
                        org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=postgres:12, imagePullPolicy=DefaultPullPolicy())
                            at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1278)
                            at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:612)
                            at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:317)
                            ... 106 more

                            Caused by:
                            com.github.dockerjava.api.exception.NotFoundException: Status 404: {"message":"No such image: testcontainers/ryuk:0.3.0"}
                                at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:241)
                                at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.post(DefaultInvocationBuilder.java:125)
                                at org.testcontainers.shaded.com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:33)
                                at org.testcontainers.shaded.com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:13)
                                at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
                                at org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
                                at org.testcontainers.shaded.com.github.dockerjava.core.command.CreateContainerCmdImpl.exec(CreateContainerCmdImpl.java:595)
                                at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:91)
                                at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:203)
                                at org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
                                at org.testcontainers.LazyDockerClient.listImagesCmd(LazyDockerClient.java:12)
                                at org.testcontainers.images.LocalImagesCache.maybeInitCache(LocalImagesCache.java:68)
                                at org.testcontainers.images.LocalImagesCache.get(LocalImagesCache.java:32)
                                at org.testcontainers.images.AbstractImagePullPolicy.shouldPull(AbstractImagePullPolicy.java:18)
                                at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:66)
                                at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:27)
                                at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:17)
                                at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:39)
                                at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1276)
                                ... 108 more
youagree commented 3 years ago

i have the same problem

image_2020_12_11T10_58_19_649Z

sparsick commented 3 years ago

Same here with Ubuntu 20.04, Docker 20.10 and Testcontainer 1.15.0. With a downgrade to docker version 19.03.14 everything is fine.

bsideup commented 3 years ago

FYI 1.15.1 is released and fixes "No such image" issue (#3574). Please only comment if you still see the issue with 1.15.1 and it is related to the TCP connectivity, not the image pulling. Thanks.

ullrichp commented 3 years ago

I found out that my issue must be related to the VPN i am connected to. When i turn if off everything works fine. When i am connected i get those connection problems. Will try to figure out whats going on with my networking.

Thanks so far.

uzoltan commented 3 years ago

I still get

Caused by: java.lang.IllegalStateException: Could not connect to Ryuk at localhost:55003 at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:198) at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:205) at org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14) at org.testcontainers.LazyDockerClient.authConfig(LazyDockerClient.java:12) at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:310)

:(

Docker version: 20.10.0 (docker for windows 3.0.0) + testImplementation "org.testcontainers:mysql:1.15.1" testImplementation "org.testcontainers:testcontainers:1.15.1"

EDIT: i'll try this: https://github.com/testcontainers/testcontainers-java/issues/3601 EDIT2: didn't help

bsideup commented 3 years ago

@uzoltan could you please share full logs?

uzoltan commented 3 years ago 
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.3.1.RELEASE)

2020-12-15 10:33:24.017  INFO [ocpi-command,,,] 8572 --- [           main] o.t.d.DockerClientProviderStrategy       : Loaded org.testcontainers.dockerclient.NpipeSocketClientProviderStrategy from ~/.testcontainers.properties, will try it first
2020-12-15 10:33:24.587  INFO [ocpi-command,,,] 8572 --- [           main] o.t.d.DockerClientProviderStrategy       : Found Docker environment with local Npipe socket (npipe:////./pipe/docker_engine)
2020-12-15 10:33:24.588  INFO [ocpi-command,,,] 8572 --- [           main] org.testcontainers.DockerClientFactory   : Docker host IP address is localhost
2020-12-15 10:33:24.623  INFO [ocpi-command,,,] 8572 --- [           main] org.testcontainers.DockerClientFactory   : Connected to docker: 
  Server Version: 20.10.0
  API Version: 1.41
  Operating System: Docker Desktop
  Total Memory: 1989 MB
2020-12-15 10:33:24.627  INFO [ocpi-command,,,] 8572 --- [           main] o.t.utility.ImageNameSubstitutor         : Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
2020-12-15 10:33:24.876  INFO [ocpi-command,,,] 8572 --- [           main] o.t.utility.RegistryAuthLocator          : Credential helper/store (docker-credential-desktop) does not have credentials for index.docker.io
2020-12-15 10:33:28.928  WARN [ocpi-command,,,] 8572 --- [containers-ryuk] o.testcontainers.utility.ResourceReaper  : Can not connect to Ryuk at localhost:55005

java.net.ConnectException: Connection refused: connect
    at java.base/java.net.PlainSocketImpl.connect0(Native Method)
    at java.base/java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:101)
    at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
    at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
    at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
    at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
    at java.base/java.net.Socket.connect(Socket.java:591)
    at java.base/java.net.Socket.connect(Socket.java:540)
    at java.base/java.net.Socket.<init>(Socket.java:436)
    at java.base/java.net.Socket.<init>(Socket.java:213)
    at org.testcontainers.utility.ResourceReaper.lambda$null$1(ResourceReaper.java:160)
    at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
    at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:158)
    at java.base/java.lang.Thread.run(Thread.java:834)

Then it's the same stacktrace a bunch of more times, like 10 times, and at the end:

2020-12-15 10:33:56.921 ERROR [ocpi-command,,,] 8572 --- [           main] o.testcontainers.utility.ResourceReaper  : Timed out waiting for Ryuk container to start. Ryuk's logs:
2020/12/15 09:33:26 Pinging Docker...
2020/12/15 09:33:26 Docker daemon is available!
2020/12/15 09:33:26 Starting on port 8080...
2020/12/15 09:33:26 Started!

Test fails with application context failed to load.

bsideup commented 3 years ago

@uzoltan thanks! Do you happen to have some VPN or anything else that might affect the network? Maybe you issue is similar to @ullrichp's: https://github.com/testcontainers/testcontainers-java/issues/3572#issuecomment-744417283

uzoltan commented 3 years ago

No VPN, docker doesn't run local kube cluster either, I only have 1 other mongo container running on 27017.

ullrichp commented 3 years ago

FYI: I solved all my issues with correctly configuring strongswan VPN with the bypass-lan plugin. Turns out its not done automatically and all local trafffic was send to the VPN.

zapho commented 3 years ago

Same issue here (1.15.1 and TCP connectivity), resolved by stopping VPN (Cisco AnyConnect) then restarting Docker for Windows then (re)starting VPN. Could not reproduce the issue after that though. Did not try 'Allow local LAN access', will do next time.

gvandeglind commented 3 years ago

Encountered the same issue (1.15.2, ubuntu 20.04 LTS with ufw enabled). It turns out that the Ryuk container is started on the default bridge network, while my other containers are started in my custom bridge network. Then the ufw firewall doesn't allow communication between these private networks, which I could resolve by adding rules. A nicer solution for me would be if I could specify in which network the Ryuk container is started, but I can't find that anywhere, such that it also starts in my custom bridge network.

luke-gee commented 3 years ago

Encountering this issue after updating docker desktop on Windows to the latest version version 3.4.0 (65384)

AlexanderArgyriou commented 1 year ago

any update here ?