testcontainers / testcontainers-java

Testcontainers is a Java library that supports JUnit tests, providing lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container.
https://testcontainers.org
MIT License
8.02k stars 1.65k forks source link

Feature request: support `docker.io/*` images to use auth config from `~/.docker/config.json` #4474

Open achaphiv opened 3 years ago

achaphiv commented 3 years ago

Version: org.testcontainers:testcontainers-bom:1.16.0

This works fine:

new GenericContainer<>("amazon/dynamodb-local:1.16.0")

I've been seeing more recommendations to prefer fully qualified docker images (e.g. podman), so I switched to:

new GenericContainer<>("docker.io/amazon/dynamodb-local:1.16.0")

And then errors started occurring during my CI build.

I eventually figured out that the auth config was not being picked up:

[Test worker] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: docker.io/amazon/dynamodb-local:1.16.0
[Test worker] DEBUG org.testcontainers.images.AbstractImagePullPolicy - Using locally available and not pulling image: docker.io/amazon/dynamodb-local:1.16.0
[Test worker] INFO 🐳 [docker.io/amazon/dynamodb-local:1.16.0] - Creating container for image: docker.io/amazon/dynamodb-local:1.16.0
[Test worker] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: docker.io/amazon/dynamodb-local:1.16.0 at registry: docker.io
[Test worker] DEBUG org.testcontainers.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /home/builder/.docker/config.json (exists) and commandPathPrefix: 
[Test worker] DEBUG org.testcontainers.utility.RegistryAuthLocator - registryName [docker.io] for dockerImageName [docker.io/amazon/dynamodb-local:1.16.0]
[Test worker] DEBUG org.testcontainers.utility.RegistryAuthLocator - No matching Auth Configs - falling back to defaultAuthConfig [null]

So I switched to index.docker.io, and it worked.

new GenericContainer<>("index.docker.io/amazon/dynamodb-local:1.16.0"):

[Test worker] DEBUG org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: index.docker.io/amazon/dynamodb-local:1.16.0
[Test worker] DEBUG org.testcontainers.images.AbstractImagePullPolicy - Using locally available and not pulling image: index.docker.io/amazon/dynamodb-local:1.16.0
[Test worker] INFO 🐳 [index.docker.io/amazon/dynamodb-local:1.16.0] - Creating container for image: index.docker.io/amazon/dynamodb-local:1.16.0
[Test worker] DEBUG org.testcontainers.dockerclient.AuthDelegatingDockerClientConfig - Delegate call to effectiveAuthConfig failed with cause: 'Invalid repository name, try "amazon/dynamodb-local" instead'. Resolution of auth config will continue using RegistryAuthLocator.
[Test worker] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: index.docker.io/amazon/dynamodb-local:1.16.0 at registry: index.docker.io
[Test worker] DEBUG org.testcontainers.utility.RegistryAuthLocator - Cached auth found: [AuthConfig{username=redacted, password=hidden non-blank value, auth=hidden non-blank value, email=null, registryAddress=https://index.docker.io/v1/, registryToken=blank}]

However, this is out of line with the docker cli, which does read from the ~/.docker/config.json.

If I change the https://index.docker.io/v1/ auth value and try to pull via:

docker -D pull docker.io/amazon/dynamodb-local:1.16.0

It fails with:

Error response from daemon: Head "https://registry-1.docker.io/v2/amazon/dynamodb-local/manifests/1.16.0": unauthorized: incorrect username or password
rnorth commented 3 years ago

Ah interesting - it sounds like there might be some magic behaviour in Docker CLI, mapping docker.io to another hostname. If this is the case, we'd have to emulate that, potentially.

Are you using hardcoded credentials in your ~/.docker/config.json file, or are you using a creds helper/store? (Looking in the file it should be fairly obvious which is in effect).

achaphiv commented 3 years ago

My CI runs:

docker login -u ${DOCKERHUB_USERNAME} -p ${DOCKERHUB_PASSWORD}
# ...
./gradlew clean build
rnorth commented 3 years ago

Depending on what's already in the config file, that docker login could be either writing hardcoded creds or pushing the creds into an helper/store.

achaphiv commented 3 years ago

This is AWS codepipeline/codebuild, so it just starts fresh each time.

I checked via cat ~/.docker/config.json, and it's just hardcoded { auths: { ... } }. No credentials store.

sschulze commented 1 year ago

In my case the docker.io registry is hardcoded in a library (specifically it's the reference to docker.io/vectorized/redpanda in the quarkus-kafka devservice). A rather dirty hack is to duplicate the auth entry in the ~/.docker/config.json to the name "docker.io". Sadly I have to do this manually with jq because docker login docker.io just creates the entry for the fully qualified hostname. This ist the line I use:

jq '.auths["docker.io"] = .auths[.auths|keys[]|select(.|test("http.*docker.io"))]' ~/.docker/config.json > ~/.docker/config.json.modified && \
mv ~/.docker/config.json.modified ~/.docker/config.json