ability to add local docker registry

[anatolykabakov]

Hi I am trying to add a local docker registry. I looked at the java docker interface you are using. There it is possible to add a docker registry by adding an authentication config RegistryConfigAuth in the constructor TcpDocker. Something like that:

Docker docker = new TcpDocker(URI.create(configService.getConfiguration().getDockerHost()), new RegistryConfigAuth( URI.create("myregistry.com"), "login", "pass")); But when I compile this code and run the script, I get an error:

bagdb_1     | 2020-11-03 18:07:14.140 [ThreadPoolTaskExecutor-2] DEBUG c.g.s.scripts.RunnableScript - Writing script to temporary directory: /scripts
bagdb_1     | 2020-11-03 18:07:14.140 [ThreadPoolTaskExecutor-2] INFO  c.g.s.scripts.RunnableScript - Pulling Docker image: ros_melodic:latest
bagdb_1     | 2020-11-03 18:07:14.140 [ThreadPoolTaskExecutor-2] ERROR c.g.s.scripts.RunnableScript - Unexpected exception
bagdb_1     | java.lang.UnsupportedOperationException: null
bagdb_1     |   at com.amihaiemil.docker.AuthHttpClient.execute(AuthHttpClient.java:114)
bagdb_1     |   at com.amihaiemil.docker.RtImages.pull(RtImages.java:91)
bagdb_1     |   at com.github.swrirobotics.scripts.RunnableScript.runScript(RunnableScript.java:203)
bagdb_1     |   at com.github.swrirobotics.scripts.RunnableScript.run(RunnableScript.java:126)
bagdb_1     |   at com.github.swrirobotics.scripts.RunnableScript$$FastClassBySpringCGLIB$$6e963ff4.invoke(<generated>)
bagdb_1     |   at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
bagdb_1     |   at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)
bagdb_1     |   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
bagdb_1     |   at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294)
bagdb_1     |   at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
bagdb_1     |   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
bagdb_1     |   at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
bagdb_1     |   at com.github.swrirobotics.scripts.RunnableScript$$EnhancerBySpringCGLIB$$820c9ae0.run(<generated>)
bagdb_1     |   at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
bagdb_1     |   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
bagdb_1     |   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
bagdb_1     |   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
bagdb_1     |   at java.base/java.lang.Thread.run(Thread.java:834)
bagdb_1     | 2020-11-03 18:07:14.198 [MessageBroker-5] DEBUG c.g.s.scripts.ScriptService - Before check: 1 scripts currently running.
bagdb_1     | 2020-11-03 18:07:14.199 [MessageBroker-5] DEBUG c.g.s.scripts.ScriptService - All scripts finished.
bagdb_1     | 2020-11-03 18:07:14.370 [http-nio-8080-exec-18] DEBUG c.g.s.scripts.ScriptController - getScriptResults

[pjreed]

Something to note is that modifying the code like that would only be necessary if you're using a registry that requires authentication. If your registry allows unauthenticated connections, you can just specify the full label for the image name, like myimage.com:5000/label:tag.

However, while testing that I also noticed that there's a bug that is preventing the script runner from parsing custom ports properly, so after I fix that, I'll test using authentication...

[pjreed]

Unfortunately, I don't think the Docker API library I'm using actually supports authenticating with a registry yet. Looking at the class that should be handling it: https://github.com/amihaiemil/docker-java-api/blob/653943a3012fe104ff5d226351cebbe52785eba5/src/main/java/com/amihaiemil/docker/AuthHttpClient.java

It looks like most of the functions there just do nothing and throw exceptions.

So, solutions for this would be to implement that functionality in amihaiemil/docker-java-api or switch to a different Docker API. When I was originally designing this, I actually also tried using docker-java, but I had some serious problems with being able to reliably capture the output from containers with it and so switched to a different one. I'm not sure if there are any other viable options...

[anatolykabakov]

Thank you for your prompt response.

please write down the docker registry settings you used when testing the docker request from the local registry. when testing, I get the error

i am using the following docker settings `bagdb_registry: restart: always image: registry:2 ports:

• "5000:5000"
  volumes:
• /dockerregistry/data:/var/lib/registry `

[pjreed]

When creating a registry like that, it will only listen on HTTP; however, by default, the docker daemon attempts to use HTTPS to connect to registries. If your registry is publicly accessible, you should configure it to use SSL and provide a valid certificate, although if it's private, it will be easiest to just configure the docker-in-docker container used by the bag database to allow an insecure registry.

My docker-compose.yml file that I'm using right now looks like this:

version: '3.6'
services:
    registry:
        image: registry:2
        ports:
            - "5000:5000"
        volumes:
            - registry:/var/lib/registry
    docker:
        image: docker:dind
        privileged: yes
        ports:
            - "2377:2377"
        volumes:
            - bags:/bags:ro # Needs to match the path in the bagdb container
            - /var/lib/tomcat9/bagdb_scripts:/var/lib/tomcat9/bagdb_scripts
            - /home/preed/public_html/bags:/home/preed/public_html/bags:ro
            - docker_cache:/var/lib/docker
            - /home/preed/src/bag-database/daemon.json:/etc/docker/daemon.json
        command: ["dockerd", "--host=tcp://0.0.0.0:2377"]
    postgres:
        image: postgis/postgis:11-2.5
        volumes:
            - postgres:/var/lib/postgresql/data
        ports:
            - "5432:5432"
        environment:
            POSTGRES_PASSWORD: letmein
            POSTGRES_USER: bag_database
            POSTGRES_DB: bag_database
volumes:
    bags:
        driver: local
        driver_opts:
            type: 'none'
            o: 'bind'
            device: '/home/preed/public_html/bags'
    docker_cache:
    postgres:
    registry:

Note that it mounts a file at /etc/docker/daemon.json inside the docker container to configure it; that file looks like this:

{
  "insecure-registries" : ["registry:5000"]
}

Then, when the Bag Database tries to pull an image in its docker container from that registry, it will use HTTP instead of HTTPS. Keep in mind that any other clients that use that registry will also need to have the insecure-registries option set with the hostname and port they use to connect to it.

Here's a very simple example that works for me:

[anatolykabakov]

thanks for helping me!

[pjreed]

Great! I've added some documentation on setting up a registry, so I'll go ahead and close this issue.