Feature: docker swarm automatically download image when not available on a node

[ziyunxiao]

Hello,

When using docker swarm, docker swarm itself will download the image when an image is not available on a node. While the Shinyproxy current behaviour is that an image on a public repo(no auth) will succeed but when on a private repo(need auth) will fail.

I will create a Pull Request for this feature. Please review and see if it can be accepted. The code need be changed on both containerproxy and shinyproxy.

Pull request

• containerproxy
• shinyproxy

TL;DR in containerproxy source code, it creates a service without authentication, and it is the root cause.

String serviceId = dockerClient.createService(serviceSpecBuilder.build()).id();

to fix that, it needs to pass authentication info like this

serviceId = dockerClient.createService(serviceSpecBuilder.build(), registryAuth).id();

To better function for shinyproxy, I am using three new container settings(container-auth-domain, container-auth-user and container-auth-password) in application.xml to give users more flexibility and fewer steps compare to use docker default configuration ~/.docker/config.json. (It will hit a bug if use docker default configuration which jersey-common version is not the same as the docker-client's dependency. org.glassfish.jersey.internal.util.Base64 package is removed from version 2.30 which the end output from shinyproxy jar)

For testing if you want to skip java building phase, the jar file can be downloaded from here the application.xml can be something like this.

proxy:
  title: ShinyProxy Test
  port: 8080
  container-wait-time: 60000
  authentication: simple
  users:
    - name: admin
      password: admin
      groups: shiny-admin
    - name: jeff
      password: password
  container-backend: docker-swarm
  docker:
    internal-networking: true
  specs:
    - id: 01_hello
      display-name: Hello Application
      description: Application which demonstrates the basics of a Shiny app
      container-cmd: ["R", "-e", "shinyproxy::run_01_hello()"]
      container-image: openanalytics/shinyproxy-demo
      container-network: shiny_shiny-net
    - id: 02_hello_private_repo
      display-name: Hello Application from private repo
      description: Application which demonstrates using private repo
      container-cmd: ["R", "-e", "shinyproxy::run_01_hello()"]
      container-image: ziyunxiao/shinyproxy-demo
      container-auth-domain: docker.io
      container-auth-user: ziyunxiao
      container-auth-password: yourpassword
      container-network: shiny_shiny-net
    - id: 06_tabsets
      container-cmd: ["R", "-e", "shinyproxy::run_06_tabsets()"]
      container-image: openanalytics/shinyproxy-demo
      container-network: shiny_shiny-net
    - id: shiny_admin_dashboard
      display-name: Shiny Admin
      description: This application used for admin only. It includes only Keycloak Admin info for now.
      container-cmd: ["./start.sh"]
      container-image: yourowndomain.com/yourownimage:lastest
      container-auth-domain: yourowndomain
      container-auth-user: admin
      container-auth-password: yourpassword
      container-network: shiny_shiny-net

logging:
  level:
    root: INFO

server:
  #useForwardHeaders: true #2.3.1
  forward-headers-strategy: native
  #useForwardHeaders: false
  servlet.session.timeout: 36000

And to deploy shiny on docker swarm, the docker-compose.yml can be something like this

version: "3.8"

services:
  shiny:
    image: yournamespace/shinyproxy:2.5.0
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /youpath/local_test/application_docker.yml:/opt/shinyproxy/application.yml
    ports:
      - 8080:8080
    networks:
      - shiny-net
    deploy:
      mode: replicated
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.role==manager
          - node.hostname==yournodename

networks:
  shiny-net:
    driver: overlay

For test deploy shiny docker stack deploy -c docker-compose.yml shiny

Thanks,

Robin

[ziyunxiao]

@LEDfan could you please take a look if this feature/update can be merged into future releases? I need this feature but don't want to maintain a copy of the code.

[LEDfan]

Hi @ziyunxiao

Thank you for your contribution! I understand the use-case and see the value. However, it will take some time for me to get though your PR and merge it. For example, I have to think about the bigger picture of this feature. When using plain Docker, there is also the issue that ShinyProxy doesn't automatically pull docker images. Maybe we should fix these two issues at the same time?

[ziyunxiao]

For plain Docker it can be fixed by adding pull image logic, there is a way to get it to work.

In file DockerEngineBackend.java around line 90, add logic to pull the image if the image does not exist.

        ContainerConfig containerConfig = ContainerConfig.builder()
                .hostConfig(hostConfigBuilder.build())
                .image(spec.getImage())
                .labels(labels)
                .exposedPorts(portBindings.keySet())
                .cmd(spec.getCmd())
                .env(buildEnv(spec, proxy))
                .build();

        List<Image> images = dockerClient.listImages(com.spotify.docker.client.DockerClient.ListImagesParam.byName(spec.getImage()));
        if (images.size() == 0){
            dockerClient.pull(spec.getImage());
        }

        ContainerCreation containerCreation = dockerClient.createContainer(containerConfig);

While this requires the user(admin) to do docker login first on the private repo. It differently requires either document or a better logic to handle this.

Regards,

[jtelleriar]

For me, this is also a key topic.

Could it be merged in order to fetch Docker images from private Docker Hub repos?

[LEDfan]

This is now part of ShinyProxy 3.0.0!