Allow multi client

[Pisamad]

Proposed change

I would like to be able to reach more than one docker server from one jupyterhub. Today, as client property of a DockerSpawner instance is a class attribute, I can not have, from the same jupyterhub, one user opening a NB container on a docker server A, and another user opening a NB container on a container B.

Alternative options

Allow the client property to be a instance attribute (is it enough ?)

[welcome[bot]]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

[1kastner]

I do not totally get what you intend to do. Maybe it is related to your wording. Do you want several docker servers as in docker engines to run in parallel? Why? Here I start guessing:

1) Do you want to distribute the workload? Is it about executing some docker containers on a remote docker engine?

2) Do you want several docker images for one user? Then maybe the docker whitelisting feature helps you here.

Or did I completely miss your point?

[Pisamad]

Hi, I want to use whitelist with, not only an images list, but a list of (remote docker server, image) :

DockerImages={
  'BaseNB on docker 01' : {
    'server': 'srv_docker_01',
    'image': 'jupyter/base-notebook:latest', 
  }, 
  'R_NB on docker 02' : {
    'server': 'srv_docker_02',
    'image': 'jupyter/r-notebook:latest',
  },
  'BaseNB on docker 02' : {
    'server': 'srv_docker_02',
    'image': 'jupyter/base-notebook:latest', 
  }, 
}

My Jupyter config is as following :

from images import DockerImages 
# Make white list images
allowImages = {}
for image in DockerImages: allowImages[image]=DockerImages[image]['image']

DOCKER_PORT='2376'

c.JupyterHub.spawner_class = 'multidockerspawner.MultiDockerSpawner'
c.MultiDockerSpawner.image_whitelist = allowImages
c.MultiDockerSpawner.extra_create_kwargs.update({'user': 'root'})

def userdata_hook(spawner, auth_state):
  if (spawner.user_options.get('image')):
    image = spawner.user_options['image']
    DOCKER_IP = DockerImages[image]['server']
    spawner.host_ip = DOCKER_IP
    spawner.client_kwargs.update({'base_url': 'tcp://'+DOCKER_IP+':'+DOCKER_PORT})
  #end
  spawn_cmd = ['/etc/jupyter/data/start.sh', 
    str(auth_state['sAMAccountName'][0]), 
    str(auth_state['uidNumber'][0]), str(auth_state['gidNumber'][0])]
  spawner.extra_create_kwargs.update({ 'command': spawn_cmd})
#end
c.MultiDockerSpawner.auth_state_hook = userdata_hook    

notebook_dir = '/home/{username}/work'
c.MultiDockerSpawner.notebook_dir = notebook_dir
c.MultiDockerSpawner.volumes = {
  'jupyterhub-data': '/etc/jupyter/data',
  '/mnt/user/{username}/work': notebook_dir
}
c.MultiDockerSpawner.remove = True

And MultiDockerSpawner is :

import docker
from docker.errors import APIError
from docker.utils import kwargs_from_env
from dockerspawner import DockerSpawner

class MultiDockerSpawner(DockerSpawner):
    @property
    def client(self):
        """single global client instance"""
        if self._client is None:
            kwargs = {"version": "auto"}
            if self.tls_config:
                kwargs["tls"] = docker.tls.TLSConfig(**self.tls_config)
            kwargs.update(kwargs_from_env())
            kwargs.update(self.client_kwargs)
            self.log.info(
              "New client %s ", kwargs
            )
            client = docker.APIClient(**kwargs)
            self._client = client
            self.log.debug('Client %s', client.base_url)
        return self._client

The problem I had is that in DockerSpawner, _client is a class attribute and, so, client is set only one time and I can switch from a docker server to another one. Setting _client as an instance attribute allows me to do that.

[1kastner]

Thank you very much for this clarification, now I get your point. I thought I could help out but I guess somebody more involved in this project needs to help out.

[manics]

I don't think we'd want MultiDockerSpawner in this repo as it's a very specialised use, but I think it's fine to make changes to help with subclassing. Do you want to open a PR and someone else can review? I don't know enough about the history of this spawner to tell whether client needs to be a class variable or whether it's fine for each instance to have it's own client (naively I'd think it's fine 😀).

[Pisamad]

@manics , Yes I will open a new PR to propose my solution. It seems that my need is very specific. As I'm new in Jupyter world, I was not sure to go to the best practice. Whatever, it seems that solution to use client instance attribute works without regression.

[minrk]

Hi! I’m going through and cleaning up old/stale issues on this repo.

I agree with @manics that subclassing is the right way to go for this. Since it looks like overriding client is sufficient, there aren't any additional actions to take on this repo.