Clarify handling of the jobs jupyter port

[gotzl]

Dear all,

my setup looks like this

• I run jupyterhub in a docker container
• some batch nodes handled by slurm

After some trail&error I've got this setup working, jupyterhub issues jobs and connects to the server started by that job. But, there is one point I don't understand.

I see that the name/IP of the execution node is retrieved by querying the job after it is started. But what about the port where the jobs server is listening? In the default configuration this port is chosen by batchspawner-singleuser (when executing the batch_script), but I don't see how this port is propagated back to jupyterhub. The effect is that the execution gets stuck in batchspawner/batchspawner.py:400.

To overcome this issue, I've extended the SlurmSpawner like this (excerpt of the jupyterhub_config.py used in the docker container)

class MySpawner(batchspawner.SlurmSpawner):
    mock_port = 50000

    req_port = Unicode('',
        help="Port where the jobs jupyterserver listens"
        ).tag(config=True)

    async def start(self):
        self.mock_port += 1
        self.req_port = str(self.mock_port)
        ret = await super().start()
        return ret
c.BatchSpawnerBase.batchspawner_singleuser_cmd = ''

and added ....{{cmd}} --port={{port}} to the batch_script.

But I wonder if I missunderstood s.t. here ...

[welcome[bot]]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

[gotzl]

Hmm.. Ok, I think I found how it is supposed to work. There is this code

    hub_auth._api_request(method='POST',
                          url=url_path_join(hub_auth.api_url, 'batchspawner'),
                          json={'port' : port})

in batchspawner/singleuser.py. This should propagate the port back to the spawner. I'll close this issue, since I think I get it now. Just need to find out why it wouldn't work before ;)

[mbmilligan]

Hi @gotzl - I'm glad to see your setup is working now. As for why it wasn't working before, in my experience, it's a common problem that firewalls or docker network routing don't allow the notebook server to connect back to the Hub's API port. It's hard to give general advice here since everyone's infrastructure is a bit different, but making sure that that connection can succeed is an aspect that often needs attention.

And like the welcome bot said, welcome to the community!