Interaction between batchspawner and profilespawner

[dstndstn]

Hi,

I'm not sure if this should be a batchspawner issue or a profilespawner issue. I have filed this batchspawner issue: https://github.com/jupyterhub/batchspawner/issues/127

I have a (Slurm) BatchSpawner wrapped in a ProfileSpawner, for a typical academic compute cluster use case -- users can either start their notebook processes on the head node, or on a compute node via Slurm.

When a batchspawner worker starts up on the remote worker node, it calls an API on the hub, which goes to a batchspawner API handler. The API handler retrieves the user.spawner object, and tries to set its .current_port attribute. It is expecting that spawner object to be a BatchSpawner, but in my setup it is actually a ProfileSpawner, so it sets the .current_port on an object other than the one that the main BatchSpawner class is polling, and it eventually thinks that the remote process timed out.

I added the following to the batchspawner API handler, here https://github.com/jupyterhub/batchspawner/blob/master/batchspawner/api.py#L12 , and it works, but is ugly:

        spawner = user.spawner
    try:
            from wrapspawner import WrapSpawner
            if isinstance(spawner, WrapSpawner):
        spawner = spawner.child_spawner
    except:
            pass
    spawner.current_port = port

Now, it seems like a cleaner solution might be for WrapSpawner to propagate some setattr requests to its child_spawner?

Open to suggestions for a clean fix. Thanks!

[dstndstn]

Hi,

I'm trying to update all the packages involved in my installation, and this remains an issue. Due to some other changes, I had to change my workaround to:

        spawner = user.spawner
        try:
            from wrapspawner import WrapSpawner
            if isinstance(spawner, WrapSpawner):
                spawner = spawner.child_spawner
        except:
            self.log.info('Exception finding wrapped spawner')
            import traceback
            traceback.print_exc()
        spawner.port = port

I really don't understand why I am encountering this issue. My setup -- ProfileSpawner offering the choice between a number of different SlurmSpawners (with different queue settings, with/without GPUs, etc) seems totally vanilla off-the-shelf exactly what it was meant for, yet out of the box, it just does not work. What gives?

[dstndstn]

Further to this -- and maybe this should be a new issue -- I am finding that in the current versions of everything, OAuth authentication is ALSO broken by ProfilesSpawner.

That is: using one of my SlurmSpawners as the spawner, everything works fine.

Using ProfilesSpawner, my singleuser server is created on the compute node, but when my web browser goes to connect to the server, it gets a 404 Not found, and in the singleuser server logs,

[D 2021-03-25 12:10:35.573 SingleUserNotebookApp auth:504] No user identified

When using SlurmSpawner directly, in the server logs I see the OAuth process going on,

Mar 25 11:59:55 mn001 bash[34760]: [I 2021-03-25 11:59:55.144 JupyterHub log:181] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-dlang&redirect_uri=%2Fuser%2Fdlang%2Foauth_callback&response_type=code&state=[secret] -> /user/dlang/oauth_callback?code=[secret]&state=[secret] (dlang@::ffff:10.40.1.17) 23.57ms
Mar 25 11:59:55 mn001 bash[34760]: [D 2021-03-25 11:59:55.300 JupyterHub provider:55] authenticate_client <oauthlib.Request SANITIZED>
Mar 25 11:59:55 mn001 bash[34760]: [D 2021-03-25 11:59:55.312 JupyterHub provider:114] confirm_redirect_uri: client_id=jupyterhub-user-dlang, redirect_uri=/user/dlang/oauth_callback
Mar 25 11:59:55 mn001 bash[34760]: [D 2021-03-25 11:59:55.313 JupyterHub provider:334] Saving bearer token {'access_token': 'REDACTED', 'expires_in': 3600, 'token_type': 'Bearer', 'scope': 'identify', 'refresh_token': 'REDACTED'}
Mar 25 11:59:55 mn001 bash[34760]: [D 2021-03-25 11:59:55.319 JupyterHub provider:189] Deleting oauth code feM... for jupyterhub-user-dlang
Mar 25 11:59:55 mn001 bash[34760]: [I 2021-03-25 11:59:55.336 JupyterHub log:181] 200 POST /hub/api/oauth2/token (dlang@172.16.1.1) 42.35ms
Mar 25 11:59:55 mn001 bash[34760]: [I 2021-03-25 11:59:55.359 JupyterHub log:181] 200 GET /hub/api/authorizations/token/[secret] (dlang@172.16.1.1) 16.11ms

while with ProfilesSpawner it does none of this.

I haven't yet figured out how to fix this, but it just makes me wonder how ProfilesSpawner ever works for anyone. I'm not trying to be rude, it just seems like this is the most straightforward case and it just doesn't work, so I feel like I'm just missing something big. Thanks.

[dstndstn]

Regarding the linked traitlets: I printed out the traits that are getting linked (https://github.com/jupyterhub/wrapspawner/blob/master/wrapspawner/wrapspawner.py#L92) and the list is:

user           
http_timeout   
db             
authenticator  
config         
api_token      
oauth_client_id
notebook_dir   
parent         
hub            
log            

Note that "port" and "server" are not among them! (related to the 404 reported by @Jon-Lillis https://github.com/jupyterhub/wrapspawner/issues/44 ?)