JUPYTERHUB_OAUTH_SCOPES and JUPYTERHUB_OAUTH_ACCESS_SCOPES not correctly set with openpbs

[danmons]

Echoing out the JUPYTERHUB_OAUTH_SCOPES and JUPYTERHUB_OAUTH_SCOPES environment variables from JupyterHub jobs created by batchspawner, these values appear incorrect.

The current solution I'm using is noted in this issue: https://github.com/jupyterhub/jupyterhub/issues/3866

I have the following in my batch_script section just before running the jupyterhub-singleuser component:

echo JUPYTERHUB_OAUTH_SCOPES "$JUPYTERHUB_OAUTH_SCOPES"
echo JUPYTERHUB_OAUTH_ACCESS_SCOPES "$JUPYTERHUB_OAUTH_ACCESS_SCOPES"

export JUPYTERHUB_OAUTH_SCOPES='["access:servers!server={username}/", "access:servers!user={username}"]'
export JUPYTERHUB_OAUTH_ACCESS_SCOPES='["access:servers!server={username}/", "access:servers!user={username}"]'

echo JUPYTERHUB_OAUTH_SCOPES "$JUPYTERHUB_OAUTH_SCOPES"
echo JUPYTERHUB_OAUTH_ACCESS_SCOPES "$JUPYTERHUB_OAUTH_ACCESS_SCOPES"

And it produces:

JUPYTERHUB_OAUTH_SCOPES [\access:servers!server=monsd/\, \access:servers!user=monsd\]
JUPYTERHUB_OAUTH_ACCESS_SCOPES [\access:servers!server=monsd/\, \access:servers!user=monsd\]

JUPYTERHUB_OAUTH_SCOPES ["access:servers!server=monsd/", "access:servers!user=monsd"]
JUPYTERHUB_OAUTH_ACCESS_SCOPES ["access:servers!server=monsd/", "access:servers!user=monsd"]

The top two lines are the default set by batchspawner, which then cause the resulting jupyterhub-singleuser instances to fail authentication. The bottom two are after I override them, and allow the instances to pass authentication and work.

Local instances of jupyterhub-singleuser, jupyter-server or JupyterLab all work fine. The problem only exists with batchspawner creating the jobs.

I'm using ProfileSpawner to build various launch profiles, and within that PBSSpawner via batchspawner.

jupyterhub_config.py snippet:

import batchspawner
c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'
c.ProfilesSpawner.profiles = [
( 'Singularity /export/home/{username}/singularity/jupyterhub.sif', 'singularity_home_jupyter_cpu', 'batchspawner.PBSSpawner', dict(req_queue = 'workq', req_nprocs = '8', req_memory = '8gb', req_runtime = '1:00:00', batch_script = '''#!/bin/bash
#PBS -l walltime={runtime}
#PBS -l nodes=1:ppn={nprocs}
#PBS -l mem={memory}
#PBS -N jupyterhub
#PBS -v {keepvars}

echo JUPYTERHUB_OAUTH_SCOPES "$JUPYTERHUB_OAUTH_SCOPES"
echo JUPYTERHUB_OAUTH_ACCESS_SCOPES "$JUPYTERHUB_OAUTH_ACCESS_SCOPES"

export JUPYTERHUB_OAUTH_SCOPES='["access:servers!server={username}/", "access:servers!user={username}"]'
export JUPYTERHUB_OAUTH_ACCESS_SCOPES='["access:servers!server={username}/", "access:servers!user={username}"]'

echo JUPYTERHUB_OAUTH_SCOPES "$JUPYTERHUB_OAUTH_SCOPES"
echo JUPYTERHUB_OAUTH_ACCESS_SCOPES "$JUPYTERHUB_OAUTH_ACCESS_SCOPES"

singularity exec --bind /scratch /export/home/{username}/singularity/jupyterhub.sif {cmd}

''')),

Tested under both Python 3.10 and 3.12 with the following (batchspawner installed from main today).

batchspawner              1.3.0.dev0
jupyter_client            8.6.0
jupyter_core              5.7.1
jupyter-events            0.9.0
jupyter-lsp               2.2.3
jupyter_server            2.13.0
jupyter_server_terminals  0.5.2
jupyter-telemetry         0.1.0
jupyterhub                4.0.2
jupyterlab                4.1.2
jupyterlab_pygments       0.3.0
jupyterlab_server         2.25.3
tornado                   6.4
traitlets                 5.14.1
wrapspawner               1.0.1

[welcome[bot]]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

[minrk]

Can you show the actual batch script it produces? Can you also test a batch script by hand with:

#PBS -v TEST_ENV
echo TEST_ENV: "${TEST_ENV}"

and submit it with:

export TEST_ENV='["item1", "item2"]'
qsub script.sh

Form the looks of it, it seems like it might be in how keepvars is implemented, which might be a pain to work around.

[danmons]

Can you show the actual batch script it produces?

Taken from the service log on the hub:

[I 2024-03-13 01:51:17.524 JupyterHub batchspawner:281] Spawner submitting job using sudo -i -u monsd qsub
[I 2024-03-13 01:51:17.524 JupyterHub batchspawner:282] Spawner submitted script:
    #!/bin/bash
    #PBS -l walltime=12:00:00
    #PBS -l nodes=1:ppn=8
    #PBS -l mem=8gb
    #PBS -N jupyterhub
    #PBS -v PATH,PYTHONPATH,LANG,JUPYTERHUB_API_TOKEN,JPY_API_TOKEN,JUPYTERHUB_CLIENT_ID,JUPYTERHUB_HOST,JUPYTERHUB_OAUTH_CALLBACK_URL,JUPYTERHUB_OAUTH_SCOPES,JUPYTERHUB_OAUTH_ACCESS_SCOPES,JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES,JUPYTERHUB_USER,JUPYTERHUB_SERVER_NAME,JUPYTERHUB_API_URL,JUPYTERHUB_ACTIVITY_URL,JUPYTERHUB_BASE_URL,JUPYTERHUB_SERVICE_PREFIX,JUPYTERHUB_SERVICE_URL

    ## Fix batchspawner bug
    export JUPYTERHUB_OAUTH_SCOPES='["access:servers!server=monsd/", "access:servers!user=monsd"]'
    export JUPYTERHUB_OAUTH_ACCESS_SCOPES='["access:servers!server=monsd/", "access:servers!user=monsd"]'

    ## Run apptainer
    apptainer exec --bind /scratch /export/home/monsd/apptainer/jupyterhub.sif batchspawner-singleuser jupyterhub-singleuser

[I 2024-03-13 01:51:17.802 JupyterHub batchspawner:285] Job submitted. cmd: sudo -i -u monsd qsub output: 8545408.pbsheadnode
[D 2024-03-13 01:51:17.802 JupyterHub batchspawner:312] Spawner querying job: sudo -i -u monsd qstat -fx 8545408.pbsheadnode

Can you also test a batch script by hand with:

$ cat script.sh
#PBS -v TEST_ENV
echo TEST_ENV: "${TEST_ENV}"

$ export TEST_ENV='["item1", "item2"]'
$ qsub script.sh

$ cat script.sh.o8545782
TEST_ENV: [\item1\, \item2\]

[minrk]

oof, so it looks like this is an actual bug in PBS/torque where it doesn't handle environment variables with quotation marks correctly. That's going to be frustrating to work around.

Can you tell the name and version of your batch system?

[danmons]

Product is Altair PBS Professional. Version installed:

# rpm -qa pbspro-server
pbspro-server-2021.1.2.20210913160841-0.x86_64

If this is a massive pain just for the positively ancient scheduler installed at this site, I'm happy enough just to run with the workaround in place. I've got it documented internally and it's working now.

I've just tested this in OpenPBS 23.06.06 (outside of JupyterHub/batchspawner, just by hand like above), and it has the same bug.

I've also tested Slurm 23.11.4, and it does not have the same bug. It's working fine.

Unfortunately I'm not set up to test Torque currently. I will pass this on to the commercial support people at Altair.

If it's too painful to work around this specific use case, I'm happy for the issue to be closed, and I can pass on these details to the relevant projects and commercial support groups.

[minrk]

I don't know how to get a test environment set up for openpbs, but this does appear to be a clear bug in openpbs incorrectly implementing environment escaping, which I found here.

The general workaround would be to have two ways to specify environment variables, one which trusts -e ENV_NAME to work (i.e. not openpbs), and one where we reimplement it by adding export... lines to the script. If we did it for all environment variables, it would leak things like the token in the script file, which is undesirable. Instead, we could detect and workaround this specific bug and check for quotation marks.

[jbaksta]

I guess I'll chime in here. We run PBS Pro and not OpenPBS, but they share some history. Our site made a bug report on PBS Pro at some point regarding how variables were not properly constructed at job launch (huge problem and not the only one we have), but ultimately I ended up base64 encoding all the JUPYTERHUB_* variables into a single one (things like brackets seem to break stuff in PBS Pro) and decoding it as part of the job_script and sourcing those vars into the job environment.

I can try to look up to see if it's fixed in our new 2022.1.5 version or 2021.1.3 which we just upgraded from if it's of interests to you, but regarding OpenPBS, it's quite a bit behind the PBS Pro versions unfortunately 😢

Anyways, not a great solution, but has worked for a bit for us.

[minrk]

@jbaksta thanks, that's useful to know. I think it's really beneficial to be able to set actual environment variables on the submitting process and include only keys in the batch script. Putting env values in the script is not ideal, because there are API keys in there.

But it seems clear that in at least some real situations values do need to be in the script, so having a flag for putting env values in the script which safely encodes them like you've done seems like it might be worth doing. It's unfortunate since it is pretty clearly a bug in PBS, but it's more important to be useful and meet people where they are than blame other projects' bugs and not do anything about it.

[jbaksta]

There are number of issues with PBS that revolve around security and JupyterHub API keys. Also, if not paying attention around the server config, often times job submitted variables are open to the world. I'm not sure of the best way to address this is without maybe implementing some encryption on things, but that might require some elements that I'm not sure would work well in our environment either.

[minrk]

Presumably such systems generally have a shared filesystem, so perhaps the simplest approach is to rely on filesystem permissions:

• make/ensure private directory on shared filesystem (like ~/.ssh)
• write environment to private file in that directory
• source the environment file in the job script