Unhandled error starting user's server: sudo: sbatch: command not found

[heavenkong]

I have run jupyterhub as root. And I found .../lib/python3.6/site-packages/batchspawner/batchspawner.py it is written like this (slurm scheduler part):

    # outputs line like "Submitted batch job 209"
    batch_submit_cmd = Unicode('sudo -E -u {username} sbatch --parsable').tag(config=True)
    # outputs status and exec node like "RUNNING hostname"
    batch_query_cmd = Unicode("sudo -u {username} squeue -h -j {job_id} -o '%T %B'").tag(config=True)
    batch_cancel_cmd = Unicode('sudo -u {username} scancel {job_id}').tag(config=True)

the error message is like this:

[I 2019-08-13 17:26:37.820 JupyterHub batchspawner:188] Spawner submitting job using sudo -E -u ghlee sbatch --parsable
[I 2019-08-13 17:26:37.820 JupyterHub batchspawner:189] Spawner submitted script:
    #!/bin/bash
    #SBATCH -p skl_v100_10
    #SBATCH --time=01:00:00
    #SBATCH --nodes=1
    #SBATCH --nodelist=gpu49
    #SBATCH --comment etc
    source /home01/ghlee/.bashrc
    conda activate my_notebook
    jupyterhub-singleuser --ip="0.0.0.0" --port=33435 --notebook-dir="/home01/ghlee" --debug --log-level=40

fea09b7030e845358604142473b716e2
[E 2019-08-13 17:26:37.851 JupyterHub user:477] Unhandled error starting ghlee's server: sudo: sbatch: command not found
[D 2019-08-13 17:26:37.870 JupyterHub user:578] Deleting oauth client jupyterhub-user-ghlee

If I change sudo -E -u {username} sbatch --parsable to sudo -E -u {username} -i sbatch --parsable, then it is submitted successfully. But I found that it can not pass the JUPYTERHUB_API_TOKEN to singleuser in the separated machine.

What is the correct command use here as root to run the jupyterhub? Please help.!

[jbaksta]

Just noting here, have a look at the sudo configuration to make sure that sudo isn't restricting PATHs. I'd suggest not running jupyterhub as root and configuring sudo properly to handle this. There are a few ways to do this as well. In addition, when using sudo, you may want to specify the full path to the Slurm commands.

[heavenkong]

Thank you. You are right. I had to put full path to the slurm commands. I will close this issue.

[miguelmarco]

I have found the same problem. How do you put the full path to the slurm commands? Is there a config option for that?

[heavenkong]

Try ‘which slurm’ to get the full path and use it.

2022년 4월 28일 (목) 오후 11:21, miguelmarco @.***>님이 작성:

I have found the same problem. How do you put the full path to the slurm commands? Is there a config option for that?

— Reply to this email directly, view it on GitHub https://github.com/jupyterhub/batchspawner/issues/152#issuecomment-1112268636, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEXEHFFGRXZLFN2ZNLDZEDDVHKNIDANCNFSM4ILIN7QQ . You are receiving this because you modified the open/close state.Message ID: @.***>

-- Gmail 모바일에서 보낸 메일

[miguelmarco]

I mean... use it where? Is there some option for that in the jupyterhub config file?

[jbaksta]

It can be combination of configuring the batchspawner submit, query, and cancel versions to include their appropriate prefix if outside of the sudoers secure_path and configuring sudo properly for the user running JupyterHub. Usually requires the use of NOPASSWD. I'd suggest reading the docs for sudoers.