hakasapl
commented
3 years ago Whoops, I seem to have submitted this report twice
Closed hakasapl closed 3 years ago
hakasapl
commented
3 years ago Whoops, I seem to have submitted this report twice
Bug description
When I try to spawn a JupyterLab instance using the Slurm spawner, the job launches according to slurm, however, the jupyterhub window just stays in the "Your server is starting up" page.
Expected behaviour
Allow me to enter the JupyterLab session that is launched
Actual behaviour
Stuck in launch screen.
How to reproduce
Your personal set up
There is a firewall between the Hub and the Lab, but I've temporarily disabled it for testing purposes. It seems like JupyterLab can communicate with JupyterHub, because when I stop JupyterHub, JupyterLab starts spitting out connection refused messages - leading me to believe that it was communicating when it was up.
Maybe there is some batchspawner configuration I'm missing with the 1.0 update?
Configuration
Slurm Spawner, slurm v 20.11.2 ```python # jupyterhub_config.py import os import importlib.machinery import batchspawner c.JupyterHub.log_level = 'DEBUG' jhub_dir = "/srv/www/jhub" slurm_config = importlib.machinery.SourceFileLoader('slurm_config',jhub_dir + '/slurm_config.py').load_module() form_config = importlib.machinery.SourceFileLoader('form_config',jhub_dir + '/form_config.py').load_module() c.JupyterHub.cleanup_servers = False # Shibboleth Authenticator c.JupyterHub.authenticator_class = 'jhub_remote_user_authenticator.remote_user_auth.RemoteUserLocalAuthenticator' c.JupyterHub.spawner_class = 'optionsspawner.OptionsFormSpawner' # This is what allows JHUB to not be at the root / folder c.JupyterHub.bind_url = 'http://:8000/panel/jhub' c.JupyterHub.hub_bind_url = 'http://:8081' c.ConfigurableHTTPProxy.api_url = 'http://0.0.0.0:8001' # Set notebook directory to the user's home directory c.Spawner.notebook_dir = '~' # Default to home directory #c.Spawner.default_url = '/lab' # Default to jupyterlab # Set admin users TODO use sudo instead c.Authenticator.admin_users = {'hsaplakoglu_umass_edu','jgriffin_umass_edu','aryamanagraw_umass_edu'} c.JupyterHub.admin_access = True c.Spawner.cmd=["jupyterhub-singleuser"] c.Spawner.ip = '0.0.0.0' c.Spawner.http_timeout = 300 c.Spawner.start_timeout = 3600 c.OptionsFormSpawner.child_class = 'batchspawner.SlurmSpawner' c.SlurmSpawner.batch_script = slurm_config.batch_script c.OptionsFormSpawner.form_fields = form_config.form_fields ``` ```python # slurm_config.py batch_script="""#!/bin/bash #SBATCH --partition={partition} #SBATCH --time={runtime} #SBATCH --ntasks-per-node={ntasks} #SBATCH --cpus-per-task={ncpus} #SBATCH --gpus={numgpus} #SBATCH --output={homedir}/.jupyterhub-slurmspawner.log #SBATCH --open-mode=append #SBATCH --job-name=JupyterLab #SBATCH --export={keepvars} source /etc/profile export MODULEPATH=$MODULEPATH:/modules/system module load R module load miniconda conda activate jupyter-3 {cmd} """ ``` ```python # form_config.py from optionsspawner.forms import ( FormField, TextInputField, NumericalInputField, CheckboxInputField, SelectField ) partition_select = SelectField('req_partition', label='Select a partition', attr_required=True, choices=[ ('cpu', "CPU"), ('gpu', "GPU"), ('cee_water_cjgleason', "CEE-Water (cjgleason)"), ('cee_water_casey', "CEE-Water (casey)"), ('cee_water_kandread', "CEE-Water (kandread)") ], default='cpu' ) runtime_input = TextInputField('req_runtime', label='Runtime (HH:MM:SS) - Your instance is killed at the end of runtime', attr_required=True, attr_value='02:00:00' #attr_pattern="[01]{1}[0-2]{1}:[0-5]{1}[0-9]{1}:[0-5]{1}[0-9]{1}" ) ntasks_input = NumericalInputField('req_ntasks', label="# Tasks per Node - For MPI Only, Leave at 1 if unsure", attr_required=True, attr_value=1, attr_min=1, attr_max=80 ) ncpus_input = NumericalInputField('req_ncpus', label='# Cores/Threads per Task', attr_required=True, attr_value=1, attr_min=1, attr_max=80 ) ngpus_input = NumericalInputField('req_numgpus', label='# GPUs - GPU Partitions Only', attr_required=True, attr_value=0 ) form_fields = [ partition_select, runtime_input, ntasks_input, ncpus_input, ngpus_input ] ```Logs
```text # JupyterHub Log [D 2021-01-13 15:58:42.111 JupyterHub pages:252] Triggering spawn with supplied form options for hsaplakoglu_umass_edu [D 2021-01-13 15:58:42.112 JupyterHub base:875] Initiating spawn for hsaplakoglu_umass_edu [D 2021-01-13 15:58:42.112 JupyterHub base:879] 0/100 concurrent spawns [D 2021-01-13 15:58:42.113 JupyterHub base:884] 0 active servers [D 2021-01-13 15:58:42.154 JupyterHub user:602] Calling Spawner.start for hsaplakoglu_umass_edu [I 2021-01-13 15:58:42.164 JupyterHub batchspawner:248] Spawner submitting job using sudo -E -u hsaplakoglu_umass_edu sbatch --parsable [I 2021-01-13 15:58:42.164 JupyterHub batchspawner:249] Spawner submitted script: #!/bin/bash #SBATCH --partition=cpu #SBATCH --time=02:00:00 #SBATCH --ntasks-per-node=1 #SBATCH --cpus-per-task=1 #SBATCH --gpus=0 #SBATCH --output=/home/hsaplakoglu_umass_edu/.jupyterhub-slurmspawner.log #SBATCH --open-mode=append #SBATCH --job-name=JupyterLab #SBATCH --export=PATH,CONDA_DEFAULT_ENV,LANG,JUPYTERHUB_API_TOKEN,JPY_API_TOKEN,JUPYTERHUB_CLIENT_ID,JUPYTERHUB_HOST,JUPYTERHUB_OAUTH_CALLBACK_URL,JUPYTERHUB_USER,JUPYTERHUB_SERVER_NAME,JUPYTERHUB_API_URL,JUPYTERHUB_ACTIVITY_URL,JUPYTERHUB_BASE_URL,USER,HOME,SHELL source /etc/profile export MODULEPATH=$MODULEPATH:/modules/system module load R module load miniconda conda activate jupyter-3 batchspawner-singleuser jupyterhub-singleuser --ip=0.0.0.0 --notebook-dir=~ [I 2021-01-13 15:58:42.220 JupyterHub batchspawner:252] Job submitted. cmd: sudo -E -u hsaplakoglu_umass_edu sbatch --parsable output: 1332952 [D 2021-01-13 15:58:42.221 JupyterHub batchspawner:274] Spawner querying job: sudo -E -u hsaplakoglu_umass_edu squeue -h -j 1332952 -o '%T %B' [D 2021-01-13 15:58:42.268 JupyterHub batchspawner:374] Job 1332952 still pending [D 2021-01-13 15:58:42.772 JupyterHub batchspawner:274] Spawner querying job: sudo -E -u hsaplakoglu_umass_edu squeue -h -j 1332952 -o '%T %B' [I 2021-01-13 15:58:43.118 JupyterHub log:181] 302 POST /panel/jhub/hub/spawn/hsaplakoglu_umass_edu -> /panel/jhub/hub/spawn-pending/hsaplakoglu_umass_edu (hsaplakoglu_umass_edu@::ffff:127.0.0.1) 1011.38ms [I 2021-01-13 15:58:43.181 JupyterHub pages:402] hsaplakoglu_umass_edu is pending spawn [I 2021-01-13 15:58:43.188 JupyterHub log:181] 200 GET /panel/jhub/hub/spawn-pending/hsaplakoglu_umass_edu (hsaplakoglu_umass_edu@::ffff:127.0.0.1) 21.05ms [D 2021-01-13 15:58:45.659 JupyterHub base:283] Recording first activity for