jupyterhub / batchspawner

Custom Spawner for Jupyterhub to start servers in batch scheduled systems
BSD 3-Clause "New" or "Revised" License
187 stars 132 forks source link

Add req_condaenv to BatchSpawnerBase #225

Closed TerenceLiu98 closed 1 year ago

TerenceLiu98 commented 3 years ago

Specify the conda environment or python environment with req_condaenv in BatchSpawnerBase

welcome[bot] commented 3 years ago

Thanks for submitting your first pull request! You are awesome! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please make sure you followed the pull request template, as this will help us review your contribution more quickly. welcome You can meet the other Jovyans by joining our Discourse forum. There is also a intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

rcthomas commented 3 years ago

Hi @TerenceLiu98 can you give an example of how this is used?

TerenceLiu98 commented 3 years ago

@rcthomas Here is am example:

c.SlurmSpawner.batch_script = '''#!/bin/bash
#SBATCH --chdir={homedir}
#SBATCH --output={homedir}/jlab_slurmspawner_%j.log
#SBATCH --partition={partition}
#SBATCH --time={runtime}
#SBATCH --cpus-per-task={nprocs}
#SBATCH --mem={memory}
#SBATCH --gres={gres}
#SBATCH --job-name=jlab-spawner

module anaconda/5.3.1
conda activate {condaenv}
{cmd}
'''

c.ProfilesSpawner.profiles = [
     ('GPU Node with PyTorch-1.9.0', 'GPU-Node #1', 'batchspawner.SlurmSpawner', dict(req_partition='GPU-Node', req_nprocs='4', req_runtime='01:00:00', req_memory='8192M', req_gres="gpu:teslaV100:1", req_condaenv="PyTorch")),
     ('GPU Node with Tensorflow 2.5.0', 'GPU-Node #2', 'batchspawner.SlurmSpawner', dict(req_partition='GPU-Node', req_nprocs='4', req_runtime='01:00:00', req_memory='8192M', req_gres="gpu:teslaV100:1", req_condaenv="Tensorflow-GPU")),
]

This is suitable for the public python environment. Each environment needs jupyter & batchspawner

TerenceLiu98 commented 3 years ago

Actually, I am now facing #217 and I don't know what to do :( But I am sure it is not caused by req_condaenv(solved

mbmilligan commented 1 year ago

Hi, I am going to close this old issue. For the benefit of future people searching for this, I would note, there is a req_prologue variable already:

req_prologue = Unicode(
    "",
    help="Script to run before single user server starts.",
).tag(config=True)

You could activate a conda or venv environment by inserting the appropriate command there.