Use custom environment

[zonca]

How do I use my own custom conda environment with the reverse proxy?

[marypthomas]

Andrea - Good question. We recommend loading modules in the batch script, so I expect you would also load the conda environment as well. We have not tried this….

On Jul 27, 2020, at 4:20 PM, Andrea Zonca notifications@github.com wrote:

How do I use my own custom conda environment with the reverse proxy?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/sdsc-hpc-training-org/reverse-proxy/issues/4, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADMDVOBGZZHX5TZSW4M4X23R5YDUXANCNFSM4PJUJO2A.

[zonca]

My environment is at:

. /share/apps/compute/si2020/miniconda3/etc/profile.d/conda.sh 
conda activate python-hpc

and jupyter at:

/share/apps/compute/si2020/miniconda3/envs/python-hpc/bin/jupyter

I tried editing batch/batch_jupyterlab.sh both adding the conda activation on top (I was getting command not found in the slurm output log) and using the full jupyter path (I get no slurm output log at all) but neither worked. In both cases I get "Pending" in the proxy interface.

Can someone familiar with the system debug this and let me know how to proceed?

This is my standard launch script works fine: https://github.com/sdsc/sdsc-summer-institute-2020/blob/master/2.1a_python_hpc/notebook.slrm

[mahidhar]

Mary, Andrea,

I already tried the batch script route for the conda installed R example I did for Paul and Mai. It works fine. Two things to keep in mind:

(a) Before you run "start_notebook" you should be in your conda environment. The start_notebook script looks for jupyter so it should be in your path at submit time.

(b) In the batch script (you can copy reverse-proxy/batch/batch_notebook.sh) I added the conda environment set up before the "# DO NOT EDIT BELOW THIS LINE" line.

With (a) and (b) I had no problems running via the reverse proxy.

Mahidhar

p.s: Mary: The function in the script should be moved after the #SBATCH lines, otherwise SLURM will ignore the #SBATCH directives.

[mahidhar]

My environment is at:

. /share/apps/compute/si2020/miniconda3/etc/profile.d/conda.sh 
conda activate python-hpc

and jupyter at:

/share/apps/compute/si2020/miniconda3/envs/python-hpc/bin/jupyter

I tried editing batch/batch_jupyterlab.sh both adding the conda activation on top (I was getting command not found in the slurm output log) and using the full jupyter path (I get no slurm output log at all) but neither worked. In both cases I get "Pending" in the proxy interface.

Can someone familiar with the system debug this and let me know how to proceed?

This is my standard launch script works fine: https://github.com/sdsc/sdsc-summer-institute-2020/blob/master/2.1a_python_hpc/notebook.slrm

Can you check to see if the job is still pending on the Comet side? If so, getting the pending message is fine. When the job runs the link will map to the right node.

[zonca]

yes, the job was running. can you please send me the scripts you used? I am quite sure I did the same and it didn't work for me.

[mahidhar]

I am trying the jupyterlab one right now and seeing the same issue. I had tried the notebook one for my test. I will look into this a bit more.

[zonca]

also, I think we should have a configuration option to the script to specify the executable for jupyter, this way I do not have to maintain my own copy of batch_jupyterlab.sh

Also it would be nice to have start_notebook installed systemwide, or at least in /share/apps/compute/si2020/miniconda3/condabin (common path to all envs), so all users can call it without having to checkout a copy of the reverse-proxy repository.

[mahidhar]

Yes, I was planning to put the reverse-proxy files in everyone's path and also provide the sample script with the SI2020 conda install path in it.

[zonca]

ok, I'll test them when they are ready, thanks

[mkandes]

@mahidhar @JamesMcDougallJr - Were you all able to get jupyter lab running? I had no trouble myself once you change 'jupyter notebook' to 'jupyter lab'. But yes, you do have to also add the environment changes for the SI2020 conda installation and loading the python-hpc environment too.

(python-hpc) [mkandes@comet-ln2 batch]$ diff batch_lab.sh batch_notebook.sh 
32,34d31
< . /share/apps/compute/si2020/miniconda3/etc/profile.d/conda.sh
< conda activate python-hpc
43c40
< jupyter lab --ip $IP --config $config --no-browser &
---
> jupyter notebook --ip $IP --config $config --no-browser &

[mahidhar]

Yes I got it working fine with James' development version. I will send the info to Andrea directly. James: You also have to change the order in the script. The #SBATCH commands must come before any other commands. Otherwise SLURM will just ignore the #SBATCH lines as comments.

[marypthomas]

All - These changes look great. We’ll sync up and review at the weekly notebook meeting this Friday (10am). Thanks, Mary

On Jul 27, 2020, at 9:08 PM, mahidhar notifications@github.com wrote:

Yes I got it working fine with James' development version. I will send the info to Andrea directly. James: You also have to change the order in the script. The #SBATCH commands must come before any other commands. Otherwise SLURM will just ignore the #SBATCH lines as comments.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/sdsc-hpc-training-org/reverse-proxy/issues/4#issuecomment-664764960, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADMDVOH6BNVJ2OP6AY5FGR3R5ZFNJANCNFSM4PJUJO2A.

[JamesMcDougallJr]

Closing as this was successfully done in the Summer Institute.