search

vsoch / forward

Port Forwarding Utility
https://vsoch.github.io/lessons/sherlock-singularity/
MIT License
52 stars 27 forks source link

adding tensorflow and singularity examples #12

Closed vsoch closed 6 years ago

vsoch commented 6 years ago

hey @raphtown ! I just worked on this (slightly modified) example that will load up the same python 2 or 3 notebook, but also with tensorflow! I have a matching tutorial to post along with it, to give a bit more context. --> https://vsoch.github.io/lessons/jupyter-tensorflow/

Also, I think I just got a wicked idea - it's pretty annoying to need to load sherlock modules (then half the time something still doesn't work) so how cool would it be to just provide things in containers? And then the user can pick / share containers with things ready to go? There is also a thing called repo2docker that would make it totally seamless to go from a jupyter notebook, to docker container, to singularity container (for sherlock). So - after this post, how about I'll give a go at "the container version?" I'd want to also include the repo2docker as part of the tutorial, so these two guys would need to be merged first:

vsoch commented 6 years ago

okay I just added an example to do the same with a Singularity container! Here is the corresponding tutorial:

https://vsoch.github.io/lessons/sherlock-singularity/

The container version is much nicer because dependencies that the user might want can be totally packaged here (although the container I provided is pretty simple :O) ) and the user doesn't have to generate a token in advance (the container generates and provides it):

image

I'm working on a bug now wrt permissions of writing new files - it might need to warrant a change to the singularity container build. I'll change the title of the post (removing the WIP) when this is done and the PR is ready for review.

vsoch commented 6 years ago

okay I figured out the issue! And we have writable notebooks!

image

The issue was that the default repo2docker container sets the user home to be /home/username, which is fine for Docker (we can write everywhere) but not for Singularity. So the fix was to enforce the user's home being... the user's home for the container at runtime.

I also added a bit to the end of start.sh to show the user how to get logs to their screen, as I noticed sometimes that the notebook doesn't output its token to the terminal, and the attempt to cat the logs doesn't catch it either. The entire execution workflow now looks like this:

vanessa@vanessa-ThinkPad-T460s:~/Documents/Dropbox/Code/srcc/forward$ bash start.sh singularity-jupyter
== Checking for previous notebook ==
No existing singularity-jupyter jobs found, continuing...
== Getting destination directory ==
== Uploading sbatch script ==
singularity-jupyter.sbatch                                                                          100%  898     0.9KB/s   00:00    
== Requesting GPU ==
== Submitting sbatch ==
sherlock sbatch --job-name=singularity-jupyter --partition=gpu --gres gpu:1 --output=/home/users/vsochat/forward-util/singularity-jupyter.out --error=/home/users/vsochat/forward-util/singularity-jupyter.err --mem=20G --time=8:00:00 /home/users/vsochat/forward-util/singularity-jupyter.sbatch 56143 ""
Submitted batch job 22399485
== Waiting for job to start, using exponential backoff ==
Attempt 0: not ready yet... retrying in 1..
Attempt 1: not ready yet... retrying in 2..
Attempt 2: not ready yet... retrying in 4..
Attempt 3: not ready yet... retrying in 8..
Attempt 4: not ready yet... retrying in 16..
Attempt 5: not ready yet... retrying in 32..
Attempt 6: not ready yet... retrying in 64..
Attempt 7: resources allocated to sh-112-05!..
sh-112-05
sh-112-05
notebook running on sh-112-05
== Setting up port forwarding ==
ssh -L 56143:localhost:56143 sherlock ssh -L 56143:localhost:56143 -N sh-112-05 &
== Connecting to notebook ==
== View Logs Like This ==
ssh sherlock cat /home/users/vsochat/forward-util/singularity-jupyter.out
ssh sherlock cat /home/users/vsochat/forward-util/singularity-jupyter.err
Container is /scratch/users/vsochat/share/repo2docker.simg
Notebook directory is /scratch/users/vsochat
[I 10:49:19.524 NotebookApp] Writing notebook server cookie secret to /tmp/jupyter/notebook_cookie_secret
/srv/venv/lib/python3.6/site-packages/IPython/paths.py:68: UserWarning: IPython parent '/home/username' is not a writable location, using a temp directory.
  " using a temp directory.".format(parent))
[I 10:49:20.123 NotebookApp] JupyterLab alpha preview extension loaded from /srv/venv/lib/python3.6/site-packages/jupyterlab
[I 10:49:20.123 NotebookApp] JupyterLab application directory is /srv/venv/share/jupyter/lab
[I 10:49:20.129 NotebookApp] nteract extension loaded from /srv/venv/lib/python3.6/site-packages/nteract_on_jupyter
[I 10:49:20.133 NotebookApp] Serving notebooks from local directory: /scratch/users/vsochat
[I 10:49:20.133 NotebookApp] 0 active kernels
[I 10:49:20.133 NotebookApp] The Jupyter Notebook is running at:
[I 10:49:20.133 NotebookApp] http://0.0.0.0:56143/?token=aaaaaaaaaaabbbbbbbbbbbbbbccccccccccc
[I 10:49:20.133 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 10:49:20.133 NotebookApp] 

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://0.0.0.0:56143/?token=aaaaaaaaaabbbbbbbbbbbbbbcccccccccc
Open your browser to http://localhost:56143

I'm pretty excited about this :) Let me know when you've had a look - I'd love to share the post soon!! If people want containers with other setups ready to go, we can get feedback on this and I can provide them in my shared folder.

vsoch commented 6 years ago

Heyo! Do you have an ETA of when you can look this over? I'd like to start on a set of containers with scientific packages ready to go, and want to share this and get feedback from users first. Wod be greatly appreciated thank you discuss soon! If you would like help to manage the code and push it's early development, or don't want me to PR here, I am ok working just on my fork but I'd much rather do it here with you!

raphtown commented 6 years ago

Hey @vsoch! I am going to take a look at this tomorrow (Monday).

Cheers.

vsoch commented 6 years ago

Awesome!

raphtown commented 6 years ago

Ok these all look great! Merging!

vsoch commented 6 years ago

woohoo awesome!

I'm making the little registry of containers now (so other users can find/share and use forward) I'll ping you when I have something to play with.