Move ebdpy helpers to esip-qhub Github

[rsignell-usgs]

@gzt5142 , I'd like to install the helpers on esip-qhub. How to proceed?

[gzt5142]

in the hytest_tutorial_notebooks repo (which is being deprecated, yes?)... there's a folder lib, and that stuff is in there. In the notebooks that I've been playing with, I just manipulate sys.path to include that folder -- I have not been adding to the standard environment via pip, conda, poetry, or similar....

I notice that you've got some other specialty libs in /usr/share/lib (or a path to that effect)... perhaps we can just put these alongside.

Let me re-bundle for this sort of side-load in a shared space, and I'll get you a link to a zip or something. I'm near the end of my day today... will prioritize for tomorrow.

[gzt5142]

@rsignell-usgs I put the HyTest helpers in /shared/users/lib, alongside the EBD code. Take a look to see if you like the way it was organized. I did it that way because /shared/users/lib has to be added to sys.path anyway for the EBD to work... so this makes it a one-time add in order to get all of the local helpers.

Look at this notebook for a quick demo of how one might use this: ''' https://jupyter.qhub.esipfed.org/hub/user-redirect/lab/tree/shared/users/gzt5142/HelperDemo.ipynb '''

Note that on configure_cluster(), it can accept the 'resource' string, much like it always did. But I added the functionality to have it make a reasonable guess if you don't specify. In this way, an unmodified notebook would run seamlessly on HPC, local, or on the QHub.

[rsignell-usgs]

@gzt5142 , could we add something to users startup file (e.g. ~/.bash_profile) that would allow us to skip this part?

 import sys
# adds the dir/folder to the system path search list
libDir = r'/shared/users/lib'
if libDir not in sys.path: #but only if it isn't already there...
    sys.path.append(libDir)

[gzt5142]

I think this will do it:

export PYTHONPATH="$PYTHONPATH:/shared/users/lib"

Let me test and verify.

[gzt5142]

Looks like the .bashrc is not user-editable. It is a symlink to a file established by the hub somehow. I'm weak on juypyterhub, so need to learn a bit more.

The more practical solution would actually be to put a symlink or redirect file in the site-packages folder within the pangeo python folder structure. I don't know if we get to edit files in those folders or not.

AND (I just learned this) -- the python environment in which the notebooks run is not spawned from the shell, so setting variables in .bashrc won't propagate. Must change the spawner (from which both the shell and the notebook server is inherited).

[gzt5142]

Looks like admin could do this -- not sure if you have that permission...

https://jupyterhub.readthedocs.io/en/stable/reference/config-reference.html

Looking for c.Spawner.environment and c.Spawner.env_keep

The way I read this, setting that option will set the PYTHONPATH for all shells and servers spawned.

[gzt5142]

https://github.com/jupyterhub/jupyterhub/issues/227

[gzt5142]

This feels like a larger question that we should answer for the HPC platform as well.

Do we want a library installed somewhere on the file system? if so, where?

One other option for helpers that we talked about a while back was a common 'setup' notebook to source into other notebooks via a %run magic. See #25

[gzt5142]

The take-away that I got from our meeting yesterday is that we'd like to use jupyter 'magics' rather than create formal packages for these common/helper bits of code. I will refactor the cluster helper on ESIP/QHUB to see how that works compared with the module approach we had been experimenting with.

[gzt5142]

I've got a mock-up of how the magic methods would be used on esip/qhub

https://jupyter.qhub.esipfed.org/hub/user-redirect/lab/tree/shared/users/gzt5142/ClustDemo.ipynb

Looking for feedback to see if that is the right direction or if I should explore other options.

[rsignell-usgs]

@gzt5142 , I think these two options are great! We can use the "run" option for tutorials, and the "load" option for custom workflows!