[BUG] Issue with spawn timeouts after local pip installation of python packages

[brantr]

After pip installing python 3.8 packages into .local, I could not spawn further instances without clearing .local first.

[frossie]

[I asked the user to file a stub ticket during in-person troubleshooting, blame me]

We support users being able to add additional packages to their notebook environment. In what is generally a rare case, the user can do something that prevents their notebook server from starting (a situation from which you can immediately recover my ticket the "Clear my .local" option when launching a new server.

We are investigating whether recent changes on how the Science Pipelines python environment is managed have made this more likely to be a problem, and also what specific package caused the issue @brantr encountered.

[athornton]

Working to reproduce the error.

[athornton]

Using recommended (currently weekly 33), I was able to do a pip install tensorflow and spawn a new container, in which I can run import tensorflow inside a notebook without an issue.

That said, it is certainly the case, particularly as the image you're using gets older relative to the current state of the world, that if you install something into the system context (which is what pip install --user effectively does, without a virtualenv), that there are going to be incompatible versions of things.

The much safer way to go about this is to create a virtualenv, and then create a kernel for that virtualenv. To use tensorflow as an example:

python3 -m venv ~/Venvs/TF
source ~/Venvs/TF/bin/activate
pip install jupyter tensorflow
ipython kernel install --name TF --user

(You need to install jupyter to be able to create a kernel from your new virtualenv.)

Then you have to shut down your pod (from the bottom of the File menu) and respawn.

Upon respawn, your new kernel will be available for use within Jupyterlab notebooks and consoles. If you activate that virtualenv, you can pip install whatever you like into it.

A similar process works for conda, with the caveat that since the root filesystem is not writeable, if you do a conda env clone and start with the lsst-scipipe-env environment as your clone source, it's going to take up a lot of space in your home directory, since what you're getting is a copy of the entire DM stack.

[brantr]

Hi @athornton, thanks very much! I had tried most of your solution but was apparently missing the last step. I can now access my software stack within a notebook after pip installing the required packages and activating the virtual environment from within the notebook. I'll mark this as closed, thanks again.