Multiple Python environments in one notebook

[zemlyansky]

Is it possible to have multiple Python environments working in the same notebook? Passing data between languages works nicely, but when I try to %get data in an environment cell it shows error:

Magic %get failed because the language module for env-tf1-python is not properly installed. Please install it according to language specific instructions on the Running SoS section of the SoS homepage and restart Jupyter server.

Having multiple environments could solve dependencies conflicts. Some libs require specific versions of other libs. In that way we could call them from their environment

upd: Installing sos-notebook in an environment doesn't help. Same error when calling %get

[BoPeng]

What exactly do you mean by "multiple Python environments". Currently SoS supports the use of multiple Python (or other kernels) instances by using the same kernel.

What you are asking seems to be using Python kernels from different environments (e.g. conda), which is supported but can be a bit tricky because you will need

1. Create a conda environment, and install a Python kernel under a different name (say p2). (python -m ipykernel install --name). Install the needed tools in that environment.
2. Switch back to the original environment, start SoS. You will see p2 displayed as a kernel, because SoS does not know what language this kernel is using.
3. Associate the kernel with a language using %use p2 --language Python3 and it will start the ipykernel from another environment.
4. Then you should be able to pass variables between the sessions.

However, because you are starting the kernel from another environment, without activating that environment, the kernel would execute correctly because the environment variables can be wrong.

I think SoS can improve this process by recognizing the language of p2 by checking the kernel spec. I will create a separate ticket for it. A related feature request could be starting kernels from docker images, which we had thought of but did not investigate further.

[zemlyansky]

Thank you, i meant virtual environments created with virtualenv or python3 -m venv. When I use the --language Python3 parameter and pass values to that kernel from Python3 it throws:

<ipython-input-8-fd83bdc29060> in <module>
----> 1 globals().update(pickle.loads(b'\x80\x03}q\x00X\x01\x00\x00\x00aq\x01K\x01s.'))

NameError: name 'pickle' is not defined

I just created a new virtual environment to reproduce that:

virtualenv env-test --python=python3
source env-test/bin/activate
python --version # -> Python 3.7.4
pip install sos-notebook sos-r sos-python
ipython kernel install --user --name=env-test
deactivate
jupyter kernelspec list # -> Available kernels: env-test ...

Then i created a new SoS notebook and repeated steps from the screenshot you shared:

What's interesting is that i can put variable from Python3 to R and then from R to env-test with no errors:

[BoPeng]

The pickle error should be a small bug that could be fixed easily. I am in ASHG meeting now but can have a look it and #259 tonight.

[zemlyansky]

Thank you, Bo! Being able to call isolated packages in their virtual environments would be an awesome feature!

[BoPeng]

Should be fixed. Please update to sos-notebook 0.20.6 and sos-python 0.18.3 and let me know if the problem has been fixed. Note that p2 should be automatically recognized as a Python3 kernel and there is no need to run %use p2 --language Python3.

[zemlyansky]

Awesome! Just updated and tested sending variables and Pandas' dataframes from Python3 kernel to env-test and back. Everything works, no errors. Thank you!