Customise the location where the kernelspec is installed (and get conda to use custom locations)

[krinsman]

Note:

This is more of a feature request than an actual issue.

Idea:

If one has installed nodejs inside of a virtual environment, ijsinstall should have an option to only install the kernel specification for ijavascript inside of the virtual environment.

(Alternatively, if it does already have this option, then could you help me to figure out which option it is? I couldn't figure out where it is in the documentation.)

Possible explanation for the behavior:

According to here, it seems that the reason why this does not currently happen is that ijsinstall installs the kernel specification "locally" (which is in this case the User's Jupyter kernelspec folder, which applies to all of the user's virtual environments, not just the one where nodejs and ijs were installed), and not "globally" (which in this case would have only been the installation location of npm, which is inside of the user's virtual environment, and not accessible from outside).

Steps to reproduce:

Let's say one has a completely brand new Miniconda installation with no packages yet installed. Then they make two conda environments, install jupyter in each, but only install npm in the first conda environment.

conda create --name ijavascript
conda create --name noijavascript

conda activate noijavascript
conda install jupyter
conda deactivate

conda activate ijavascript
conda install jupyter
conda install nodejs
conda deactivate

Doing the following does not lead to any output from the terminal:

conda activate noijavascript
which npm
conda deactivate

But when one does the following, one confirms that npm is working inside of the other environment. Good:

conda activate ijavascript
which npm
conda deactivate

The problem does not seem to be with npm -- it does seem to install packages only within its installation directory (at least when one uses the -g flag, which is what the ijavascript documentation thankfully recommends that one do), which in this case is the virtual environment.

Specifically, one can do:

conda activate ijavascript
npm install -g ijavascript
which ijs
which ijsinstall
conda deactivate

conda activate noijavascript
which ijs
which ijsinstall
conda deactivate

which ijs
which ijsinstall

The first block shows ijavascript is installed inside of the ijavascript virtual environment, and the second block shows ijavascript is not installed/not accessible from within the noijavascript virtual environment. The third code block shows that ijavascript is also not accessible outside of any virtual environment (in the "base" virtual environment).

However, if one does (INSTALL 1)

conda activate ijavascript
ijsinstall
conda deactivate

conda activate noijavascript
jupyter notebook

One will see the ijavascript kernel as an option, even though we are in the noijavascript virtual environment. This is (seemingly) not desired behavior, and occurs because by default ijsinstall installs the kernel spec. "locally", which then applies to all of the user's virtual environments, even those which aren't supposed to be affected.

Even though using npm's "global" install option (-g) allows one to avoid the analogous problem with ijs and ijsinstall, the analogous flag for ijsinstall does not allow one to avoid this problem.

Specifically, the default behavior, corresponding to (INSTALL 1) above, is for the ijavascript kernel specification to be installed at (on Mac OS X):

/Users/${USER}/Library/Jupyter/kernels

but if one does instead (INSTALL 2):

conda activate ijavascript
ijsinstall --install=global
conda deactivate

conda activate
jupyter notebook

one will still have the same problem, because in this case ijsinstall puts (on Mac OS X) the kernel specification in:

/usr/local/share/jupyter/kernels

so that the ijavascript kernel will be visible in all of the virtual environments of all users, not just all of the virtual environments of one user. (This StackOverflow thread was helpful for me in beginning to understand this issue.)

Desired Behavior:

The kernel specification would be installed in:

<conda installation directory>/envs/ijavascript

or something similar to this, which allows the ijavascript kernel to be visible ony in that virtual environment, but not others which aren't supposed to have access to ijavascript.

[n-riesco]

The location of the kernelspec is determined by Jupyter. You can get the current location by running jupyter --data-dir.

I guess both conda environments, ijavascript and noijavascript, use the same folder to store kernelspecs. You can confirm this by running:

conda activate ijavascript
jupyter --data-dir
conda deactivate

conda activate noijavascript
jupyter --data-dir
conda deactivate

Jupyter supports the environment variable JUPYTER_DATA_DIR to customise the folder where kernelspecs are stored. For example, I can do this:

$ jupyter --data-dir
/home/user/.local/share/jupyter

$ JUPYTER_DATA_DIR=/home/user/.local/share/ijs jupyter --data-dir
/home/user/.local/share/ijs

In your case, I'd configure the conda environments, ijavascript and noijavascript, so that they set JUPYTER_DATA_DIR with different locations. See here conda's documentation to do so.

Please, let me know if this works for you.

[krinsman]

@n-riesco Sorry for the extremely slow reply. You're right, this does seem to entirely be a problem on the Jupyter, maybe conda, side of things, since after installing Jupyter in a new conda environment, the result of jupyter --data-dir isn't a directory specific to the conda environment in which it is installed.

In particular, your suggestion works perfectly at least for me. I'll outline what I did step-by-step for clarity as well as reproducibility for anyone in the future with a similar issue.

conda create --name ijavascript
conda activate ijavascript
conda install nodejs
conda install jupyter
cd <ijavascript environment location>
mkdir ./Jupyter
mkdir -p ./etc/conda/activate.d
mkdir -p ./etc/conda/deactivate.d
touch ./etc/conda/activate.d/env_vars.sh
touch ./etc/conda/deactivate.d/env_vars.sh

Then I used a text editor to make the contents of <ijavascript environment location>/etc/conda/activate.d/env_vars.sh to be the following:

#!/bin/bash
export JUPYTER_DATA_DIR=<ijavascript environment location>/Jupyter

and then I used a text editor to make the contents of <ijavascript environment location>/etc/conda/deactivate.d/env_vars.sh to be the following:

#!/bin/bash
unset JUPYTER_DATA_DIR

Finally, I did (I wanted to exit and then re-enter the environment to make sure that the changes took effect and that the script env_vars.sh was actually called):

conda deactivate
conda activate ijavascript
npm install -g ijavascript
ijsinstall

(I apologize in advance if I made any errors in transcribing my workflow here.) (Also I did #!/bin/bash because my shell is BASH, but I guess if one's shell is SH then one should start the shell scripts the way that the conda documentation suggests, #!/bin/sh.)

This was a little annoying to do, but: (1) it works perfectly, and (2) makes fairly clear that, to the extent that this is an issue, it is an issue with either conda or Jupyter's defaults.

In any case, if anyone is knowledgeable enough about virtual environments to even notice this as being an issue, they probably have the skills to implement the above solution. It is inconvenient, but definitely possible to work around.

Thank you @n-riesco for clarifying and solving this issue for me! I really appreciate it.

[krinsman]

@n-riesco Do you think this is worth the time raising as an issue with Jupyter (or with conda)? It seems relatively unimportant, although I thought Jupyter and conda are related projects and like to play nicely with each other, and an automatic implementation of the above would be a nice feature. On the other hand, I am not sure if it would actually be a quick fix for them to implement.

[n-riesco]

@krinsman Thank you for the thorough explanation. I think this is going to be very useful to other conda users. I'd really appreciate a contribution to update https://github.com/n-riesco/ijavascript/blob/master/doc/install.md and add a section for conda with the instructions above.

[n-riesco]

@krinsman

Do you think this is worth the time raising as an issue with Jupyter (or with conda)?

I reckon conda should address this need (especially considering that Jupyter already provides JUPYTER_DATA_DIR for this purpose).

[krinsman]

@n-riesco Sorry about the slow response -- yes I will try to do that ASAP.

Update: Here it is: https://github.com/n-riesco/ijavascript/pull/160