How to specify path to jupyter kernels when in different directory than jupyter itself

[abalter]

Cross-posted here.

We are running jupyter hub installed locally on a server serverA. This server has web access. We would like the notebooks to use kernels located in a conda environment on a different server, serverB, that does not have web accesss. However, it is where we keep our code base.

The conda path on serverB with the kernels is mounted as an NFS share on serverA.

While we keep our code on serverB, we would like to execute the actual computation on serverA, where jupyter is installed.

            browser                
               /\
               ||
               ||  web access
               ||
               \/
            serverA           |              serverB
-------------------------------------------------------------------
                              |
      opt/conda/bin/jupyter   |
                /\            |
                ||            |
    computation ||            |
                ||            |
                \/            |
            <kernel> <------------------ .../conda/bin/ <kernels>
                         NFS Mount
                              |
                              |
------------------------------------------------------------------- 

Our sysadmins have determined that it would be unstable to run the jupyter web service over a mapped drive. Therefore we need to run the jupyter service on serverA but use the kernels on serverB.

EDIT There are very good instructions for accessing a kernel over ssh. We could use this approach, but this seems unnecessary in our case because the drive to the executables is actually mapped.

[takluyver]

I'm not sure what you mean - these two statements seem incompatible:

While we keep our code on serverB, we would like to execute the actual computation on serverA, ... Therefore we need to run the jupyter service on serverA but use the kernels on serverB.

The computation is done in the kernels. If your kernels are on B, the computation happens on B.

[abalter]

When I run python on serverA, the python executable in the NFS Mount serverB:/pathB/condaB/bin loads into the memory on serverA and runs there. If I type "which python" on serverA, it shows me /pathB/condaB/bin/python. If I run python on serverA and look at top, I see the process running on serverA. However, the binary executable is from serverB.

For stability, we need to run jupyter from serverA:/pathA/condaA/bin to be accessible from https://serverB/jupyter. However, if I open a notebook and type "!which python", I need to see "/pathB/condaB/bin/python", not "/pathA/condaA/bin/python."

The reason for this is that serverB:/pathB/condaB/ is on the drive attached to our cluster. This is where we keep all of our software packages. serverA is a machine for development and interactive processing for jobs that may later be run on the cluster. So, they need to be developed with the exact same binaries as we would run on the cluster.

Also, ports 80 and 443 are closed on the cluster. However those ports are open on serverA in order to run jupyter and other apps.

We could have a process to regularly match up condaA with condaB. But it would be much simpler to have jupyter simply call the binary in /pathB/condaB/.

Sorry for the lengthy explanation. I just want to make sure I'm explaining the situation well enough.

EDIT Also, the right-to-left arrow above "NFS Mount" is to indicate the binary on serverB is being loaded into serverA.

EDIT 2 When I first wrote this, I completely switched A and B. I think it is correct now.

[blink1073]

Hi @abalter, making a file like the one below should allow the notebook to run your remote kernel.

cat /usr/local/share/jupyter/kernels/serverb_python/kernel.json
{"argv": ["/pathB/condaB/bin/python", "-m", "ipykernel", "-f", "{connection_file}"], "display_name": "Python (Remote)", "language": "python", "mimetype": "text/x-python", "name": "conda_remote"}

You can test it on the command line as jupyter console --kernel=conda_remote.

[abalter]

I will definitely give that a try. Am I right that I would need to add sections for all the kernels I use (R, bash, ...)?

[blink1073]

Yes, that is correct.

[steffenheyne]

There is a package called "remote_ikernel" to make remote kernel configuration easier. We use this a lot and it works great. https://bitbucket.org/tdaff/remote_ikernel