Proxy notebook

[OliverEvans96]

We would like to run a notebook server on a login and launch a notebook which is connected to a kernel on a compute node.

[OliverEvans96]

Option 1: IPyParallel

By following http://ipyparallel.readthedocs.io/en/latest/process.html#starting-the-controller-and-engines-on-different-hosts we can do the following:

1. Start a controller on login

2. Copy the json config to compute

3. Start the engine on compute, specifying the json config. The engine and controller will communicate & register.

4. Open a notebook on login.

5. Run:

   import ipyparallel as ipp
   rc = ipp.Client()
   %autopx

   (ipp magics docs)

6. Any following commands will be run on compute.

Advantages:

• We can do it

Disadvantages:

• login must have an ip address visible to compute. This is fine on a cluster, but if login is a laptop behind a router, then we'll have to be more creative.
• Since the kernel is remote, there are a few jupyter artifacts
  • Out[0:{num}]: will be prepended to every output (where {num} is the output number on compute)
  • No tab completion since the kernel running on login has no variables defined or modules imported.

[OliverEvans96]

@shreddd said that @rcthomas has been working on spawning remote kernels for notebooks, which would be an excellent solution here depending on how far along the idea is.

[mlhenderson]

Not sure yet if this is compatible with existing NERSC constraints, but worth checking on: https://bitbucket.org/tdaff/remote_ikernel

[OliverEvans96]

Great find! This looks like basically the idea we're going for.