jupyter/nbconvert/nbformat script setup

[dhomeier]

I have just submitted PR #391 as first of a two-part attempt to replace the stale update in #164 to the IPython + Notebook system. The actual jupyter-*, nb* and ipython-py and notebook-py updates themselves with their mess of circular dependencies are to follow in a second PR. In the course of this I have tried to understand the function of the various jupyter packages and scripts a bit better and found the current setup rather complicated and possibly confusing.

Very briefly, jupyter-core-pyXX provides the script jupyter[XX], which acts as a launcher or (thin?) wrapper for the various commands provided by jupyter-client-pyXX, nbformat-pyXX, nbconvert-pyXX and notebook-pyXX. It can detect and launch any commands in $PATH under the name of jupyter*. E.g. with the current py27 and py35 versions of the packages installed, one would find

 jupyter2.7 -h                                                                                                                         
usage: jupyter2.7 [-h] [--version] [--config-dir] [--data-dir] [--runtime-dir]
...
Available subcommands: bundlerextension kernel kernelspec kernelspec2.7
kernelspec3.5 migrate migrate2.7 migrate3.5 nbconvert2.7 nbconvert3.5
nbextension run serverextension trust trust2.7 trust3.5

> jupyter2.7 kernel -h                                                                                                                  
jupyter: 'kernel' is not a Jupyter command

Here things begin becoming messed up, since some of the scripts are not managed under update-alternatives (and some currently are not working anyway). But even where they are, calling a versioned jupyter launcher with a command will execute the highest installed version of that command script, e.g. in the above example jupyter2.7 nbconvert would launch the jupyter-nbconvert3.5 script (under python3.5), since the underlying command.py is calling

    command = 'jupyter-' + subcommand
    try:
        _execvp(command, sys.argv[1:])
    except OSError as e:
        sys.exit("Error executing Jupyter command %r: %s" % (subcommand, e))

Maybe not a critical issue, since apparently all the subcommand scripts can be called directly, but I'd at deem it at least confusing.

My plan for the new update is to put all scripts under update-alternatives administration and also standardise their names to the -pyNN scheme. This would still leave the above behaviour of launching the highest installed version (unless called explicitly like jupyter-py27 kernel-py27), but it would require moderate patches to command.py to make jupyter see and execute exclusively the commands of its own Python version (so jupyter-py27 kernel would always launch a python2.7 kernel). Thoughts?

[schwehr]

I ran screaming from this mess. I think if you can make it work at all, that's good enough for now. Part of the solution is aim for removing the py27 variants as soon as possible. Crossing py2 and py3 is a great way for things to implode. Patching so that it gets the correct version is technically the best thing to do, but it's not going to be fun.

🧨💥😱

[dhomeier]

Thanks for the feedback! Getting the versions straightened out turned out to require only a few patches, so I moved forward with that option. py27 is included in #413 with the LTS version 5.x, but I've removed all py34 variants (there is another LTS branch ipython 6.x for that, but it hopefully should not be needed by anyone).