prompt_toolkit breaks redirected input

[dsblank]

I believe that prompt_toolkit has broken this use of jupyter-client:

$ jupyter console --kernel metakernel_python < generate_help.py
Traceback (most recent call last):
  File "/usr/local/bin/jupyter-console", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.4/dist-packages/jupyter_core/application.py", line 267, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/traitlets/config/application.py", line 595, in launch_instance
    app.initialize(argv)
  File "<decorator-gen-113>", line 2, in initialize
  File "/usr/local/lib/python3.4/dist-packages/traitlets/config/application.py", line 74, in catch_config_error
    return method(app, *args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/jupyter_console/app.py", line 137, in initialize
    self.init_shell()
  File "/usr/local/lib/python3.4/dist-packages/jupyter_console/app.py", line 110, in init_shell
    client=self.kernel_client,
  File "/usr/local/lib/python3.4/dist-packages/traitlets/config/configurable.py", line 412, in instance
    inst = cls(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/jupyter_console/ptshell.py", line 252, in __init__
    self.init_prompt_toolkit_cli()
  File "/usr/local/lib/python3.4/dist-packages/jupyter_console/ptshell.py", line 404, in init_prompt_toolkit_cli
    self.pt_cli = CommandLineInterface(app, eventloop=self._eventloop)
  File "/usr/local/lib/python3.4/dist-packages/prompt_toolkit/interface.py", line 81, in __init__
    self.input = input or StdinInput(sys.stdin)
  File "/usr/local/lib/python3.4/dist-packages/prompt_toolkit/input.py", line 67, in __init__
    assert self.stdin.isatty()
AssertionError

[takluyver]

You mean jupyter console, right? That's a separate package from jupyter_client. I don't think piping text into jupyter console was ever something we deliberately supported, but I believe it used to work and no longer does.

It should be straightforward to write a little program using jupyter_client that takes a filename and a kernel name, starts the desired kernel, and runs the given file in it.

[dsblank]

Yes, jupyter console as shown. Regardless of whether it was intentional, it used to work and now doesn't for any kernel, including ipython consoles. As you note, it should be straightforward to add a little code to jupyter console to handle this for all kernels. Where should this code go? Should it test for self.stdin.isatty() and if not True, attempt to read from it, process it, and exit?

[dsblank]

Alternatively (or in addition to), we could add a -c "CODE" flag (like python and ipython have). Although in my usecase, redirect would be easier.

[takluyver]

I think I'd do it as a separate command, not part of jupyter console. It should be simple to build on top of jupyter_client.

[takluyver]

e.g. it might be something like jupyter run --kernel foo generate_help.py

[dsblank]

I see that there is a PipeInput class in prompt_toolkit/input.py that doesn't appear to be used by any program. Was the intention for that for input from a pipe? A run subcommand would be fine.

[dsblank]

Where should the code for a jupyter-run subcommand live?

• its own repo
• with jupyter_client
• with jupyter_console

I guess it will build on jupyter_console since it has a subset of the same flags (--kernel, --existing, --ip, etc)... everything but the shell options. But jupyter_console is setup to be a single use repo. jupyter_client makes sense, but then I would have thought jupyter_console would be in jupyter_client too.

[minrk]

I think putting it in jupyter_client makes sense. I think we should have a lot more complete utilities for running code in jupyter_client (#176). jupyter_console is a peer of qtconsole, in that it is a specific application built on top of the protocol, whereas jupyter_client is just the lower-level tools for building such applications. run is drastically simpler than a full terminal client, since it's really just a single execute call.

I don't think it does need to inherit flags from jupyter_console. --kernel and --existing make sense, but I think that's close to the extent of it, and all of the logic for those arguments is implemented in jupyter_client, not jupyter_console. I don't think things like --ip should be in jupyter run, though.

[dsblank]

Sounds like a plan! I'll work on a PR.

[dsblank]

Here is an initial stab:

https://github.com/dsblank/jupyter_client/commit/08d6c300dbd08f0ef14ea1bb62608c84ee2504a4

I used ZMQTerminalInteractiveShell which doesn't sound like the right class, but seemed to be the class with all of the communication methods. Feedback welcomed!

[takluyver]

jupyter_client shouldn't import jupyter_console, because that is a circular dependency. You probably want jupyter_client.BlockingKernelClient.

[dsblank]

With a couple more lines, this could run notebooks too. Perhaps a simpler method than using nbconvert for just executing cells. Would allow you to run a notebook's cells with a different kernel.

[takluyver]

I'd rather not. jupyter_client currently does not know anything about notebook documents, and I'd prefer to keep it that way. I know it's a bit awkward that nbconvert is the officially supported way to run a notebook at the command line, but this isn't the right place for it either.

[dsblank]

It looks like JupyterConsoleApp already wraps up BlockingKernelClient. But what is missing are all of the methods that ZMQTerminalInteractiveShell implements, such as the handling of the messages. Should a class exist that contains those methods for use by ZMQTerminalInteractiveShell, and this non-interactive shell use? Or should I subclass ZMQTerminalInteractiveShell? Or am I missing something?

[takluyver]

A lot of the functionality won't be needed (like handling complete_reply). There will probably be some duplicated functionality, but let's get the code written before trying to abstract pieces out. Premature deduplication is just as bad as premature optimisation.

For another relatively simple consumer of jupyter_client, see nbconvert's execute preprocessor.

[dsblank]

Ok, I'll take a look at duplicating methods. Thanks; will get back to this soon.

[minrk]

What prompted me to open #176 was reimplementing this yet again in dask.distributed (https://github.com/dask/distributed/pull/370/files), so I really think we should have an implementation of the full execute/wait/output display in jupyter_client, either as functions, or as a new method on KernelClients. It's quite small. What I think is in scope:

1. function or Client method for running a code block
2. in a plain script, display plain-text output
3. if run from an IPython kernel, properly republish rich output

If we do it carefully with hooks for output, we could make this the canonical implementation that nbconvert's execute preprocessor uses, instead of it being a duplicate implementation.

[dsblank]

@minrk I think this is a good idea. These parts make a coherent subset that can be used in a few places.

(Also, I think it will take me a long time to package this up... I am not sure what is required and the implications of each of the pieces.)

[minrk]

I think we can start with the simple self-contained version, and think about what nbconvert might want to use after that. My implementation in dask should be most of the way to what I was thinking already.

[Jerrypiglet]

This issue also breaks PyCharm remote debug feature when initializing remote python interpreter.