fix --simple-prompt mode to support multiline input?

[dhmunro]

A side effect of --simple-prompt mode is to turn off even rudimentary multiline input handling, including all continuation lines. This mode is required for running jupyter_console under any of the emacs comint modes, which means that jupyter_console cannot be used in any of the emacs-based IDEs. This defect was fixed in ipython 5.4 and 6.1, and a similar small patch fixes jupyter_console. The problem with emacs comint modes is that they are strictly line-based; emacs handles all line editing and passes the subprocess single complete lines. There is no concept of a cell or a suite of lines as a single block, so that prompt_toolkit functionality is lost. However, conventional continuation lines present no problem; they simply are not treated as a single block. Making --simple-prompt mode recognize traditional continuation lines is relatively simple using the same machinery prompt_toolkit requires to tell when multiline input is complete.

The ZMQTerminalInteractiveShell.init_prompt_toolkit_cli method in ptshell.py can be patched to make jupyter_console handle continuation lines in --simple-prompt mode for any kernel implementing an is_complete method, including ipython. It is very similar to the equivalent ipython init_prompt_toolkit_cli method. Here are the modified lines of ptshell.py in context:

import types
...
def init_prompt_toolkit_cli(self):
    if self.simple_prompt or ('JUPYTER_CONSOLE_TEST' in os.environ):
        # Simple restricted interface for tests so we can find prompts with
        # pexpect.  Multi-line input may not be supported.
        def prompt(myself):
            prompt_text = 'In [%d]: ' % myself.execution_count
            prompt_continuation = ' '*(len(prompt_text) - 2) + ': '
            line = cast_unicode_py2(input(prompt_text))
            if myself.use_kernel_is_complete:
                while True:
                    more, indent = myself.check_complete(line)
                    if not more:
                        break
                    cline = cast_unicode_py2(input(prompt_continuation
                                                   +indent))
                    line = u'\n'.join([line, indent + cline])
            return line
        self.prompt_for_code = types.MethodType(prompt, self)
        self.print_out_prompt = \
            lambda: print('Out[%d]: ' % self.execution_count, end='')
        return
    ....

I don't understand the check_complete() semantics well enough to be sure this works as intended in all cases, but it seems to behave exactly as without --simple-prompt in tests with the ipython kernel. The logic is essentially the same as the analogous ipython code. And it makes emacs comint modes running jupyter_console --simple-prompt feel very nearly the same as running in an ordinary terminal without --simple-prompt. (The only thing you can't do is edit a previous line in a multiline block before sending it.)

[dhmunro]

I think the indent treatment in the above patch is incorrect -- there isn't any way for the subprocess to indicate a de-indent, so it would be better to simply ignore the indent string returned by check_complete. That requires the user to add the indent by hand, instead of the auto-indent provided by prompt_toolkit, but it's a small price to pay for being able to properly enter multiline suites with multiple indent levels like a for loop with an embedded if test. Plain python doesn't auto-indent continuation lines (probably for the same reason), so simply input(prompt_continuation) and join([line, cline]) makes this prompt_for_code method handle continuation lines the same as plain python.

[dstrozzi]

I want to add a strong +1 to adopting this patch (or an equivalent one). I run the Jupyter console --simple-prompt under emacs on Windows, and this would be a big help. I have to run jupyter console and not ipython to get matplotlib figure windows to work, which is a whole separate thread.

[dsblank]

is this still an issue with --simple-prompt and jupyter console?

[dstrozzi]

I verified on windows 7, Python 3, using the most up-to-date Anaconda distribution, that the problem persists. From a DOS prompt, type 'jupyter console --simple-prompt', then 'if 1:' and it barfs:

In [2]: if 1: File "", line 1 if 1: ^ SyntaxError: unexpected EOF while parsing

I am not willing to play with getting things from git repos and building, I bet that will be a time suck. The relevant file, ptshell.py, was modified on Aug. 2, 2018.

[usnigg]

Hi, I seem to be in a similar situation as dstrozzi. I use Elpy for my python development. When using ipython6.4.0 --simple-prompt (which supports multi-line editing) and emacs26.1 on windows however I can't get the interactive shell to plot stuff (i.e. when using %pylab magic and then trying to plot something, the figure window opens but is frozen). The only workaround I found so far is to use jupyter console --simple-prompt instead of ipython. This fixes the plot window freezing issue but no longer enables multi-lien editing.

[dstrozzi]

To follow on to @usnigg, I originally was using elpy and had this issue. It took me a while to figure out it has nothing to do with elpy, but the general way emacs on Windows "launches" ipython / matplotlib windows as subprocesses. Ipython and Jupyter-console differ in this regard, and I don't understand the details. I just use the plain out-of-the-box emacs python env, I get the same issue as with elpy. So the dilemma is "jupyter console + plots work but multi-line input broken", or "ipython + plots broken but multi-line works".

Since my OS preference is "Linux is nice but I prefer emacs", I'd imagine there's a fair number of users running Python within emacs on Windows.

[aiguofer]

I would really like this as well... when I want to run some multi-line piece of code I currently have to type it into a file and send to shell in Emacs, which is kind of annoying. I'd love to have multi-line support again as when I use ipython --simple-prompt.

[Carreau]

Trying to revive this repository, I've cleanup a bit the codebase; I also found that there is a lot of assumption this runs Python code;

If one of you want to send a PR i'll be happy to try to get it in.

There is also likely logics that can be borrowed from qtconsole.

[dstrozzi]

I am glad to hear this! Before discussing a PR, just FYI I have an up-to-date Anaconda install (2020.07 distro, with updates just done, python 3.8.5) on my Win10 laptop. From CMD prompt:

jupyter console --simple-prompt
In [3]: if 1==1:
  File "<ipython-input-3-f50d32018107>", line 1
    if 1==1:
            ^
SyntaxError: unexpected EOF while parsing

So - the same error persists.

[dstrozzi]

So, I don't have a patch to contribute, but just wanted to encourage someone to look at this. The current code has changed enough since dhmunro's original patch that it can't directly be applied. Running jupyter console within emacs is one use case, but not the only.

[Dr-NAME]

Hi, Based on the code initially proposed by @dhmunro, I've written a patch compatible with the current code. I tested it on Windows 10 with emacs with the current version of jupyter-console (6.6.3). It allows to handle expressions on several lines. However, it suffers from the same limitation as before, where the indent must be inserted manually.

in ptshell.py :

...
    def init_prompt_toolkit_cli(self):
        if self.simple_prompt or ('JUPYTER_CONSOLE_TEST' in os.environ):
            # Simple restricted interface for tests so we can find prompts with
            # pexpect. Multi-line input not supported.
            async def prompt():
                prompt = 'In [%d]: ' % self.execution_count
                # Multiline handling based on @dhmunro code
                promptContinuation = "...: ".rjust(len(prompt), " ")
                line = await async_input(prompt)
                while True:
                    more, indent = self.check_complete(line)
                    if not more:
                        break
                    cline = await async_input(promptContinuation)
                    line = "\n".join((line, cline))
                return line

            self.prompt_for_code = prompt
            self.print_out_prompt = \
                lambda: print('Out[%d]: ' % self.execution_count, end='')
            return
...

In the few tests I've carried out with different expressions, I haven't come across a situation where it didn't work.