If environment variable TERM is set to "dumb", don't use escape sequences

[mkoeppe]

This is so that so that ipython works with an Emacs inferior shell. See https://www.reddit.com/r/Python/comments/4w5d4e/psa_ipython_5_will_break_emacs_heres_how_to_fix_it/ http://emacs.stackexchange.com/questions/24453/weird-shell-output-when-using-ipython-5 https://trac.sagemath.org/ticket/21227

(I've also reported this as an IPython issue: https://github.com/ipython/ipython/issues/9886)

[jonathanslenders]

Hi @mkoeppe, Thanks for reporting this issue. But if we're not allowed to use escape sequences, how would we ever be able to handle cursor movements?

If escape sequences are not allowed, then probably there's not much of prompt_toolkit that's going to work. (No highlighting, no incremental search, no custom key bindings, etc.)

If Emacs wants to handle the cursor movements itself before delivering the input to the application, then --simple-prompt has to be used, as mentioned in the stackexchange post. (IPython could maybe detect the $TERM variable, and switch to the simple prompt.)

Jonathan

[mkoeppe]

On Tue, Sep 13, 2016 at 1:34 PM, Jonathan Slenders notifications@github.com wrote:

But if we're not allowed to use escape sequences, how would we ever be able to handle cursor movements?

If escape sequences are not allowed, then probably there's not much of prompt_toolkit that's going to work. (No highlighting, no incremental search, no custom key bindings, etc.)

If Emacs wants to handle the cursor movements itself before delivering the input to the application, then --simple-prompt has to be used, as mentioned in the stackexchange post. (IPython could maybe detect the $TERM variable, and switch to the simple prompt.)

On dumb terminals, prompt toolkit should fall back to a simple mode of interaction, similar to readline.

[edited quote markup]

[mkoeppe]

By the way, --simple-prompt is not sufficient because it does not even handle multiline input. It's ok for automatic testing, but not much else.

It seems that the IPython people have essentially delegated all prompt interaction to your package. That's why it's important that it provides basic prompt functionality on dumb terminals.

[jonathanslenders]

Do you have somewhere a link to documentation of what a "dumb" terminal accepts as output and what it sends as input? I think that even Vi and ncurses send escape sequences when $TERM is dumb.

According to this page: http://unix.stackexchange.com/questions/43945/whats-the-difference-between-various-term-variables "dumb" means that everything is broken...

[mkoeppe]

http://invisible-island.net/ncurses/terminfo.src.html

[mkoeppe]

I think that even Vi and ncurses send escape sequences when $TERM is dumb

Yes, but they have no other choice. After all, vi is the visual editor. A command-line based system such as IPython, on the other hand, can (and should) gracefully degrade to more basic interaction when the terminal is not fully functional.

[jonathanslenders]

Thank you for the link! I'll have a look at this and see what is possible.

For the multiline editing part, I'm afraid, because without cursor positioning (for moving the cursor upwards) I don't see any way to work on a "dumb" terminal.

Another question: why would you use the "inferior shell" instead of the built-in terminal emulator (M-x term)?

[mkoeppe]

On Tuesday, September 13, 2016, Jonathan Slenders <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

For the multiline editing part, I'm afraid, because without cursor positioning (for moving the cursor upwards) I don't see any way to work on a "dumb" terminal.

It would be enough to provide the equivalent of the multiline prompting of plain Python. For example, the user types: def f(x): And the system prompts for another line: ...

Another question: why would you use the "inferior shell" instead of the

built-in terminal emulator (M-x term)?

The real application are various comint-derived emacs modes, rather than just the inferior shell. They expect to exchange line-based input/output with Python.

Sent from my phone

[jdemeyer]

Just a small comment: in my experience working with other programs, unsetting TERM (i.e. removing the environment variable) is the standard way to communicate a "dumb" terminal. See also discussion at https://trac.sagemath.org/ticket/12263

[serv-inc]

Hi, is there any update on the greater problem of emacs integration? Even a regexp for emacs to ignore the control characters might work...

[dcolascione]

A well-behaved terminal program really should degrade gracefully with TERM=dumb. prompt_toolkit 2.x doesn't seem to do that. The problem of multi-line input can be solved by doing what readline does: force horizontal scrolling.

I'm also surprised that prompt_toolkit barfs if stdin or stdout isn't a PTY. A well-behaved program should fall back to very simple IO in this case, and it shouldn't be the job of the main application, above prompt_toolkit, to do that.

[intelfx]

This is still an issue, and right now it makes it impossible to e. g. interact with programs based on prompt_toolkit using expect.

If escape sequences are not allowed, then probably there's not much of prompt_toolkit that's going to work.

Exactly.

If a dumb terminal (or a non-tty output altogether) is encountered, prompt_toolkit should disable most of its facilities and gracefully degrade to become a simple command interpreter framework.

But if we're not allowed to use escape sequences, how would we ever be able to handle cursor movements?

You simply don't.

Do you have somewhere a link to documentation of what a "dumb" terminal accepts as output and what it sends as input?

For output, I think you're limited to plain ASCII in this case, or even less than that. For input, a dumb terminal doesn't send anything on its own — just what the user has typed.

In the end, that terminfo database linked above is the authoritative source on meaning of various $TERMs (after all, $TERM is by definition a selector into the terminfo database, see term(7)).

[glynnc]

FWIW, you shouldn't compare $TERM against some hard-coded list. Ideally, any escape sequences you send to the terminal should have been obtained from termcap or terminfo. Similarly for recognising specific input sequences as representing cursor/function/etc keys.

But if that's too much trouble, at least check for the existence of some common operations which any curses-capable terminal will support. A good candidate is "cub1" (cursor_left), which is the character which should be sent to move the cursor to the left. All VTs will have this (it's usually code 8, backspace), dumb terminals may not. Any terminal which lacks this features is going to be very dumb. Another candidate is "rmkx" (keypad_local). Also, if a terminal lacks the "lines" numeric capability, it's either a dumb terminal or a hardcopy terminal.

[jonathanslenders]

Hi everyone, Thank you for your patience! If you have some time, can you check out the following PR: https://github.com/prompt-toolkit/python-prompt-toolkit/pull/1035

It should handle dumb terminals correctly and won't render any escape sequences to the output if PromptSession().prompt() is used. Most up to date prompt_toolkit applications like IPython do that, also all the examples in the examples directory.

I tried with the Emacs inferior shell, and it seems to work fine, but maybe I'm still missing some stuff.

[mkoeppe]

Thanks very much for working on this! Looks good (but I haven't tested it).

[bestlem]

Seems to work in emacs. But the xonsh documentation is not up to date https://xon.sh/editors.html#xonsh-comint-buffer says

Make sure you install xonsh with readline support and in your .xonshrc file define

$SHELL_TYPE = 'readline'

I think the last bit is not needed anymore

[pythonwood]

in xfce4-terminal, django with ipdb debugging. always: WARNING: your terminal doesn't support cursor position requests (CPR).

[ssbarnea]

I just found this with latest iterm, which was a bit of a surprice.