search

oils-for-unix / oils

Oils is our upgrade path from bash to a better language and runtime. It's also for Python and JavaScript users who avoid shell!
http://www.oilshell.org/
Other
2.82k stars 154 forks source link

Detect when the user types past the last column in the terminal #257

Closed jyn514 closed 2 years ago

jyn514 commented 5 years ago

This has a symptom similar to https://github.com/oilshell/oil/issues/252 and that's how I found it, but the cause I think is different. When completing a line which wraps to the next line, the completed line overwrite the suggestions:

overwrite

When we backspace, it deletes the suggestions on the current line:

deletes the suggestions on the current line.

If we try to tab-complete again, it gets stranger: the cursor goes to the right of the screen and parts of the prompt get written on the new line.

image

This is probably caused by core/comp_ui.py not taking into account the line wrap.

jyn514 commented 5 years ago

Related to https://github.com/oilshell/oil/issues/240

andychu commented 5 years ago

Yeah there are a lot of bugs in this area... I am going to fix some known ones, and then we'll have to do another round of testing to see what's left :-/

Right now OSH has --completion-display=nice or =basic. I think I have to revert to the "basic" one unfortunately. The "nice" one is more like zsh, but probably has more bugs.

Both of them have to known the terminal width, which is related to the SIGWINCH issue. I noticed that the Python interpreter, OSH, and readline are all fighting over SIGWINCH.

andychu commented 5 years ago

I think we fixed most of the low hanging display bugs, and I plan to make a release today or tomorrow, so we can do another round of testing.

However this one is somewhat fundamental. To restate the issue clearly:

Temporary workarounds:

andychu commented 5 years ago

Hm I noticed this rl_event_hook() function (which CPython also uses.) It might be possible to do something super hacky and poll for being in the last column. Doesn't sound very appealing though...

https://tiswww.case.edu/php/chet/readline/readline.html#IDX238

jyn514 commented 5 years ago

I'm not sure this is related to going off the end of the screen actually. The problem still occurs in miniature on one line. I remember seeing an issue about extra spaces on OSX, this looks related.

image

andychu commented 5 years ago

Hooks that might help solve this problem (?)

https://lists.gnu.org/archive/html/info-gnu/2014-02/msg00011.html

k.  New application-settable variable: rl_input_available_hook; function to be
    called when readline detects there is data available on its input file
    descriptor.

l.  Readline calls an application-set event hook (rl_signal_event_hook) after
    it gets a signal while reading input (read returns -1/EINTR but readline    
    does not handle the signal immediately) to allow the application to handle
    or otherwise note it.  Not currently called for SIGHUP or SIGTERM.
andychu commented 4 years ago

Discussion where I referenced this:

https://oilshell.zulipchat.com/#narrow/stream/121540-oil-discuss/topic/interactive.20features

I just tried out ble.sh and it seems to do the right thing. And I'm very surprised it can do the right thing in pure bash! I want to chat with @akinomyoga about how this works, as I think this is one of the things that's keeping me from using osh!

The bug right now is if you do TAB completion in osh, and the screen isn't wide enough, the current line "draws over" the completion candidates. In ble.sh the candidates move to the next line correctly.

And yes the first screenshot in this issue shows this pretty much.

andychu commented 4 years ago

note: yash also solves this problem and appears to do something pretty smart.

andychu commented 4 years ago

Not essential since we're cutting the interactive shell for 2020

andychu commented 2 years ago

Sorta related

https://github.com/romkatv/powerlevel10k#horrific-mess-when-resizing-terminal-window

akinomyoga commented 2 years ago

This is a nice summary of the problem of the terminal text reflowing. I also received multiple issues related to the terminal text reflow in ble.sh: https://github.com/akinomyoga/ble.sh/issues/114#issuecomment-850737131, https://github.com/akinomyoga/ble.sh/issues/142, and https://github.com/akinomyoga/ble.sh/issues/154. Every time, I need to explain that this is caused by the undefined behavior of the terminal in text reflowing, and there is no perfect solution at the line-editor side. I think I can point the user to this section of powerlevel10k README next time I receive the same report.

andychu commented 2 years ago

Yeah now that I look more carefully at the history of this issue, and at powerlevel10k, that is a related but distinct issue.

  1. Reflowing after window resize changes the vertical distance between the cursor and the start of the prompt.
  2. So does typing a long line off the end of the screen, which doesn't involve a window resize (that is what this issue is about)

I think you're saying that in case 1 it's impossible to predict what the new distance is?

But I think it's possible to handle case 2 if you control the whole line editor, which OSH doesn't. OSH is using GNU readline. (And I still haven't looked into the readline hooks I mentioned above)

So I guess to sum up this issue, we can

  1. either find some GNU readline hook to fix it
  2. Go back to a bash style of printing a new prompt every time you hit TAB, which means you never have to go "up" again. This is very easy to implement but I think a bad experience :-(
  3. maybe: just document it as a known issue? You can sort of avoid it by having a wide enough terminal :-/

I still would like to run ble.sh but I think that is further away unfortunately

andychu commented 2 years ago

I guess I would add that situation #2 is more common than #1.

That is, it is common to type a long line and have a narrow terminal, causing wrapping

But if I resize my window, I'm used to having to do something to "reset" the state... i.e. using vim or tmux. Usually I tend to open new windows rather than resize existing ones, but some people may have different work habits

akinomyoga commented 2 years ago

I think you're saying that in case 1 it's impossible to predict what the new distance is?

Yes, it's impossible for the general terminal.

But, of course, if the terminal and its version are specified, we can test the behavior of the terminal and write the code for that specific terminal. I have once checked the behavior of several terminals on the text reflow, but the problem is that every terminal has its own behavior slightly different from any other terminal. Some terminals remember that whether the automatic line folding at the end of the line has occurred or not for each line. Some terminals do not distinguish the whitespaces at the end of the line from the empty terminal cell (the state when nothing has been written to the cell). Some terminals try to become smarter and do something complicated, for which it is hard to predict the exact behavior from the external applications, etc.

But I think it's possible to handle case 2 if you control the whole line editor,

Yeah. Anyway, full control of the line editor is needed for both cases 1 and 2.

I guess it is hard to solve the problem of this issue (case 2 for completion candidates) as far as we are using GNU readline.

I still would like to run ble.sh but I think that is further away unfortunately

We could actually once run the core part of ble.sh on osh (cf Running ble.sh With Oil) [ Note: I have recently tested it on the latest version of oil, but it became not working although I guess the adjustment is not so hard]. So, actually, I guess, after a few hooks for the user input are provided by Oil, it is not so hard for me to adjust ble.sh (and probably to submit additional small fixes to Oil) so that it works as a line editor of Oil.

However, the problem that turned out then was that the initialization and the response to the user inputs were very slow with osh which is written in Python. I think another problem was the footprint (i.e., the memory use) of Oil after loading ble.sh and related modules (which currently have more than 40k SLoC in total). I thought that these problems would be solved in oil-native. I actually have never tried oil-native, but how's the current progress of oil-native?

andychu commented 2 years ago

So, actually, I guess, after a few hooks for the user input are provided by Oil, it is not so hard for me to adjust ble.sh (and probably to submit additional small fixes to Oil) so that it works as a line editor of Oil.

Oh really that would be very interesting -- do you have an idea of the missing features? Maybe we can discuss more on #653 or start a new issue for "ble.sh 2022 :)"

oil-native is making slow progress -- as of the latest release there are 1131 tests passing, out of 1900 or so.

https://www.oilshell.org/release/0.9.6/test/spec.wwz/cpp/osh-summary.html

What is good is that we can just write code in Python and that number goes up automatically.

But what is bad is I haven't had time to work on it much since March 2021, since I've been working more on the Oil language, documentation, etc.

So right now I am trying to accelerate the project by hiring a compiler engineer. I applied for 50K euros from this open internet fund:

https://nlnet.nl/news/2021/20211201-call.html

I think we will get a response by February 1st. And then I am setting up Github Sponsors / Open Collective now so we can take donations. There have been many readers for many years, so I think at least some of them will donate.

I think it helps that the project is very concrete and it can absolutely be done with enough effort... there is no "risk".

There are even some benchmarks of oil-native on small programs here: https://www.oilshell.org/release/0.9.6/benchmarks.wwz/compute/

But it still can't run large programs.

I think it is slightly slower than bash, but it is "hilariously unoptimized". I have no doubt that we could optimize it in C++ and get 2x speedup easily, and comparable memory usage to bash (maybe a little higher because of GC), but that has to happen after the 1900 spec tests pass!

I noticed we stopped running ble.sh tests in our build awhile ago, let me re-enable it, which is #762

andychu commented 2 years ago

Hm this GNU readline configuration might let us avoid the issue? It can just scroll?

Then if people want to avoid that, they can just hit \ and enter?

horizontal-scroll-mode
    This variable can be set to either `on' or `off'. Setting it to `on' means that the text of the lines being edited will scroll horizontally on a single screen line when they are longer than the width of the screen, instead of wrapping onto a new screen line. This variable is automatically set to `on' for terminals of height 1. By default, this variable is set to `off'.
andychu commented 2 years ago

Horizontal scroll mode seems to do a good job, feedback welcome: https://www.oilshell.org/release/0.9.8/