(consoleapp) Remember command history in REPL #182

Closed perlun closed 2 weeks ago

Moved to GitLab

Please continue to the new version of this issue here: https://gitlab.perlang.org/perlang/perlang/-/issues/182. The GitHub page you are currently reading will not contain the latest information on this issue.

GNU readline has a nice feature: it remembers the command line history between invocations of a given program. This can be really useful sometimes, for example when typing a particular program in the REPL which exhibits a particular bug, doing a change, recompiling & rerunning the REPL again to see how it changes the program in question => up-arrow would be super-nice.

We don't use GNU readline at the moment, but maybe we should? How hard would it be to integrate it with Perlang as it stands? I'm not at all convinced that the "managed only" approach that most Java and C# programs take is the way to go, and we really want Perlang (programs) to be able to easily interoperate with random C libraries in fact. Maybe this would be a good candidate to trying out how hard it would be.

Caveat

GNU readline is GPL-licensed. We must be extremely careful with using GPL-licensed software as part of our tooling, since it might pull our whole codebase into it because of how the GPL works. The Perlang REPL is technically isolated from the actual interpreter in Perlang, but I still think that something like BSD's libedit would be a safer choice if we want to avoid "the GPL trap" (sorry for this wording, I'm tired and it's late in the evening. I love the accomplishments of the FSF and their emphasis on freedom etc; I just don't vibe with the GPL in this regard.)

Alternatives to GNU readline

ReadLine: https://github.com/tonerdo/readline
- C# GNU-Readline like library
- MIT-licensed
- Unmaintained, no commits pushed since June 2018
- Works on both Unix-like systems and Windows.
BSD libedit: https://www.thrysoee.dk/editline/, git repo exists at https://salsa.debian.org/debian/libedit
- C library, requires P/Invoke
- BSD-licensed (3 clause)
- Probably only works on Unix-like systems (=not Windows)
Linenoise: https://github.com/antirez/linenoise
- C library, requires P/Invoke
- BSD-licensed (2-clause)
- Only works on VT100-compatible systems, i.e. Unix.
- Reputable author; @antirez is the original author of Redis

Another option

Given the above, I'm considering that we should author a library of our own to handle this, meeting the following requirements:

C#-based (which lets us avoid the complexity inherent in any P/Invoke-based solution. The more platforms we add, the more awkward it will become to cross-build an .so/.dll dependency for numerous OS:es and CPU architectures)
Support for both Linux/macOS and Windows. (Should be doable as long as we wrap Console.ReadKey.)

The most reasonable approach would probably be to write a clean-room implementation of Linenoise in C#, something like a "Linenoise.NET". We could pretty much copy Salvatore's design pretty much straight off, and just implement something similar to it C#.

The end result should be something which is published under a liberal license (pretty much anything else than GPL is fine; LGPL would actually be OK if we want to prevent closed-source forks) and published as a public NuGet package. Who knows, if we decide to make a library like this, it may end up being more popular than Perlang itself. :sweat_smile:

Some other nice features in GNU readline that I'm missing in our current implementation:

Navigation word-by-word. The current library we use does support Ctrl-A and Ctrl-E + Home/End-style navigation (i.e. "Emacs-style" and MSDOS/Windows-style). It would be nice to be able to navigate between words using Alt-B/Alt-F (Emacs-style) and perhaps with Ctrl-Left and Ctrl-Right as well to make people like @slovdahl happy. :sunglasses:
Alt-Backspace to delete a whole word. This is especially convenient when deleting a long word (or number).

While this might not be the most important thing in the world, it does feel quite sub-par at the moment, as already mentioned above. Especially for me as a long-term Bash (=GNU Readline) user, I find it limiting to not be able to jump forward and backward between words using M-b and M-f (Alt is the Meta key in my case).

I think we should try to give this a look within the next few iterations. :+1:

An interesting detail about BSD editline and GNU readline is that the mysql CLI seems to use the former:

[21:28:41] per@delorean:~$ ldd /usr/bin/mysql
    linux-vdso.so.1 (0x00007ffd08baf000)
    libedit.so.2 => /lib/x86_64-linux-gnu/libedit.so.2 (0x00007f2875122000)
    libncurses.so.6 => /lib/x86_64-linux-gnu/libncurses.so.6 (0x00007f28750f8000)
    libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007f28750c6000)
        ...

...whereas the psql (Postgres CLI) seems to use the latter:

[21:28:34] per@delorean:~/git/tmp/libedit-20221030-3.1/examples$ ldd /usr/lib/postgresql/14/bin/psql
    linux-vdso.so.1 (0x00007ffc9b9f9000)
    libpq.so.5 => /lib/x86_64-linux-gnu/libpq.so.5 (0x00007f3930b6c000)
    libreadline.so.8 => /lib/x86_64-linux-gnu/libreadline.so.8 (0x00007f3930b14000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3930a35000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3930854000)
        ...

This post led me to discover https://github.com/mono/mono/blob/master/mcs/tools/csharp/getline.cs; a minimal, self-contained Readline implementation in C# by @migueldeicaza. It seems like one of the better options for this; I'll start looking into incorporating it into our CLI now.

354 takes us quite far, but the most important missing features so far are:

Missing keys
- ^U: delete the whole current line
- ^H: backspace
- ^J: line feed. Ignores the current line and starts a new, empty line.
- Ctrl-backspace: handled as regular backspace, at least on GNU/Linux
Semantic details
- When using ^R to perform a reverse-search, the backspace key can be used in normal readline programs to erase the last character typed. With getline.cs as it currently stands, the whole line is deleted.
- When using Alt-backspace multiple times to delete whole words, sometimes getline.cs gets stuck and does not delete all the way to the beginning of the line.

(I haven't checked in Miguel's comments if any of these are already "known" issues or not. Some of the ones above should be easy to fix so I'm planning on looking into it in a subsequent MR, once #354 is merged.)

As a side note, it's amazing how many small details like this you tend to rely on when having used Linux & readline-based programs (like bash) for a long time... :slightly_smiling_face:

354 takes us quite far, but the most important missing features so far are:

Some of these has been fixed as of #356.

The REPL will be (temporarily, but probably for a few years) dropped as of #406:

I am currently (2023-11-03) leaning towards dropping (parts of) the REPL soon, perhaps in the 0.5.0 or 0.6.0 release. This will make things simpler and free us from having to keeping it working all the time, since it won't be working in compiled anyway (for quite a long time, realistically speaking). Once the Perlang compiler is mature enough to be able to interface with LLVM to generate machine code for an arbitrary Perlang expression tree, we can reimplement the REPL on top of this.

Because of the above :point_up:, I'll move this to the Later milestone for now, making it clear that this should be seen as "eternally backlogged" in the currently planned releases.

perlang-org / perlang