Defer support of "visual programs" (ncurses, etc.) to another terminal

[Ambrevar]

Running ncurses applications with cmd is no good, and this is to be expected :) However, we could be a bit smarter here and (semi-)automatically catch the command in advance to start it with an external application, such as Xterm, or, let's be crazy, Emacs' vterm :)

Here is a proof of concept:

(defvar *visual-command* '("htop"))
(defvar *command-wrappers* '("sudo" "env"))
(defun visual-command-p (command)
  "Return true if the COMMAND list runs one of the programs in `*visual-command*'.
`*command-wrappers*' are supported, i.e.

  env FOO=BAR sudo -i powertop

works."
  (labels ((basename (arg)
             (namestring (pathname-name arg)))
           (flag? (arg)
             (str:starts-with? "-" arg))
           (variable? (arg)
             (and (< 1 (length arg))
                  (str:contains? "=" (subseq arg 1))))
           (first-positional-argument (command)
             "Return the argument that's not a flag, not a variable setting and
not in `*command-wrappers*'."
             (when command
               (if (or (flag? (first command))
                       (variable? (first command))
                       (find (basename (first command))
                             *command-wrappers*
                             :test #'string=))
                   (first-positional-argument (rest command))
                   (first command)))))
    (sera:and-let* ((cmd (first-positional-argument command)))
      (find (basename cmd)
            *visual-command*
            :test #'string=))))

(defun vterm-terminal (cmd)
  (list
   "emacsclient" "--eval"
   (let ((*print-case* :downcase))
     (write-to-string
      `(progn
         (vterm)
         (vterm-insert ,(str:join " " cmd))
         (vterm-send-return))))))

(defvar *terminal* '("xterm" "-e")
  "The terminal is either a list of arguments after which will be prepended to
the visual command to run, or a function of one argument, the list of commands,
returning the new list of commands.")

(defun maybe-launch-visual-command (cmd)
  (if (visual-command-p cmd)
      (cmd:cmd
       (if (functionp *terminal*)
           (funcall *terminal* cmd)
           (append *terminal* cmd)))
      (cmd:cmd cmd)))

Try it out with

(maybe-launch-visual-command '("htop"))

To use Emacs Vterm, just set

(setf *terminal* #'vterm-terminal)

What do you think?

[jcguu95]

Slightly related is #13: for programs that require some interactive response, how did you handle it @Ambrevar ?

[Ambrevar]

I haven't at the moment, but it's a high priority for me too :)

The most common kind of interaction for me is sudo. For this, I just added this to ~/.slynkrc:

;; To get "sudo" to work in SLY.
(let ((askpass (format nil "~a/.local/bin/emacs-askpass" (uiop:getenv "HOME"))))
  (when (uiop:file-exists-p askpass)
    (setf (uiop:getenv "SUDO_ASKPASS") askpass)))

With emacs-askpass being:

#!/bin/sh
emacsclient -e '(read-passwd "sudo password: ")' | xargs

[jcguu95]

I figured that too, given your wonderful article on lisp repl > shell.

For sudo, it's doable. Do you have any slight idea for the general case? I noticed that for #'uiop:run-program, there's an :interactive action that would work in a terminal emulator. However, for slime/slynk the situation is much trickier. I couldn't think of a non-adhoc solution. Any idea or hint would be highly appreciated!

[Ambrevar]

I think we should investigate what Emacs Eshell and `M-x shell'.

Off the top of my head, I think they open a PTY when they execute a process. If I understand correctly, interactive shell programs use a PTY to collect input from the user.

[jcguu95]

@Ambrevar what a nice hint! I was pessimistic about this problem due to some previous discussions with others actually. But recalling that M-x shell works just fine fires the hope up again!

Here are some notes after digging into M-x shell a little bit. First of, the magic is provided by comint.el. comint is an abstraction for the users to interactive with a subprocess in emacs. According to this,

.. comint has handles all the nitty-gritty stuff like handling input/output; a command history; basic input/output filter hooks; and so on. In other words, it’s the perfect thing to build on if you want something interactive but want more than just what comint has to offer.

So the user can just interact with a subprocess easily with it! Some quick tests confirms the claim above. Namely, all of the followings work for me (though they need some tuning).

(comint-run "/usr/bin/bash")
(comint-run "/usr/bin/pacman" '("-Q"))
(comint-run "/usr/bin/sudo" '("pacman" "-S" "alacritty"))
(comint-run "/usr/bin/python")
(comint-run "/usr/bin/sage")

The interactive i/o are handled decently.. which is quite mind blowing! What's best is that M-x shell and sly are already based on comint. So most the work can start with comint indeed.

I am not entirely sure, but I'd guess that sly asks comint to call sbcl. But if this the case, what follows is very weird:

With sbcl run in a terminal emulator, the following calls a python repl within the lisp repl as expected.

* (uiop:run-program "/usr/bin/python" :output :interactive :input :interactive)
>>> 1+1
2
>>> def f(x):
...     return 2*x
...
>>> f(f(3))
12

However, the evaluating the same form in sly does not work. No output from sbcl is sent to sly, and while attempt to input, I received the message [sly] REPL is busy. This suggests that sly isn't just a comint calling sbcl naively.. as then I should be able to interact with sbcl decently as in the examples above (pacman, python, sudo, bash.. etc).

EDIT

Indeed, sly is more than that. It actually calls inferior-lisp, creates a slynk-server in it, and connect to it (see sly and sly-attempt-connection and how they related in the source). Another observation is that M-x inferior-lisp works fine with * (uiop:run-program "/usr/bin/python" :output :interactive :input :interactive).

So the problem must lie between inferior-lisp and sly! This looks like a rabbit hole as one has to study how they connect to each other, and why certain I/O doesn't get passed between them - essentially, how slynk protocol works.

Other resources

• https://emacs.stackexchange.com/questions/30047/how-do-i-reliably-get-output-from-a-process
• https://emacs.stackexchange.com/questions/692/asynchronously-wait-for-output-from-a-comint-process
• https://www.gnu.org/software/emacs/manual/html_node/elisp/Processes.html
• https://www.masteringemacs.org/article/comint-writing-command-interpreter

[Ambrevar]

I don't know the details, but SLY runs SBCL in an "inferior Lisp" buffer, and communicates with it via a mrepl channel.

<1:CL-USER> (bt:all-threads)
(#<SB-THREAD:THREAD "main thread" RUNNING {1000FE8103}>
 #<SB-THREAD:THREAD "Slynk Sentinel" RUNNING {1006AEA763}>
 #<SB-THREAD:THREAD "control-thread" RUNNING {1006D17B13}>
 #<SB-THREAD:THREAD "sly-channel-1-mrepl-remote-1" RUNNING {1009130513}>
 #<SB-THREAD:THREAD "slynk-indentation-cache-thread" RUNNING {1006D18243}>
 #<SB-THREAD:THREAD "reader-thread" RUNNING {1006D17E53}>)

Beside the main thread, all other threads are fire up for the REPL.

I suspect this is what's blocking here.

[jcguu95]

(A reminder that I had edited my post - you might have missed it as you seemed to be using email.)

Sure, I think it is clear now that the data flow is

sbcl <---> inferior-lisp <---> sly

Here, sly talks to inferior-lisp by starting a slynk-server in sbcl, and then sly-connect. In general, we cannot expect that whatever is printed in inferior-lisp is also printed in sly, and this is our current issue. Indeed, while (uiop:run-program ... :output :interactive :input :interactive) works as expected between sbcl and inferior-lisp, what's correctly printed in inferior-lisp is not automatically printed in sly. Same applies to the other direction; though we can interactively input in inferior-lisp with sbcl, there's simply no supporting protocol for this between inferior-lisp and `sly.

I think it's now clear that this must be done by hacking the sly-slynk protocol. If it is even possible to achieve, one might want to sync inferior-lisp and sly once they are connected by the sly-slynk protocol.

[Ambrevar]

Back to topic: @ruricolist How do you feel about this? Should I send a patch?

[ruricolist]

Yes, you can send a patch, although I would prefer that no commands be marked as visual by default.

[vindarel]

Hey there,

For sudo, it's doable. […] I noticed that for #'uiop:run-program, there's an :interactive action that would work in a terminal emulator

it wasn't obvious for me to find out so here it is: this works for sudo and visual commands such as htop (and vim, ncdu etc) but not interactive ones such as fzf:

(uiop:run-program '("sudo" "htop") :output :interactive :input :interactive)

my 2c

[Ambrevar]

Didn't know about this, great tip! Thanks for sharing, @vindarel !

[ruricolist]

I added a :<> redirection operator, so (from a REPL running a terminal) you can do:

(cmd "sudo -S ls" :<> :interactive)

And this will prompt for a password.