A clearer intro for the README

[hmelman]

I have a friend that was looking at embark and didn't find the README that helpful. I told him I looked at it and found it reasonable and asked what he didn't like about? He said "I didn't find an obvious on-ramp for newcomers".

I don't find the text difficult but I do realize that embark uses a bunch of its own terminology, action, target, indicator, etc. It does define these terms but it is a little wordy. And I find the introduction of "a sort of right-click contextual menu" confusing since the mouse isn't involved. I think it might be good to introduce the concepts via standard emacs terms. What if it began with something like:

Embark makes it easy to choose a command to run based on what is near point. Unlike many emacs features it works well both in a regular buffer and in the minibuffer (in a way helm and counsel users will be familiar with). Bind the command embark-act to a key and it acts like prefix-key for a keymap of actions (commands) relevant to the target around point. With point on an url in a buffer you can open the url in a browser or eww. With a file candidate in the minibuffer you can rename or copy it, With point on a lisp function you can copy, get help for it, or trace calls to it. If while switching buffers you spot an old one, you can kill it right there and continue to select another. And there are a bunch of functions for working on an active region. It can also collect all the candidates in a minibuffer to an occur-like buffer or export them to a buffer in a suitable mode like dired for a set of files, ibuffer for a set of buffers, or customize for a set of variables.

[oantolin]

I'm convinced the README problem is unsolvable, it's always too abstract or not abstract enough, it's a wall of text or it's not detailed enough, the metaphors it uses are too vague or it should use more metaphors that people can relate too even if they are not precise, etc.

So, I'm completely out of ideas and am willing to adopt any text that people wish to contribute, with the idea that you cannot possibly be worse at writing that README than I am.

[oantolin]

(In fact reading people's comments about the README feels a lot like reading student evaluations, where I am simultaneously the best math prof one student in the class has ever had and another student wonders how the university lets me teach at all ---I wish that were hyperbole but it's almost direct quotes! 😆)

[oantolin]

I'm not sure that a clearer overview is what your friend means by an "on-ramp for newcomers". Isn't that more likely to mean quick installation and configuration instructions followed by the very basics of how to use the thing?

[oantolin]

Well, whether or not your friend would consider the intro text you wrote "an on-ramp", I think it's better than what we have, so I will change it.

[oantolin]

Please take a look, @hmelman. I only added body text to the previously empty ""Overview section, and changed the first sentence of the section "Acting of targets".

I worry the overview is still too wordy, maybe I should have followed your text more closely. What do you think?

[hmelman]

I sympathize with the difficulty of describing this package, its a bit unusual in its usage. By on-ramp my friend meant a quick overview of what the package does so that he can figure out if he wants to keep reading. For that reason brevity was what was missing at the start. The rest has plenty of detail and as I said, I thought it was pretty good.

So I tend to agree what I had was a better overview :) My friend and I iterated on it a few times so I know it addressed his concerns. In fact I'd even break my text into a second paragraph where I start with the examples and maybe even remove that second part, since you get to examples pretty quickly. But I found I wanted a bridge to get to the collect part of my description which I think is important. And I wanted to include a hint for people coming from ivy or helm as I think that's still a big driver of users to embark.

I'm not sure what the big wins are with embark. I use it for embark-collect and also to copy a symbol at point. I've installed a titlecase package and rather than finding a keybinding for its command, I added it to embark-region-map and it's worked great for me. I'm trying to use embark more. If you have particular examples you're fond of, use them.

[oantolin]

OK, I can hew more closely to your text, specially since it is what your friend considers an "on-ramp" (I think the usage I mentioned of "on-ramp" is more standard, it's good that you knew what your friend meant).

So you don't think it's important in the overview to explain that Embark doesn't just let you call commands but it also enters information at their prompt for you? I think that's the main thing I added to your text.

How do you feel of using this next text as the body of the previously empty "Overview" section?

[hmelman]

So you don't think it's important in the overview to explain that Embark doesn't just let you call commands but it also enters information at their prompt for you? I think that's the main thing I added to your text.

Hmmm. I could be missing something but it feels like an architectural detail for later. Kind of like the difference between when a function acting on several items gets called once with the whole collection, or several times each with one item. It's important to understand the difference, but maybe not at first. The general idea is more important to get across (particularly in an intro), that the command will "act on" the target near point.

I think for intro there are enough ideas to get across in: it works in the minibuffer and in a regular buffer, it can work on one target or collect several, and it knows the type of the target and self-configures its choices based on that. If I were to add a fourth idea it would be that uses standard keymaps and commands and doesn't need wrappers to do its magic. And this is leaving out indicators and embark-become (which feels more special case).

[oantolin]

Well, I was worried that since you made it sound like it just calls Emacs commands someone might think, "Oh, I put point on a file name, call embark-act, choose copy-file and now I have to type the freaking file name point was on? This is stupid."

[hmelman]

Fair point. Then I think having good examples in what was my "middle section" is important. You probably have better ones than me. E.g., I know that to get help on a lisp function I could do s-; h (I bind embark-act to s-;) but that's not really a win over the standard C-h o. I use s-; w because point is usually some place in the target and embark includes the whole target for me, so it's a time saver. Maybe while mentioning a rename-file example include a parenthetical "(embark enters the target as the source argument for you)" as quick way to make the point.

Like my titlecase example I find embark really useful to "find keys" to put commands on, or as a good way to access infrequently used commands that might not have a binding or one that I remember (like calc-grab-sum-across on a region). But I'm not sure how to mention these in the intro without derailing the thought with the details of the infrequent command. Maybe they just make good examples for a section further down.

[oantolin]

The non-Embark way to get help on the function at point is not C-h o but C-h o RET, so it is a tiny, tiny victory for Embark. 😛

[hmelman]

Agreed. Lots of "tiny, tiny" victories add up to a good package. :)

[oantolin]

Like my titlecase example I find embark really useful to "find keys" to put commands on, or as a good way to access infrequently used commands that might not have a binding or one that I remember (like calc-grab-sum-across on a region).

This is also a big part of why I like Embark, but I agree it's probably hard to work into the intro.

As a related point, I really like recycling keybindings for similar operations in different contexts. With Embark the single binding t transposes regions, sentences, paragraphs and s-expressions. By default Embark uses h for Emacs Lisp help, and for display-local-help on non-elisp identifiers. You can remap display-local-help to the appropriate command in other major modes (which eglot will do for you if you have it), so I've configured it to also give me man pages in shell scripts, perldoc in Perl programs and in eshell buffers it's great: on Emacs Lisp symbols it give me Emacs help, and on shell commands it gives me man pages. So h just means "help!" wherever I am now.

[oantolin]

The point you make about s-; w working anywhere within the target is nice. But you save even more time when doing s-; DEL, which deletes without polluting the kill-ring, because, for example, to delete a word or sentence without putting it on the kill-ring you'd need to move to the beginning of it, mark it, and then call delete-region which is 3 steps.

[hmelman]

I think those benefits are good ones and they're also a bit non-obvious. Perhaps they belong in a Tips and Tricks section. They're the kind of thing that experienced users realize but intermediate ones might not.

[minad]

There is also the wiki for tips and tricks! I think the intro got better, but I liked the keyboard context menu analogy - it is a great analogy for everyone who is familiar with the usual GUI paradigms. You could also mention the discoverability aspect. Embark let's you discover many "hidden" commands which are relevant to the target at point but which are otherwise unbound, given that the Emacs are so crowded.

[oantolin]

See if you like 34e54d19bb94030c6fc307100c40d828f8cb5c6b any better. I made the overview a single paragraph following your text much more closely. I removed some of your examples and just said there are hundreds of preconfigured actions and you can add more. I also brought back the right-click metaphor that @minad liked, but added "keyboard-based" which I think takes care of your objection to it @hmelman.

[hmelman]

I added a Modest Embark Benefits section to the wiki home page. Maybe we could refine this and put it in the README at some point. Moreso than just Tips and Tricks, I think these collectively offer justification for using Embark.

[hmelman]

See if you like 34e54d1 any better.

Yes I like it. 😄

[oantolin]

I added a Modest Embark Benefits section to the wiki home page. Maybe we could refine this and put it in the README at some point. Moreso than just Tips and Tricks, I think these collectively offer justification for using Embark.

Thanks! I like it.

[hmelman]

By default Embark uses h for Emacs Lisp help, and for display-local-help on non-elisp identifiers. You can remap display-local-help to the appropriate command in other major modes (which eglot will do for you if you have it), so I've configured it to also give me man pages in shell scripts, perldoc in Perl programs and in eshell buffers it's great: on Emacs Lisp symbols it give me Emacs help, and on shell commands it gives me man pages. So h just means "help!" wherever I am now.

I've just started using eglot (just for python now). Could you be more specific about the configuration for these? Maybe put it in the wiki?

[oantolin]

I think no configuration is needed for the h action with eglot: Embark has it bound to display-local-help and eglot remaps that to something useful. In fact we bound h to display-local-help exactly so eglot users got this functionality without any configuration, on @astoff's suggestion.

I don't use eglot for shell scripts (though there is a bash LSP server!) nor in eshell buffers, so there I manually remap display-local-help to man. Thanks to Embark entering input for you, you can remap display-local-help to either a function that displays help for the thing at point or to a command like man, which prompts you for what you want help on. That's another kind of uniformity you get from using Embark.

[oantolin]

If you're satisfied with overview, @hmelman, we can close this issue now. (And we can keep discussing, of course.)

[hmelman]

I don't use eglot for shell scripts (though there is a bash LSP server!) nor in eshell buffers, so there I manually remap display-local-help to man. Thanks to Embark entering input for you, you can remap display-local-help to either a function that displays help for the thing at point or to a command like man, which prompts you for what you want help on. That's another kind of uniformity you get from using Embark.

So you do:

(define-key eshell-mode-map [remap display-local-help] #'man)

[oantolin]

Exactly, and I am adding that tip to the wiki as you suggested.

[oantolin]

It's on the wiki.

[minad]

Remapping h to man in eshell doesn't make sense in all cases. We could do better and detect if a symbol is indeed a man page or an Elisp symbol. There are also the variables eshel-prefer-lisp-variables/functions.

[oantolin]

You didn't try the eshell remapping, @minad, it magically only applies to non-elisp symbols!

[minad]

Indeed I didn't try it, I just saw the discussion. How is this possible? Does the same happen also in Slime buffers? So it is first checked if it is an Elisp symbol and only fallback to the local help otherwise.

I tried this:

(with-eval-after-load 'esh-mode
  (define-key eshell-mode-map [remap display-local-help] #'man))

Then I press C-. h but it says Not bound. I think the Elisp symbol transformer must be improved. It should check if a symbol is indeed defined in some sense, e.g., if it is bound, fbound, a face or has some documentation attached.

EDIT: I think intern-soft is not good enough.

https://github.com/oantolin/embark/blob/34e54d19bb94030c6fc307100c40d828f8cb5c6b/embark.el#L882-L885

EDIT2: I think a better check is this.

(when-let (sym (intern-soft name))
  (or (boundp sym) (fboundp sym) (symbol-plist sym)))

[oantolin]

I mean, sure there are some small number of false positives, but in general symbols don't tend to get created unless they are bound to something, so I find intern-soft a pretty good approximation. We can definitely change it if necessary. I've been using the remapping for a while in eshell buffers and I don't think I noticed any examples of it giving me the wrong kind of help (of course, maybe you interned a bunch of unbound symbols in your session for some reason).

[oantolin]

I have no problem with tightening up the check, but I am surprised you have so many unbound symbols interned, that does not seem to be the case in my Emacs usage.

[minad]

The problem is probably my installation, since I have a package which stores command argument completions (like pcmpl-args). The commands are stored as symbols. This renders the eshell help ineffective. In my setup:

(intern-soft "cat") -> cat
(boundp 'cat) -> nil
(fboundp 'cat) -> nil
(symbol-plist 'cat) -> nil

Another alternative but then it gets really ugly:

(when-let (sym (intern-soft name))
  (or (boundp sym) (fboundp sym) (facep sym)
      (documentation sym)
      (documentation-property sym 'variable-documentation)
      (documentation-property sym 'group-documentation)
      (documentation-property sym 'face-documentation)))

[oantolin]

Mmh, do you expect many symbols to have variable-documentation but not be boundp?

[minad]

I think checking boundp, fboundp and symbol-plist is the right approach, since these are essentially all the namespaces we've got in Elisp. All other symbols are just interned but have no information directly attached. Can we try this tightened check for a while and see if it works well or if we get complaints?

(when-let (sym (intern-soft name))
  (or (boundp sym) (fboundp sym) (symbol-plist sym)))

[oantolin]

Do face names pass that test? We probably want to include face names.

[oantolin]

It seems faces have a face-defface-spec property so the symbol-plist test catches them. I'm satisfied, let's try that test.

[minad]

Yes, from what I've seen groups and faces and everything also use symbol-plist behind the scenes. The docstrings are also attached via such properties.

[oantolin]

OK, the test has been tightened up: 857c8403529f54bfeecf460fe98b1a0b8e206802

[oantolin]

The problem is probably my installation, since I have a package which stores command argument completions (like pcmpl-args).

What package is that?

[minad]

What package is that?

A private one. It is unpublished since there exists pcmpl-args and I didn't want to publish a replica. I made this before I became aware of pcmpl-args. I still use it since I find pcmpl-args a bit overengineered and I have difficulties fixing bugs there.

;;; pcmpl-help.el --- Parse --help output for shell completion -*- lexical-binding: t -*-

;; Copyright (C) 2022  Free Software Foundation, Inc.

;; Author: Daniel Mendler <mail@daniel-mendler.de>
;; Maintainer: Daniel Mendler <mail@daniel-mendler.de>
;; Created: 2022
;; Version: 0.1
;; Package-Requires: ((emacs "27.1"))
;; Homepage: https://github.com/minad/pcmpl-help

;; This file is part of GNU Emacs.

;; This program is free software: you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.

;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with this program.  If not, see <http://www.gnu.org/licenses/>.

;;; Commentary:

;; Complete command line options by parsing the --help output of commands.

;;; Code:

(require 'pcomplete)
(eval-when-compile
  (require 'subr-x)
  (require 'cl-lib))

(defvar pcmpl-help--cache nil)

(defsubst pcmpl-help--consume (regexp &optional back)
  "Consume REGEXP and move BACK."
  (when (looking-at regexp)
    (goto-char (- (match-end 0) (or back 0)))))

;; TODO This can be written in a nicer style. PRs welcome.
;; Extend this if slightly different --help output formats are discovered
(defun pcmpl-help--parse-options ()
  "Parse help output and return list of candidates."
  (let (options case-fold-search) ;; case-sensitive!
    (while (< (point) (point-max))
      (let (help opts arg)
        (pcmpl-help--consume "[ \t]*-" 1)
        (while
            (and
             (pcmpl-help--consume "\\(--?[[:alnum:]#+][[:alnum:]#_\\.+-]*\\)=?")
             (push (match-string 1) opts)
             (cond
              ;; Separated argument with equal sign, `-opt=arg`
              ((= (char-before) ?=)
               ;; Allow spaces in quoted arguments!
               (when (pcmpl-help--consume "\\(\\(?:<.+?>\\|{.+?}\\|'.+?'\\|\".+?\"\\|\\[.+?\\]\\|[[:alnum:]_.,-]+\\)+\\|[^ \n\t,<{['\"]+\\)\\(?:,[ \t]*\\|[ \t]or[ \t]\\|[ \t]+\\|[ \t]*$\\)")
                 (setq arg (match-string 1))))
              ;; Next option
              ((pcmpl-help--consume "[ \t]+-" 1))
              ;; Space separated argument and help, `-opt arg  help`
              ((pcmpl-help--consume " \\([^ \n\t,]+\\)[ \t][ \t]+[^ \n\t,-]" 1)
               (setq arg (match-string 1)))
              ;; Uppercase argument, `-opt ARG`
              ((pcmpl-help--consume " \\([0-9A-Z][0-9A-Z_,.-]*\\)\\(?:,[ \t]*\\|[ \t]+\\|[ \t]*$\\)")
               (setq arg (match-string 1)))
              ;; Quoted args, allow spaces!
              ((pcmpl-help--consume "\\(?:,?\\|[ \t]*\\)\\(\\(?:<.+?>\\|{.+?}\\|'.+?'\\|\".+?\"\\|\\[.+?\\]\\)+[.[:alnum:]]*\\)\\(?:,[ \t]*\\|[ \t]or[ \t]\\|[ \t]+\\|[ \t]*$\\)")
               (setq arg (match-string 1)))
              ;; Comma separator
              ((pcmpl-help--consume ",\\([ \t]*\\|$\\)"))
              ;; Comma or space separated options `-opt1arg -opt2', `-opt1 arg, -opt2`, `-opt1 arg -opt2'
              ((pcmpl-help--consume "[ \t]*\\([^ \n\t,]+\\)[ \t,][ \t]+-" 1)
               (setq arg (match-string 1)))
              ;; Trailing comma `-opt1 arg,'
              ((pcmpl-help--consume "[ \t]*\\([^ \n\t,]+\\),[ \t]*$")
               (setq arg (match-string 1))))))
        (setq help (point))
        (end-of-line)
        (setq help (and opts (buffer-substring help (point))))
        (when (< (point) (point-max)) (forward-char))
        (when opts
          (setq arg (string-trim (or arg "") "[ \t,]+" "[ \t,]+") )
          (setq arg (replace-regexp-in-string "[ \t]+" " " arg))
          (when (> (length arg) 20)
            (setq arg (substring arg 0 20)))
          ;; When no help string is available and the next line does not
          ;; start with - use the next line as help string.
          (when (and (equal help "") (looking-at "[ \t]+[^ \n\t-]"))
            (setq help (buffer-substring (point) (progn (end-of-line) (point))))
            (when (< (point) (point-max)) (forward-char)))
          (setq help (string-trim help))
          ;; Only take first sentence
          (when-let (pos (string-match-p "\\.\\(?: \\|\\'\\)" help))
            (setq help (substring help 0 pos)))
          ;; For example Python separates the help text with :
          (when (string-match "\\`[:-][ \t]+" help)
            (setq help (substring help (match-end 0))))
          (setq help (replace-regexp-in-string "[ \t]+" " " help))
          (when (> (length help) 60)
            (setq help (substring help 0 60)))
          (dolist (opt (nreverse opts))
            (unless (assoc opt options)
              (let ((long (car (last opts))))
                (push
                 (if (and (equal help "") (= (length opt) 2) (> (length long) 2))
                     ;; When no help string is available, use the long option as help.
                     (list opt arg (format "same as %s" (string-remove-suffix "=" long)))
                   (list opt arg help))
                 options)))))))
    (nreverse options)))

(defun pcmpl-help--format-options (options)
  "Format OPTIONS."
  (cl-loop
   with max-arg = (cl-loop for (_opt arg _help) in options maximize (length arg))
   with max-help = (cl-loop for (_opt _arg help) in options maximize (length help))
   for (opt arg help) in options collect
   (progn
     (put-text-property 0 (length arg) 'face 'font-lock-variable-name-face arg)
     (put-text-property 0 (length help) 'face 'font-lock-comment-face help)
     (cons opt
           (concat " "
                   arg
                   (and (> max-arg 0) (make-string (- max-arg -1 (length arg)) ?\s))
                   help
                   (make-string (- max-help (length help)) ?\s))))))

(defun pcmpl-help--options (cmd)
  "Compute options for CMD."
  (or (alist-get cmd pcmpl-help--cache nil nil #'equal)
      (setf (alist-get cmd pcmpl-help--cache nil nil #'equal)
            (pcmpl-help--format-options
              (with-temp-buffer
                (call-process cmd nil t nil "--help")
                (goto-char (point-min))
                (pcmpl-help--parse-options))))))

(defvar pcmpl-help--commands
  '(automake awk basename bash bzcat bzip2 cat cc chgrp chmod chown chroot cksum clang cmp comm convert cp cpp csplit ctags curl cut cvlc date dd df diff dir dircolors dirname du echo egrep emacs env etags expand expr factor false ffmpeg fgrep file find fmt fold gawk gcc gdb gpg grep groups gunzip gzip head hostid hostname htop id install jar java join killall last lastlog ld ldd link ln locale logname ls lua m4 make man md5sum mkdir mkfifo mknod mktemp more mount mv netstat nice nl nm nohup nvim objcopy objdump od passwd paste patch pathchk perl pgrep pinky pr printenv printf pstree ptx pwd python rake readelf readlink renice rg rgrep rm rmdir rsync ruby sdcv sed seq sha1sum sha224sum sha256sum sha512sum shellcheck shred sleep sort split ss stat strings strip stty su sudo sum sync tac tail tar tee texindex touch tr traceroute true truncate tsort tty umount uname unexpand uniq unlink uptime users vdir vim vlc w wall watch wc wget whatis whereis who whoami xargs xev xmodmap xxd xz yes zcat)
  "Enable --help parsing for all the commands in this list.")

;;;###autoload
(defun pcmpl-help--completions ()
  "Return a list of completions for the current argument position."
  (catch 'pcomplete-completions
    (when (pcomplete-parse-arguments pcomplete-expand-before-complete)
      (if (= pcomplete-index pcomplete-last)
          (funcall pcomplete-command-completion-function)
        (let* ((cmd (funcall pcomplete-command-name-function))
               (sym (intern-soft cmd))
               (opts (and sym
                          (memq sym pcmpl-help--commands)
                          (string-match-p "\\`--?" (pcomplete-arg 'last))
                          (pcmpl-help--options cmd))))
          (if opts
              (lambda (str pred action)
                (if (eq action 'metadata)
                    `(metadata (annotation-function
                                . ,(lambda (cand) (cdr (assoc cand opts)))))
                  (complete-with-action action opts str pred)))
            (ignore
             (pcomplete-next-arg)
             (funcall (or (pcomplete-find-completion-function cmd)
                          pcomplete-default-completion-function)))))))))

;;;###autoload
(advice-add 'pcomplete-completions :override #'pcmpl-help--completions)

(provide 'pcmpl-help)
;;; pcmpl-help.el ends here

[minad]

The symbols are in pcmpl-help--commands :-P