Handling of dynamic minibuffer-completion-table

[clemera]

When minibuffer-completion-table is a function the candidates might be created dynamically. Currently selectrum--normalize-collection gets the candidates from the table only once but this fails if the candidates are computed dynamically. On the other hand it is slow and unnecessary to recompute big tables like help--symbol-completion-table on each input change.

[raxod502]

Hmmm... how often are dynamic completion tables used? There was one in find-file, which I worked around by implementing a custom selectrum-read-file-name. After all, if the completion table is dynamic, there's a good chance that it's doing something fancy and specific to the default completing-read paradigm, and really needs to be ported. Perhaps this is rare enough that we can simply implement all of them individually and be done with it?

[clemera]

how often are dynamic completion tables used?

I don't know, I stumbled across this issue when someone on reddit shared a command for locate.

Perhaps this is rare enough that we can simply implement all of them individually and be done with it?

It is probably rare but I think it's kind of bad/annoying if commands using default completing-read don't work as expected. Maybe recomputation could be done with an idle timer and use while-no-input, which could be smooth enough.

An other easier workaround would be to provide a command similar the recently introduced selectrum-exhibit function. This could be used to trigger the recomputation of the candidates manually via a keybinding.

[raxod502]

Your suggestions are good. Any of that would be fine. Another option would be to simply recompute the table on every input change, but have a customizable list of table symbols that are known to be static.

[clemera]

I have experimented and while the timer approach works I don't like that the filtering behaves like helm (the output is noticeable delayed). It feels much more snappy with the current approach. I think dynamic minibuffer tables where the candidates can change with input are to rare to justify this decline in UI experience. I would prefer to default to the current behaviour and then maybe having the option to declare some special tables as needing to use the timer approach. But maybe having a command to recompute the current table on demand would suffice already (this is also how those tables where designed to be used because with default completion they are only recomputed on TAB press, too).

When a candidates function gets passed to selectrum-read we currently compute it on every input. This isn't problematic as long those dynamic functions are fast enough. When we decide to have a variable which declares completion tables to be computed with a timer this option should probably also affect the computation of dynamic candidate functions so we have a single way to configure timer based computation.

[raxod502]

Ok, good to know about the timer approach. I agree we don't want to go the Helm way. There are a couple of good options, as you point out -- it's a balance between making things work out of the box / without extra keystrokes being required, allowing extensibility for people to deal with cases not supported properly, and not having more options than are really needed. I think you've got more context than me about what the right thing to do is.

[minad]

What is needed to support completing-read with a dynamic generation function in selectrum? I cannot implement consult-buffer based on completing-read it seems, since the generation function is only called once.

Independent of that, you discuss timer based solutions - I would prefer not to do such a thing on the level of the selection framework. I think rather the generating function should take care of caching. But I don't know of current uses of generation functions, are they all fast or should the completion system handle potential slowness of the generation function?

[clemera]

What is needed to support completing-read with a dynamic generation function in selectrum? I cannot implement consult-buffer based on completing-read it seems, since the generation function is only called once.

Essentially the dynamic function needs to be called on each input change. This is what we do when you pass a function to selectrum-read. It works when the completion function doesn't require a lot of computation but otherwise this will make things stutter. The idea with the timer is that you wait for a time slice where the users stops typing and then do the computation. Are you able to make consult-buffer work with helm (using regular completing-read)?

But I don't know of current uses of generation functions, are they all fast or should the completion system handle potential slowness of the generation function?

Some built-in tables like help--symbol-completion-table used by C-h f can get quite big/slow, a timer solution would work around that but would also introduce a delay before candidates are displayed as in helm.

[minad]

I haven't tried helm for a long time. But slowness was certainly a big issue. For now I am not concentrating on consult-buffer.

[clemera]

I suggested it as it should work with all dynamic completion tables AFAIK. But I probably should have suggested icomplete which is probably be the most API compliant incremental narrowing framework as of today.

[ackerleytng]

I'm trying to implement something like helm-projectile-ag (or rg/grep, etc), and I believe having being able to dynamically generate the completion list would be nice too!

For now, I have something like this (I'm calling it projectile-ag because it depends on projectile to find within the project, and my understanding is that one of selectrum's tenets is that people should just be able to use completing-read, hence there shouldn't be a dependency on selectrum - but names aside...)

(require 'projectile)

(defun projectile-ag--marked-input ()
  "Return the marked input (anything highlighted in buffer), nil if nothing is marked."
  (when (use-region-p)
    (buffer-substring-no-properties (region-beginning) (region-end))))

(defun projectile-ag--search-input ()
  "Get the input for search."
  (or (projectile-ag--marked-input)
      (thing-at-point 'symbol t)
      ""))

(defun projectile-ag--do-ag (s path)
  "Call ag with search string S on PATH and return all results.  Return a list of results (strings)."
  (with-temp-buffer
    (let ((exit-code (process-file "ag" nil t nil "--nocolor" "--nogroup" s path)))
      (if (= exit-code 0)
          (split-string (buffer-string) "\n" t)
        '()))))

(defun projectile-ag--projectile-ag ()
  "Call ag with search input on projectile's project root."
  (let ((input (projectile-ag--search-input)))
    (completing-read
     "ag: "
     (projectile-ag--do-ag input (projectile-project-root))
     nil t input)))

And then I realized that completing-read can actually take a function to generate the list of completion options? so I tried something like this

(defun projectile-ag--search-results (string pred flag)
  "Stop STRING PRED FLAG."
  pred
  flag
  (message string)
  (if (equal string "111")
      (message "yay")
    (message "nay"))
  (if (equal string "111")
      '("foo" "bar")
    '("baz" "quux")))

(completing-read
 "ag: "
 #'projectile-ag--search-results
 nil t "1")

And although the function is executed every time (the messages appear) the completion options in the minibuffer don't change.

Also, I agree with @minad that the generating function should take care of the caching if necessary.

[clemera]

And then I realized that completing-read can actually take a function to generate the list of completion options?

Yes, but note that the completion table gets called multiple times to query information from it, so you need to take care that the process isn't invoked unnecessarily. To help with this there is the helper function completion-table-dynamic and there is also a wrapper which handles caching for you named completion-table-with-cache:

(completing-read
 "Locate: "
 (completion-table-with-cache
  (lambda (str)
    (process-lines
     "locate" "-i" "-l" "10" (concat "*" (replace-regexp-in-string " +" "*" str) "*")))))

But there is a problem with this as I just realized recently. Selectrum wants do its own matching on the results while the completion code in Emacs very much assumes the default completion UI (with the assumption that your input is a prefix of the matches). completion-styles also come into play but even before that the candidates above get filtered before using all-completions. This means for example when calling the above table with bin as the input you just don't get a match:

(funcall
 (completion-table-with-cache
  (lambda (str)
    (process-lines
     "locate" "-i" "-l" "10" (concat "*" (replace-regexp-in-string " +" "*" str) "*"))))
 "bin" nil t)

But calling it with the empty string will call the process with the empty string which is not what you would want. With Selectrum we would want to get the candidates returned by the process and then filter them afterwards according to user settings. I don't have a solution for this currently :(

[clemera]

There is also the problem that there exists no real try-completion action with tables such as above. Emacs default UI expects to first insert a common substring (if there is one) before presenting you all completions. Because of this the example above will trigger errors with some completion-styles in default completion. I came up with one example which works with the default UI and also would work with Selectrum when we would pass the string to the table by using a table specific completion-style:

(defun check-try (string _table _pred point &optional _metadata)
  (cons string point))

(defun check-all (string table pred _point)
  (all-completions string table pred))

(defun test (str pred action)
  (setq-local completion-styles '(check))
  (setq-local completion-styles-alist
          '((check check-try check-all "Checking")))
  (when (eq action t)
    (all-completions 
     ""
     (process-lines
      "locate" "-i" "-l" "10" (concat "*" (replace-regexp-in-string " +" "*" str) "*"))
     pred)))

(completing-read "Test: " 'test)

But this is ugly and also for other cases we wouldn't want to pass the string like for example help--symbol-completion-table would prefilter the symbol on prefix while in Selectrum we want to get all symbols and decide how to filter them. I start to think that the way completion and dynamic tables currently work in Emacs itself are not suited for dynamic candidate sets we are looking for here at all.

[minad]

@clemera I think we could try to first get the simple cases to work? I thought about writing a consult-match function which expects the user to enter a regexp/string, then the candidate set is generated dynamically. Maybe after pressing space, the dynamic generation is stopped and the normal filtering continues - or do everything completely dynamic. I started something here https://github.com/minad/consult/tree/consult-match, but it is static as of now.

[clemera]

I don't have any simple examples I could test with, do you have any tables at hand that work with default completion but not with selectrum?

[minad]

No, but I would for example try to implement this consult-match thing I mentioned above. And then we could see if that is possible to support in selectrum.

[clemera]

Sounds good, I will look out for more examples, too.

[clemera]

BTW I just noticed the completion-styles hack would also be an alternative way to implement consult--buffer I think.

[minad]

BTW I just noticed the completion-styles hack would also be an alternative way to implement consult--buffer I think.

You mean to fix the completion style such that it has special support for the prefix used by consult--buffer? I thought about something like this in this old discussion https://github.com/oantolin/orderless/issues/24#issuecomment-739492540

But I like the solution we are having now for narrowing very much - did you see the improvements I added regarding backspace and showing a fancier narrowing indicator. I don't think there is much to be improved here. The only thing missing from the current implementation is the feature to show invisible buffers by pressing SPC - this will require dynamic candidates. But I don't think I will add support for that. How often do you switch to an invisible buffer? I rather concentrate on a good solution for the most common cases.

[clemera]

But I like the solution we are having now for narrowing very much - did you see the improvements I added regarding backspace and showing a fancier narrowing indicator.

No I did not, I have to check. I also like the current implementation its simple and robust, I think, also less expensive because the table does not need to be requeried.

[minad]

Yes, I agree. The more things we can do statically, the more robust it will be. And yes, also potentially more performant. But for example for icomplete I don't except a difference, even if there is a recomputation it will just return the same cached list.

But still - I think at some point we need a solution for dynamic candidates, e.g. for consult-match and consult-rg/grep. There are not that many other commands on the wish list of consult which would need it, but these two I think are reasonably useful. Furthermore the question is also what other dynamic commands are there in the Emacs code base or inside commonly used packages.

[clemera]

But for example for icomplete I don't except a difference, even if there is a recomputation it will just return the same cached list.

It does? I thought it only caches the very last computation of completions but I may be wrong.

But still - I think at some point we need a solution for dynamic candidates, e.g. for consult-match and consult-rg/grep. There are not that many other commands on the wish list of consult which would need it, but these two I think are reasonably useful. Furthermore the question is also what other dynamic commands are there in the Emacs code base or inside commonly used packages.

Yes, we should collect some examples here and then go from there.

[clemera]

But still - I think at some point we need a solution for dynamic candidates, e.g. for consult-match and consult-rg/grep.

My experiments above indicate that it might be quite hard to get such a dynamic completion table working with the default API. Maybe a different interaction model could also make sense for these commands. You could read the string passed to the process first and then get the candidates which you can then narrow with the completion UI. This would also make sense regarding the search syntax which is different between completion UI and the external process. As long as it is easy to edit the search string passed to the process I think I would like this.

[ackerleytng]

Here's my implementation, which prompts for a search input if it there isn't anything marked or at the cursor

https://gist.github.com/ackerleytng/290c29ac951c18d859593a9414f88fe7

I think in terms of UX it isn't that bad since most people know what they're searching for. It's probably a nice tradeoff for simplicity and performance.

Aside: I'm new to elisp, please comment on my coding style and let me know what naming conventions I should follow, or which repo you think I should contribute this to! Thanks!

[minad]

@ackerleytng Just to be clear - you ask first for the search string and then you present the options as a static list of candidates? This is a good first step and certainly an option when we cannot get dynamic candidates to work.

Aside: I'm new to elisp, please comment on my coding style and let me know what naming conventions I should follow, or which repo you think I should contribute this to! Thanks!

Please contribute this to Consult. But we have to wait a bit until we merge this since it makes sense to first explore if we can make it work with dynamic tables.

[ackerleytng]

Yes, the search string step (first step) is completely separate from the searching and filtering (second step).

I'm not actually using anything from consult though - the magic of consult is in "previewing" the search with context in the buffer, right? The implementation doesn't actually do any previewing - how will consult handle opening files that have not previously been opened by emacs?

[minad]

@ackerleytng We have to figure that out - currently I avoided that and only preview already open buffers in order to avoid expensive loading. But technically there is no problem to open files and preview them.

If you don't want to make it part of consult that's also okay, but the plan is to add a consult-rg command at some point so maybe you can help with that. And maybe some of the other consult commands turn out to be useful for you.

[ackerleytng]

I'd be happy to contribute - let's continue the discussion at https://github.com/minad/consult/pull/68

[clemera]

Just to be clear - you ask first for the search string and then you present the options as a static list of candidates? This is a good first step and certainly an option when we cannot get dynamic candidates to work

@minad I would even prefer this approach UI wise (because of search syntax mismatch and responsiveness). Getting dynamic candidates to work and also have a grep command using it would of course be nice nevertheless.

[minad]

From my perspective it is not necessary to implement this. We manually trigger refreshing of the ui in consult for asynchronous commands. We do this for both icomplete and selectrum. It is solid and works well.

[jyp]

This bug is affecting the wordnut package. The relevant call is:

(defun wordnut--completing (input)
  (let ((completion-ignore-case t))
    (completing-read "WordNut: "
             (completion-table-dynamic 'wordnut--suggestions)
             nil nil input 'wordnut-completion-hist)))

I'd like to insist that this bug should be fixed. One of the main advantage of selectrum wrt. similar packages is that it respects the standard emacs interface. But this bug shows otherwise.

[alphapapa]

@clemera This is off-topic, but thanks to this example you posted, I was finally able to make a dynamic completion command for Org QL: https://github.com/alphapapa/org-ql/commit/4f5fbc49e6546939685aa7ff2c07ce609fc1d08e Thank you very much for sharing it.

May I suggest that you contribute it more permanently somewhere, like in the Elisp manual, or even just a blog post? I had to do a bit of Web searching before I finally found this issue (using Google). The minimal example was key to my being able to make it work; I would have had to dig through source code and Edebug for probably hours more before coming up with that, because the Elisp manual section on programmable completion just isn't enough.