Allow using `completion-styles` in Helm and implementation of own helm style

[joaotavora]

Expected behavior

Helm's completion engine can use SLY's completion functions as a backend. The original issue is here: https://github.com/joaotavora/sly/issues/214

Actual behavior (from emacs-helm.sh if possible, see note at the bottom)

The value for completion-at-point-functions is sly-complete-symbol which does the completion itself. This does not trigger the problem. It only happens in other situations such as when SLY's sly-read-symbol-name calls completing-read which Helm overrides via completing-read-function. Actually it used to happen because I've turned off that overriding in that particular context context.

But the root problem remains and I guess some users would like to use Helm's UI for completing SLY symbols. However, when trying

(helm-comp-read "foo " (sly--completion-function-wrapper sly-complete-symbol-function))

Nothing interesting happens. sly--completion-function-wrapper is supposed to produce a reasonably well-behaved completion-function/table. It works well with SLY's own sly-complete-symbol

But it probably isn't well behaved enough, or not to Helm's expectations. I want you to help me understand what's missing.

In particular SLY's lower-level sly-flex-completions and sly-simple-completions functions, which request completions over the network, will return an empty list of completions when the pattern is the empty string, i.e. they only start operating once you feed them at least a character.

Perhaps Helm expects the full completion list to be handed to it up front. But that is impractical because it's frequent that a running SLY session has many thousands of symbols available. Is there a way for Helm to behave more incrementally, a little bit like company does?

Also sly-flex-completions will perform "scatter-match" completion instead of "prefix" completion, which is what sly-simple-completions does.

Steps to reproduce (recipe)

./emacs-helm.sh
(add-to-list 'load-path "~/Source/to/sly")
(require 'sly-autoloads)
M-x sly ; this requires a `lisp` program somewhere in your path, possibly sbcl

Backtraces if any (M-x toggle-debug-on-error)

N/A

Describe versions of Helm, Emacs, operating system, etc.

MELPA helm-20190627.758 A reasonably fresh Emacs 27 Windows 10

Are you using emacs-helm.sh to reproduce this bug (yes/no):

yes

[thierryvolpiatto]

João Távora notifications@github.com writes:

(metadata (display-sort-function . identity) (category helm-completion sly-completion))

Ok, so this is wrong as confirmed by Stefan.

SLY's completion table is very limited by the very nature. I don't think it can magically work with any other category that links to a style that is not backend.

What I was expecting is that if backend style fail to match (e.g. with pattern=="foo bar") the next cycle in completion-styles is tried until one style (helm style is able to match "foo bar") match something.

With current helm code and helm-completion-style set to emacs, if in emacs-27 you add flex to completion-styles, both helm and flex styles work happily together:

You can in same session enter "def key", "key def" "defkey" etc.., all are accepted and match among other things define-key, so that's fine, but in sly, only sly completion is working (e.g. as soon you enter a space, you have no more completion).

-- Thierry

Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[joaotavora]

Thierry Volpiatto notifications@github.com writes:

What I was expecting is that if backend style fail to match (e.g. with pattern=="foo bar") the next cycle in completion-styles is tried until one style (helm style is able to match "foo bar") match something.

I see. See my other mail to Stefan (you are in CC, I think).

The thing is: once you pass "foo bar" to SLY's Common Lisp part (the "backend"), it gives all the completions that match it according to its matching heuristics. It doesn't know, indeed it can't know if and how you want to do filtering later.

The only solution to that would be to request all symbols from the Common Lisp part. And we've already established that that is impractical.

The problem is repeated exactly in Eglot/LSP.

With current helm code and helm-completion-style set to emacs, if in emacs-27 you add flex to completion-styles, both helm and flex styles work happily together:

That's great! But notice that helm's styles and Emacs's styles are completing things -- file names, Elisp symbols, elements in a list -- that are in memory or at least "not very far away". The integral set of such things is fast to fetch, and small in relative terms.

In the "backend" situation, the set is much "farther away" and may be really really big (believe me, there are Lisp applications larger than Emacs, even an Emacs with many packages).

The "backend style"" is really a "non-style", a "sorry-but-you-have-to-give-up-on-styles" style. Do I explain myself?

[thierryvolpiatto]

João Távora notifications@github.com writes:

With current helm code and helm-completion-style set to emacs, if in emacs-27 you add flex to completion-styles, both helm and flex styles work happily together:

That's great! But notice that helm's styles and Emacs's styles are completing things -- file names, Elisp symbols, elements in a list -- that are in memory or at least "not very far away". The integral set of such things is fast to fetch, and small in relative terms.

In the "backend" situation, the set is much "farther away" and may be really really big (believe me, there are Lisp applications larger than Emacs, even an Emacs with many packages).

The "backend style"" is really a "non-style", a "sorry-but-you-have-to-give-up-on-styles" style. Do I explain myself?

Yes, thanks.

Another question about flex style:

Is there a sorting function for it? I couldn't find it (I expected it in metadata) and had to write one (using the completing-score prop). Did I miss something?

Thanks.

-- Thierry

Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[joaotavora]

Another question about flex style:

I suppose you mean Emacs's "flex" style, not SLY's right?

Is there a sorting function for it?

There wasn't, but there is now, very recently in Emacs master. However, as Stefan found out, you must arrange for a non-nil metadata object to be passed to completion-all-completions (or completion-try-completions). Then, after that call, you will find the correct sorting function you want to use in the display-sort-function of the metadata alist. That is the sorting function that your should use to present the candidates. Company has this problem (which really wasn't a problem until recently in Emacs master) and I hope @dgutov will fix it soon.

[thierryvolpiatto]

João Távora notifications@github.com writes:

Another question about flex style:

I suppose you mean Emacs's "flex" style, not SLY's right?

Is there a sorting function for it?

There wasn't, but there is now, very recently in Emacs master.

Ah ok, I have updated emacs and now I have it.

However, as Stefan found out, you must arrange for a non-nil metadata object to be passed to completion-all-completions (or completion-try-completions).

Ok done.

Then, after that call, you will find the correct sorting function you want to use in the display-sort-function of the metadata alist.

Ok I see it.

That is the sorting function that your should use to present the candidates.

I don't understand, if the metadata is specified in the completion-all-completions call, the candidates should be already sorted with the display-sort-function ?

Did I miss something?

Here is the function I was using to sort flex candidates:

(defun helm-completion-in-region-sort-flex-candidates (candidates _source)
  "Sort CANDIDATES computed with flex completion-style."
  (sort candidates (lambda (s1 s2)
                     (let* ((str1 (if (consp s1) (car s1) s1))
                            (str2 (if (consp s2) (car s2) s2))
                            (scr1 (get-text-property 0 'completion-score str1))
                            (scr2 (get-text-property 0 'completion-score str2)))
                       (if (and scr1 scr2)
                           (> scr1 scr2)
                         (helm-generic-sort-fn s1 s2))))))

I have actually better results if I use it than if I let completion-all-completions doing the filtering with display-sort-function, but I am unsure if it really does it.

Company has this problem (which really wasn't a problem until recently in Emacs master)

Same here, no problem until now.

Thanks.

-- Thierry

Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

Ah, I see, you are adding the sort fn after computing candidates, so the sort fn is intended to use for the display function (as its name says), it is unfortunately unusable for helm because it doesn't handle cons cell candidate i.e. (DISPLAY . REAL) candidates, but it's fine I can use the function I wrote for this, it does anyway more or less what yours does. Thanks.

[joaotavora]

I can use the function I wrote for this, it does anyway more or less what yours does.

Hmm, not sure what you mean. display-sort-function existed before I added the possiblity for a style to adjust it. You "must" honour it. It takes, as it has always done, as input a list of (possibly propertized) strings.

If you are returning a cons cell, that is a little unfortunate (you should use text properties for this). But you can (funcall emacs-display-fn (mapcar #'car candidates)) and then use a result to reconstruct the associations to the "real" things.

[thierryvolpiatto]

João Távora notifications@github.com writes:

I can use the function I wrote for this, it does anyway more or less what yours does.

Hmm, not sure what you mean. display-sort-function existed before I added the possiblity for a style to adjust it. You "must" honour it. It takes, as it has always done, as input a list of (possibly propertized) strings.

Too bad, I expected that the display-sort-function ran before i.e. in the completion-all-completions call, it could be done in completion--nth-completion.

If you are returning a cons cell, that is a little unfortunate (you should use text properties for this).

No, I am not returning a cons cell, but helm filters may do it.

But you can (funcall emacs-display-fn (mapcar #'car candidates)) and then use a result to reconstruct the associations to the "real" things.

No, this is not usable, also the display-sort-function is also corrupting helm style.

Will see what I can do with all this, thanks.

-- Thierry

Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[joaotavora]

No, this is not usable,

It's not very hard. Just make a new list of strings where some property, say, helm-original-comppoints to your cons-or-maybe-string thing. Then apply display-sort-function to that new list. Then (mapcar (lambda (s) (get-text-property 0 'helm-original-comp s)) new-list) and go on with the processing. The only downside is a new N-long list, but shouldn't be terrible performance wise, it is dwarfed by the O(NLogN) of the sorting itself.

also the display-sort-function is also corrupting helm style.

This I didn't understand. The display-sort-function for helm style shouldn't be provided by the flex style.

@monnier can you weigh in quickly on this issue?

[monnier]

also the display-sort-function is also corrupting helm style. This I didn't understand. The display-sort-function for helm style shouldn't be provided by the flex style. @monnier can you weigh in quickly on this issue?

Sorry, I didn't understand either.

[dgutov]

it is unfortunately unusable for helm because it doesn't handle cons cell candidate i.e. (DISPLAY . REAL) candidates

Couldn't you do something like this?

(defun dsf-to-helm-sort (dsf)
  (lambda (e1 e2)
    (let ((s1 (if (consp e1) (cdr e1) e1))
          (s2 (if (consp e2) (cdr e2) e2)))
      (funcall dsf s1 s2))))

[thierryvolpiatto]

Dmitry Gutov notifications@github.com writes:

it is unfortunately unusable for helm because it doesn't handle cons cell candidate i.e. (DISPLAY . REAL) candidates

Couldn't you do something like this?

(defun dsf-to-helm-sort (dsf) (lambda (e1 e2) (let ((s1 (if (consp e1) (cdr e1) e1)) (s2 (if (consp e2) (cdr e2) e2))) (funcall dsf s1 s2))))

It is what I usually do, but this time this is not usable. The problem is when switching in same session from one sort fn (e.g. flex) to another (helm), also signature is not the same. So I am actually sorting flex candidates with flex sort fn from the result of completion fn and the helm candidats from a transformer (faster because there is less candidates). But I may find some thing better soon.

Thanks.

-- Thierry

Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

Hi Joao,

João Távora notifications@github.com writes:

No, this is not usable,

It's not very hard. Just make a new list of strings where some property, say, helm-original-comppoints to your cons-or-maybe-string thing. Then apply display-sort-function to that new list. Then (mapcar (lambda (s) (get-text-property 0 'helm-original-comp s)) new-list) and go on with the processing. The only downside is a new N-long list, but shouldn't be terrible performance wise, it is dwarfed by the O(NLogN) of the sorting itself.

also the display-sort-function is also corrupting helm style.

This I didn't understand. The display-sort-function for helm style shouldn't be provided by the flex style.

I can pass the display-sort-function for helm style using your completion--adjust-metadata, that's ok.

The problem is the flex display-sort-function; No doubt it is working fine with flex only (it does), however when switching to another style (even one of the basic emacs styles, not necessarily helm) it fails because your sort function doesn't find the text properties to properly sort, for this I suggest to fall back to an alternative fn, this way we can't have error when going out of flex scope, something like this: (This is working perfectly with helm, it allow transition between helm style and flex style and vice versa during the small lap of time when switching to a different style)

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 9a8db078193..c0ef93f0674 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -3481,6 +3481,7 @@ that is non-nil."

 (put 'flex 'completion--adjust-metadata 'completion--flex-adjust-metadata)

+(defvar completion--alternative-sort-fn #'string-lessp)
 (defun completion--flex-adjust-metadata (metadata)
   (cl-flet ((compose-flex-sort-fn
              (existing-sort-fn) ; wish `cl-flet' had proper indentation...
@@ -3491,10 +3492,15 @@ that is non-nil."
                         completions)))
                  (sort
                   res
-                  (lambda (c1 c2)
-                    (or (equal c1 minibuffer-default)
-                        (> (get-text-property 0 'completion-score c1)
-                           (get-text-property 0 'completion-score c2)))))))))
+                  (lambda (s1 s2)
+                     (let* ((str1 (if (consp s1) (car s1) s1))
+                            (str2 (if (consp s2) (car s2) s2))
+                            (scr1 (get-text-property 0 'completion-score str1))
+                            (scr2 (get-text-property 0 'completion-score str2)))
+                       (if (and scr1 scr2)
+                           (> scr1 scr2)
+                         (funcall completion--alternative-sort-fn
+                                  s1 s2)))))))))
     (let ((alist (cdr metadata)))
       (setf (alist-get 'display-sort-function alist)
             (compose-flex-sort-fn (alist-get 'display-sort-function alist)))

Thanks.

-- Thierry

Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[joaotavora]

Dmitry Gutov notifications@github.com writes:

it is unfortunately unusable for helm because it doesn't handle cons cell candidate i.e. (DISPLAY . REAL) candidates

Couldn't you do something like this?

(defun dsf-to-helm-sort (dsf) (lambda (e1 e2) (let ((s1 (if (consp e1) (cdr e1) e1)) (s2 (if (consp e2) (cdr e2) e2))) (funcall dsf s1 s2))))

'dsf' is "display sort function"?? Then this doesn't make sense right? It's not a binary test predicate. It could be, but isn't (probably would have another name if it were).

 `display-sort-function': function to sort entries in *Completions*.
 Takes one argument (COMPLETIONS) and should return a new list
 of completions.  Can operate destructively.

[dgutov]

@joaotavora My bad, you're right, of course.

Though I wonder if there's a good reason behind that choice, and whether test predicate would be more useful, without loss in power.

[joaotavora]

@dutov, no problem. I think @thierryvolpiatto has solved it anyway, there was some confusion between the new flex and helm styles.

(There is also a lot confusion, because there is another off-list email thread with the same subject that includes Stefan, so you are missing pieces of the puzzle. I started that, sorry).

Regarding your test predicate question, :man_shrugging: . No loss in power, but it is much harder to think in terms of test predicates when you want something simple, say, move some element to the top of the list and keep all the remaining ones.

[dgutov]

you are missing pieces of the puzzle

It's no trouble.

No loss in power, but it is much harder to think in terms of test predicates when you want something simple, say, move some element to the top of the list and keep all the remaining ones

I wouldn't say "much". It's a bit more difficult, but display-sort-function is likely never used this way, because they come from completion tables, and this sounds more like user customization. And if a user can write a completion table with a sorting function, then surely they could stomach that bit of complexity. So if performance is the same, we should probably move towards sorting predicates in some future version of c-a-p-f.

[joaotavora]

So if performance is the same

But it's not though, right? Doing something simple as I suggested is O(N) while full sorting is at best O(NlogN). Your "likely never" will likely not hold "forever".

[dgutov]

I'm saying that use case is probably never exercised.

Anyway, this is the first time we've seen this problem, so OK. Maybe wait until we see a second example before coming to conclusions.

[joaotavora]

I'm saying that use case is probably never exercised.

Yes, I understood. Tho if it is, you'll be stuck with sub-optimal perfomance.

Anyway, this is the first time we've seen this problem, so OK. Maybe wait until we see a second example before coming to conclusions.

There isn't a real problem here with d-s-f here, the problem was a confusion between the flex and helm completion styles. Something that I believe @thierryvolpiatto already as a solution for.

[thierryvolpiatto]

João Távora notifications@github.com writes:

I'm saying that use case is probably never exercised.

Yes, I understood. Tho if it is, you'll be stuck with sub-optimal perfomance.

Anyway, this is the first time we've seen this problem, so OK. Maybe wait until we see a second example before coming to conclusions.

There isn't a real problem here with d-s-f here, the problem was a confusion between the flex and helm completion styles. Something that I believe @thierryvolpiatto already as a solution for.

There is no problem with the d-s-f now, I solved the problem by sorting the candidates in the completion function instead of sorting it later in filtered-candidate-transformer. The goal is to be able when in helm to use both flex and helm styles together, e.g. being able to type in minibuffer "defkey" and have appropriate results with flex, and then delete backward and type "def key" and have appropriate results with helm. I think you understood this but wanted to be sure. I actually use the completion--adjust-metadata like flex in minibuffer and wait your implementation with cl-defgeneric (or whatever) to update my code. For now it is working perfectly like this.

• completing-read-multiple => Ok
• completion-at-point in lisp buffers => Ok
• completion-at-point in shell-mode => Ok
• completion-at-point in Sly => Ok (only sly completion as expected)
• completion-at-point in eshell (27) => No (file completion broken dunno why) hopefully we can use helm-completion-styles-alist to workaround this.

Thanks all for your help on this!

-- Thierry

Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[joaotavora]

I think you understood this but wanted to be sure.

Yes, I understand. That's a great achievement!\

Thanks all for your help on this!

This monster issue is fantastic. Look at how far we've come!

EDIT: I wrote some months ago:

"I didn't do a good job of selling you the API. But if you used it, you would share that fancy exclamation mark pattern matching with all the other compliant UIs.."

And now you did! Congratulations.

[monnier]

• completion-at-point in eshell (27) => No (file completion broken dunno why) hopefully we can use helm-completion-styles-alist to workaround this.

If you send me a recipe to reproduce the problem I promise to try and track down the source of the problem ;-)

[thierryvolpiatto]

monnier notifications@github.com writes:

• completion-at-point in eshell (27) => No (file completion broken dunno why) hopefully we can use helm-completion-styles-alist to workaround this.

If you send me a recipe to reproduce the problem I promise to try and track down the source of the problem ;-)

1) Install and run emacs with helm:

git clone https://github.com/emacs-helm/helm.git cd /path/to/helm make && sudo make install helm

Note: If your emacs-27 is not installed, use helm -P /path/to/emacs-27

2) Eval in scratch

(setq helm-completion-styles-alist nil)

to remove the workaround.

While writing helm-completion-styles-alist in scratch you can try completion with M-tab ;-)

3) M-x eshell

Try to cd somewhere hitting tab to have completion.

4) Reenable workaround:

(setq helm-completion-styles-alist ' ((eshell-mode . helm)))

Do the same as in 3), completion in file is now working.

Thanks!

-- Thierry

Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[monnier]

2) Eval in scratch

(setq helm-completion-styles-alist nil)

to remove the workaround.

While writing helm-completion-styles-alist in scratch you can try completion with M-tab ;-)

3) M-x eshell

Try to cd somewhere hitting tab to have completion.

Hmm... here's what I did:

src/emacs -Q                                                    \
          -l ~/src/emacs/elpa/packages/async/async-autoloads.el \
          -l ~/src/emacs/elpa/packages/helm/helm-autoloads.el   \
          --eval '(setq helm-completion-styles-alist nil)'      \
          -f helm-mode                                          \
          -f eshell

then

cd /home/ TAB

and it "worked" but incorrectly: it showed me the list of completions for the pwd (in my case the worktree of my master branch for Emacs) rather than the completions for "/home/".

If I ask for a backtrace with M-: (debug) RET I get something like:

Debugger entered: nil
  eval((debug) t)
  eval-expression((debug) nil nil 127)
  funcall-interactively(eval-expression (debug) nil nil 127)
  call-interactively(eval-expression nil nil)
  command-execute(eval-expression)
  read-from-minibuffer(#("Pattern: " 0 9 (face helm-minibuffer-prompt)) nil (keymap ...) nil nil #("/home/" 0 1 (rear-nonsticky (arg-begin arg-end) arg-begin t) 5 6 (rear-nonsticky (arg-end arg-begin) arg-end t)) t)
  helm-read-pattern-maybe("Pattern: " nil nil noresume (keymap ...) nil nil)
  helm-internal((... ...) nil "Pattern: " noresume nil "*helm-mode-completion-at-p..." (keymap ...) nil nil)
  apply(helm-internal ((... ...) nil "Pattern: " noresume nil "*helm-mode-completion-at-point*" (keymap ...) nil nil))
  helm((... ...) nil "Pattern: " noresume nil "*helm-mode-completion-at-p..." (keymap ...) nil nil)
  apply(helm ((... ...) nil "Pattern: " noresume nil "*helm-mode-completion-at-point*" (keymap ...) nil nil))
  helm(:sources (... ...) :input nil :default nil :preselect nil :prompt "Pattern: " :resume noresume :keymap (keymap ...) :allow-nest nil :candidate-number-limit 100 :case-fold-search smart :history nil :buffer "*helm-mode-completion-at-p...")
  helm-comp-read("Pattern: " #f(compiled-function (str predicate action) #<bytecode 0x1576b62b35a1>) :name "completion-at-point" :nomark t :marked-candidates nil :initial-input nil :buffer "*helm-mode-completion-at-point*" :fc-transformer (helm-cr-default-transformer helm-completion-in-region-sort-fn) :match-dynamic t :fuzzy nil :exec-when-only-one t :quit-when-no-cand #f(compiled-function () #<bytecode 0x1576b624b321>) :must-match t)
  helm--completion-in-region(#<marker at 52 in *eshell*> 58 #f(compiled-function (string pred action) #<bytecode 0x1576b62b4f3d>) nil)
  apply(helm--completion-in-region (#<marker at 52 in *eshell*> 58 #f(compiled-function (string pred action) #<bytecode 0x1576b62b4f3d>) nil))
  #f(advice-wrapper :override completion--in-region helm--completion-in-region)(#<marker at 52 in *eshell*> 58 #f(compiled-function (string pred action) #<bytecode 0x1576b62b4f3d>) nil)
  completion-in-region(#<marker at 52 in *eshell*> 58 #f(compiled-function (string pred action) #<bytecode 0x1576b62b4f3d>) nil)
  completion-at-point()
  funcall-interactively(completion-at-point)
  call-interactively(completion-at-point nil nil)
  command-execute(completion-at-point)

and if I trace completion-all-completions and all-completions I see something like this:

======================================================================
1 -> (completion-all-completions "" #<bytecode> nil 0 (metadata (cycle-sort-function . #3#) (category . file) . #4#))
| 2 -> (all-completions "" #<bytecode> nil)
| 2 <- all-completions: ("test/" "lwlib/" "autom4te.cache/" "m4/" "src/" "doc/" "nextstep/" "etc/" "lib/" "build-aux/" "nt/" "oldXMenu/" "info/" "admin/" "msdos/" "lisp/" "lib-src/" "modules/" "leim/")
1 <- completion-all-completions: ("test/" "lwlib/" "autom4te.cache/" "m4/" "src/" "doc/" "nextstep/" "etc/" "lib/" "build-aux/" "nt/" "oldXMenu/" "info/" "admin/" "msdos/" "lisp/" "lib-src/" "modules/" "leim/" . 0)

so it seems that Helm drops the "/home/" prefix somehow (although in the above trace it does re-appear when helm-read-pattern-maybe calls read-from-minibuffer, so apparently it's stored somewhere, maybe in some global variable?) and doesn't move it into the default-directory either (which would be another way to preserve it).

At this point, I'm not exactly sure where to keep digging, because I don't quite understand how it's supposed to be working. But it seems that the input var might not be properly passed down from helm--completion-in-region to helm-comp-read.

Does that ring a bell?

    Stefan

[thierryvolpiatto]

monnier notifications@github.com writes:

2) Eval in scratch

(setq helm-completion-styles-alist nil)

to remove the workaround.

While writing helm-completion-styles-alist in scratch you can try completion with M-tab ;-)

3) M-x eshell

Try to cd somewhere hitting tab to have completion.

Hmm... here's what I did:

src/emacs -Q \ -l ~/src/emacs/elpa/packages/async/async-autoloads.el \ -l ~/src/emacs/elpa/packages/helm/helm-autoloads.el \ --eval '(setq helm-completion-styles-alist nil)' \ -f helm-mode \ -f eshell

If you installed from Melpa, you can cd to ~/src/emacs/elpa/packages/helm/ and then ./emacs-helm.sh -P /path/to/emacs/src/emacs Once in emacs M-x eshell

then

cd /home/ TAB

and it "worked" but incorrectly: it showed me the list of completions for the pwd (in my case the worktree of my master branch for Emacs) rather than the completions for "/home/".

Yes same here.

If I ask for a backtrace with M-: (debug) RET I get something like:

[...]

so it seems that Helm drops the "/home/" prefix somehow (although in the above trace it does re-appear when helm-read-pattern-maybe calls read-from-minibuffer, so apparently it's stored somewhere, maybe in some global variable?) and doesn't move it into the default-directory either (which would be another way to preserve it).

At this point, I'm not exactly sure where to keep digging, because I don't quite understand how it's supposed to be working. But it seems that the input var might not be properly passed down from helm--completion-in-region to helm-comp-read.

Does that ring a bell?

Hmm, not yet, I will try to dig into this ASAP.

Thanks.

-- Thierry

Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

monnier notifications@github.com writes:

Does that ring a bell?

Yes, thanks it helped, it is fixed now, I was setting base-size at each call of the candidate function whereas it have to be set one time for all at first call, also input should be plain and not the basename of pattern. You can see the changes in dev branch, I will merge soon if I see no bug in next hours (or days).

Thanks a lot for your help.

-- Thierry

Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

Now everything looks ok, eshell completion is fixed, initial completion popup using proper style depending of position of cursor in buffer. Now for sorting what did you decide @joaotavora and @monnier ? Do you want something lke this for minibuffer.el:

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 43dd277a2e4..e4cddef3d8f 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -946,13 +946,21 @@ This overrides the defaults specified in `completion-category-defaults'."
                                    string table pred point)))
                (and probe (cons probe style))))
            (completion--styles md)))
-         (adjust-fn (get (cdr result-and-style) 'completion--adjust-metadata)))
+         (adjust-fn
+          (completion--adjust-metadata (cdr result-and-style))))
     (when (and adjust-fn metadata)
       (setcdr metadata (cdr (funcall adjust-fn metadata))))
     (if requote
         (funcall requote (car result-and-style) n)
       (car result-and-style))))

+(cl-defgeneric completion--adjust-metadata (style)
+  "Returns a suitable function to adjust metadata for STYLE."
+  nil ; Always return nil when no method found for STYLE.)
+
+(cl-defmethod completion--adjust-metadata ((_style (eql flex)))
+  #'completion--flex-adjust-metadata)
+
 (defun completion-try-completion (string table pred point &optional metadata)
   "Try to complete STRING using completion table TABLE.
 Only the elements of table that satisfy predicate PRED are considered.

[thierryvolpiatto]

I would like to have this fixed in emacs-27 before releasing a new helm version.

Thanks.

[joaotavora]

Now for sorting what did you decide @joaotavora and @monnier ?

Good question. We haven't decided yet. But it was going to be something like that, minus the -- of course (beware the --!).

I think the only blocker here is that Stefan is thinking of redesigning the whole thing, and we wants something future proof. Generic functions seem nice enough for this. But if it turns out that in that future redesign it doesn't make any sense for styles to influence metadata objects (say, because metadata objects disappear altogether), then it pays to think as much as possible about this early on.

[thierryvolpiatto]

João Távora notifications@github.com writes:

Now for sorting what did you decide @joaotavora and @monnier ?

Good question. We haven't decided yet. But it was going to be something like that, minus the -- of course (beware the --!).

Yes, it's just a dirty patch, kind a proof of concept.

I think the only blocker here is that Stefan is thinking of redesigning the whole thing, and we wants something future proof. Generic functions seem nice enough for this. But if it turns out that in that future redesign it doesn't make any sense for styles to influence metadata objects (say, because metadata objects disappear altogether), then it pays to think as much as possible about this early on.

Ok, IOW this will not happen soon, right?

-- Thierry

Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[joaotavora]

Ok, IOW this will not happen soon, right?

No I didn't say that. I believe something will happen before the 27 branch is cut. Let's wait for Stefan.

[thierryvolpiatto]

João Távora notifications@github.com writes:

Ok, IOW this will not happen soon, right?

No I didn't say that. I believe something will happen before the 27 branch is cut. Let's wait for Stefan.

Ok, thanks.

-- Thierry

Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[monnier]

Ok, IOW this will not happen soon, right? No I didn't say that. I believe something will happen before the 27 branch is cut. Let's wait for Stefan.

Yes, something will happen soon (probably later this week). It's probably going to look pretty much like what you wrote. I'll let you know when it's done.

[thierryvolpiatto]

monnier notifications@github.com writes:

Ok, IOW this will not happen soon, right? No I didn't say that. I believe something will happen before the 27 branch is cut. Let's wait for Stefan.

Yes, something will happen soon (probably later this week). It's probably going to look pretty much like what you wrote. I'll let you know when it's done.

Great thanks.

-- Thierry

Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

Now the completing-read's are also using completion-styles by default, only the ones specified in helm-completing-read-handlers-alist use something else.

I saw a conversation in emacs-devel (https://lists.gnu.org/archive/html/emacs-devel/2019-11/msg00217.html) speaking of the advantage of flx with initials, note that using both flex and initials in completion-styles is even better and (much) faster than using flx. I think completion-styles is not understood by many (me first some weeks ago) due to its lack of documentation.

[thierryvolpiatto]

Now completion-styles can be used in any helm sources (for now helm-mode have to be activated to have the helm style).

(helm :sources (helm-build-sync-source "test"
                 :candidates (helm-dynamic-completion
                              '(foo bar baz foab)
                              'symbolp)
                 :match-dynamic t)
      :buffer "*helm test*")

[thierryvolpiatto]

Also helm-M-x can now use completion-styles and have both flex (emacs-27 only) and helm styles. Of course standard M-x (execute-extended-command) have completion-styles when helm-completion-style is 'emacs (you have to remove it from helm-completing-read-handlers-alist).

[manuel-uberti]

Thank you for working on this!

Beside setting helm-M-x-use-completion-styles to t and completion-styles to '(flex), is there anything else to do? I already have helm-mode active, by the way.

[thierryvolpiatto]

Manuel Uberti notifications@github.com writes:

Thank you for working on this!

Glad you try this :-)

Beside setting helm-M-x-use-completion-styles to t and completion-styles to '(flex), is there anything else to do?

No, be sure to be on emacs-27 to have flex style.

I already have helm-mode active, by the way.

Yes, for now you need helm-mode active as well.

-- Thierry

Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[manuel-uberti]

So far everything seems to be working fine with these settings. I noticed that setting completion-styles to '(flex) breaks company-mode in CIDER, but I'll report it there not here.

[thierryvolpiatto]

Manuel Uberti notifications@github.com writes:

So far everything seems to be working fine with these settings.

Thanks for testing!

I noticed that setting completion-styles to '(flex) breaks company-mode in CIDER, but I'll report it there not here.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.*

-- Thierry

Get my Gnupg key: gpg --keyserver pgp.mit.edu --recv-keys 59F29997

[thierryvolpiatto]

I noticed that setting completion-styles to '(flex) breaks company-mode in CIDER, but I'll report it there not here. @dgutov and @bbatsov ?

[manuel-uberti]

I reported it here: https://github.com/company-mode/company-mode/issues/932

[joaotavora]

@thierryvolpiatto @manuel-uberti

If you read up to the very first few comments of this enormous thread, you will see that CIDER will suffer from the exact same problem as SLY, i.e. it can't support arbitrary Emacs completion styles. Or rather, the only way it can possibly support them is if it decides to fetch the whole (presumably very large) complete symbol space from the Clojure process that supports it.

So basically, CIDER's tables should be using the backend completion style, which unfortunately is not yet in Emacs 27 (or GNU Elpa). The backend completion could be named a give-up-on-styles-and-use-whatever-the-backends-style-is, but the word "backend" was found to be shorter.

In the meantime, CIDER can define the backend style itself, just like SLY does.

[manuel-uberti]

Thank you for the information @joaotavora. I did see CIDER mentioned, you're right. I hope your details can be of any help to @bbatsov then.

[dgutov]

but the word "backend" was found to be shorter

We could call it raw?

[joaotavora]

We could call it raw?

That's better, indeed. Myabe a bit too dry, or ...raw. The name "bypass" might be even better.

[monnier]

We could call it raw? That's better, indeed. Myabe a bit too dry, or ...raw. The name "bypass" might be even better.

The name matters because it is something that end users can choose by placing it (or removing it) from their completion-styles and completion-category-overrides. So we should choose the name such that it can make sense to the user.

As a user I prefer "backend" to "raw", but I'm probably not the most qualified users to choose. Other options could be "external", "non-emacs", "native", ...

I think "raw" and "native" both suffers from the fact that depending on what's your point of view you can either expect them to be "raw/native to Emacs" or "raw/native to the backend".

-- Stefan "who still remembers endless hours of discussions around what raw meant in some software design of his"

[dgutov]

On 20.11.2019 15:43, monnier wrote:

As a user I prefer "backend" to "raw", but I'm probably not the most qualified users to choose. Other options could be "external", "non-emacs", "native", ...

I think "raw" and "native" both suffers from the fact that depending on what's your point of view you can either expect them to be "raw/native to Emacs" or "raw/native to the backend".

I'd say "raw to Emacs" doesn't have the same ring (or the degree of possible confusion), but I'm fine with "backend" as well. Or "external", for that matter.