counsel-company doesn't work well with the new "flex" completion style in Emacs 27.

[AmaiKinono]

I am using Emacs 27.0.50. A new "flex" completion style is added in Emacs 27 recently:

An implementation of popular "flx/fuzzy/scatter" completion which matches strings where the pattern appears as a subsequence. Put simply, makes "foo" complete to both "barfoo" and "frodo". Add 'flex' to 'completion-styles' or 'completion-category-overrides' to use it.

So I added 'flex to completion-styles. Here's it in action using counsel-company:

You can see that "package" didn't disappear after selecting the candidate.

[basil-conto]

AFAIK, Ivy lacks support for various built-in completion features such as programmed completion, completion-styles, completion-regexp-list, completion-category-overrides, :exit-function, display-sort-function, etc.

Some of these could probably be added without much difficulty. For example, support for :annotation-function was added not too long ago. Others have (unfortunately) been deliberately ignored in the design of Ivy.

AFAICT, Ivy reinvents completion-styles via ivy-re-builders-alist; see (ivy) Completion Styles. As such, you're probably looking for (ivy) ivy--regex-fuzzy.

[AmaiKinono]

Thanks. I have noticed ivy has several completion styles, but they may not meet my requirement.

Let me explain it. Sometimes we can't remember a symbol's name accurately, and it would be great if we can get it by one or two words in it, maybe in the middle of it. ivy--regex-ignore-order fits this situation perfectly, but, we need to have candidates first, then counsel-company can do its filtering stuff on them.

With a "plain" completion-at-point function (i.e. without fuzzy style), we have to type the beginning several letters of the name, then we have the candidates. So even with ivy--regex-ignore-order, you still have to know the beginning part of the symbol. This is why I am trying to use counsel-company with the flex style, which is obviously not perfect, since I only want "you can type the middle part first", not the fuzzy stuff.

It will be great if you have any thought on this!

[basil-conto]

It will be great if you have any thought on this!

I'm not familiar with company or counsel-company, but I can tell you that the following:

With a "plain" completion-at-point function (i.e. without fuzzy style), we have to type the beginning several letters of the name, then we have the candidates. So even with ivy--regex-ignore-order, you still have to know the beginning part of the symbol.

is a shortcoming of ivy-completion-in-region, as compared to the default completion-in-region-function (see completion--do-completion).

[AmaiKinono]

I've found someone created a better counsel-company. I read his code, and found that:

• add :predicate (lambda (candidate) (string-prefix-p prefix (car candidate))) fixes this problem.
• add :caller 'counsel-company fixes #2051.

I am not familiar with ivy-read, so I am not sure if these changes are reasonable. @basil-conto would you like to look into it? If this is the right thing to do, I will (try to) create a PR.

[basil-conto]

I've found someone created a better counsel-company.

@fuxialexander Would you be interested in contributing your improvements upstream?

I am not familiar with ivy-read, so I am not sure if these changes are reasonable. @basil-conto would you like to look into it? If this is the right thing to do, I will (try to) create a PR.

Sorry, I'm not familiar with company or counsel-company, and only @abo-abo controls the source. If you think you have a solution, please PR.

[abo-abo]

@basil-conto

Others have (unfortunately) been deliberately ignored in the design of Ivy.

It wasn't deliberate, as far as I remember. Ivy is based on icomplete-mode (500 lines of code, no support for mentioned features). I don't oppose adding those features to Ivy as long as they have a use case.

[abo-abo]

@AmaiKinono

So I added 'flex to completion-styles. Here's it in action using counsel-company:

Please show more precisely what this means. And any extra config as well.

[AmaiKinono]

@abo-abo

Please show more precisely what this means.

It means:

(add-to-list 'completion-styles 'flex)

If you are using a lower version, you can do this instead:

(add-to-list 'completion-styles 'substring)

It allows you to type bar and have foobar as a candidate. The problem is the same.

And any extra config as well.

I didn't do much:

• company: I have not do much about it, just (global-company-mode 1) and some tweaks for the appearance. Didn't do anything with the backends.
• counsel: I have keybindings to various counsel-commands, including counsel-company. counsel-mode is not used.
• ivy: ivy-mode is used, and:

  (setq
  ivy-height 7
  ivy-dynamic-exhibit-delay-ms 250)

Please tell me if more information is needed.

[abo-abo]

@AmaiKinono It seems this issue was fixed in #2051. Please test.

[AmaiKinono]

@abo-abo I think no. I pulled the latest master branch, tried both "flex" and "substring" style, and the behavior is exactly the same: In emacs-lisp mode, type (package, call counsel-company, type ^use, and press RET. It becomes (packageuse-package.

[abo-abo]

With emacs-26.1, I did make plain and used this config:

(setq ivy-re-builders-alist
      '((t . ivy--regex-ignore-order)))
(setq completion-styles '(substring basic partial-completion emacs22))

I entered (package and pressed C-M-i. Entering use RET works as expected.

[AmaiKinono]

The problem still exist on my machine, both in Emacs 27 and 26.2. I think maybe I did something wrong with my config, so I created a minimal init.el and it reproduces the problem:

;; I use straight.el as my package manager, here's where it builds the packages into. These are latest.
(add-to-list 'load-path "~/.emacs.d/straight/build/ivy/")
(add-to-list 'load-path "~/.emacs.d/straight/build/swiper/")
(add-to-list 'load-path "~/.emacs.d/straight/build/counsel/")
(add-to-list 'load-path "~/.emacs.d/straight/build/company/")

(require 'counsel)
(require 'company)

(setq ivy-re-builders-alist
      '((t . ivy--regex-ignore-order)))

(setq completion-styles '(substring basic partial-completion emacs22))

(global-set-key "\C-t" 'counsel-company)

Now start Emacs, go to scratch buffer, type something like (company, press C-t (minibuffer goes up), type counsel, RET, and it becomes (companycounsel-company.

Edit: I think with make plain, you are calling completion-at-point with C-M-i, that command actually works fine. But I am having problem with counsel-company.

[basil-conto]

Others have (unfortunately) been deliberately ignored in the design of Ivy.

It wasn't deliberate, as far as I remember.

See, for example, the discussion in #255.

I don't oppose adding those features to Ivy as long as they have a use case.

The use case is handling the Emacs completion system properly instead of reinventing it.

[basil-conto]

@abo-abo @AmaiKinono

the following:

With a "plain" completion-at-point function (i.e. without fuzzy style), we have to type the beginning several letters of the name, then we have the candidates. So even with ivy--regex-ignore-order, you still have to know the beginning part of the symbol.

is a shortcoming of ivy-completion-in-region, as compared to the default completion-in-region-function (see completion--do-completion).

The reproduction is simple, and requires no external packages.

Substring

1. emacs -Q
2. (add-to-list 'completion-styles 'substring) C-j
3. (message C-M-iC-M-i

Note that format-message is included in the list of completions.

1. make plain
2. (setq ivy-re-builders-alist '((t . ivy--regex-ignore-order))) C-j
3. (message C-M-i

Note that format-message is not included in the list of completions.

4. C-hfDELmessage

Note that format-message is included in the list of completions.

Flex

1. emacs -Q
2. (add-to-list 'completion-styles 'flex) C-j
3. (ssag C-M-i

Note that format-message is included in the list of completions.

1. make plain
2. (setq ivy-re-builders-alist '((t . ivy--regex-fuzzy))) C-j
3. (ssag C-M-i

Note that ivy-completion-in-region finds no completions.

4. C-hfDELssag

Note that format-message is included in the list of completions.

[abo-abo]

@basil-conto For substring with make plain you forgot (add-to-list 'completion-styles 'substring). When I add it, I can select format-message.

[basil-conto]

For substring with make plain you forgot (add-to-list 'completion-styles 'substring). When I add it, I can select format-message.

The same is true in the flex scenario, but why should this be needed for in-buffer completion and not minibuffer completion? AFAIK Ivy has never explicitly heeded completion-styles, so having to customise a separate completion mechanism to get in-buffer completion to work in the same way as minibuffer completion sounds a bit suspicious, as if it might lead to unforeseen inconsistencies or problems. Or am I overreacting? If so, perhaps the connection with or dependency on completion-styles should be advertised more prominently.

[abo-abo]

I think the relationship to completion-styles is via all-completions which Ivy uses to get a list of candidates.

[basil-conto]

I think the relationship to completion-styles is via all-completions which Ivy uses to get a list of candidates.

Sure, but that is an implicit implementation detail, not something the user is aware of. Ivy says "if you want out-of-order substring filtering, use ivy--regex-ignore-order." It does not additionally say "also customise completion-styles."

Furthermore, that still doesn't explain the difference in behaviour between ivy-completing-read and ivy-completion-in-region. If one doesn't heed completion-styles, then neither should the other. Conversely, if one depends on completion-styles, then so should the other. At least, that's what I would expect as a user. This is why I'm saying that there's something up with ivy-completion-in-region.

[abo-abo]

Sure, but that is an implicit implementation detail, not something the user is aware of. Ivy says "if you want out-of-order substring filtering, use ivy--regex-ignore-order." It does not additionally say "also customise completion-styles."

Thing is, I didn't know about completion-styles before. I just assumed that completion-at-point is prefix-based, so you can't get use-package out of (package-|.

completion-styles can be seen as part of the collection function, i.e. outside of ivy. Once ivy gets a list of strings, we really know what to do, and we don't need completion-styles any more. But with completion-at-point, this initial list is computed using completion-styles.

[AmaiKinono]

Sure, but that is an implicit implementation detail, not something the user is aware of.

In fact I didn't find that confusing (as an average user). My understanding is, it's ivy's work to do filtering on the candidates, but the candidates themselves are provided by mechanisms of Emacs. You tweak the behavior of the former using ivy-re-builders-alist, the latter using completion-style. It's reasonable and straightforward.

The problem is the way ivy do its action doesn't work well with "flex" and "substring" completion style, as is shown in the beginning and https://github.com/abo-abo/swiper/issues/2054#issuecomment-489656170. This problem is more urgent in terms of the user experience.

[basil-conto]

Sure, but that is an implicit implementation detail, not something the user is aware of.

In fact I didn't find that confusing (as an average user).

I never implied it's confusing, rather that it isn't required for any other Ivy or Counsel feature that I'm familiar or have used in the last few years as an Ivy user. I don't see why in-buffer completion should be different to minibuffer completion in this regard.

My understanding is, it's ivy's work to do filtering on the candidates, but the candidates themselves are provided by mechanisms of Emacs.

Not always, and there is often more than one mechanism, as seen in this discussion.

You tweak the behavior of the former using ivy-re-builders-alist, the latter using completion-style. It's reasonable and straightforward.

Again, this is the first time I come across a need for tweaking completion-styles in Ivy. I'm not saying this situation is unacceptable, but it's definitely uncommon/unusual, and this warrants some further (low-priority) investigation into whether there is an alternative approach that could be taken.

[AmaiKinono]

@basil-conto I think I don't fully understand this discussion, because I am not familiar with ivy itself. I am a long time spacemacs user and just started to create my own configuration from scratch not long ago :joy: I am having trouble with counsel-company so I am just focusing on it, hoping my problem being solved. but seems you are talking in a bigger perspective. So sorry I can't offer further viewpoint about it.

[basil-conto]

So sorry I can't offer further viewpoint about it.

There's nothing to apologise for, indeed parts of this discussion belong in their own issue, which this issue could link to.

[abo-abo]

Thanks, please test.

[AmaiKinono]

Yes, that fixes it! :tada: