Completion like flx?

[hjudt]

I am trying to switch from ido to vertico. I've spent quite some time configuring ido, and so far it worked great for my taste, but vertico and other completion systems seem to be the new fad (no offense meant), resulting in some issues with ido in conjunction with e.g. packages that somehow expect the user uses vertical-style completions (like org-roam). So I've tried to somehow replicate my current ido setup with vertico. One thing I miss is flx completion, which is a great enhancement to ido. Where it shines is e.g. I enter plp and it automatically finds package-list-packages, without me having to enter p-l-p, which is more awkward/inconvenient. Is something like that available via completion styles or vertico extensions? I have found flex but this is not flx?

[minad]

I am trying to switch from ido to vertico. I've spent quite some time configuring ido, and so far it worked great for my taste, but vertico and other completion systems seem to be the new fad (no offense meant), resulting in some issues with ido in conjunction with e.g. packages that somehow expect the user uses vertical-style completions (like org-roam).

First, if something works great for you, I don't recommend switching. Note that org-roam also works perfectly fine with vertico-flat-mode which provides a Ido-like horizontal style display. So I don't see why Ido-completing-read (horizontal display) cannot handle this?

Is something like that available via completion styles or vetico extensions? I have found flex but this is not flx?

Filtering is implemented by the completion-styles. My personal recommendation is @oantolin's excellent Orderless completion style. For me this is perfect since in most cases, it just matches the right set of candidates. Orderless can also be configured to support flex-filtering, however without sorting.

But it is understandable if this is not for everyone and if you prefer a pure flex style filtering and sorting. As you already figured out, this is implemented by the Emacs builtin flex completion style. (Unrelated to flex there are multiple other styles you may want to try, e.g., initials, partial-completion and substring.)

I have found flex but this is not flx?

Please elaborate - what do you mean by "this is not flx"? I close this issue since it is not a Vertico issue. Feel free to continue the discussion. I hope my answer is helpful!

[hjudt]

I am trying to switch from ido to vertico. I've spent quite some time configuring ido, and so far it worked great for my taste, but vertico and other completion systems seem to be the new fad (no offense meant), resulting in some issues with ido in conjunction with e.g. packages that somehow expect the user uses vertical-style completions (like org-roam).

First, if something works great for you, I don't recommend switching. Note that org-roam also works perfectly fine with vertico-flat-mode which provides a Ido-like horizontal style display. So I don't see why Ido-completing-read (horizontal display) cannot handle this?

I actually intend to switch to a vertical-style display completion extension, trying to get accustomed to it. After all, it often only requires a change in habits. The new version of org-roam seems to be designed for vertical-style completion, because certain data will be shown only when using this, and you have to install workarounds for ido. See here for more info: https://github.com/org-roam/org-roam/issues/1778

Now that is all fine for me, I'm always eager to try new things and can quickly switch back if it doesn't work for me. I'm also interested in the benefits that vertico and related packages can give me.

Is something like that available via completion styles or vetico extensions? I have found flex but this is not flx?

Filtering is implemented by the completion-styles. My personal recommendation is @oantolin's excellent Orderless completion style. For me this is perfect since in most cases, it just matches the right set of candidates. Orderless can also be configured to support flex-filtering, however without sorting.

Maybe I can get the desired behaviour if I spend more time on configuring completion styles or categories. I need to look further into this.

But it is understandable if this is not for everyone and if you prefer a pure flex style filtering and sorting. As you already figured out, this is implemented by the Emacs builtin flex completion style. (Unrelated to flex there are multiple other styles you may want to try, e.g., initials, partial-completion and substring.)

I have found flex but this is not flx?

Please elaborate - what do you mean by "this is not flx"? I close this issue since it is not a Vertico issue. Feel free to continue the discussion. I hope my answer is helpful!

Yes, your answers are indeed highly appreciated. I have not found another place to discuss this, so I started an issue here.

As for flx, I mean this: https://github.com/lewang/flx The page has a link to a screencast demonstrating the package (with flx-ido), you might skip a bit forward on this though to get at the interesting part (start at 2 min or 2:30 so). Anyway, I'll try orderless or maybe other styles and see if they give me the same convenient feedback.

[hjudt]

It seems orderless does what I want, but I need to type spaces between the initials. E.g. package-list-packages => "p l p" instead of "plp". I guess this is ok too, especially since recent completions are being remembered.

[minad]

As for flx, I mean this: https://github.com/lewang/flx

flx is obsolete in the context of completing-read, with flex being the substitute in the form of a completion style.

It seems orderless does what I want, but I need to type spaces between the initials. E.g. package-list-packages => "p l p" instead of "plp". I guess this is ok too, especially since recent completions are being remembered.

You can enable the orderless-flex matching style (see the orderless readme) or the orderless-initialism matching style. Alternatively the flex completion style supports the matching you want to achieve (plp matching package-list-packages). So I am not sure what you are asking for. Try the configuration as given in the README.

;; option 1 
(setq completion-styles '(basic substring flex))

;; option 2
(setq completion-styles '(orderless)
         orderless-matching-styles ...)

See also https://www.gnu.org/software/emacs/manual/html_node/emacs/Completion-Styles.html

[hjudt]

Thanks for your help, this is exactly what I have been looking for. Now happily using orderless-matching-styles...

[hjudt]

One last question: How can I find out which completion categories there are so that I configure overrides? The emacs manual only tells about styles.

[minad]

Good question. Grep? Emacs itself does not define many categories. We define many as part of Marginalia, see https://github.com/minad/marginalia/blob/7608acbe6f8cca3c484d05b5d21d27319ba1ed4f/marginalia.el#L82.

[hjudt]

Great, thanks for the hint. So far, I was able to configure most vertico/consult stuff to my liking with category overrides and style dispatchers. I will stay with company-flx instead of orderless/simple capf though, as the nifty scoring it does improves usability in programming modes a lot. Orderless just presents too many useless options, which flx gets right immediately without any configuration (though it has other quirks that orderless does not have in other use cases).

[minad]

Personally I use Corfu instead of Company but I am biased of course ;)

But Corfu with Orderless also does not yet remember the latest matches.

[oantolin]

@hjudt If you like flex completion but dislike orderless's opinionated "completion styles should not do any sorting" stance, you could try the hotfuzz completion style. It prioritizes matches at word (or camelcase component) boundaries.

[hjudt]

I am unsure what I like more, hotfuzz or flx, and will continue trying hotfuzz for some time to see how that feels. hotfuzz seems to sometimes have more exact matches than flx regarding what I expect, sometimes it is the other way round. I will keep using orderless for M-x and buffer switching and probably everything else because I've been using smex till now which allowed me to select recently used entries, and orderless has that feature too, while hotfuzz seems not to. I would miss this very much.

I also need to use company for lisp development with slime/sly, otherwise I wouldn't mind switching to corfu.

After all, I also think my previous ido setup has really been quite good and effective, nevertheless the other packages can nearly achieve the same with some tweaking and are better in some aspects because of higher configurability. I also like very much that they depend on built-in functionality, learning about all this gave me more insight into the completion system and now I understand a lot of things better, and I am able to tweak company/corfu/whatever.

[oantolin]

I've been using smex till now which allowed me to select recently used entries, and orderless has that feature too

Orderless doesn't do any sorting at all, not even putting recent matches at the top. Since this is the Vertico repo I assume you are using Vertico, and then Vertico is what is putting the recent matches at the top.

[minad]

I've been using smex till now which allowed me to select recently used entries, and orderless has that feature too, while hotfuzz seems not to. I would miss this very much.

Sorting by history is actually provided by Vertico as documented in the README.

I also need to use company for lisp development with slime/sly, otherwise I wouldn't mind switching to corfu.

I am not saying you should switch but lisp development should work with Corfu. If something does not work, I'd appreciate a bug report.

After all, I also think my previous ido setup has really been quite good and effective, nevertheless the other packages can nearly achieve the same with some tweaking and are better in some aspects because of higher configurability.

So what is missing from your perspective? Note that the packages like Vertico+Orderless+Embark+Marginalia+Consult etc. give you much more than Ido, and in a more general and better integrated form. However replicating Ido or any kind of other system was and is not a goal. Each feature should be justifiable on its own. So it would be odd if you wouldn't miss some details from Ido, in particular if you used Ido for a long time and got accustomed to all its features and quirks.

[oantolin]

I am not saying you should switch but lisp development should work with Corfu.

I've used Corfu with both Slime and Sly, and it works just fine, or at least did when I tried (I mostly use consult-completion-in-region).

[minad]

@oantolin So what do you dislike about Corfu? Always looking for well-reasoned criticism... ;) My main gripe I have with it are occasional display issues due to the Emacs display engine. Maybe I should experiment with these weird double buffering settings that Emacs has. Besides that I put my hope into the recently merged pgtk display.

[oantolin]

The only very minor issue I had with Corfu was that if I was near the bottom of the window the popup could flip-flop between being above or below the current line. But that almost never happens to me since I tend to center the function I'm working on. I think I mainly don't use Corfu just because consult-completion-in-region works just fine, I already have consult installed, and it gives me the same familiar completion interface I always use. So it's not really a case of their being anything about Corfu that doesn't work for me, it's just that I find consult-completion-in-region satisfactory so why switch?

[minad]

@oantolin Makes sense! Corfu is also an oddity in comparison to the more conventional Emacs buffer displays. The flipflop is a good point. Maybe the window should stay above, is this what you meant?

[oantolin]

Also, I have probably years of getting used to having completion in region use the minibuffer: if I recall correctly consult-completion-in-region is based on a function from my configuration I used for a very long time. It's entrenched, I'm not used to having a popup, I'd have to get used to it.

[oantolin]

@oantolin Makes sense! Corfu is also an oddity in comparison to the more conventional Emacs buffer displays. The flipflop is a good point. Maybe the window should stay above, is this what you meant?

It should stay on one side, doesn't matter which. I guess a good rule would be to stay above if point is in the bottom 10 lines, otherwise stay below. Something like that.

[minad]

Indeed, you contributed consult-completion-in-region to Consult.

[minad]

It should stay on one side, doesn't matter which. I guess a good rule would be to stay above if point is in the bottom 10 lines, otherwise stay below. Something like that.

I will try that.

[hjudt]

I am not saying you should switch but lisp development should work with Corfu. If something does not work, I'd appreciate a bug report.

Ok, thanks I will try after I have explored vertico and related packages a bit more.

After all, I also think my previous ido setup has really been quite good and effective, nevertheless the other packages can nearly achieve the same with some tweaking and are better in some aspects because of higher configurability.

So what is missing from your perspective? Note that the packages like Vertico+Orderless+Embark+Marginalia+Consult etc. give you much more than Ido, and in a more general and better integrated form. However replicating Ido or any kind of other system was and is not a goal. Each feature should be justifiable on its own. So it would be odd if you wouldn't miss some details from Ido, in particular if you used Ido for a long time and got accustomed to all its features and quirks.

At the moment, I cannot think of anything I am missing, but I need to gain more experience. I only wanted to say that I was quite satisfied with my ido setup, even without fiddling around with it too much. I know replicating ido behaviour is not the goal of vertico et al, still I will use my old setup as a base for comparison, because I was very content with it. I think that the new packages can give me more than ido and its related packages too, but I will need to gain more experience before being able to judge what I like and what not. So thanks for your ideas and suggestions, I will try them and find out, and after a while probably report back from my "journey". What's more, thanks for writing/maintaining vertico and your help.

[hjudt]

After two days of usage without really changing my initial configuration, here are some impressions:

• hotfuzz is a bit slower than flx. In fact, it makes completion-at-point a bit laggy. This is especially noticeable when you delete characters backwards with completion active. company + company-flx shows nearly no lag or at least less lag when writing/deleting.
• When using hotfuzz for command completion, vertico will not display recent commands. Why the heck is that? Anyway, I only tested this and will stick to using orderless+initialism for this.
• consult-buffer: I needed to turn off previews for recent files because it seems to automatically establish a lot of ssh remote connections, which I do not want (depending on location, remote hosts might be offline or respond slowly which will make emacs hang for some time). In general, I probably will turn off other previews too because it causes too much flashing for my taste, though some are really useful. Gladly this is highly configurable, maybe I can also restrict previews to non-remote resources. Otherwise it replaces ido + virtual buffers for fast buffer switching quite nicely. I keep sticking to ibuffer for heavy-duty management of buffers because it displays in a bigger buffer and has great filtering capabilities.

[hjudt]

@minad, @oantolin Thank you for your discussions about company, corfu and consult-completion-in-region. I remember before trying company years ago, I used hippie-expand and found it cool enough ;-) Then I saw company and that it displayed completion candidates at point like many modern IDEs do and found it looked/worked cool that way. After trying vertico et al, I think I should reconsider this and start trying each of those various options for some time. I am not sure about having completions shown in other buffers or the mini-buffer, because it might be quite some way off the screen area I might be working on. I guess that's one of the main points of company/corfu, to have it presented immediately where you type. On the other hand, it might just take some time for getting used to it. Anyway, I guess I should try embark next.

BTW: Seems org-roam fixed their bug so it works for ido users (and also users of other non-vertical completion frameworks). So theoretically I could switch back, but now both my horizontal and vertical views of reality have been spoiled ;-)

[minad]

After trying vertico et al, I think I should reconsider this and start trying each of those various options for some time.

Seems generally like a good approach.

I am not sure about having completions shown in other buffers or the mini-buffer, because it might be quite some way off the screen area I might be working on. I guess that's one of the main points of company/corfu, to have it presented immediately where you type.

Yes, this is why I made Corfu for in-buffer completion.

Anyway, I guess I should try embark next.

Note that all these packages are quite orthogonal and independent. Embark is one of my favorite packages. Even if you drop the vertical packages in the end going back to your more familiar setup, you may still want to keep Embark, if you like actions at point. But from what I've heard and from my own experience, Embark takes a while to get accustomed to, despite the intuitive general idea.

[hjudt]

There is one ido feature that I miss and that I'd like to replicate with vertico/consult: If you open a file, or want to create a directory, then I can "escape" the completion system using a key (don't remember exactly, but like hitting C-f again), and thus switch back to the original find-file or dired-create-directory, without the completion system getting in my way. I could then hit the up key, for example, to recall a previous value from the history very quickly. Do consult/vertico also allow this somehow? I know I can switch buffers and get a recent file, but that's not the same.

[minad]

Yes I recall that Ido offers a way to disable itself somehow. This feature does not exist in Vertico but you could write a similar command which disables the Vertico display and keymap. However such a feature seems pointless since Vertico already supports everything the default completion system supports, e.g., walking through the history. Are you sure this is not an XY problem? What do you want to achieve?

In other words, there shouldn't be a scenario where the completion system gets in your way. If there is, it is likely you are using it wrong or you trying to achieve something very special. But maybe there is already a solution, if you tell me precisely what you want to achieve.

[hjudt]

Ok. One thing I often do:

1. Find a file. C-x C-f ~/my-new-file.txt
2. Save, kill file buffer.
3. Now I often notice I forgot something, need to reopen exactly this file. Easy in ido, just C-x C-f, then again C-f to exit ido and call traditional find-file, hit up key to get to recent entry.

How can I achieve step 3 without having to reenter the file path? Imagine the file is not at an easy accessible location, but 20 directories deep (~/my/very/nested/directory/my-new-file.txt). With vertico, I could probably go back using C-x b and select virtual buffer for recently closed file, but that also doesn't work always, and I am very used to being able to do it with C-x C-f. Maybe I did not save the file, for example, but just killed it, but now I decided I still want to create it.

[hjudt]

Ah ok, I can use M-n and M-p.... It seems I have read that somewhere but forgot about it...

[minad]

Ah ok, I can use M-n and M-p.... It seems I have read that somewhere but forgot about it...

Yes, exactly. All completion features remain available with Vertico as is, except for some keybindings which are overwritten on purpose, like in this case the up/down key. Fortunately the history search and history cycling still exist on other keys.

[minad]

You may also want to use consult-history, which you could bind to C-c h (or M-s/M-r only in the minibuffer) to search through the history with completion.

[hjudt]

Thanks for the hint, I'll try that.

[hjudt]

I've now tried corfu, and it rocks! I've replaced company with it, so thanks for that good stuff. Also, I've become more accustomed to orderless and am quite satisfied with the results, so I do not need that flx stuff anymore.

[ghosty141]

@hjudt If you like flex completion but dislike orderless's opinionated "completion styles should not do any sorting" stance, you could try the hotfuzz completion style. It prioritizes matches at word (or camelcase component) boundaries.

Sorting is a huge pita right now...

1. hotfuzz doesn't do highlighting as of now (if you don't use selectrum) -> this is a dealbreaker for me.
2. writing your own sorting function that plugs into vertico doesn't seem to be easy, there is basically no documentation on "how" to do this. Especially one which is as fancy as hotfuzz'.
3. Orderless stance would be more ok if the ecosystem would support it but sadly it's far from that. Everything that does sorting is kinda tied to a completion-system (prescient -> selectrum , flx -> ido/ivy, helm in it's own little world, etc. etc.).

@minad It would be great if you could explain this topic a bit. I'd be interested in getting some experience with elisp and the whole completion system.

[oantolin]

Have you tried the built-in flex completion style, @Ghosty141? I think you might like its sorting. (In general I highly recommend trying packages built-in to Emacs before looking to external packages, because there are plenty of "forgotten" gems that come with Emacs.)

[minad]

@Ghosty141 You are mixing up older and unrelated projects, which may lead to the impression that the situation is worse than it actually is. My recommendation is to follow the description here: https://github.com/minad/vertico/#complementary-packages. Rely only on the builtins and packages which are compliant with the Emacs APIs!

1. I consider hotfuzz as an experimental completion style. It is pretty new and recently got a dynamic module since the initial implementation proved as too slow. Highlighting should be fixed on the side of hotfuzz. In order to implement this efficiently, lazy highlighting is required as performed by Vertico, see vertico--all-completions. Please ask the author of hotfuzz about this if he is interested in adding highlighting.
2. You can overwrite the default Vertico sort function. By design, you shouldn't use this mechanism to sort by candidate ranking (how well the candidate matches the input). The Vertico sort function exists purely to determine the primary ordering of the candidates. This candidate order can then be altered by completion styles like flex or hotfuzz such that the best match appears at the top.
3. Orderless is supported by completion systems and completions tables which are compatible with the default Emacs completion machinery. Please don't mix up different projects here. Selectrum+Prescient is an alternative completion system which does not use the default Emacs machinery. Instead it implements its own filtering infrastructure which is incompatible with many completion commands. I developed Vertico in particular due to the complexity entailed by that and the compatibility issues. Ido is an obsolete completion system which also uses its own machinery. Note that Ido by default only supports file and buffer completion, it is hardly comparable to the UIs which implement full completing-read. Ivy also uses its own sorting and regexp compiler for filtering. Helm in comparison uses completion styles as far as I know. This makes it possible to use Orderless with Helm (https://github.com/oantolin/orderless/issues/105). Vice versa, one could use the Helm completion style with Vertico or the default completion UI.

To summarize the situation - there is an Emacs default API and completion machinery (implemented in minibuffer.el) which is used and supported by most commands. It is your best bet to use that and use a completion UI which is fully compliant. The fully compliant completion UIs currently are: Vertico, Mct, builtin Icomplete and builtin default completion. My projects try to adhere closely to these bases APIs provided by Emacs to ensure best compatibility. Personally I use Vertico+Orderless. Orderless doesn't alter the candidate ordering, which I prefer. I still get recent candidates at the top thanks to Vertico's sorting by history.

[ghosty141]

@minad Thanks for the write up :) Looking back, my comment got a bit too rambly, sorry for that.

2. Ah that makes sense. So basically it's the completion styles "job" to handle meaningful/advanced sorting and vertico only does slight "post-processing".

3. I absolutely agree with your stance on the scope of vertico and it's goal, I really like it and it's truly a breath of fresh air. According to this issue: orderless_#80 the completion source or UI should handle sorting and orderless purely provides candidates -> my impression is, that the source should handle sorting. Is this correct?

In that case hotfuzz would be a good example right? (obviously just meant as an example which I know of)

@oantolin Yeah I did. I too follow that mindset but flex sadly doesn't offer the functionality I want.

For example: I have a custom defun "e-emacs-config". If I search for "e-e" I get "hide-entry" while "e-emacs-config" is only at the fourth place. What kinda works is partial-completion but that doesn't show results where the query-string is in the middle of the word (at least in my case I gotta move the cursor to the first place or add a *).

[minad]

2. Yes, but I would say "pre-processing", since Vertico sorts first and then the completion style may alter that order to move better matches higher up.
3. I am not sure what you want to achieve. If the "source" (also called a completion table) knows the most recent candidates or the most frequently used candidates, the completion table can pass the sorted candidates to completing-read. Furthermore the completion table should ensure that Vertico does not apply default sorting by setting the display-sort-function and cycle-sort-function to identity in the completion metadata. In contrast, the sorting which is performed by flex (or hotfuzz) is not related to the completion table and happens afterwards when Vertico performs filtering/sorting via the completion style.

[ghosty141]

My goal is basically: fuzzy searching in the minibuffer where the sorting is based on the quality of the match. Vertico provides the UI while flex (for example) would provide the fuzziness. The problem for me is that the sorting flex does, cannot be configured the way I like. I ended up here beacause in the issue I linked above the completion UI (vertico) was also mentioned as a component which can do sorting.

Right now I'm wondering what needs to be done if I'd want custom sorting for a completion style like flex. From what I've understood so far, the completion table should implement sorting, so bascially a plugin or smth is needed to do this.

[minad]

Right now I'm wondering what needs to be done if I'd want custom sorting for a completion style like flex. From what I've understood so far, the completion table should implement sorting, so bascially a plugin or smth is needed to do this.

No, the solution is not to adjust the completion table. The correct solution is to implement a completion style which is better than flex, with better ranking and thus better sorting. Hotfuzz is one such attempt at a better completion style.

[oantolin]

What kinda works is partial-completion but that doesn't show results where the query-string is in the middle of the word (at least in my case I gotta move the cursor to the first place or add a *).

partial-completion doesn't do any sorting either, in this example it simply doesn't match the hide-entry candidate.

[hjudt]

@Ghosty141 I have also found hotfuzz and flx more convenient in the beginning and thought they would give me better results. Well, in many cases, they really do. However, that might just be a matter of habit, and what I find more useful now are two things: First, the speed of "unordered" completion and the ability to narrow the search; Because there can always be candidates which do not get listed in a position I would expect them to be. So being able to refine your search seems more important to me than to get proper hits in the first results. Next, there might always be candidates which the fuzzy/flex search styles can miss or appear at the far bottom of the list, but which might be exactly what you have been looking for. What certainly is a very important feature is the sorting of recent entries at the top, for easy accessibility. Although sometimes I wish there would be a way to get rid of some of these too because they have only been a one-time lookup.

[minad]

@hjudt I agree with you. There is mostly no need for flex matching. It is better to have a deterministic filtering which does not alter sorting. Orderless is the perfect solution here and therefore I recommend it wherever I can.

What certainly is a very important feature is the sorting of recent entries at the top, for easy accessibility. Although sometimes I wish there would be a way to get rid of some of these too because they have only been a one-time lookup.

Yes, sorting by recency is provided by Vertico for that reason. Furthermore Embark allows you to remove entries from the history with the action embark-history-remove. This allows you to get rid of such one time lookups.

[hjudt]

I have one other minor issue: When renaming a file in dired (hitting R for rename), I get completion candidates pre-selected that are existing files and that somehow match (maybe because I use partial completion for files?). If I do not want that, I can press C-a to go to the beginning of the prompt, and the candidates will vanish and I can simply hit enter to confirm the new name. This feels a bit awkward. Is there some other way to deal with this and similar cases?

Apart from that, I've recently tried embark, and this really presents a great and refreshing new way of working!

[hjudt]

Ok, I got so confused by the many options and features that I missed what is written on the main page:

Non-existing candidates can be submitted with M-RET or by moving the point to the prompt.

This of course solves a few of my problems ;-)

[hjudt]

Furthermore Embark allows you to remove entries from the history with the action embark-history-remove. This allows you to get rid of such one time lookups.

Somehow, this does not work or does not give the expected results. Actually, if I examine the minibuffer-history variable, I cannot find the string there after embark-history-remove. But if I then type M-x and look at the completion candidates, then that candidate is still listed there. Do I need to alter something else to finally get rid of these one-time lookups?

[minad]

There is a flx-completion package proposed for MELPA: https://github.com/melpa/melpa/pull/8029. By turning flx into a completion-style it will become usable by Vertico. The advantages compared with the builtin flex style are unclear to me.