Idea: Async CAPFs = ACAPFs

[jdtsmith]

I see you have as a TODO supporting company's async-flavoring. An interesting idea is to extend this to vanilla CAPFs as well, with a simple extra :async property, to make... async-capf's or ACAPFs.

The idea would be to extend the CAPF mechanism to allow it to return (if it thinks it can eventually complete something there) a null table, but include an :async callback-receiver property in the return:

(list start end nil :async callback-receiver)

This would "break nothing" for non-supporting UIs (though you'd get no completions). A supporting UI would pass the callback-receiver function it receives its own (unique!) callback function — a function which will accept just one argument: capf-result. The unique callback could be for example a closure that includes a request-id or some other disambiguator, so the UI knows which request is giving it results.

When the ACAPF eventually has some candidates (from a process or remote server, say), it can complete the CAPF operation by calling this callback with a regular CAPF response ((funcall callback `(,start ,end ,table ,@extra-props))), and then everything proceeds as a normal CAPF would have. The table could be a function (if it wants to specify some metadata and do something fancy) or just a simple collection, like always. The UI can either drop this request if the buffer has in the meantime changed, such that the completions are "no longer relevant", or pop-up the UI, if they have arrived "in time". Or a UI could allow the ACAPF to itself make that call in its table function (e.g. if the original region text includes a prefix of the current probe text), and only provide "relevant" completions. I'd suspect the former would be more robust.

To support both async-capable and "regular-old" completion UI's, the CAPF-author could wrap the async CAPF with a simple "polling" wrapper which supplies the ACAPF its own simple callback, and polls for completion (with timeout), before returning the full capf-result, seemingly synchronously.

[jdtsmith]

Do you envision backend authors need to tailor themselves to support either sync or async? Backend authors can either write synchronous or asynchronous backends. Both could be supported by Corfu/Cape/Company. So the answer is yes, but I don't know what you are actually asking.

I think you figured me out despite my poor wording, but I'll rephrase perhaps more simply: are you authoring a new CAPF API for general use by many future UIs, or a single point of entry wrapper to translate ACAPFs -> CAPFs, such that UI's don't even know what happens (or both)?

So yes, who will wrap with cape-async-capf? A frontend like corfu, automatically? The user, in their config file? The backend author? And if the latter, are they then stuck with a backend that will otherwise NOT work well with e.g. company or default completion? Will you hope/expect company, for example, will directly support ACAPFs internally, such that backend-wrapping is counter-indicated? Or is it more desired to produce a CAPF which any current user-chosen UI (most of which will not interrupt) can make use of — "wrap once, use anywhere"? That will be important to document to get any adoption I suspect.

I'm happy to try out my (still unreleased) ipython mode as an async guinea pig once this settles down. I think I can see how to do it: set a buffer-local cancelled flag in the cancel callback, and if set, just discard process output until the prompt is seen, then reset everything. I don't have a way to ask the process to stop processing (though I could send it an interrupt signal I suppose).

It might also be sensible to make a fake process-facing backend which supplies random completions with arbitrary random delays to see how it performs (would also give bragging rights if it does much better than "just waiting" and is less error-prone than "interrupting without notice").

For this I have the invincible cape-noninterruptible-capf,

I meant can't a backend just (let (throw-on-input) (accept-process-output ....))) as kryptonite?

[minad]

So yes, who will wrap with cape-async-capf? A frontend like corfu, automatically? The user, in their config file? The backend author? And if the latter, are they then stuck with a backend that will otherwise NOT work well with e.g. company or default completion? Will you hope/expect company, for example, will directly support ACAPFs internally, such that backend-wrapping is counter-indicated? Or is it more desired to produce a CAPF which any current user-chosen UI (most of which will not interrupt) can make use of — "wrap once, use anywhere"? That will be important to document to get any adoption I suspect.

Most likely if we succeed - Company will support ACAPF directly since it has to be careful to integrate with its existing infrastructure. For Corfu one could just push Cape as a middleware during registration. But maybe I will simply detect asynchronous capfs in Corfu and also support them directly, moving that part from Cape to Corfu.

I'm happy to try out my (still unreleased) ipython mode as an async guinea pig once this settles down. I think I can see how to do it: set a buffer-local cancelled flag in the cancel callback, and if set, just discard process output until the prompt is seen, then reset everything. I don't have a way to ask the process to stop processing (though I could send it an interrupt signal I suppose).

Great. But instead of buffer local variables you should probably capture everything in closures.

I meant can't a backend just (let (throw-on-input) (accept-process-output ....))) as kryptonite?

Then it is non-interruptible and it will be slow and unresponsive. If that is the goal?

[minad]

@jdtsmith Btw, I am proposing to move to a five argument calling convention for the completion tables for safety reasons, see https://github.com/minad/cape/pull/14#issuecomment-981629266. Doing that would also make it easy to detect asynchronous completion tables (call with five args, if it fails treat the table as synchronous, otherwise continue to treat it as asynchronous).

[jdtsmith]

I meant can't a backend just (let (throw-on-input) (accept-process-output ....))) as kryptonite?

Then it is non-interruptible and it will be slow and unresponsive. If that is the goal?

The goal is to solve the problem that unanticipated interruption leads to unwanted hidden output appearing in the inferior process buffer — a much worse problem than an extra 15ms pause. But now I'll have a much better way to solve it :).

But seriously part of the communication issue will be to (gently) convince backend authors that they either join the async API bandwagon, or risk being interrupted by brute force, with unpredictable consequences; i.e. async is the easier road. Which other UI's interrupt again?

[dgutov]

Most likely if we succeed - Company will support ACAPF directly since it has to be careful to integrate with its existing infrastructure.

But company-capf calls completion-all-completions, to have all completion requests go through the completion styles infrastructure.

For it to be able to use the asynchronous tables, won't that require completion-all-completions, completion-try-completion, etc (and that means the completion styles' code) to gain awareness of asynchronous tables?

Otherwise, company-capf--candidates would have to "synchronize" each such table first. Which would mean no possibility to use it in parallel with company-dabbrev-code, for example. Or company-yasnippet, or company-files.

[minad]

@dgutov Indeed.

For parallelization you would have to treat company capf specially. Parallelization multiple acapfs could work via a cape-super-acapf. To parallelize an acapf together with a company backend you would also have to handle the parallelization a level deeper and basically invert the handling. You could introduce a list of async calls to resolve and then the acapf synchronizer calls those too during synchronization and returns the results synchronously in the end.

The other approach would be to change the completion style infrastructure or to not make use of it. You could also implement your own asynchrouns completion styles in company. But this seems like more work and incompatibility with upstream?

See also my comment https://github.com/minad/cape/pull/14#issuecomment-981226820

[minad]

@dgutov I m convinced that this can still be implemented in a reasonable clean way in Company. The only ugliness is really that when you resolve a group of backends with comoany-capf, then company-capf is in charge of synchronizing everything. If you synchronize a group of non-acapf backends you can synchronize them directly since you don't go through the synchronous completion style infrastructure. Arguably this promotes the company-capf backend to a bit of a special status.

But in practice it wouldn't be to bad. If you keep a variable company-async-call-list around which is a list of "jobs"/asynchronous calls, as soon as synchronization happens, this list is processed in parallel. This processing can then happen in the capf backend too and also at another place where we want to synchronize to get access to the results. But the code can be reused.

[dgutov]

The other approach would be to change the completion style infrastructure or to not make use of it. You could also implement your own asynchrouns completion styles in company. But this seems like more work and incompatibility with upstream?

I suppose that would be the way forward: to drop the company-backends support, basically. But that event has a whole list of requirements, as you probably recall.

The only ugliness is really that when you resolve a group of backends with comoany-capf, then company-capf is in charge of synchronizing everything. If you synchronize a group of non-acapf backends you can synchronize them directly since you don't go through the synchronous completion style infrastructure.

But company-capf works with the value of completion-at-point-functions, right? The value which may have been modified by the user, but more likely has been changed with add-hook by major and/or minor modes. And that very same value needs to work with M-x completion-at-point, Counsel, Corfu, and other frontends for CAPF. Right? Because otherwise we might as well go back to using company-backends.

And if M-x completion-at-point must be able to handle it, and it doesn't support async completion tables natively, then whatever completion tables are produced by said completion-at-point-functions elements, they must already be "synchronized". Maybe grouped at the same time, using a function combinator.

Which is to say, it probably can work, but company-capf won't be able to tell that those completion tables (or some of them) are "asynchronous" internally, and so no particular support for them would be possible or needed.

[minad]

I suppose that would be the way forward: to drop the company-backends support, basically. But that event has a whole list of requirements, as you probably recall.

Hmm, I would prefer to follow the approach with minimal impact as I proposed here instead of changing all the existing infrastructure.

But company-capf works with the value of completion-at-point-functions, right? The value which may have been modified by the user, but more likely has been changed with add-hook by major and/or minor modes. And that very same value needs to work with M-x completion-at-point, Counsel, Corfu, and other frontends for CAPF. Right? Because otherwise we might as well go back to using company-backends.

This is certainly an issue. But my idea would be to first support this in Company and Corfu (breaking support upstream if async capfs are registered in completion-at-point). Please keep in mind that this is reasonable, since people use older Emacsen. There is basically no way around this. But if you use Company or Corfu, there won't be an issue. In particular one could install an advice to the capf wrapper which fixes completion-at-point such that it works with acapfs if Company/Corfu are used.

And if M-x completion-at-point must be able to handle it, and it doesn't support async completion tables natively, then whatever completion tables are produced by said completion-at-point-functions elements, they must already be "synchronized". Maybe grouped at the same time, using a function combinator.

Multiple ways out:

1. Fix via advice from outside
2. We could also introduce a new variable completion-async-at-point-functions

You are right that you could just stick to company-backends. But this way one loses the advantage of inventing an API which is conceptually closer to the upstream infrastructure. I find this valuable and as we discussed it is possible to extend/overhaul this API gracefully with your idea of :async return values plus an adjusted filter argument to the table.

Which is to say, it probably can work, but company-capf won't be able to tell that those completion tables (or some of them) are "asynchronous" internally, and so no particular support for them would be possible or needed.

One could also do that - leave the entire support to a supplementary library like Cape. No special support in Corfu or Company needed, Cape would handle the synchronization and parallelization. In fact this is what the current implementation does - this is the MVP. The only question is if this would impede adoption. I am sure it would lead to friction. Probably it depends on if some major backends like lsp-mode are interested in this convention.

So I see all your points, you are right about them. I only disagree with the minimal vs major impact changes, where I would prefer the minimal impact approach even if it requires some unorthodox stacking of the capf transformations. The MVP I have in Cape would work as is, it leaves the entire process out of the existing infrastructure (Does not touch Corfu, Company or minibuffer.el). This is certainly nice and the whole idea of my Cape library - Cape is supposed to provide Capf transformers and a few useful Capfs. So from this point of view I am happy with Cape. I am also happy to maintain the async prototype infrastructure here.

What do you propose as next steps?

[dgutov]

I only disagree with the minimal vs major impact changes, where I would prefer the minimal impact approach even if it requires some unorthodox stacking of the capf transformations.

I suppose one reason for my disliking is it will stack a new and different design on top of an existing, already-complex one. Which would make the total sum more complicated further. To extend/develop/etc. Especially for future developers.

minibuffer.el is very ossified in its ways, and any new layer on top (instead of an approach favoring some big breakage) would encourage that ossification further. Whereas if we tried to bring "proper" futures into the core, it could encourage some constructive rewrites, better documentation, cutting off of edge cases, and so on. And, together with utility functions acting on futures, would encourage their use in other features that use asynchronous computations.

Anyway, it's a big job which I'm not itching to do myself, so I can't demand that of others either.

We could also introduce a new variable completion-async-at-point-functions

All right. But company-capf, as well as any other frontend, would still need to "pipe" the completion function through the completion styles, and that's usually the first thing they'd do. So aside from specifying the symbol that interrupted completion requests would throw, there's not much opportunity for it to benefit from the asynchronous nature of the table. company-capf would "synchronize" it and then pass to completion-all-completions and friends. It might as well be "synchronized" in advance by whatever piece of code set up the hook.

Then what's left of company-capf integration with the present proposal, is to just add a new dynamic binding.

Speaking further of completion-async-at-point-functions: one of the bits of information that a hook (such as completion-at-point-functions) stores is the ordering of functions. If some major/minor modes would use this hook, and others - the -async- one, the ordering between them can get lost.

[minad]

Okay, let's not pursue this here then.