Autocomplete Issues when Typing Quickly

[amadeus]

This is probably going to be a bit long winded of a post, so bear with me.

The high level issue I've been encountering is that the popup completion window seems to appear quite inconsistently for me. Sometimes it shows, sometimes it doesn't and I finally took the time to today to sit down and figure out why, which I will outline below.

I made a quick video here that shows the issue in action -- when I type quickly, I get no autocomplete results. If I type slowly, I get the expected results:

https://user-images.githubusercontent.com/83376/169717120-3055c36e-15c6-42ad-8c13-4963237c13a4.mp4

For context, my minimal configuration for this:

Plug 'vim-denops/denops.vim'
Plug 'Shougo/ddc.vim'
Plug 'Shougo/ddc-matcher_head'
Plug 'Shougo/ddc-sorter_rank'
Plug 'Shougo/ddc-converter_remove_overlap'
Plug 'statiolake/ddc-ale'

call ddc#custom#patch_global('sources', ['ale'])

call ddc#custom#patch_global('sourceOptions', {
  \ '_': {
    \ 'matchers': ['matcher_head'],
    \ 'sorters': ['sorter_rank'],
    \ 'converters': ['converter_remove_overlap'],
    \ 'timeout': 30000,
  \ },
  \ 'ale': {
    \ 'mark': 'lsp',
    \ 'forceCompletionPattern': '\.|:|->|\./',
    \ 'timeout': 30000,
    \ 'isVolatile': v:true,
  \ },
\ })

Most of the projects I work on are very large Typescript projects, so results can be slow to return.

Anyways, given this issue, I started tracing both the ale plugin and inside dcc to see if I could figure out where the issue was coming from.

The situation that I will document my findings is from the example video of trying autocomplete magerAsset. Basically, if I type ma and wait, then I will get results back perfectly. If I quickly type mag -- a language server task will be kicked off to ale to search for the results for ma, but before the results get back the g for mag will be typed and no results will be shown.

I added a few different logs around the two plugins to trace what was going on:

• ddc-ale
  • I put a log in the gatherResults function
• ddc.vim
  • here
  • here
  • here
  • inside this if

And what I found when logging, was the following:

What you can see happening here is when we get TextChangedI for mag in vim (that's the log line that says LOCKED, btw) -- we prevent any subsequent searches for results from appearing, or new searches to get queued up. I have to wait until results have returned before I can type again to get new/improved results

I get the sense that locked has some architectural significance here, but at the very least, it feels like we should be queuing up new searches to perform on the updated input if those updates come in while locked. Forcing one to type slowly on large codebases that may take a while to lookup seems a bit fragile?

[amadeus]

I did notice, if I just modify this line like so:

  " before:
  if get(g:, 'ddc#_locked', v:true) || !ddc#_denops_running()

  " after:
  if !ddc#_denops_running()

Everything appears to work great, although I don't know if this is a safe change or not

[Shougo]

The feature is intended behavior due to the performance.

[amadeus]

Forcing devs to type slowly to get autocomplete results doesn't feel like the ideal use case? Like, any project that is medium size or larger is going to suffer under this limitation. Furthermore I feel like any autocompletion tool out there today allows you to type as quickly as possible, eventually it will catch up and show you results if you are willing to wait, it won't just throw everything away and never show you a result.

I feel like the ideal experience would look something like this (assuming there's a minimum of 2 characters to start a query for autocomplete results):

• User types ma
  • We begin a fetch for results, but that might take a second
• Meanwhile as results are fetching, the user types an additional g
  • ddc sees that a request is in progress, so it queues a new search for when the current one completes.

From here there are a couple possibilities -- lets assume for a second the dev types an additional letter e before we've gotten the results back from the original search ma

• User type an e, bringing our full input up to mage
  • ddc sees that we both have a result fetch in progress, AND that there's one queued
  • The smart thing to do here would be to throw away the current one in the queue and replace it with this one instead

Now when the first batch of results come in, we can immediately perform the next search for with the latest input mage, forgetting about any intermediary searches that no longer matter.

With the hack fix I provided above, we get much more reliable results, but I do believe it will still perform all intermediary searches, which isn't great.

Maybe this is something I can try to put together as a PR? I feel it's quite an important feature for a robust autocomplete library.

I've attached a video of how much more reliable the autocomplete feels with removing the ddc#_locked check:

https://user-images.githubusercontent.com/83376/169968722-a2bac782-ea1d-4ade-8d7b-6007f41879cf.mp4

No matter how fast or slow I type, the results window ALWAYS shows.

In contrast, here's the video from yesterday showing the bug -- the fact that I can't see any results despite having refined my query (especially when there's no UI to tell me we are searching or not -- there's no way for me to know how fast or slow to type):

https://user-images.githubusercontent.com/83376/169968856-b704d67b-058a-474d-8d57-85ee55e1f5fc.mp4

[amadeus]

Here's a quick code mockup that I threw together that should do this pretty well:

https://github.com/Shougo/ddc.vim/compare/main...amadeus:queued-events?expand=1

I'm just not sure if there are other areas that might need to be updated as well.

I threw in some logs for _onEvent in app, along with the context.input that shows pretty well how it works:

[Shougo]

The smart thing to do here would be to throw away the current one in the queue and replace it with this one instead

I know it is smart. But ddc does not have the cancel previous state feature.

[Shougo]

Thanks. I have merged your code.

[amadeus]

Oh awesome, thanks!