Imports kill DCD performance

[HK47196]

After using the profile feature of vim, roughly 90%+ of the completion time is spent in functions.importPaths in dub.vim. Completions are taking 1+ second each(making it near unusable,) I modified the function to return an empty array and manually added my imports and the completions are basically instant.

Also, repeatedly inserting the import paths slows down DCD drastically, start it up in a terminal and monitor the logging - it will get slower linearly as the same paths repeatedly get added.

[idanarye]

That's weird. When I start dcd-server in terminal and add the same paths over and over again I don't see the same files added over and over again - DCD remembers the paths it already imported. It does appear get slower the more I add them though - not linearly though.

At any rate, you are the first one reporting this problem, and I don't want to remove the automatic import registration altogether, so this is the plan:

1. Add a new option - g:dutyl_dcd_reduceImportPathsRegistration - that can be one of the following:
   • 0(fallback): Current behavior, import paths are detected and registered before each completion.
   • 1(default): Import paths are detected each time, but only new paths are registered. Dutyl will keep track on dcd-server's process ID and clean the list of already registered paths when the process ID is changed.
   • 2: Import paths are detected and registered only when :DUDCDstartServer or :DUDCDrestartServer are called.
   • 3: Don't automatically detect and register import paths.
2. Add a new command - :DUDCDregisterImportPaths. This command will detect and register the import paths, because even if you don't want it done every time you autocomplete, when you need to register import paths you still want to be able to order Dutyl to do it.

[idanarye]

I've opened an issue on DCD about DCD getting slow when adding the same import paths over and over, so I want to wait for it to get fixed before drastically changing Dutyl's main feature like thist.

In the meanwhile, we might want to work on the long time it takes to get the import paths from DUB. It's pretty quick on my machine, but maybe my project's simply don't have that much dependencies. How long does it take to run call dutyl#core#requireFunctions('importPaths').importPaths() on your project?

If this is the bottleneck, I might be able to speed things up by caching the result of this function.

[HK47196]

My project isn't very large, it just pulls in a lot of dependencies from dub. Without the imports, it's near instant. Used reltime() for getting the time.

with just gfm in my dub.json dependencies dutyl#core#requireFunctions('importPaths').importPaths() takes 0.263~ seconds per invocation

With an empty dub.json dependency, dutyl#core#requireFunctions('importPaths').importPaths() takes 0.044~ seconds per invocation

What's the reason that includes are collected every omnicomplete invocation? Wouldn't an autocmd on bufwritepost for files named dub.json suffice?

[idanarye]

I simply don't want to assume that the current instance of Vim is the only thing that's ever going to touch DUB & DCD. I like to use Vim like an IDE, but many Vim users use the shell as their IDE and only use Vim for editing files. Since I can not assume I have exclusive ownership over the dub.json file and the dcd-server process, I prefer to always assume something else touched them since they last time a Dutyl function was invoked.

That being said, seeing that importPaths takes over 1/4 of a second for you, I should probably add some caching. I can't assume exclusivity over dub.json and dcd-server, but I can use the last modification time and the process ID to check if something else touched those.

[idanarye]

@rsw0x I've created a new branch - feature/reduce-import-paths-registration-overhead - where Dutyl caches the import paths it queries from DUB so it doesn't have to re-query them unless dub.json, package.json or dub.selections.json's last modification time changes.

Please check out this branch and see if it makes things fast enough for you. If it's still too slow, I'll cache the import paths sent to DCD as well, but this is a much riskier change that I'd rather avoid if Dutyl is fast enough without it.

[HK47196]

Fork works great, without imports it's about ~30% faster, with imports it's about three times as fast (0.3s -> 0.09s per full completion), definitely more responsive. It's about the same amount of time to complete both now which is great.