counsel-projectile-find-file extremely sluggish in larger repositories.

[dminuoso]

When switching the project to https://github.com/NixOS/nixpkgs (~25,000 files), or calling M-x counsel-projectile-find-file inside the same repository, typing a few letters (say "postfix") causes emacs/ivy to hang for a few seconds on my laptop until the ivy minibuffer refreshes and displays the filtered strings. In comparison, projectile-find-file is much more snappy, which seems to respond within a hundred milliseconds or so.

I also tried to set counsel-projectile-find-file-more-chars to 2 to no avail.

My counsel/ivy/projectile config is completely left at default.

Here's the output of CPU and memory profilers:

CPU profiler

``` - command-execute 4889 57% - call-interactively 4889 57% - funcall-interactively 4770 55% - counsel-projectile-switch-project 4719 55% - ivy-read 4717 55% - ivy-call 4622 53% - counsel-projectile-switch-project-action 4622 53% - counsel-projectile-switch-project-by-name 4622 53% - # 4620 53% - counsel-projectile 4620 53% - ivy-read 4617 53% - read-from-minibuffer 4521 52% - ivy--queue-exhibit 4471 52% - ivy--exhibit 4471 52% - ivy--update-minibuffer 4464 52% - ivy--filter 3017 35% + counsel-projectile--matcher 1719 20% + ivy--recompute-index 1297 15% ivy--regex-plus 1 0% + ivy--format 1447 16% + ivy--insert-minibuffer 6 0% + minibuffer-inactive-mode 2 0% + timer-event-handler 2 0% undo-auto--add-boundary 1 0% + ivy--reset-state 64 0% + projectile-project-p 1 0% + projectile-prepend-project-name 1 0% + hack-dir-local-variables-non-file-buffer 2 0% + read-from-minibuffer 75 0% + ivy--reset-state 5 0% + projectile-project-p 1 0% + projectile-project-root 1 0% + execute-extended-command 51 0% + byte-code 119 1% - ... 3673 42% Automatic GC 3673 42% + evil-repeat-post-hook 1 0% + evil--jump-hook 1 0% + direnv--maybe-update-environment 1 0% + redisplay_internal (C function) 1 0% + timer-event-handler 1 0% undo-auto--add-boundary 1 0% ```

Memory profiler

``` - command-execute 3,380,453,797 99% - call-interactively 3,380,453,797 99% - funcall-interactively 3,377,042,321 99% - counsel-projectile-switch-project 3,373,287,970 99% - ivy-read 3,373,284,421 99% - ivy-call 3,373,145,049 99% - counsel-projectile-switch-project-action 3,373,145,049 99% - counsel-projectile-switch-project-by-name 3,373,145,049 99% - # 3,373,122,051 99% - counsel-projectile 3,373,122,051 99% - ivy-read 3,373,111,234 99% - read-from-minibuffer 3,313,649,682 98% - ivy--queue-exhibit 3,313,561,434 98% - ivy--exhibit 3,313,561,434 98% - ivy--update-minibuffer 3,313,181,018 98% + ivy--format 1,742,675,931 51% - ivy--filter 1,570,505,087 46% - ivy--recompute-index 1,502,558,612 44% - ivy--completing-fname-p 1,502,557,556 44% - counsel-projectile--project-buffers-and-files 1,502,557,556 44% - projectile-current-project-files 1,498,212,421 44% - projectile-project-files 1,498,035,757 44% - projectile-dir-files-alien 1,498,035,757 44% - projectile-files-via-ext-command 1,463,755,470 43% - shell-command 1,400,129,372 41% call-process-shell-command 1,400,084,492 41% make-temp-file 3,152 0% + delete-file 3,048 0% + split-string 61,078,085 1% string-trim 94,191 0% generate-new-buffer 28,821 0% + projectile-get-sub-projects-files 134,115 0% + projectile-project-vcs 16,276 0% + projectile-acquire-root 176,664 0% + counsel-projectile--project-buffers 3,170,679 0% remove 946,176 0% + projectile-project-root 183,804 0% file-relative-name 44,476 0% + ivy--preselect-index 1,056 0% + counsel-projectile--matcher 67,944,371 2% + ivy--regex-plus 2,104 0% + ivy--insert-minibuffer 318,648 0% + ivy-set-text 59,672 0% ivy--input 2,096 0% + redisplay_internal (C function) 36,424 0% + command-execute 6,128 0% + direnv--maybe-update-environment 3,168 0% + ivy--reset-state 59,435,965 1% + ivy--update-prompt 1,024 0% + projectile-prepend-project-name 8,779 0% + projectile-project-p 1,488 0% + projectile-maybe-invalidate-cache 550 0% + hack-dir-local-variables-non-file-buffer 21,929 0% + read-from-minibuffer 105,273 0% + ivy--reset-state 29,600 0% + ivy--update-prompt 1,024 0% abbreviate-file-name 2,048 0% + projectile-prepend-project-name 585 0% + projectile-project-p 458 0% + projectile-project-root 458 0% + execute-extended-command 3,754,351 0% + byte-code 3,411,476 0% + redisplay_internal (C function) 18,740 0% evil-repeat-pre-hook 8,188 0% ... 0 0% ```

[dminuoso]

The immediate problem appears to be https://github.com/ericdanan/counsel-projectile/blob/06b03c1080d3ccc3fa9b9c41b1ccbcf13f058e4b/counsel-projectile.el#L1632

This collection function is called each time a new character is added to the input (but not when characters are removed, I guess ivy-read simply caches the filtered results).

In a non-trivial repository this can incur several hundred milliseconds on a single character stroke, so if one rapidly types in foo.nix, this will call the function 6 times in rapid succession and hanging for a few seconds.

[huangfeiyu]

I am experiencing the same issue. I use counsel-projectile to open file, when I type the file name, the issue occurs. But if I just paste the file name without typing, I get the result quickly.

[ericdanan]

Hi, thanks for reporting. I guess you are right that the issues comes from the use of a collection function. I didn't notice it because I don't use large repositories. It is probably worth trying to change this, but unfortunately it will probably take a while until I can look at it, sorry about that (PRs welcome of course).

[nicodebo]

Hello, I just want to say that I'm having the exact same issue. The counsel-projectile command is very laggy with a folder of about 200 files while projectile-find-file feels very responsive.

[laustbn]

Hi,

I also see this issue on macOS. If I watch active processes on my machine whilst running counsel-projectile-find-file, counsel-projectile is causing Projectile to re-index the project for every(?) keystroke I do in the mini-buffer. When Projectile indexes a project it will call external commands like git, find, etc. and I believe this is where the slowness comes from. The CPU profile above doesn't reveal this, probably because Emacs is simply waiting for external commands to finish. shell-command does show up in the memory profile, however. I would guess the root cause is the constant re-indexing, but I didn't look into the code to figure out why that happens.

As a workaround I found that if I enable caching of project files in Projectile (with (setq projectile-enable-caching t)), it stops calling external commands and responsiveness becomes similar to projectile-find-file, at least for the projects I work on (few thousand files).

[DerGuteMoritz]

@laustbn

As a workaround I found that if I enable caching of project files in Projectile (with (setq projectile-enable-caching t)), it stops calling external commands and responsiveness becomes similar to projectile-find-file, at least for the projects I work on (few thousand files).

Great tip, thanks for sharing! I recommend reading the docs on caching to understand how/when you would need to manually purge the cache.

Another thing I noticed that it was still sluggish when switching between projects with counsel-projectile-switch-project. Turns out that this is due to the default action being counsel-projectile-switch-project-action which also adds buffers to the set of completion candidates. Setting it to counsel-projectile-switch-project-action-find-file instead makes it snappy, too.