helm-projectile-find-file is very slow on a large project

[corngood]

This change:

https://github.com/bbatsov/helm-projectile/commit/4b6685a4f5e07d34329c3d30924a5a64a4742948#diff-d681abcff22239ad92721bedc89d3de7R520

causes helm-projectile-find-file to go from ~1s before being reponsive, to >30s on my project (git repo with 130k files). I've reproduced this on multiple machines, with versions of emacs from 24 to 27 (master), on cygwin and linux.

@xiongtx I'll try to dig a bit deeper, but in the meantime do you have any thoughts on improving this?

[corngood]

Here's a profiler capture (http://termbin.com/vj8v), but it doesn't seem that interesting:

- command-execute                                             3629459  99%
 - call-interactively                                         3629459  99%
  - funcall-interactively                                     3629459  99%
   - eval-expression                                          3629459  99%
    - eval                                                    3629459  99%
     - progn                                                  3629459  99%
      - helm-projectile-find-file                             3629263  99%
       - let                                                  3629263  99%
        - helm                                                3629263  99%
         - apply                                              3629263  99%
          - helm                                              3629263  99%
           - apply                                            3629263  99%
            - helm-internal                                   3629263  99%
             - helm-read-pattern-maybe                        3626586  99%
              - helm-update                                   3619715  99%
               - helm--collect-matches                        3619115  99%
                - helm-compute-matches                        3619115  99%
                 - helm-get-cached-candidates                 3616819  99%
                  - helm-get-candidates                       3616819  99%
                   - helm-interpret-value                     3616819  99%
                    - helm-funcall-with-source                3616819  99%
                     - apply                                  3616819  99%
                      - #<lambda 0xbfcf67c894>                3616819  99%
                       - if                                   3616819  99%
                        - progn                               3616819  99%
                         - save-current-buffer                3616819  99%
                          - let*                              3616819  99%
                           - while                            3596832  99%
                            - setq                            3592437  98%
                             - nconc                            82668   2%
                              - list                            80867   2%
                                 cons                           79067   2%

[xiongtx]

Thanks for posting the CPU profile 👍; but you're right that it doesn't look that interesting. Can you capture the memory profile as well and link to it here?

By "being responsive", do you mean the amount of time it takes for the list of files to show up, or the amount of time for the list to update after you type something to filter the list?

And what's your OS?

[corngood]

I actually get a similar delay before the list appears the first time, and after each time the filter is changed. In both cases Emacs is maxing out a CPU core, and no child processes are involved. So it's consistent with expensive lisp code I guess.

The profile I posted is from Emacs 25.3.1 on Linux x64. I also get similar behaviour with the same project on cygwin.

I'll grab a memory profile in a bit.

[xiongtx]

It seems that the issue is the use of collect...into.

This is nearly instantaneous:

(progn
  (cl-loop for i in (number-sequence 0 130000)
     collect (cons (number-to-string i) i))
  :done)

while this seems to hang (I didn't have the patience to wait ~30 s 😛):

(progn
  (cl-loop for i in (number-sequence 0 130000)
     collect (cons (number-to-string i) i) into pairs)
  :done)

[corngood]

Here's a memory profile: http://termbin.com/seh8

- command-execute                                          23,462,730 100%
 - call-interactively                                      23,462,730 100%
  - funcall-interactively                                  23,462,730 100%
   - eval-expression                                       23,462,730 100%
    - eval                                                 23,462,730 100%
     - progn                                               23,462,730 100%
      - helm-projectile-find-file                          21,612,770  92%
       - let                                               21,607,476  92%
        - helm                                             21,607,476  92%
         - apply                                           21,596,776  92%
          - helm                                           21,596,776  92%
           - apply                                         21,596,776  92%
            - helm-internal                                21,596,776  92%
             - helm-read-pattern-maybe                      8,956,236  38%
              - helm-update                                 5,356,848  22%
               - helm--collect-matches                      4,504,570  19%
                - helm-compute-matches                      4,504,570  19%
                 - helm-get-cached-candidates               4,049,292  17%
                  - helm-get-candidates                     4,049,292  17%
                   - helm-interpret-value                   4,049,292  17%
                    - helm-funcall-with-source              4,049,292  17%
                     - apply                                4,049,292  17%
                      - #<lambda 0xbfcf67c894>              4,046,188  17%
                       - if                                 4,046,188  17%
                        - progn                             4,044,872  17%
                         - save-current-buffer              4,044,872  17%
                          - let*                            4,044,872  17%
                           - while                          4,036,684  17%
                            - setq                          4,036,684  17%
                             - nconc                        4,036,684  17%
                              - list                        4,036,684  17%
                                 cons                       4,036,684  17%

This time instead of my own project I ran it on the linux repository, which isn't quite as big but still shows the problem. You should be able to repro with:

• emacs ~/src/linux/. to open dired in the root of linux
• make sure projectile caching is enabled
• (projectile-find-file) (run it twice if you need to populate the cache, should be instant the second time)
• helm-projectile-find-file) (should be quite a bit slower)

[corngood]

@xiongtx Yeah, that's consistent with what I saw. If I remove everything from cl-loop starting with into, it's as fast as it used to be.

[xiongtx]

Can you try 7fe0ccd? It should be on MELPA in a few hours but you could just pull down the code.

[corngood]

That's much better. It's definitely < 1s now to show helm, and almost instantaneous for filtering.

Thanks for getting that done so quickly. It took me longer to read the cl-loop docs.

[xiongtx]

If you want to learn more about (cl-)loop, I recommended Peter Seibel's introduction from Practical Common Lisp 🙂.