Projectile makes new file creation extremely slow

DarkArc commented 2 years ago

Expected behavior

New file creation (or opening of a "new to projectile" file) is similar to vanilla emacs (virtually instantaneous).

Actual behavior

New file creation (and opening of files that are "new to projectile") takes several seconds.

Steps to reproduce the problem

Create a new file within a projectile directory, or open a file generated by another program that projectile hasn't indexed.

Profile of Opening the File

- command-execute                                                3852  93%
 - call-interactively                                            3852  93%
  - funcall-interactively                                        3852  93%
   - counsel-find-file                                           3787  92%
    - counsel--find-file-1                                       3787  92%
     - ivy-read                                                  3787  92%
      - ivy-call                                                 3760  91%
       - counsel-find-file-action                                3760  91%
        - find-file                                              3760  91%
         - find-file-noselect                                    3760  91%
          - find-file-noselect-1                                 3759  91%
           - after-find-file                                     3758  91%
            - run-hooks                                          3749  91%
             - projectile-find-file-hook-function                3745  91%
              - projectile-cache-files-find-file-hook               3742  91%
               - projectile-cache-current-file                   3742  91%
                - projectile-serialize-cache                     3733  91%
                 - projectile-serialize                          3684  89%
                  - write-region                                  149   3%
                   + select-safe-coding-system                    149   3%

Environment & Version information

Projectile version: 2.6.0-snapshot
Emacs: 27.2
Operating System: Fedora 35

bbatsov commented 2 years ago

I haven't noticed such a slowdown myself so I assume it happens only in certain cases (e.g. in your stacktrace it seems you have caching enabled; if you're cache is big serializing it is quite expensive). You can just use remove-hook to remove the problematic function from find-file-hook.

DarkArc commented 2 years ago

Upon examination, it looks like my cache has grown to nearly 200MB. Maybe this would be nightmarish, but perhaps each project should have its own file/file set?

My rational is I don't really use the global projectile search, I just jump to the project, and search there. Most of my projects are small, but I have a lot of projects checked out, and a few very large ones.

Other ideas:

Debounce the writes (that way a huge file can't end up being written for every file opened that isn't in cache)
Use quick append writes to a "catch up log", upon startup if there's a catch up file, merge it into the in memory cache to create a the fast in memory cache, and update the on disk cache. (I like this the best from a perspective of keeping the editor as responsive as possible, despite cache sizes)

Example large projects:

llvm-project/llvm

Even my "core" projects are enough to start to notice the performance hit with a roughly 6MB file (~150k files).

bbatsov commented 2 years ago

Yeah, I agree that the global cache was a mistake, but I haven't had the time to make it project local (e.g. a projectile.cache file in the root of each project). Perhaps someone would be me to this eventually - shouldn't be a complex change, as the global search doesn't really depend on having a global cache.

DarkArc commented 2 years ago

Sleeping on this I'm not sure that per-project caches is enough. With projects like LLVM there are 100k+ files, writing a cache with 100k+ entries everytime a new file is added/opened isn't great.

Having some kind of append log, and then merging that into the startup (after so long idle?), seems like not only the better thing for performance, but for the disk health.

Dushistov commented 3 months ago

But per cache definitely improve performance when working with small projects. My projectile.cache is 73M. But even I working with small projects:

❯ git ls-files | wc -l
153

it takes several seconds to save new file. And profile for such small projects looks exactly the same:

        1844  97% - command-execute
        1536  80%  - funcall-interactively
        1406  74%   - find-file
        1406  74%    - find-file-noselect
        1403  73%     - find-file-noselect-1
        1399  73%      - after-find-file
        1388  73%       - run-hooks
        1385  72%        - projectile-find-file-hook-function
        1374  72%         - projectile-cache-files-find-file-hook
        1371  72%          - projectile-cache-current-file
        1304  68%           - projectile-serialize-cache

At first I thought that projectile just so slow, but then I find out that this is global cache for all projects.

bbatsov / projectile