Expanding directory super slow on large project

[ydzhou]

It happens on a regular basis... I mainly program in Golang. For the project of size similar to Kubernetes, it gets very slow.

I tried to do profiler and here is my results:

         376  63% - command-execute
         343  57%  - byte-code
         343  57%   - read-extended-command
         343  57%    - read-extended-command-1
         343  57%     - completing-read-default
         343  57%      - apply
         343  57%       + vertico--advice
          33   5%  - funcall-interactively
          26   4%   + treemacs-single-click-expand-action
           7   1%   + pixel-scroll-precision
         208  34% + redisplay_internal (C function)
           4   0% + timer-event-handler
           4   0% + substitute-command-keys
           2   0%   mouse-fixup-help-message
           1   0%   treemacs--post-command
           0   0%   ...

Any idea?

[Alexander-Miller]

Could be git-mode. How many lines of output do you get when you run git status --porcelain --ignored=matching .?

I'll be able to clone kubernetes later today. Any particular directory where this happens or it there no pattern?

[ydzhou]

Looks like if I disable git mode, it runs faster.

Is git mode enabled by default?

[ydzhou]

I am comparing the speed of treemacs verus dired or dired-sidebar. I think both dired and dired-sidebar are faster when expanding the same large folder. Is there any configuration I can disable more to speed up?

[ydzhou]

28 lines when I run your command

[ydzhou]

Another run of profiler.

I think this is the root cause. Baseically if I already opened a lot of folders under a large folder, I close that large folder and re-expand... it can take few seconds to open.

         789  88% - command-execute
         688  77%  - funcall-interactively
         688  77%   - treemacs-single-click-expand-action
         688  77%    - treemacs-toggle-node
         678  76%     - treemacs--expand-dir-node
         583  65%      - treemacs--reentry
         535  60%       - treemacs--reopen-node
         535  60%        - treemacs--expand-dir-node
         363  40%         - treemacs--reentry
         354  39%          - treemacs--reopen-node
         354  39%           - treemacs--expand-dir-node
         316  35%            + treemacs--reentry
          15   1%            + treemacs--parse-collapsed-dirs
           9   1%            + treemacs--collapsed-dirs-process
           1   0%            + run-with-timer
           9   1%          + treemacs-goto-node
          72   8%         + treemacs--parse-collapsed-dirs
          42   4%         + treemacs--collapsed-dirs-process
           4   0%         + treemacs--start-watching
           2   0%         + #<compiled 0x11b120cc65eb4e40>
           1   0%         + -separate
          48   5%       + treemacs-goto-node
          38   4%      + treemacs--flatten-dirs
          15   1%      + treemacs--parse-collapsed-dirs
          14   1%      + treemacs--git-status-process
          12   1%      + treemacs--collapsed-dirs-process
           2   0%        concat
           2   0%      + treemacs--filter-files-to-be-shown
          10   1%     + treemacs--collapse-dir-node
         101  11%  + byte-code
          84   9%   redisplay_internal (C function)
          13   1% + timer-event-handler
           2   0%   treemacs--post-command
           1   0%   pfuture--append-output-to-buffer
           0   0%   ...

[ydzhou]

I think this getting much worse with file-follow-mode on... I have 20+ file opened and the folder expanded everywhere.

[Alexander-Miller]

Is git mode enabled by default?

Yes. Specifically the version that waits 500ms for the async git process to finish before using its output for highlighting.

28 lines when I run your command

Then that's not the problem.

Baseically if I already opened a lot of folders under a large folder, I close that large folder and re-expand... it can take few seconds to open.

That's the issue then. Treemacs remembers what was open where, and re-opens all the previously expanded sub-sub-sub-directories when you do that. Depending on the amount of directories to open that can be a lot of work (mostly calling directory-files on everything and rendering the results).

Emacs is single-threaded, what little async io capabilities it has I already use for git-mode. But it's not possible to do all the work in the background and finally just render the result up front, you have to block the UI. Add enough sub-directories and you start running into hard limits like that.

Is there any configuration I can disable more to speed up?

A few.

Git-mode is probably not the problem because it is async already, but if you do notice slowdown because e.g. treemacs always needs to through 10000 ignored files you can look at treemacs-max-git-entries and treemacs-git-command-pipe. Also make sure to use deferred git-mode.

You can set treemacs-collapse-dirs to 0.

Turn treemacs-sorting back to a default if you've changed it. File sorting by size or modification date is a lot slower than alphanumeric.

To disable re-opening press TAB with a prefix argument. That way treemacs forget about opened sub-directories.

If file-follow-mode is slow you can still manually find the current file in treemacs with treemacs-find-file.

Also when do the slowdowns happen - when you open things yourself, or because treemacs is doing something in the background on its own?

[Alexander-Miller]

Correction: even git-mode is a problem if you have enough sub-directories because treemacs needs to start a new process for every one of them.

I tried things out on my own laptop (an pretty old Thinkpad), using kubernetes to render about 1000 lines distributed over 100+ sub-directories. Disabling git-mode and collapse-dirs features my time to render everything went from ~6s down to 200ms.

Looks like plain file io is not the problem, it's the amount of processes I am starting. I'll see about fixing that, but in the meantime turning off these features should fix your speed issues.

[ydzhou]

Also when do the slowdowns happen - when you open things yourself, or because treemacs is doing something in the background on its own?

It slows down when I open things myself.

To disable re-opening press TAB with a prefix argument. That way treemacs forget about opened sub-directories.

This is one of my faviorite feature haha

[ydzhou]

Can confirm. Making treemacs-collapse-dirs to 0 improves speed hugely! :) Then disabling treemacs-git-mode smooth out the last bit and its speed is now on par with dired based plugins.

Why does treemacs-collapse-dirs slow things down so much?

Also is this the right way to disable git-mode, (treemacs-git-mode -1)? I get a Git integration prompt when starting the emacs.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity (this bot only works as a reminder, it will not close issues).