Mini.files lags a bit when navigating over directories with a decent amount of files

[GitMurf]

Contributing guidelines

• [X] I have read CONTRIBUTING.md
• [X] I have read CODE_OF_CONDUCT.md
• [X] I have updated 'mini.nvim' to latest version

Module(s)

Mini.files

Description

I am not sure if this is expected or not but it’s been slightly bugging me for a while now and figured I would confirm whether this is expected or something in my setup that I can investigate to help make mini.files more performant.

Assuming a file structure like this:

foo/
      File1
      File2
bar/
      << a couple hundred files / folders >>
baz/
      File3
      Folder1

When I navigate with j/k over folders foo, bar and baz, I get a bit of lag / delay when I hit the “bar” directory which has a couple hundred items underneath it.

So as I navigate I get a little delay / jumpiness anytime there is a folder with a lot of items under it. A common example is node_modules folder in JavaScript repos. But the same thing happens like in my obsidian vault when I navigate over a folder that has a fair amount of notes in it.

It is not horrible by any means (which is why I have delayed mentioning it) but it has gotten frustrating enough that I remember to try and do a 2j or 2k when I am over an item just before a larger folder so that I hop over it and don’t get the little delay over the node_modules folder for example haha ;)

Is this expected? Curious if there is a way to debounce the fetching of directory sub items when navigating quickly with j/k or something?

Or only grab the first 30 child items or something (enough to fill the next window) and then if you navigate into the folder (L) it loads the full amount (ok if the bit of lag happens then… the frustrating part is just when it is unexpected as you are navigating the directories above and don’t even have interest in the child items in that directory because you are navigating somewhere else.

Hope this makes sense and look forward to hear your thoughts!

Neovim version

0.11

Steps to reproduce

Explained in detail above…

Expected behavior

No response

Actual behavior

Explained in detail above. I expect (or hope) that there is no real lag / delay when navigating over a folder that has a decent amount of child items.

[echasnovski]

Thanks for the issue!

It looks like you're also using 'mini.icons' together with 'mini.files'. If yes, then this is an expected behavior and should only happen for the first time the directory is previewed during the entire Neovim session (not current explorer session). Does preview after the first instance of delay also show the delay?

At the moment I think having this "only happens the first time" delay is somewhat acceptable. But I must admit that I also encounter this from time to time and it is indeed slightly annoying. And the delay is visibly smaller in 0.11 since I've spent time optimizing vim.filetype.match() in Neovim core.

Curious if there is a way to debounce the fetching of directory sub items when navigating quickly with j/k or something?

Or only grab the first 30 child items or something (enough to fill the next window) and then if you navigate into the folder (L) it loads the full amount (ok if the bit of lag happens then… the frustrating part is just when it is unexpected as you are navigating the directories above and don’t even have interest in the child items in that directory because you are navigating somewhere else.

Any adjustment in 'mini.files' is limited by the "don't really want to increase line count and complexity further" mentality. Hence debouncing is most certainly out of the picture.

I remember briefly looking at limiting the number of items in the preview and it also required considerable amount of changes. I'll take a look at this again more closely, just to be sure.

[GitMurf]

@echasnovski thanks for the quick response and thoughtful reply! Unfortunately it is not the icons issue. 1) it happens everytime and not just the first time and 2) I had already tested by disabling icons all together and same experience and 3) a great example of the problem is node_modules folder which all it has is child folders (no files) so the only unique icon is the folder icon.

Let me know if you have other ideas to test!

[echasnovski]

@echasnovski thanks for the quick response and thoughtful reply! Unfortunately it is not the icons issue. 1) it happens everytime and not just the first time and 2) I had already tested by disabling icons all together and same experience

The only problem with delay I encountered is because of 'mini.icons'. Other than that even with about 4500 entries the delay is at most 100 ms and is due to the sheer amount of entries processed from disk.

... and 3) a great example of the problem is node_modules folder which all it has is child folders (no files) so the only unique icon is the folder icon.

This has nothing to with uniqueness of icons. Each unique path shown in explorer results in the call to MiniIcons.get(), which has potentially slow-ish first call and very fast next ones.

Even without 'mini.icons', the previewed child paths are cached and previewing after first time should be faster.

Let me know if you have other ideas to test!

The best thing to do is to provide more details about the issue: how many files/directories, OS, HDD or SSD, etc. Ideally provide an example of directory with which I can try to reproduce the issue (like for example create a fixed set of files in the directory with which you can observe a delay).

If you have the means, I'd also ask to make a screecast of the persistent delay (i.e. not only the first preview, but the subsequent ones) with not enabled 'mini.icons'.

[echasnovski]

Latest main branch now computes prefix only for visible items in preview (with recomputation if/when the previewed path is focused). As this is the only slowdown of preview I know about, I am going to close this. If there is more info that helps to reproduce this, feel free to comment here.

[GitMurf]

Thanks @echasnovski … I’ll give it a try and let you know!

[GitMurf]

@echasnovski I figured it out! Well at least the exact line of code that is the culprit. I am hoping you know why this would be so slow (takes over 1,100 ms for one of my node_modules folders everytime my cursor navigates over the folder).

https://github.com/echasnovski/mini.files/blob/b545d5c9271f4c92524bd49c93b93bb54a572433/lua/mini/files.lua#L2297

Let me know if you have some ideas of how you want me to debug / log around that set buffer call. I am presuming/assuming the issue is because of the amount of data it is loading into the win/buffer... but it seems odd this would be this slow 🤷🏻‍♂️ I feel like it has to be an autocmd or something being triggered behind the scenes on the loading of the buffer into the window because I can't think of any reason it would take over a second just to change the buffer loaded to the nav window.

Thoughts? Questions? Ideas?

Thanks!

[GitMurf]

@echasnovski fyi if it helps, I dumped the contents of the buffer that is being loaded for my node_modules example and there are 1,101 items (all directories). Here is an example of what it dumped out as in case there is something weird in there you see.

As a side note, it looks like it indeed did only include the directory icon on the first 50 items (based on your recent optimization for loading the icons)... you can see below that the end of my list does not have the icon.

{
  "/0045/󰉋 /.bin",
  "/0046/󰪺 /.cache",
  "/0047/󰉋 /.pnpm",
  "/0048/󰉋 /.vite",
  "/0049/󰉋 /@alloc",
  "/0050/󰉋 /@ampproject",
  "/0051/󰉋 /@asamuzakjp",
  "/0052/󰉋 /@atjson",
  "/0053/󰉋 /@aws-crypto",
  "/0054/󰉋 /@aws-sdk",
  "/0055/󰉋 /@babel",
  "/0056/󰉋 /@bcoe",
  "/0057/󰉋 /@commitlint",
  "/0058/󰉋 /@corvu",
  "/0059/󰉋 /@cspotcode",
  "/0060/󰉋 /@electron",
  "/0061/󰉋 /@electron-forge",
  "/0062/󰉋 /@esbuild",
  "/0063/󰉋 /@eslint",
  "/0064/󰉋 /@eslint-community",

-- skipped a bunch

  "/1125//ws",
  "/1126//xml-name-validator",
  "/1127//xmlbuilder",
  "/1128//xmlchars",
  "/1129//xterm",
  "/1130//xterm-addon-fit",
  "/1131//xterm-addon-search",
  "/1132//y18n",
  "/1133//yallist",
  "/1134//yaml",
  "/1135//yaml-eslint-parser",
  "/1136//yargs",
  "/1137//yargs-parser",
  "/1138//yarn-or-npm",
  "/1139//yauzl",
  "/1140//yn",
  "/1141//yocto-queue",
  "/1142//z-schema",
  "/1143//zwitch",
  "/1144//.modules.yaml"
}

[echasnovski]

Thoughts? Questions? Ideas?

I am fairly certain that showing buffer in a window would not result in such delay on its own. This means that there is indeed some other plugin/configuration that affects this.

I feel like it has to be an autocmd or something being triggered behind the scenes on the loading of the buffer into the window ...

Yes, this would be my initial guess indeed. I'd suggest bisecting plugins (disable halve and try to reproduce, etc.). Maybe looking at :au BufWinEnter (and maybe :au BufWinLeave) can give a clue on which plugin may be causing this.

[GitMurf]

Ok @echasnovski I just figured it out! It is because the buffer is filetype minifiles and for whatever reason (autocmds or whatever else may be going on with that filetype) when loading the buf to a window for larger numbers of items it is slow. I confirmed by setting the filetype to "" nothing, setting the buf to the window and then setting the filetype back and then everything runs super smooth / fast!

Here is the rough code I got working... not sure what other implications this could have and I'm sure you probably have a more graceful way to do this ;)

  local orig_filetype = vim.api.nvim_get_option_value('filetype', { buf = buf_id })
  vim.api.nvim_set_option_value('filetype', '', { buf = buf_id })
  -- this is the original line that caused the problem
  vim.api.nvim_win_set_buf(win_id, buf_id)
  vim.api.nvim_set_option_value('filetype', orig_filetype, { buf = buf_id })

[echasnovski]

Ok @echasnovski I just figured it out! It is because the buffer is filetype minifiles and for whatever reason (autocmds or whatever else may be going on with that filetype) when loading the buf to a window for larger numbers of items it is slow. I confirmed by setting the filetype to "" nothing, setting the buf to the window and then setting the filetype back and then everything runs super smooth / fast!

Here is the rough code I got working... not sure what other implications this could have and I'm sure you probably have a more graceful way to do this ;)

That's great, but I am afraid it is not figured out. Instead of finding an issue inside 'mini.files', please, focus on creating a real world reproducible setup. I.e.: "I'd suggest bisecting plugins (disable halve and try to reproduce, etc.)". This will both potentially help fix an issue in some other plugin and provide a test case for possible future fix in 'mini.files'.

Setting filetype indeed may cause an issue if there are many autocommands for FileType event, so the fact that setting it twice fixes the issue is a surprise. So at least this is not an issue I can remember dealing with and it probably involves some not common code being executed when any buffer is shown in the window.

[GitMurf]

ok @echasnovski I think I have figured it out and potentially could be a bug in Neovim core. Would be great if you could try to repro what I am going to show you and then maybe help figure out who to confirm with from the Neovim core team (assuming you have some connections as I have none, haha).

So basically Neovim will always look for a "parser" for any filetype. As the help docs say for treesitter-parsers, it will look in all runtime paths to find a parser looking in ../parser/minifiles.* (or whatever file type it is). I set verbose level to 16 (highest) and below is what you will find when a buffer enters a window:

Searching for "parser/minifiles.*" in runtime path
Searching for "C:\Users\USER_NAME\AppData\Local\nvim-kickstart\parser/minifiles.*"
Searching for "C:\Users\USER_NAME\AppData\Local\nvim-kickstart-data\lazy\lazy.nvim\parser/minifiles.*"
Searching for "C:\Users\USER_NAME\AppData\Local\nvim-kickstart-data\lazy\arrow.nvim\parser/minifiles.*"
Searching for "C:\Users\USER_NAME\AppData\Local\nvim-kickstart-data\lazy\nvim-lint\parser/minifiles.*"
Searching for "C:\Users\USER_NAME\AppData\Local\nvim-kickstart-data\lazy\gitsigns.nvim\parser/minifiles.*"
Searching for "C:\Users\USER_NAME\AppData\Local\nvim-kickstart-data\lazy\barbar.nvim\parser/minifiles.*"
Searching for "C:\Users\USER_NAME\AppData\Local\nvim-kickstart-data\lazy\conform.nvim\parser/minifiles.*"
Searching for "C:\Users\USER_NAME\AppData\Local\nvim-kickstart-data\lazy\todo-comments.nvim\parser/minifiles.*"
Searching for "C:\Users\USER_NAME\AppData\Local\nvim-kickstart-data\lazy\persistence.nvim\parser/minifiles.*"
Searching for "C:\Users\USER_NAME\AppData\Local\nvim-kickstart-data\lazy\dressing.nvim\parser/minifiles.*"
Searching for "C:\Users\USER_NAME\AppData\Local\nvim-kickstart-data\lazy\nvim-notify\parser/minifiles.*"
Searching for "C:\Users\USER_NAME\AppData\Local\nvim-kickstart-data\lazy\nui.nvim\parser/minifiles.*"
Searching for "C:\Users\USER_NAME\AppData\Local\nvim-kickstart-data\lazy\noice.nvim\parser/minifiles.*"
Searching for "C:\Users\USER_NAME\AppData\Local\nvim-kickstart-data\lazy\cmp-nvim-lsp\parser/minifiles.*"
Searching for "C:\Users\USER_NAME\AppData\Local\nvim-kickstart-data\lazy\mason-lspconfig.nvim\parser/minifiles.*"
Searching for "C:\Users\USER_NAME\AppData\Local\nvim-kickstart-data\lazy\mason-tool-installer.nvim\parser/minifiles.*"
Searching for "C:\Users\USER_NAME\AppData\Local\nvim-kickstart-data\lazy\mason.nvim\parser/minifiles.*"
Searching for "C:\Users\USER_NAME\AppData\Local\nvim-kickstart-data\lazy\nvim-lspconfig\parser/minifiles.*"
Searching for "C:\Users\USER_NAME\AppData\Local\nvim-kickstart-data\lazy\nvim-treesitter-textobjects\parser/minifiles.*"
Searching for "C:\Users\USER_NAME\AppData\Local\nvim-kickstart-data\lazy\nvim-treesitter-context\parser/minifiles.*"
Searching for "C:\Users\USER_NAME\AppData\Local\nvim-kickstart-data\lazy\nvim-treesitter\parser/minifiles.*"

And then no parser is found because there is not a parser for minifiles type (which is totally fine). That is not the problem... everything above is normal and expected. The problem is that those checks (20 or so different runtime paths) repeat for each line in the buffer for some reason! Instead of just checking for the filetype parser, not finding it and moving forward, it checks it 20n times! So this is why it gets slow when there are large number of items in a MiniFiles directory because for my node modules folder for example, it checks 20 1,000+ paths! 20 runtime paths and 1,000 lines checking repeatedly for each line.

So my debug log when simply moving my cursor over the node modules folder in MiniFiles is over 40,000 lines long (there are other repeated problems too but the majority are for the parser check).

TLDR - I believe the logic here is flawed / mistaken as Neovim should only be checking one time per buffer/file and not repeating the check for N number of lines in the file.

Does this make sense? Any questions? Let me know if you are able to repro. As a tip, this happens in any buffers and really has nothing to do with MiniFiles specifically... so you can just enew a blank buffer, do a set filetype=minifiles, add a few thousand random lines of text (can be anything) and then turn verbose logging on and simply bnext to another buffer and bnext back to the buffer you are testing. You should get 10's of thousands of the debug log lines shown above.

[GitMurf]

And as a further way to "prove" this is the issue, the problem completely goes away for MiniFile if I simply register the minifiles type as another valid parser like markdown with this: vim.treesitter.language.register('markdown', { 'minifiles' }). I am NOT saying that is the solution because I don't know what implications that may have on the rendering (seems to work fine though) but more just to further support the issue is with neovim searching for a valid parser to use WAY too many times.

[echasnovski]

So basically Neovim will always look for a "parser" for any filetype. ... And then no parser is found because there is not a parser for minifiles type (which is totally fine). That is not the problem... everything above is normal and expected.

I am afraid this is not the built-in behavior. And I can not reproduce.

All log lines I have when previewing directories in 'mini.files' are lines indicating looking for 'minifiles' filetype and indent plugins. Like these:

Searching for "ftplugin/minifiles[.]{vim,lua} ftplugin/minifiles_*.{vim,lua} ftplugin/minifiles/*.{vim,lua}" in runtime path
...
Searching for "indent/minifiles[.]{vim,lua}" in runtime path
...

This is done once per every setting of filetype (i.e. set filetype=minifiles or vim.o.filetype = 'minifiles') which is only on the directory's first preview. I.e. not for every line for every time directory preview is shown.

---

That's great, but I am afraid it is not figured out. Instead of finding an issue inside 'mini.files', please, focus on creating a real world reproducible setup. I.e.: "I'd suggest bisecting plugins (disable halve and try to reproduce, etc.)". This will both potentially help fix an issue in some other plugin and provide a test case for possible future fix in 'mini.files'.

Have you tried narrowing down your active plugins? Or at least reproduce this with only 'mini.files' enabled?

[GitMurf]

@echasnovski ok the mystery is solved! The reason trying the plugin deactivation exercise was not returning good results figuring it out was because it was not a plugin option! It was a Option setting! Can you try the test again with the following set and see if you are able to reproduce all those "searching for parser ...." logs?

vim.opt.foldmethod = 'expr'
vim.opt.foldexpr = 'v:lua.vim.treesitter.foldexpr()'

[GitMurf]

Also @echasnovski I believe this may be related to https://github.com/LazyVim/LazyVim/issues/1581 and https://github.com/neovim/neovim/pull/24230

Originally it was thought to only be an issue pre-0.10 nightly but it appears that it has either returned or just another scenario that was never caught / fixed completely. I am posting and cc'ing @folke over in that old LazyVim issue as well.

[echasnovski]

@echasnovski ok the mystery is solved! The reason trying the plugin deactivation exercise was not returning good results figuring it out was because it was not a plugin option! It was a Option setting! Can you try the test again with the following set and see if you are able to reproduce all those "searching for parser ...." logs?

vim.opt.foldmethod = 'expr'
vim.opt.foldexpr = 'v:lua.vim.treesitter.foldexpr()'

Yes indeed, with this I can reproduce. Both the delays in 'mini.files' and many lines log about trying to find a parser.

All windows in 'mini.files' have folds disabled, so the fact that 'foldexpr' is still executed seems suspicious. So I don't think this is a 'mini.files' issue.

That said, if 'nofoldenable' will prove to be not enough, here are some counter-measures for the future:

• Modify 'mini.files' to set vim.wo[win_id].foldmethod = 'manual' when creating 'mini.files' windows.
• Suggesting users to set it themselves via (equivalent of) vim.cmd('au User MiniFilesWindowOpen setlocal foldmethod=manual').

[GitMurf]

so the fact that 'foldexpr' is still executed seems suspicious. So I don't think this is a 'mini.files' issue.

I completely agree. I posted in LazyVim on an old related issue (since LazyVim uses the same fold expression) and will be opening up a new issue likely in the core neovim repo. Would you mind if I cc you on it in case they have any questions or if you have any 2cents / context to add? I am not familiar at all with neovim core so trying to speak the language of what should or should not be happening is difficult for me ;-)

[echasnovski]

Would you mind if I cc you on it in case they have any questions or if you have any 2cents / context to add? I am not familiar at all with neovim core so trying to speak the language of what should or should not be happening is difficult for me ;-)

Sure. But this should be enough.

[GitMurf]

FYI for anyone else who may come across this, any further conversation on this issue will be discussed here: https://github.com/LazyVim/LazyVim/issues/1581#issuecomment-2310709609 (or if moved somewhere else will be documented / mentioned there).

No further discussion needed here related to MiniFiles other than maybe a follow-up confirmation when it gets fixed upstream.

[pkazmier]

@GitMurf Thank you so much for pursuing this.

While browsing my home directory, I typically cross a notes directory of mine that has about 2k files in it. When I do, there is a significant delay of about 5 seconds where the UI is frozen entirely. I've just not had time to look into this.

There was hope the other day when @echasnovski added the optimization for showing icons only for the visible buffer as I thought to myself, maybe that was the root cause of my delays, but alas it made no significant difference.

Then I saw your discovery today about using treesitter as the fold expression, which is exactly what I do (having come from LazyVim, I really liked that behavior as it got me using folds for the first time ever). So, I just added an autocommand to change the fold method to manual for MiniFiles windows, and viola! No more delays!

Thank you!

[echasnovski]

I believe this now should be fixed on latest main. The reason I decided to add this is because I realized that even if vim.treesitter.foldexpr() is made "smarter" in Nightly, the issue would still exist on Neovim<0.11 (or maybe on Neovim<0.10.2 if it is backported). And as it is a popular setting (which I'd argue against, personally), this seems to be worth it.

Also this prompted all other modules that use interactive helper floating windows to have this fix. So, @GitMurf, your time was worth more than 'mini.files' fix. Thanks for the research!

[pkazmier]

Thanks @echasnovski!

Curious as to how you use folds if not a fan of treesitter as the foldexpression. My main use case is to fold all functions at a particular level so I can see an overview of that level (most often the top-level). I suppose I could just use MiniPick with LSP symbols to do the same though. Are folds a key part of the vim experience that I'm missing out on?

[echasnovski]

Don't get me wrong, folds are immensely useful and I utilize them all the time (for quicker vertical scroll and hierarchical overview).

To me 'foldmethod=indent' is the best compromise as a default fold method. It can work in any buffer and usually provides that hierarchical structure as you'd expect.

There are some small issues I experience from time time (like automatic fold closing when editing at the top of indent level), but they might not be a 'foldmethod=indent' fault.

[GitMurf]

@echasnovski you've inspired me to give indent folding a try! As you say, mostly that's all I want anyways. And I've had enough PTSD from this treesitter issue 🤣 that frankly I want to try and minimize the amount I rely on different tree sitter components. There is just so much surface area for exposure to little inefficiencies and "breaks" given every file format is essentially its own plugin you have to rely on in essence.