Searching sub-directories slows down/hangs app when searching large directory trees

[mattstermiller]

Describe the bug When searching a large directory tree, the app slows down and can eventually become unresponsive. It becomes slower as more folders/items are searched. This happens regardless of the number of items matched.

To Reproduce Steps to reproduce the behavior:

1. Go to a folder with lots of sub-directories and items in those directories (such as C:)
2. Search for any term, with Sub-folders enabled
3. As it searches, notice the app becomes less and less responsive. This can be tested by pressing Tab to focus the path box and Esc to unfocus, or moving the window around with the mouse, noting how much time it takes to respond.

Expected behavior The app should not become unresponsive when searching large directory trees.

Desktop (please complete the following information):

• OS: Windows 10 build 2004
• Koffee Version: 0.6.0 commit c93120a

Additional context My hunch about the problem is that VinylUI's comparison of the model iterations slows down as the SubDirectories list property grows. I'm not sure what to do about this except find a way to speed up comparison of this property (use a different collection type?) or possibly even move the SubDirectories data out of the model.

[mattstermiller]

I should try write a test to perform a recursive search on a generated deep hierarchy in FakeFileSystem to observe the slow behavior.

I still believe it's being caused by the SubDirectories property on the model and is churning comparing the items in it and concatenating long lists. After thinking about this more, the SubDirectories property is really just a cache and probably doesn't belong on the model. I think the best thing to do, regardless of whether it is the main source of recursive search slowdown, is to create a separate caching mechanism that remembers directory listings for a short time, such as 1-5 minutes, and cleared with an explicit refresh. This would result in a better user experience too, since you could re-use cached results after clearing the search, canceling a partially completed search, or flipping back and forth searching different trees, etc.

[mattstermiller]

I can't seem to reproduce the extreme unresponsiveness or hanging anymore. I checked out the referenced commit and cherry-picked various commits that updated SDK and tooling references in order to compile. When testing, I am using the same PC on the same parent directory I used when I wrote this issue and, while the UI does seem to become slightly less responsive, it's still acceptable (seems to be about 1/3 second between updates at its worst) and it actually finished the search. This is far better than my previous experience where it ground to a halt and never finished.

It's hard to say what could have "fixed" or improved the situation, but it might have to do with improvements in the SDK, F# compiler, .NET Framework fixes, Windows fixes or removing the dependency on FsXaml.

All that said, I did see a noticeable improvement in UI responsiveness by disabling updates to the SubDirectories property when its size reaches the hundreds of thousands. I would still like to implement the caching mechanism I described in my previous comment for this reason and also being able to reuse results for a time, but since this seems to no longer be a blocker, it has lower priority.