Issue: Large Folder bottleneck

[dmgolembiowski]

Known issue

Doesn't handle large folders well. Referenced at https://github.com/imsnif/monocle#known-issue. I'll fork a branch and attempt to see what performance I can squeeze out.

[dmgolembiowski]

At present, what seems like a promising way to stop this large-directory plugin crashing + slowness seems to be some combination of:

• glidesort for sort_by drop-in replacement on the search results
• https://github.com/GoogleChromeLabs/wasm-bindgen-rayon to take advantage of rayon-wasm for splitting apart the work to scan directories
• a per-search counter-generating FnOnce to spawn new AtomicUsize's (at 0) to have each worker fetch_add when they contribute a discovered SearchResult, otherwise breaking their search off when max_results is compared to the AtomicUsize and the latter hits the upper limit
• Using a lazily-initialized queue to have workers push found search results to the UI

Nice to Have:

• Using either cberner/redb or spacejam/sled for disk-caching recursive directory tree's files via the file path mapping to the md5 of the contents and successful search result query terms -> search results (represented as seek positions and span lengths)

[dmgolembiowski]

Closing this issue until some single-threaded wasm rayon support exists.