search

olson-sean-k / wax

Opinionated and portable globs that can be matched against paths and directory trees.
https://glob.guide
MIT License
112 stars 10 forks source link

performance #51

Open fdncred opened 11 months ago

fdncred commented 11 months ago

I've been working on ways to make globbing with wax a little faster with rayon and I keep coming up empty. I'm beginning to wonder if it needs to be built in somewhere so that it globs a directory and then all the directories inside get their own thread to glob their contents, up to some limit. Any thoughts on this?

olson-sean-k commented 10 months ago

make globbing with wax a little faster with rayon

Yeah, I've also thought about this a bit. Wax depends entirely on walkdir for this, so I think any such features would need to be implemented upstream. The major exception to this is filtering, as Wax is basically a glorified filter atop walkdir. #49 is a great example of a serious performance problem concerning this filtering, wherein directory trees were not discarded and would be read from the file system for no reason (despite what the documentation claims). I just landed a fix for that in 76afaa1.

fdncred commented 10 months ago

I'm wondering how hard it would be to switch from walkdir to jwalk which already supports streaming, sorting, threads, etc. and is also built on walkdir and ignore. https://github.com/Byron/jwalk

olson-sean-k commented 10 months ago

Oh, cool, I wasn't aware of jwalk! Thanks for pointing it out! It's not clear to me how easy it would be to replace walkdir with jwalk, but I'll take a look.