BurntSushi
commented
7 years ago OK, I'm just going throw a bunch of stuff against the wall and see what sticks.
The primary motivation behind globset is in the name itself: I need a way to match many (possibly hundreds) of globs against a single file path very quickly. The naive solution of iterating over each glob and checking if it matches simply doesn't fly (the time complexity itself is not good enough). For that reason, and because RegexSet is a thing, it made a lot of sense to do the "convert the glob to a regex" dance. However, it doesn't stop there. It turns out that the regex engine doesn't optimize common globs as well as it could, so I ended up developing a set of matching strategies that make glob matching even faster. I dare say that most globs don't touch the regex engine at all. But when they do, they're still reasonably fast.
For that reason alone, you might consider globset to be relatively niche. It's important and the performance matters, but perhaps it's not common to need that kind of performance at that kind of scale.
With that said, it turns out that if you're implementing glob matching for a set of globs, it's also pretty easy to implement straight-forward single-glob matching as well. So globset provides that. I also made sure to use most (all?) of the test suite from the glob crate inside of globset (and then some), so the functionality should be roughly equivalent.
Tangentially, the glob crate can't match file paths that don't fit into a &str, and this turns out to be a real problem that people face in practice (there are some issues on the ripgrep repo for it, IIRC). I'm not familiar enough with the implementation in glob to know how easy or hard it is to fix that, but from a brief glance, it looks like the matcher pretty much assumes valid UTF-8 in that it's based on char. So it would probably require some sizeable attention to fix that.
I personally don't have the bandwidth to fix the glob crate any time soon. The globset crate is something that must exist for ripgrep to be as fast as it is, so regardless of what we do with glob, I think it's somewhat safe to say that globset will continue to work. However, here are some caveats:
- Since
globsetusesregex, as @bstrie points out, using it requires pulling inregex.regexhas (perhaps deservedly) gotten a reputation for being a big dep because it takes a long time to compile. - Since it uses
regex, there is some overhead associated with compiling the glob since it requires compiling a regex. It's probably more expensive to compile a regex than it is to compile a glob in theglobcrate, although I haven't benchmarked it. - In my incomplete experiments, the matching performance of
globsetfor a single glob is roughly on par with theglobcrate, and tends to get better as the paths get longer. globsetdoesn't (yet) have an analogousglobfunction that returns an iterator over paths that match.globsetdoesn't support therequire_literal_leading_dotoption, mostly because it seemed a little tricky to implement, but I honestly haven't given it much thought.
4 and 5 seem like things that can be fixed. 1 and 2 are things that are probably just facts of life for the globset crate. 3 doesn't seem like a blocker.
So I think there's a key trade off here... The globset crate brings significantly more implementation complexity and larger compile times, but also fixes some of shortcomings of the glob crate as it is.
bstrie
alexcrichton
Keats
KodrAus
rivy
Gilnaa
pablosichert
Per the original rust-lang-nursery RFC ( https://github.com/rust-lang/rfcs/blob/master/text/1242-rust-lang-crates.md ):
I'm not sure who the "original crate author" of glob is, as I'm guessing this crate was split out from the stdlib during the great pre-1.0 purge, but it doesn't appear that there's any real champion trying to advance this crate through the stabilization process.
I'm interested in starting a discussion as to the future of this crate. Sadly I can't find the original petition arguing why this crate belongs in the nursery in the first place, so I'm not even sure how to begin evaluating the aspirations of this crate that keep it from being 1.0 today.
Let's focus on answering three concrete questions:
I don't consider myself an expert in either crate, but the biggest potential pitfall that I see when comparing glob to globset is that the latter relies on the complete regex crate, and transitively all of regex's dependencies. In contrast, glob is small self-contained, but appears to be less featureful than globset, and I'm unsure if it handles both Unicode and non-Unicode as well as the regex crate. Some microbenchmarks would be interesting to see as well.
Tagging @burntsushi, as the author of globset and who's familiar with the rust-lang-nursery process. And tagging @alexcrichton, who I'm guessing doesn't particularly care about this crate but who's the only active point of contact listed for this crate on crates.io. :P