I've seen that your lib is 40% slower compared to RobotFileParser from Python versions < 3.13 . I suspect this is because of re module compilation and matching.
pyre2 is a drop-in replacement for re which is faster for simple patterns which are exactly what robots.txt relies on. pyre2 falls back to re if it doesn't support some RegExp features (like lookarounds) but it won't be the case here.
My claims about potential speedup should be tested with your lib of course but nonetheless I think these are worth a consideration.
Just throwing up a far future idea.
I've seen that your lib is 40% slower compared to
RobotFileParserfrom Python versions < 3.13 . I suspect this is because ofremodule compilation and matching.pyre2is a drop-in replacement forrewhich is faster for simple patterns which are exactly what robots.txt relies on.pyre2falls back toreif it doesn't support some RegExp features (like lookarounds) but it won't be the case here.My claims about potential speedup should be tested with your lib of course but nonetheless I think these are worth a consideration.