Open lpereira opened 1 year ago
Nice audit. Which of these looks like it would be the quickest to implement? (Maybe the first?)
I'm already looking at the first item.
At this point I don't want to change the representation of SRE opcodes, so, to store a string literal, I'm adding a literals array to the pattern object and referencing the string by its index; at a later time, making it inline or storing it in the info block (to improve prefetching, etc) is an option that I'll probably look at.
I have a working prototype in this branch. Needs quite a bit of cleanup but it seems to be working as expected.
This is cool. I think you probably already know this, but wanted to highlight in advance that the regex benchmarks in pyperformance aren't very representative of real world usage. I think they will be fine as guardrails against any unintended regressions, but I don't think they will be super useful for creating a good sense of "X faster". That said, there's lots of interesting regexs in the stdlib that do things that people probably do all the time (email, json, toml, csv, urllib, etc.) A benchmark over all the regexes that actually appear in the stdlib would probably be a much better approximation. Pulling all of the regexes across all the IETF standards would be an interesting exercise, too, since those tend to get used a lot of places in real code.
Other things I've found that look like good candidates:
\\n\\n\\n\\n+
, which appears in the stdlib, could be folded into \\n{4,}
, avoiding a lot of dispatches[0-9]
could be folded to \d
\s+
, could have an specialized opcode. Currently, this regex generates 5 instructionsFor instance,
[0-9]
could be folded to\d
That's not entirely correct since other Unicode digits can also match \d
. According to the docs [0-9]
is equivalent to \d
only if we're using a bytes pattern or the re.ASCII flag is used. My intuition would be that the vast majority of regex executions don't fall into those categories.
- Similar to the previous possible optimization: matching something like
[0-9]*-[a-z]{1,4}
should look for-
first using a fast byte scan, backwards matching[0-9]*
, and then finally[a-z]{1,4}
.
A generalization of this is to find a "plain" substring in the regex that you can look for with strstr
as "prematcher"/"bloom filter". We've had some success with this approach.
Any progress on this?
I paused the work on this but I intend to resume it soon.
I've spent some time looking over the
re
and_sre
modules and identified some things that could be improved:_sre
as there's an instruction dispatch per character, rather than using something likememcmp()
. This can be further improved because each character is stored in its ownSRE_CODE
, which is large enough to store aPy_UCS4
-- so, for the common case where regexes are matching ASCII strings, there's quite a bit of wasted space there.A|B
whereA
andB
are literals could leverage from a better strategy (e.g. using the Teddy algorithm, as used by Hyperscan).[0-9]{2}/[0-9]{2}/[0-9]{4}
, common when looking for dates, could be changed to something that searches for the/
characters in the right places, using a fast byte search, and then checked for the ranges. As a special case, consider[0-9]/
the same as[0-9]{1}/
so that we can also apply this optimization.[0-9]*-[a-z]{1,4}
should look for-
first using a fast byte scan, backwards matching[0-9]*
, and then finally[a-z]{1,4}
._sre
module for a regex: we can generate/eval some Python code that uses methods fromstr
directly. These methods are already quick, and even if they're not, optimizing them further would improve two things at a time. For instance, a regex like^(foo|bar|baz)
could be implemented asstr.starts_with(('foo', 'bar', 'baz'))
; this would be a pessimization if Teddy were to be used in the_sre
module, but it could be implemented instr
instead.There are many other things that could be improved, but I think this is a good representation of what could be done in the
re
module to improve things on a first pass.