Open tomyeh opened 2 years ago
Likely reason is that the RegExp implementation remembers each capture (meaning each character since the RegExp her captures each individual matched character) in case it needs to backtrack from the *
. It doesn't recognize that it actually never needs to backtrack the *
, since the continuation is irrefutable.
That makes the backtracking stack linear in the input, and for AoT, that stack appears to be limited in size. So, working "as intended", you're hitting a natural limit of the runtime.
The solution would be to rewrite the RegExp.
I'd probably just remove the parentheses to avoid the capturing entirely. If you really want to find the last matched character, I'd do it afterwards as
var input = "....";
var match = regex.matchAsPrefix(input);
String? lastMatchedChar;
if (match != null && match.end > match.start) lastMatchedChar = input[match.end - 1];
Otherwise, specify formally what you want to match and capture, then it should be possible to create a RegExp which does that more efficiently.
Well, it is only a simplified pattern to illustrate the issue, and we did manage to work around this limitation. However, it implies using parenthesis to capture a substring from user's input is unsafe. That said, as long as parsing the input provided by user, we can not use parenthesis at all!? After all, it is easy to run out of stack with even only tens thousands of characters.
I can't give absolute rules (the RegExp implementation may change at any time, and this is an implementation dependent issue).
In general, you need to be aware that the JavaScript RegExp specification is inherently backtracking.
Any time a RegExp can make a choice (|
, quantifiers like *
, +
, or ?
), it remembers the state before continuing, so if the first option fails, it can backtrack and try the other option. This is the backtracking stack.
How much state is stored depends on what can happen during the optional matching. For *
loops containing captures, since the next repetition can do the same capture again, it also needs to store the captures that might be overwritten.
That doesn't mean you can't use captures. Even without the capture inside the *
, you'd still have a lot of potential backtracking, just less state per character. Sometimes the RegExp engine can recognize patterns that allows it to drop some backtracking information, when it's clear that it won't ever be part of any matching result. That's an optimization, but not something I'd rely on if I can avoid it.
But yes, using captures inside a greedy quantifier is something that can blow up (and probably disables some optimizations that could otherwise help with the greedy quantifier backtracking).
Always be careful when you have
(?:[ab]*b)+
is bad, (?:[ab]*b)
matches the same thing more efficiently.If possible, make it easy for the RegExp engine to see where a *
ends, for example by having a quick negative look-ahead which ensures that doing one more iteration wouldn't work anyway.
For the current example, r'([ ABab\d])*?(?!=[ ABab\d])'
.
Here I use a non-greedy match followed by a negative lookahead. That's a pattern which heeds to maintain only minimal backtracking information.
(I guess volumes can be written about RegExp optimization).
Understand and thanks for your detailed explanation. However, running out of stack with a simple regex (([A-Z])*
and tens thousands of characters is, IMO, too limited.
BTW, I don't think ending with a negative look-ahead will help, since it'll run out of stack before reaching it. For this example, r'([ A-Za-z0-9])*(?!=[ A-Za-z0-9])'
still not able to run.
It does seem that the AoT stack (or heap) is extremely small.
If I run the code
void main() {
var re = RegExp(r"([a-z])*");
var s = "abcdefghijklnmopqrstuvwxy";
while (s.length < 5000) {
s += s;
}
while (s.length < 10000000) {
s += s;
var sw = Stopwatch()..start();
var m = re.matchAsPrefix(s);
sw.stop();
print("${s.length}: ${sw.elapsed}");
}
}
then it always ends at some point.
On the web (DartPad), it ends at
3276800: 0:00:00.025900
Uncaught RangeError: Maximum call stack size exceededError: RangeError: Maximum call stack size exceeded
(That's a JavaScript stack overflow from the JavaScript RegExp engine.)
Running it on the native VM, I end up at:
838860800: 0:00:11.292660
Exhausted heap space, trying to allocate 34359738400 bytes.
Unhandled exception:
Out of Memory
#0 _growRegExpStack (dart:_internal-patch/internal_patch.dart:152:24)
The RegExp engine used by both Dart, and most other browsers, uses heap memory for the RegExp stack, not the system stack, precisely because the RegExp engine would otherwise easily overflow the system stack with backtrack information.
For a dart compile exe
I get:
12800: 0:00:00.000275
Unhandled exception:
Stack Overflow
#0 _RegExp._ExecuteMatchSticky (dart:core-patch/regexp_patch.dart)
That's a wastly lower range, almost as if the compiled version cannot grow the RegExp stack at all.
It does look like a bug (or "implementation issue").
I can't give absolute rules (the RegExp implementation may change at any time, and this is an implementation dependent issue).
In general, you need to be aware that the JavaScript RegExp specification is inherently backtracking. Any time a RegExp can make a choice (
|
, quantifiers like*
,+
, or?
), it remembers the state before continuing, so if the first option fails, it can backtrack and try the other option. This is the backtracking stack. How much state is stored depends on what can happen during the optional matching. For*
loops containing captures, since the next repetition can do the same capture again, it also needs to store the captures that might be overwritten.That doesn't mean you can't use captures. Even without the capture inside the
*
, you'd still have a lot of potential backtracking, just less state per character. Sometimes the RegExp engine can recognize patterns that allows it to drop some backtracking information, when it's clear that it won't ever be part of any matching result. That's an optimization, but not something I'd rely on if I can avoid it. But yes, using captures inside a greedy quantifier is something that can blow up (and probably disables some optimizations that could otherwise help with the greedy quantifier backtracking).Always be careful when you have
- nested quantifiers. Can cause exponential blowup.
(?:[ab]*b)+
is bad,(?:[ab]*b)
matches the same thing more efficiently.- Multiple ways to match the same string (if something afterwards doesn't match, it'll have to go back and try all those ways to match the string, and then fail again).
If possible, make it easy for the RegExp engine to see where a
*
ends, for example by having a quick negative look-ahead which ensures that doing one more iteration wouldn't work anyway. For the current example,r'([ ABab\d])*?(?!=[ ABab\d])'
.Here I use a non-greedy match followed by a negative lookahead. That's a pattern which heeds to maintain only minimal backtracking information.
(I guess volumes can be written about RegExp optimization).
just removed + from my RegExp and it was not giving stackoverflow error anymore. Thank you so much. You save my day.
Any update about this issue?
I have a related bug, RegExp.allMatches
throws stackoverflow on flutter release mode:
const String pattern = r'("(\\.|[^\\"\r\n])*"(?=\s*:))|([{}[\],:])|(")|(\b(true|false|null)(?!\.)(?=\b|\s))|((-?)(\b0[xX][a-fA-F0-9]+|(\b\d+(\.\d*)?|\.\d+)([eE][-+]?\d+)?))|(//)|(/\*)|(\S)';
const String input = '{"data":"JABFwwAAADLQNgS4XtL1yngIEEjDAH0TSvg39tIS/x98JGP7fwnXPM5jagwBtknnUGIIAABFwksAACADbS8HeETSI5KOZvG1ucUf6F648f+hz/iXpIVqvBzgLjU+CYWXAwBAgXAHAABkqOVOgFXNOwGukbS4268TULsAfRVS+Df204z/vfs09v6HMrb/z91Mc6RHm3cC7JD0gcSdAABQBAwAACBjcx0CBNYtQN+FGv8f731No+9/0Pf4n8VxpkOAuyS9LzEEAIC8YwAAAD3QNgS4QdJVkk5r/+cC6xXAi9DCv7Gn3Md/6qic+V8k3S3pPYkhAADkGQMAAOiR6YYAAbYK4A3x34P/fvf/CkMAACgIBgAA0EOtQwDXGAJcKWmJ73UBRRdi+MtIru40+r6f+J/nkaZDgO9K+qPEEAAA8ohPAQCAHko/HcBJ7yfGPCzp/yvpkO91AUXlXKjxb+TqSVHjX5IWybjLJG2WtEbi0wEAII+4AwAA+uAvW+4EMM7dKOkKcScA0JUgw19qxn9do+9/UNT4b3WkeSfAvZLelbgTAADyhAEAAPRJhyHAtyUt9b0uIO+CDX/Ja/z38FjTIcB9kv4gMQQAgLzgJQAA0Ce7my8HkPS+M+bvJP1XSZ/5XheQZ8R/b/T4WBc3Xw5wh6S1Ei8HAIC84A4AAOizljsBVpvGGwNyJwDQJujwl0KO/1ZH5Mz/LOl+Sb+XuBMAAHxjAAAAHjAEADoLPvyl5rv9J32Pf09HyxAAAHKEAQAAeMIQAJgqiviX5JL+f9Sf56NlCAAAOcEAAAA8YggAxBP+jb06jb4XVfynjsiZf5L0oKTfSQwBAMAHBgAA4BlDAMSM+O/hf8/3hk92uHknAEMAAPCEAQAA5ABDAMQmpvBv7Df6+E8xBAAAjxgAAEBOtAwBVhnnbpR0haQlvtcFZCm28G/smfhvwxAAADxhAAAAOcIQACEj/nsb/wU7XoYAAOABAwAAyBmGAAhNjOHf2Hcj/j/avU9jH35I/J+MIQAA9BkDAADIobYhwA2SrpJ0mu91Ad2INfwbe2+J/w+a8d+Dq64AjpghAAD0EQMAAMgphgAoMuL/fX20e6/GPvho6jP/GV55BXTEDAEAoE8YAABAjnUYAlwtabHvdQGdxBz9E1rif7Q9/lPzvPoK9JjTIcADkn4vMQQAgF5gAAAAOdc2BLhe0rWSFvleF9CK+Ndk/O/aq7GPPpJMepnV4XJrjldggR/zkeYQ4H4xBACAnmAAAAAFwBAAeUX4N7XE//GW+J88nvkPASI5aoYAANBDDAAAoCDahgDXSbpe0kLf60KcCP8Wafy/ukfHP/q4Ef8tV1jzHQJEeNTpEOB7kv4gMQQAgKwwAACAAmkZAqxs3gnAEAB9R/y36BT/qQyGABEfNUMAAOgBBgAAUDBtQ4DrJN0oadj3uhAH4r/FTPGfavsfuU7/w2n/uegdkTP/i6R7Jb0rMQQAgPliAAAABdRhCHCTpCHf60K4CP+Tjf7xvUb8f/zJzP9gl0MAjnoKhgAAkCEGAABQUG1DgGsl3SyGAMgY4d/Z6B/f04e/2q0Tn3w6u3/h/9/encVIch/2Hf/+e2Z3KXFJSuIhAwES5FGKKFGyZMv2u4C8WI8J5Fh5sANLFA/JkqzbVizrsgLkkvMUwbYkG84rgUCBnwLIEs89xCWN+CUBzN3Z2WuW5B4zO+T0Pw/d1VPdU31WVdf1/QDU7s701NT8a9RVv1//+18LlAAx43ECDkuAbwIXwRJAklblaUaSGiyjBPgccFfV+6XmM/hPd3trmyunlwj/iRklQJzxOAFwc7gmwJ8A22AJIEmr8BQjSQ2XLgGAx0KMn8cSQDkY/qe7vbXNlVNnuLPz6mpXURklQCzgFoEdcXM4E+AbWAJI0kp6Ve+AJCmfsx8Zzfq/DHw/hvA94E7V+6XmidHwP8th+L8OxNXerB+z/hpnPk4jJwnx48DXgHcC3PVbf171PklSo9gvS1JLTMwE+FSI8QvAiar3S/Vn6J/v9tY2V144w53rk9P+w0pXU3HFuwMIGMwE+BHwdeBynz77f/U7Ve+TJDWCpxVJapFUCfAgg7cDfBE4XvV+qZ4M/ou5vbXN1RdOc+f6q1NemF+8BJj2Xv+Y9cEZH+q6GOPNCKMSALAEkKQFeEqRpJZ55G93CTESQ3iQwUyAL2EJoBSD/+LS4T+xagkQp3zZ0c9bAswSD3+BkxLgj4ArYAkgSfO4BoAktczZj7yFGALX7r/rCvBnMYRvA/tV75fqwfC/uKzwD9Oy+PQ1ASIz3tKf+kTI+uCMD3VNjDEd/gFOBvg48MfAAwDHf+sHVe+mJNWaBYAktdT91/aIIaRLgDeq3idVxwX+lnN7a5urzw/Df8a4LVoCLDTklgBzxem/vHcH+G0GdwawBJCkOZxQJkktlawHEEMgxJi8HeDLwLGq903rY+hf3ij8v/rqeOjOuGqa9naAmPMWgb4d4FBc7Jf41vDtAF8FroFvB5CkLB07hUhSt2SUAI+GGL+CJUDrGfxXM5r2v/Nq9gMWKAFGK/pbAuS2YPhPWAJI0hwdOX1IUnel7gwAg7sDWAK0mMF/dXPDf2JKCRCzHmgJsJIlg3/arQg/Br7ch50elgCSlNbiU4ckKZFRAvh2gJYx+OdzGP6vs9Dl0cRDpi/0ZwmwrBzhPzEqAYAdsASQpEQLTxuSpCxTSgBvEdhwBv/8Dhf8u5766GIlQPb0/4wHWgIspIDwn7AEkKQMLTplSJLmmbIwoCVAAxn8izEK/zvD8D92ZTT7MilOeYglwPIKDP5pR0qAeO3/8cb/+pOqf1xJqoy3AZSkDjn7kbcAEGKcvEXgftX7psV4O7/ijE/7Hxob2+yBHpvun+MWgQvpwC0CSwr/MLhF4L8BvgW8HSDc/8+r/nElqVIWAJLUMVNKgO9gCVB7Bv+CRNjd2ubaC2e4c+165uen/CM7Z1sCrKzE8J9ISoBvMywBjv/WD6r+sSWpMhYAktRB6RJg4+DgCvD9YQlwp+p901G+6l+gGNm9uM3VF85w59rO9JnzEyXA9EX+sh4/UGgJkLndZpcAawj/iXQJ8A6wBJDUXQ1/x5gkKa9H/nY3mQ3wEIM1Ab4AnKh6v2ToL1yM7F68xNXnB+E/+/31E1+yxJoA0x5S2JoAGXcemPpNa36Ft8bwn+bCgJI6r+anB0nSOqTuEPAQ8FiI8fPAXVXvV1cZ/EuQhP8XznDnyjUIw0ugGSXA6N+WAIWqKPwnRiVAhJ2AJYCkbqnpqUGStG6WANUz+JdkWvhPTJQAmYfBEqAQFYf/hDMBJHWWawBIkoDDdQGAy8D3gf8A7K68QS3M9/iXaBj+r71wNjv8w2Lvm1/g7gDztlnYmgATj2/CmgAxxrqEfxi/O8D94JoAkrqjZr2wJKlq6ZkAIcbHgd8H3lr1frVRffJQS8XI7tYlrp06w96VqxCmv+4RYaE1AZwJsLwaBf9JtyL8CPgqcA2cCSCp/ZwBIEkak54JEEP4r8B/BG5XvV9t4iv+azAK/2fZu3w1+5X/5KFH/sLCdweYvx9HP9SlmQA1Dv8wmAnw28CfAA+AMwEktZ8zACRJmSZmAjwBfBq4u+r9arJ6Z6EWiZHdrW2unfoFu5euEHrpV9dnxuTJhzgTYEU1D/6TkpkAXwOugjMBJLWXBYAkaaqJEuBJ4EksAZbSrBzUAmPh/zKhl5rsOLzqiUsGdkuA5TQs/CeSEuAPgStgCSCpnSwAJEkzpUqAdw5nAjwBnKx6v+qumRmo4ZLwf/oX7G5PhP/kIRkzAaayBFhKQ4N/2q0IPwS+zmAxVEsASa1jASBJmmuiBHgSeAy4p+r9qqPmZ6BmijGyt7XNtdMvsrt96Uj4HzsslgAL7+qiWhD+EzeHMwG+jiWApBayAJAkLcQSYLb25J/mif3I3sVtdobhn1T4nx/aLQHyalH4TyQlwL8HLoElgKT2sACQJC1sogT4NPAocG/V+1Wl9mWfBgkQD8bDf+j1RoF37qGxBFh4V7O0MPin3Ry+HeAbwDZYAkhqB28DKElaWOoWgZdiCP8J+G/A61XvVxW8lV/FQsgM/0vJuv/f3Md6i0BoffgHOBng4wzuDPBL4C0CJbWDMwAkSUvLmAnwSeC+qvdrHdqfexogBOLBAXsXL7Fz5kV2Lx6G/9HxGa34v8j2jvxlgcd2dyZAB8J/2s0Ifwl8E7gIzgSQ1GwWAJKklUyUAJ8BPkGLS4BuZZ4amwz/W5cIG73s42MJkLntVUuAjgX/tKQE+BawBZYAkprLAkCStLK2lwDdzTs1NRb+f8HuVrLg34zLGUuAzG0vWwJ0OPwnbgzXBLAEkNRoFgCSpFwySoDfA95W9X7lYdapoanhf/SAGV87+MMSYNp2p5cABv8xSQnwbeACWAJIah4LAElSbm0pAcw6NTUM/7sXL3H99C+4PZz2n/HAGdsY/GEJMG27RzcUV1kdsP1uDN8O8B0sASQ1kAWAJKkQqRLgl4YLAzamBDD411gS/rcywn/mVYwlwLLbzioBRq/8e6WYJSkBvgucB0sASc3h07okqTBNKwEM/jWXCv87p38xWvBv/DGZXzhjm4M/LAGytztrMUWNsQSQ1Eg+pUuSClX3EsDQ3xCLhP/RYxf+4NinLAHG/9mfvIfikrvaQYclQByWAH9tCSCp3nw6lyQVro4lgMG/QQLEg/5i4T/1NQt+cOxTlgDjC/2teovADrsR4S+IfA94BSwBJNWbT+WSpFLUpQQw+DdPvx/Z29rm+ulfsHvxEqHXWzKoz/3g2Ke6XAJkLfRnCbCcGHmdwUwASwBJtefTuCSpNFWVAIb+5hqF/1PD8J965b/yEmD0+OaXAEdu77fCLQI19lxjCSCpEXwKlySVKuMWgZ8A7ivjexn8m+lwAbrI3oVtrp86y+7Fy4Te0dX+LQGOfmjZEiBO+z+KJcDCpgyhJYCk2uv407ckaR3KLAEM/c01duhiZPfCNjunzrJ38RKhtzH+YEuAmQ9ZpASIi/6fJXOfLQESc4bREkBSrXX0qVuStG4ZJcAngXtX3Z7Bv5kyD1uM7G5ts/NCEv6Taf8Tlym1LAHmbHPR/S65BFg4/M/c526XAEsMoSWApNrq0NO2JKlqEyXAp4FHWaIEMPQ329Twf2EY/rfT4T9hCbDMzza57Vl39ltm+10vAVZ47rEEkFRLHXjKliTVySolgMG/2aYeviT8P3+GvUuXM8J/whJgmZ8tvabC2AMtAVaS4/nHEkBS7bT46VqSVFcZJcCngHsmH2fwb7aZhy8d/reH4X/mVYklwKI/W4yxkFsEzt7nbpQABTwHJSXAnwLnwRJAUrVa+FQtSWqCiRLgSeBx4KShv/nmHsIk/D93hjuXr0BIXY5YAmRsYvHLtXT0twRYXcHPQzcYlADfxRJAUsVa8jQtSWqisRKgH5+MwxKg6v3SahbKTLPCf8ISIGMTs7cXYyzkFoGL7vPYtkO7LidLKiEtASTVQi//JiRJWs3Zj7xl8JfIpdgP/xn4L8CtqvdLy1su/J+eHv7nbixO/edCMTQu/MGxTy0cceORv8zdj7DAY2Ztb/Re/4yHZG87LtFozNiFEAz/i7sH+LfAF4B/AnD8Yz+o+seV1EHtetaWJDXSIz8ZzQR4iBCfBJ4E7q56vzTfwnlpLPxfXSw4OhMgYxOH/5h6a7+yZgIEiC0L/GlrevtRMhPgO8AFcCaApPVq77O4JKlRJkqAJ4DPAG+ter+UbamsFCO7Fy5y/dkz7F1ZMPwnLAEyNzE3rBZVAgSIyQNaetVYwbojN4AfAt8CtsASQNL6tPSpXJLURBklwO8Db6l6v3Ro6awUYffCFtefOc2dK9dG4T8u+crzwp9scwlAepG/fLcIPPLAMP7Ptof+0XhUt+ioJYCkSrT8aV2S1DQTJcDjwGexBKjcqjlp9/wWO8+cYv/qTuqjlgDz9iNr20cX+iuwBAjhcHp/R64Oa3DHkaQE+CZwESwBJJWvI0/xkqQmySgBPgfcVfV+dVGejJQd/hOWAPP2I9n2kff6F1UChDD8c/g/HbkqrEHwT7vJoAT4BrANlgCSytWRp3pJUtOMlQDwGCF+HkuAtSov/CcsAebtR3+hhf4WLwHi2FswuhP6EzUL/wlLAElr07GnfUlSk2SUAH8AnKh6v9oub0baPb/FztOn2L+2s8CjLQGyxNQWl707QNbjYgitX8V/npqG/0RSAvwxcAksASSVo7tnAUlSI0yUAJ8ixC9gCVCKIvLRIPy/wP7V64MPLHSlYQmQNpryn+MWgaOF/HrDPzt8xVfz4J92E/gR8HXgMlgCSCpeh08HkqSmSJUADzKYCfBF4HjV+9UmhYX/n7/A/rXr45+wBJixqcNtxqykulQJMFzIz9A/0qDwn7AEkFQqTw2SpEZ45Ce7ECLE8CCDmQBfwhIgt6Ly0dTwn7AEmP74eV8wb78DxN4w/Hd4iv+kBob/RFIC/BFwBSwBJBXHs4QkqTEe+cku7B+H4/uWADkVmY1G4X/n+uwNWwIccfhe/yXvDpAO/b7aP6bBwT/tFoMS4GvAVbAEkFSMXtU7IEnSUo7vQwxXgD8jhm8Db1S9S00SKSP8Pz8//CfffKE9HEz2WOqHWvSTqX8ulJnjwh8c+9S8bccYB1P+4wLbTD6dhP7NHv2NDWKvN3jF3/A/0pLwD3A38NsM7gzwAMDxj/2g6n2S1AKeMiRJjTJaDyAGCDGZCfBl4FjV+1ZnZeSi3fNbXP/Z89zZeTX1HVa7L/20B7VxJsDs9/of3c/klf5o2J+pReE/LZkJ8FXgGjgTQFI+nkYkSY2TUQI8SohfwRLgiLIy0SD8v8D+zg7EyaBuCZD+VLLtOC+hhuH/GPqX0tLgn2YJIKkwnlIkSY2UujMADO4OYAmQUmYmGoX/azvj39MSIPNTMcb52w4QN3qG/iV1IPwnbgE/Br4M7IAlgKTVeHqRJDVWRgnQ+bcDlJ2Hds9vcf3nL7B/dSfz85YAE+MR4+y3A/TC8L+eV2VL6lD4T1gCSMrNU40kqdGmlACduzvAOrLQvPA/2hdLgKPT/VMlgKE/nw4G/zRLAEm5eNqRJDXelIUBO1ECrCsL7b6yxfWn54f/0X51tgTICKiBwdT+jUHoX2r/Nabj4T9xpAR443/+IfG1C1Xvl6QG8BQkSWqFLpUAa81AceKV/yWuHLpWAhy+6h8OQ39vGPyDl1x5GPyPSEqALwHXwZkAkhbj2UiS1BqjEqDfg17/QeAxQvwiLSkB1p6BYmT3wsVB+L+SeuXfEmD8Z0w/aji9vz9c0E/5Gf6nsgSQtDTPTJKkVhmVAL0+9HtJCfAF4ETV+7aqSvJPjOyev8j1p0+xf+Xa0c9bAgw/FyEkoX/4vn4VxvA/l2sCSFqKBYAkqZUe+ckuhAgxPMSgBPgDGlYCVJZ95oX/RIdLgGQxv/5GIG4Y+otm8F/KqAQIsNMH3rAEkDSFBYAkqbVSdwhISoDPA3dVvV/zVJp90uH/8hUIc8Jtl0qAAP3hYn59V/AvjeF/JbeAH8f0woCWAJIyeOqSJLXaRAnwOCF+jpqWAJXnnmH4f/WZU9y5dAVG72Gfc7nQ1hIgDF/p3+jRH77ar/IY/HO7FSfvDmAJIGmCZzJJUuuNlQAhPg58FnhL1fuVqEXuScL/06fYv3SF2Ju8ROhICRAG/5Ms5NfvBW/btwaG//yGQ5i8HeArwDWwBJA0zlOaJKkTJkqAJ4DPAG+tcp9qk3lSr/zvbx++8n80+La4BAiB2GOwev9wBf9oKl0Lhzm/iSG8BfwI+CqWAJImWABIkjqjDiVA7bLOKPyfZn/7cmra//DTbS0Bklf6w3CK/2ZvbNaD4b98DnExpgxjUgJ8DbgKlgCSBiwAJEmdMlECPAk8Cdxd9vetZdZJh/+Ll2DKLezaVALEEAbv7d/oDV/tnyg8TKVr4TAXY84wWgJIOsICQJLUORklwBPAyTK+V21zTn8Q/l979jR7Fy8R5ty/vtElQBi+0t8Lg1f6M27bZ/BfD4e5GEsMY1IC/CFwBSwBpK6zAJAkdVKqBHjn8O0AhZYAtc456fC/NQz/C1wRNKkEGLzSP1i5Pw5f6Y8h+4sM/+vhMBdjhWG8BfwQ+DpwGSwBpC6zAJAkddZECfAk8BhwT55t1jrjBOCgz+75bV579gx7W9vjr/w3vAQYTe/vDd/TPyP0g8F/XRzm4uQYypsMZgJ8HUsAqdMsACRJnVZECdCIfBMC8eCAvfMXh+F/yrT/hpUAsZeE/nC4gn9v9tcZ/NfHoS5OAUN5k8FMgD8GLoElgNRFFgCSpM6bKAE+DTwK3Dvv6xqTbY6E/+SV/ymXAXUvAcLg1f70e/rnhX4w+K+bw12MgocxKQG+AWyDJYDUNRYAkiSxXAnQqFwzGf4vbBPGFsFrSAmQhP4QBlP7NzfoD1/9X2h7ptG1caiLU9JQWgJIHWYBIEnSUEYJ8EngPmhY6E+EAMPw/2pm+B89cMrXz/8WZZcAsReG//WImz36Cy5YONqGaXTtHPJilDyMN4G/BL4JXARLAKkrLAAkSUqZKAE+E+ETDEuARkmF/9eePcPu+VT4zzz716QESE3vjxs9+sP/lr1iMfivn0NejDUO4w0GMwG+BWyBJYDUBRYAkiRNSJcAMcTP0LQSYDjt/04y7f/8RcLGxniwqFkJkH6lv7+5WugHg39VHPZiVDCMlgBSx1gASJKUIaME+D3gbVXv11xTwn+iTiVA3EiF/o1Af3Mj15WJ4X/9HPLiVDiUSQnwbeACWAJIbWYBIEnSFI0rAeaE/0SVJcD4K/1h4RX8Z35PU2glHPbi1GAobzBYE+A7WAJIrWYBIEnSDKkS4JfiYGHAepYAE+/5nxb+E+ssAWKvByHQ7wX6x3qDBf028l+CGPyr4bAXp2ZDmZQA3wXOgyWA1EYWAJIkzVH7EiAJ/6+kwv/mxtwvK7UEGL7S3+8F4mZvtKBfEQz+1XHoi1PTobQEkFrOAkCSpAXUtgRIv/L/zMQr/4sE9bFtZX6DKd83e19iLwzez78xvG1fQaEfDP5VcuiLVfPhtASQWswCQJKkBdWuBJgV/kePmb+ZXCVAYLSIX94V/Kfun+mzUg5/cRo0lDeAvwC+B7wClgBSW1gASJK0hNqUAAE46E+E/x6Zp/YSSoDYC/STqf3DP2MJVxWG/+o49MVq4HC+zmAmgCWA1CIWAJIkLSmjBPgEcN/admAs/J9m7/z2MPynH5DxNXPMKwFiL5nWP3y1/1iPGMq5lDD4V8ehL16Dh9QSQGoZCwBJklaQcYvA9ZUA/cje+S1ee/o0excmw3+imBIgDhfx62/0RlP88962b+b3N31WyuEvVkuG0xJAahELAEmSVlRJCTAZ/nuz3m+/WgnQ74Wxqf1lh34w+FfN4S9ey4bUEkBqCQsASZJyyCgBPgncW8o3i5G9Vwbh/86FbeilXvnPUwJEiBthdKu+QfAPh4v5lZRkDP314GEoVouH0xJAagELAEmScpooAT4NPErRJUAS/n9+mjtbw1f+WWTxvumfiL0wCPqbqVf7Sw79gx+lxRGpQTwMxevAkFoCSA1nASBJUgEySoBPAfcUsvFR+D/Fna1Lo/A/+nT6H7NKgAj0GAX+/sZw9f6NUNpifuM/RgfiUQN4GMrRoWG1BJAazAJAkqSCTJQATwKPkbcESIX//a1L49P+0w9L/yNMfCIkoX/j8JZ9awr9gx+hQ9Go5jwUxevokCYlwJ8C58ESQGoKCwBJkgqUUQI8DpxcaWMxsvfKRV772QvsX7wMcxbimywBDqf2bwxu3bdR/mJ+R3+EjsajmvEwlKPjw3qDQQnwXSwBpMawAJAkqWCFlABJ+P+759nfvgxh+Mr/nDN37AUOjm1wcGyDuBmIvfWH/sHudzwa1YiHohwOK2AJIDWOBYAkSSWYKAGeAJ4E7l7oi4+E/+R0Hcb+GD28F+gf2+Bgs0dM3ttfQegf7LqxqC48FOVwWI9ISoDvABfAEkCqMwsASZJKkioBHhrOBJhfAozC/3Psb19Jhf/E4N+D0N8bvtLfI/Z69DeqO60b/OvFw1EOh3UqSwCpISwAJEkq0UQJ8ATwGeCtmQ+eE/5jGLzS39/coH9ssJBfv9er9Gxu8K8XD0d5HNq5bgA/BL4FbIElgFRHFgCSJJUsowT4feAtYw+KDFb7/+lz7F+6OnaGPji+OVjQ79jGYPX+3vpW8J/G4F8vHo7yOLRLSUqAbwIXwRJAqhsLAEmS1mCiBHgc+CypEmDvH7d47afPDsP/8JX+YxscHOuNVu+vOvSDwb+OPCTlcWhXMioBYuRijJGDv/ndqvdJ0lD1VxKSJHVEugQAHo8hfg64a+8fL/DaT59j7/qr9Dc3ODjeG63eX9VifpMM/vXjISmPQ5tPjPFmjPwQ+AawDVgCSDVRj6sKSZI6YmwmQP/gsVv/+Mrnr58+d9fe9VeHi/nVJ/SDwb+OPCTlcnjzST1nWAJINdSregckSeqSs/9yNOv/8hs3bn1/5x/+4Xu3b924c3Bi8D7/uoT/GKPhv4Y8JOVyePOZeM44GQIfB74GvBNg41//96p3Ueo8CwBJktYsKQF+5V89dPn1Cxe+Hzf4LoE7Ve9XwuBfPzEa/ssUMfznNeV5IykB/ojBW58sAaSKbVS9A5IkddH2X32T//N//ym9zc1bwMsE3oDwa1R0bjb015eHplwOb35znj+Oh8C7gHcAzwO3eu/5TeJLT1W921InWQBIklSRN198is33fRRiuE3g5cEsgPDrrPH8bPCvLw9NuRzeYiz4HJIuAZ4DblsCSNWwAJAkqUKDEuA3YfPN2/R7aysBDP715aEpn0NcjCWfR46HwLuB+4FnsQSQKmEBIElSxTbf91Ho9yDE2xCSEuA3KOE8bfCvNw9P+Rzi/HI8jyQzASwBpIpYAEiSVLHRWwEIpZUABv968/CUzyEuRgHPJekS4Blg1xJAWh8LAEmSamBKCbCXtwQw+Nebh2c9HOZiFPh8kpQAD2AJIK2VBYAkSTUxVgLAbeDvVy0BDP715yEqn0NcnBKeU5IS4EHg51gCSGthASBJUo0clgDAYQmw8NsBDP715yFaD4e5GCU/p1gCSGtmASBJUs1klAAz7w5g6G8GD9P6ONTFWNNziyWAtEYWAJIk1dBYCTC+MOCoBDD4N4OHaX0c6uKs+fnFEkBaEwsASZJqKnNhQLgTI5kzAVQ/hv/1caiLUWGxmC4BfgbsWQJIxfPiQZKkGkuXAL1/xu14nZdDZJ/Ah4HNqvdP2Qz+6+NQF6cGs4qSEuAhnAkglcICQJKkmktKgLgD/R63Q+TlAG9YAtRP9fmpWxzu4tQg/Cd8O4BUIgsASZIa4M0Xn+LNc09x7D0fheHdASwB6qM+2akbHO5i1Sj8JywBpJJYAEiS1CBvnnuKzYc/CnALRjMBfhVLgMrULzu1m8NdnBoG/7SxEiAEdoMlgJSbBYAkSQ2TUQIcEPgQcKzqfeuSemen9nG4i1Xz8J9wJoBUMAsASZIaaLIEIHAAlgDr0Izc1C4OebEaEv4T6RLgaSwBpFwsACRJaqh0CRAGJUAf+CCWAKVoVmZqB4e8WA0L/mlJCfAA8AyWANLKLAAkSWqwiRLgJQIRS4DCNTc3NZdDXqwGh/9EugR4FrhtCSAtzwJAkqSGyygBAD4AHK9635qu+ZmpeRzy4rUg/CeSEuB+LAGklVgASJLUAlNKgPdjCbCS9uSlZnHYi9Wi4J+WlADvAJ7DEkBaigWAJEktkbEmQMASYCntzEv157AXr6XhP3E8BN7NoAR4HrhlCSAtxgJAkqQWyZgJEIBHgBNV71udtTsr1ZtDX7yWh/9EeiaAJYC0IAsASZJaJqME6GEJkKkbOameHPpydCT8J5IS4O3AKSwBpLksACRJaqGMEmADeC+WACPdykn14tAXr2PBPy0pAd4GnAZuWgJI01kASJLUUpYA2bqbk6rn0Jejw+E/YQkgLcgCQJKkFptSAjwM3FX1vq2bGalaDn85DP8jSQlwH3AGSwApkwWAJEktl1ECHKNDJYD5qFoOfzkM/pnSJcBZ4IYlgDTOAkCSpA6YKAHOdaEEMB9Vz0NQDsP/TCeGJcC9WAJIR1gASJLUEVNKgPfQshLAbFQ9D0E5DP4LS0qAe4AXsQSQRiwAJEnqkDaXAGaj6nkIymP4X5olgJTBAkCSpI6ZsiZAY0sAc1E9eBjKY/hf2QkYlQDngNctAdR1FgCSJHVQqgS42eQSwFxUPQ9BeQz+q0sNXVICnMQSQLIAkCSpq5pcApiL6sHDUB7D/+oyhs4SQBqyAJAkqcOaVgKYierBw1Auw//qZgxdUgLcDbyEJYA6ygJAkqSOa0IJYB6qBw9DuQz++SwwfJYA6jwLAEmSNFkCJHcHeJiKSwDzUH14KMpl+F/dkkNnCaBOswCQJEnA1FsEVlICmIXqw0NRPsP/6lYcOksAdZYFgCRJGqmyBDAD1Y+HpFwG/3xyDp8lgDrJAkCSJI3JKAE2gfcyuGAunBmofjwk5TL451Pg8FkCqHMsACRJ0hETJcBLBDYouAQwA9WPh6R8hv98Shg+SwB1igWAJEnKNKUEeB85SwDzT/14SNbD8J9PicNnCaDOsACQJElTZZQAPeARVigBzD714yFZD4N/fmsYwnQJcA5LALWUBYAkSZppSgnwfuD4Il9v9qknD8t6GP7zW+MQngDeDZzEEkAtZQEgSZLmyigBAnNKAHNPPXlY1sPgn19FQ5jMBLAEUCtZAEiSpIVklAAAHyBVAph56stDsz6G//wqHsJ0CfAicMMSQG1hASBJkhY2pQT45Rg5VvW+aTrj6HoY/ItRk2FMSoB7sARQi1gASJKkpUyWADEQgQ+CJUDd1CNHdYPhP78aDmFSAtwLnMUSQC1gASBJkpaWLgH68FIYZM0PYQlQC/XLUe1m+M+vxkOYLgHOADctAdRkFgCSJGklb557ijAsARiUAAdYAlSuvjmqfQz+xWjAMCYlwH1YAqjhLAAkSdLK+ueeYuOwBHgZOAjwK8Bm1fvWRfXPUe1h+C9Gg4YxuUXgfcBpLAHUUBYAkiQpl4wS4M0AH8YSYG2ak6Gaz+BfjIYO43EGMwHeBpwCblkCqGksACRJUm4ZJcAblgDr0cwc1UyG/2I0fBgtAdRoFgCSJKkQGSXAviVAeZqdoZrF4F+clgxlUgK8HXgeSwA1iAWAJEkqzJSZAL+G1xyFaUd+ag7DfzFaOIxJCfAO4DngtiWAmsCTsSRJKlRSAgS4zaAEuBPg1/G6I5f25ad6M/gXp8VDaQmgxvFELEmSCtc/9xS9hz/KAdzuWQLk1t78VE+G/+J0YCiPM7g7wP3As1gCqOY8CUuSpFJsPPxRekCE2+GwBPgNvP5YWPuzU70Y/IvVoeFMZgJYAqj2PAFLkqRSpN4KYAmwpO7kpvow/Beno0OZLgGeAXYtAVRHnnwlSVJp0iUAh2sC7FkCZOtmbqqWwb9YHR/OpAR4AEsA1ZQnXkmSVKrUnQFgUAL8PZYAY7qdmapj+C+WwwkclgAPAj/HEkA140lXkiSVbkoJ4NsBMPxXweBfLIfzCEsA1VanT7iSJGl9MkqATt8dwMxUDcN/sRzOqSwBVEudO9lKkqTqpEuAiYUBO1MCmJeqYfAvnkM6lyWAaqcTJ1pJklQfM+4O0OoSwKxUHcN/sRzOpaRLgJ9F2AuWAKpQa0+ykiSpvtIlQP+wBNgP8GFgs+r9K5JZqToG/+I5pCs5DrwrwkMMZwJYAqgqFgCSJKkS6RIgHK4J8EabSgCzUnUM/8VzSFcXicOZAGH0dgBLAFXBAkCSJFWmf+4p+ueeojdYFyC5O0DjSwBzUnUM/sVzSPOJh88IlgCqnAWAJEmqXGpxwFsczgT4VRpWApiTqmX4L55Dmk88+qxgCaBKWQBIkqRayCgBDgJ8CDhW9b7NY0aqlsFfdRNnPytYAqgyFgCSJKk2JkuAAAfUuAQwdlbL4F8eh3Z1cbFnhnQJ8DSWAFoTCwBJklQrGSVAH/ggNSoBzEbVM/yXx6Fd3YLhP5GUAA8Az2AJoDWwAJAkSbUzUQK8FAaZu/ISwFxUPYN/eRzafJYM/4l0CfAscNsSQGWyAJAkSbWUUQIAfIDBBfNamYvqwfBfHod2dSsG/7SkBLgfeA5LAJXIAkCSJNXWlBLg/aypBDAT1YPBv1wO7+oKCP+JpAR4B5YAKpEFgCRJqrWMNQECaygBzET1YPgvj0ObT4HhP3EcePewBHgeuGUJoKJZAEiSpNrLmAkQgEeAE0V/LzNRPRj8y+Xw5lNC+E+kZwJYAqhwFgCSJKkRMkqAHgWWAOahejD4l88hXl2JwT8tKQHeDpzCEkAFsgCQJEmNkVECbADvJUcJYBaqD8N/uRzefNYU/hNJCfA24DRw0xJARbAAkCRJjVJUCWAWqg+Df/kc4nzWHP4TlgAqnAWAJElqnCklwMPAXfO+1hxUHwb/8jnE+VUU/hNJCXAfcAZLAOVkASBJkhopowQ4xowSwBxUL4b/8jnE+VUc/hPpEuAscMMSQKuyAJAkSY01UQKcyyoBanH5rhGD/3o4zPnUJPinnWBQAtyLJYBysACQJEmNNqUEeE9c4O0AWi/Df/kc4vxqGP4TSQlwD/AilgBagQWAJElqvMkSgGEJgCVALRj818Nhzq/G4T9hCaBcLAAkSVIrTK4JgCVA5Qz+6+Ew59eA4J+WLgHOAa9bAmhRFgCSJKk1UiXATSwBKmX4Xw+HOb+Ghf9EUgKcxBJAS7AAkCRJrWIJUC2D/3o4zMVoaPhPpEuAl7AE0AIsACRJUutYAqyfwX99HOpiNDz8J5IS4G4sAbQACwBJktRKlgDrY/hfH4c6v5YE/zRLAC3MAkCSJLXWRAmQ3B3gYSwBCmHwXx+HuhgtDP8JSwAtxAJAkiS12pRbBFoC5GDwXy+HuxgtDv8JSwDNZQEgSZJazxKgGAb/9XK4i9GB4J9mCaCZLAAkSVInZJQAm8B7GVwwawaD/3o53MXpWPhPWAJoKgsASZLUGRMlwEsMroUsAaYw+K+fQ16cjob/hCWAMlkASJKkTplSArwPS4Axhv/1criL0/Hgn2YJoCMsACRJUudklAA94BEsAQz+FXDIi2P4PyIpAU4yeOuTJUDHWQBIkqROmlICvB84XvW+VcHgv34OebEM/1NZAmjEAkCSJHVWRgkQ6FgJYPCvhsNeHIP/QiwBBFgASJKkjssoAQA+QAdKAMP/+jnkxTL8LyVdArwI3Aj/4jeJL1sCdIkFgCRJ6rwpJcAvA8eq3rcyGPyr4bAXy/C/khMx8q5AuIekBHiPJUCXWABIkiSRWQJE4IO0qAQw+FfDYS+WwX91w9/FE8C7AuFe4CyWAJ1iASBJkjQ0pQT4EA0vAQz+1XHoi2X4X93E72K6BDgD3LQE6AYLAEmSpJSMEuCAhpYABv/qOPTFM/yvbsrvY1IC3IclQGdYAEiSJE2YKAFeZlAC/AqwWfW+LcrwXx2HvlgG/3zm/D6eAN49LAFOYwnQehYAkiRJGTJKgDeBD1PzEsDgXx2HvniG/9Ut8ft4nMFMgLcBp4BblgDtZQEgSZI0RUYJ8AY1LQEM/tVy+Itn+F/dCr+PlgAdYQEgSZI0Q0YJsE+NSgCDf7Uc/uIZ/PPJ8TuZlABvB57HEqCVLAAkSZLmmDIT4Neo+FrK8F8th794hv98CvidTEqAdwDPAbctAdrFAkCSJGkBqRLgNoMS4A7w61R0PWX4r45DXw7Dfz4F/l4eZ7AwoCVAC1kASJIkLagOJYDBv1oOf/EM/vmU9DuZlAD3A89iCdAaFgCSJElLGBYAMF4C/AYlX1cZ/Kvl8JfD8J9Pyb+XydsBLAFaxAJAkiRpCalZALCGEsDgXy2HvzyG/3zW9LuZLgGeAXYtAZrNAkCSJGlJU0qAPQosAQz+1fMQlMPgn9+afzeTEuBB4GksARotVL0DkiRJTXXsYz9I//NB4FHgS0Bv1W0a/KvnISiP4T+fin83bwXCj4GvAK/3/8fvVj0cWoEFgCRJUg6HJUAM0LsP+FAM/ZNLbygC/eQvUjv527260VNEtW4D/xu4w9/8u6r3RSuwAJAkScppUAIEBrP/DzZj72D5a6wIHFT9k0iqqxo9RbwJRAuAZtqsegckSZKa7o2//h2OfezPGb4+9+ZKG/GlUUmSJEmSJElSXv8fMgSQ9g0bI/sAAAAldEVYdGRhdGU6Y3JlYXRlADIwMTktMDktMDVUMTU6NTA6MTEtMDc6MDCgLHNtAAAAJXRFWHRkYXRlOm1vZGlmeQAyMDE5LTA5LTA1VDE1OjUwOjExLTA3OjAw0XHL0QAAAABJRU5ErkJggg=="}';
final RegExp regex = RegExp(pattern);
final Iterable<RegExpMatch> matches = regex.allMatches(input, 0);
I am translating highlightjs with dart, highlightjs has their regex rules, I can't optimize the regex string to avoid this issue.
I'd actually consider going back and optimizing the RegExp in highlightjs, if that's possible. (It might not be, it seems they somehow combine existing RegExps into a single one, in a not always optimal way.)
If possible, it should help them too, since they're using the same RegExp engine as Dart on most browsers. It might not stack-overflow as easily, but that just means it has a bigger stack, the RegExp is still written in an inefficient way.
It is actually slightly embarrassing for the RegExp compiler, because the fix to avoid the stack overflow is to change
("(\\.|[^\\"\r\n])*"(?=\s*:))
("(\\.|[^\\"\r\n])*?"(?=\s*:))
The compiler should be able to see that the all the choices inside the *
are disjoint from each other (starts with \
vs. cannot start with \
) and from the continuation (starts with "
), and recognize that you don't need backtrack points at all.
Also, the RegExp shouldn't be capturing sub-matches unnecessarily, so ("(?:\\.|[^\\"\r\n])*?"(?=\s*:))
is likely even better. Captures in previous rounds go on the stack too, so they can be restored on backtrack. Don't capture inside an *
unless necessary, It's very rarely necessary.
Here, making the inner parentheses non-capturing also removes the stack overflow.
(Does mean that the capture indices change, but ("(?:\\.|[^\\"\r\n])*?"(?=\s*(:)))
would capture something inconspicuous instead.)
There are other rewrites one can do, to avoid backtracking across the different joined |
s that highlightjs seem to combine.
One option is to do ((?=pattern))\1
instead of just (pattern)
, because look-aheads are atomic, so this will be an atomic match. It either matches or not, but won't leave anything on the backtrack stack for later alternatives.
Highlight.js maintainer here.
...optimizing the RegExp in highlightjs, if that's possible. (... they somehow combine existing RegExps into a single one, in a not always optimal way.)
Yes, this is how we search for the next match "concurrently"... take the stack of potential matches and build a large "either" (|
) regex. Example:
(?:(abc)|(xyz)|(123))
We need the inside captures here so we can determine which of the expressions was matched. The first one to have actual content in its capture group.
the RegExp is still written in an inefficient way.
If it could be shown this is clearly the case (in Node or recent browsers) we might be open to some changes, but when I plug the below example into regex101.com it says that the *?
variant is worse, with 2800 vs 2100 iterations for a long string. Now true this is with the PCRE2 engine (which is the only one that can provide such stats)... but I'd be very hesitant to start making micro performance changes like this based on someone's word alone - without clear, measurable stats.
We do already security scan for Catastrophic Regex and fixing any of those that could be found would be a high priority.
Here, making the inner parentheses non-capturing also removes the stack overflow.
It's possible we capture in our "sub matches" a lot more than necessary... capture vs not-capture is not something we usually think or care about... it's much easier to read/write groups without having to add the ?:
prefix everywhere... I wouldn't want to force grammar authors to think about that unless it was shown there was a large performance impact. If a large performance benefit was shown I'd first consider rewriting the regexes dynamically during our "compile time" phase to add the ?:
where possible. But as noted above this would only remove captures from individual rule matches, not our large compiled "either" regexes which purposely use captures.
There are other rewrites one can do, to avoid backtracking across the different joined |s that highlightjs seem to combine. One option is to do ((?=pattern))\1 instead of just (pattern), because look-aheads are atomic, so this will be an atomic match. It either matches or not, but won't leave anything on the backtrack stack for later alternatives.
I don't think you can capture inside look-ahead though? Or did you mean ((?=pattern)\1)
. Nevermind, that doesn't work - you need the capture for the \1... so I'm not sure what exactly you'd proposing here.
RegExp(r'([ A-Za-z0-9])*');
Just out of curiosity is +
any better than *
here?
it says that the *? variant is worse, with 2800 vs 2100 iterations for a long string.
I wouldn't necessarily trust PCRE to have the same behavior as Irregexp. Both are backtracking regexp implementations, but there are differences in details, and optimization. (Someone should build an instrumented Irregexp that says how many loads, comparisons and backtracks it does, and the stack sizes needed. Shouldn't be too hard... He says, without any real clue.)
Whether *
or *?
is more efficient depends on a lot of things in the context.
An *?
will try the continuation first, before trying to take another trip in the loop. If the continuation often matches for a while before backtracking, then that can be expensive. On the other hand, if the continuation fails quickly, the *?
doesn't need to remember the state of the previous loop iteration, and you avoid needing a stack the size of your input.
The best thing to do for a RegExp is still to ensure that every branch is decided quickly, if possible. Use *?
if there are captures inside the loop, if it doesn't change the meaning.
I don't think you can capture inside look-ahead though?
You can in ECMAScript. Possibly not in PCRE.
RegExp(r'([ A-Za-z0-9])*');
Just out of curiosity is + any better than * here?
By itself, not really. It's just one, fairly empty, stack frame less. But it's always dangerous to have a RegExp that can match the empty string, especially if it's combined with other RegExps. So I'd prefer the +
, and be ready to handle no-match instead of an empty match.
But if you really want to match any number of letters, digits or space, and capture the last one, you can write it as:
RegExp(r"[ A-Za-z\d]*([ A-Za-z\d])(?![ A-Za-z\d])")
That matches all but the last character without any captures, then captures the last one, and checks that it's the last one.
Or use look-ahead to make the first loop atomic, to match everything eagerly, and then a look-behind with a capture to capture the last character:
RegExp(r"(?=([ A-Za-z\d]+))\1(?<=([^]))");
That's probably overkill (but it shows that you can capture in both look-ahead and look-behind).
If the capture is accidental, just make it non-capturing. Never capture unless you need the substring. Never capture inside a repetition, unless you really know what you're doing.
Whether or ? is more efficient depends on a lot of things in the context.
True. The behavior would be exactly the same so long as the loop and the continuation are disjointed, yes?
I'd guess that'd often be the case with our regex... when you have a *
loop that's not disjointed from the continuation seems to be where you quickly get into exponential backtracking in worst case scenarios. So here is another case I wonder if perhaps our compiler couldn't optimize on-the-fly...
Anyways I just wanted to respond here since Highlight.js was mentioned/linked. If someone is interested in pursuing any of these ideas (particularly compilation or engine changes) with-in the scope of the Highlight.js project we could talk about it over there. I've certainly never had this type of issue reported though, so it sounds like perhaps the default stack size in Dart is simply too small - and that might be an easy fix - perhaps say matching the stacks size used by common browsers or Node.js?
There are other rewrites one can do, to avoid backtracking across the different joined |s that highlightjs seem to combine.
I'd love it if you had time to share perhaps some examples of this though... this is a pretty core piece of our engine (multi-search) and if there were some easy wins there with real performance gains that'd surely improve the entire parser. Something the size of the example I provided earlier might be nice: (abc)|(xyz)|(123)
. Again, we need the inside captures so we can known which of the expressions matched.
Earlier it sounded like you were suggesting something like:
((?=(abc))\1)|((?=(xyz))\3)|((?=(123))\5)
But I'm not understanding how that would be an improvement? It seems like within each look-ahead (however complex) the amount of back-tracking would be exactly the same... no?
so it sounds like perhaps the default stack size in Dart is simply too small
It is when AoT compiled, as mentioned in comment 6.
I think it still is, which is why the issue is still open. Running the example program up there stack-overflows on AOT after the first printed line, at a string with length 25600. The non-AoT version runs until:
838860800: 0:00:10.964360
Exhausted heap space, trying to allocate 34359738400 bytes.
Unhandled exception:
Out of Memory
#0 _RegExp._growBacktrackingStack (dart:core-patch/regexp_patch.dart:312:22)
That's trying to allocate a 32+Gb backtrack stack in order to hold at most 838860800 backtracks, at least 40 bytes per backtrack. A backtrack contains one capture (start/end index) and a character position, which is up to 24 bytes on a 64-bit CPU, and maybe some extra state information, so not completely out of the ballpark, even if a little on the large side.
Even if you need the captures for something, putting the regexp with captures inside a *
suggests that you only need the last capture. (Unless the continuation has backrefs to the captures.)
What is significant is that if I change the RegExp to the equivalent, which only does captures on the last match:
var re = RegExp(r"(?:[a-z]*([a-z]))?"); // No capture state in backtrack.
then it runs to "completion" on both native and AoT, even with the minimal backtrack stack.
("Completion" being when the string concatenation s += s;
overflows the allowed string length.)
That's not just because there is no capture in the backtrack, most likely the compiler recognizes that the regexp being repeated always matches the same length, so backtracking doesn't even need to store the position, it can just subtract the length of the match, and doesn't need any stack-storage at all.
The time above was how long it took to do the 838860800 length string, about 11 seconds. With the non-capturing RegExp, the similar time is about 1 second ("0:00:00.934905"). All that allocation, and if it's not a realloc then also copying from old allocation to new, takes time.
var re = RegExp(r"(?:[a-z]+(?<=([^])))?");
is even slightly faster, but that only works when you know the length of the match.
Earlier it sounded like you were suggesting something like:
((?=(abc))\1)|((?=(xyz))\3)|((?=(123))\5)
But I'm not understanding how that would be an improvement?
It was an idea if you need this regexp to be insider a *
, because it might reduce the backtrack stack usage.
It won't solve all problems, but if the abc
introduces any backtracks, then it might be worth getting rid of them when we have a complete match.
Or maybe it's a bad idea, if we want to go back and see if abc
can match in a different, longer, way too if what comes after doesn't match.
My guess is that a regexp like this will usually be happy after having found a parsing of one of the options. And in that case, making the match atomic would at least reduce the number of backtrack stack-frames introduced per iteration.
But(!) that assumes a backtracking implementation to begin with. I can see that the V8 people is/have been working on a non-backtracking compiler: https://v8.dev/blog/non-backtracking-regexp Adding back-references is exactly the kind of thing that would make V8 fall back on the backtracking RegExp implementation, and keeping yourself within the regexp subset that can be handled by the non-backtracking compiler may be more efficient than optimizing the backtracks. So, everything may change in the future. (The Dart RegExp compiler doesn't, as far as I know, have the non-backtracking compiler enabled yet.)
It was an idea if you need this regexp to be insider a *, because it might reduce the backtrack stack usage.
It's not inside a *
or +
, it's just a single regex that we try to execute at whatever part of the string we're currently at in the parsing process...
Are there any workarounds without having to rewrite how regexes are created?
My application depends on the liquid_dart library which uses quite a few
https://github.com/kingwill101/liquid_dart/blob/master/lib/src/parser/lexer.dart
@kingwill101 I would recommend rewriting regexps (or not using regexp's at all). If regexp is doing so much backtracking that it exhausts the stack - this means this regexp is likely doing too much work and is much slower than it could be.
@kingwill101 Looking at the liquid_dart/parser/lexer.dart
library, there are some small tweaks that one could do on some of the RegExp
s, but the only one that really stands out is the markup
RegExp.
You can try changing the markup
RegExp to
final markup = _TokenCreator(TokenType.markup, r'(?:[^\s{]|\{(?![{%])|\s+?(?!\s|\{[{%]-))+');
and see if that helps with the stack behavior.
@kingwill101 Looking at the
liquid_dart/parser/lexer.dart
library, there are some small tweaks that one could do on some of theRegExp
s, but the only one that really stands out is themarkup
RegExp.You can try changing the
markup
RegExp tofinal markup = _TokenCreator(TokenType.markup, r'(?:[^\s{]|\{(?![{%])|\s+?(?!\s|\{[{%]-))+');
and see if that helps with the stack behavior.
Thank you, it works without throwing a stackoverflow. I'm not a regex expert so I'll have to spend a bit of time trying to understand why it does
The idea is to the backtrack stack small by having mutually exclusive options at each alternative. The RegExp compiler is smart enough to recognize that, and not create a backtrack point when it knows that the other alternative cannot possibly also match.
So the expression inside the +
has three alternatives, one which is [^\s{]
, so any non-whitespace, non-{
character,
another which stats with {
, and a third which starts with \s
. No character can matche more than one of these, so a single character look-ahead is sufficient to choose the unique possible continuation, which means no backtracking information needs to be stored.
The {
-start checks that it isn't {{
or {%
, in which case the {
is matched.
The whitespace-start grabs all the whitespaces, then checks that it's not followed by another whitespace (I meant all the whitespaces!), or a {{-
or {%-
.
The \s*?(?!\s|....)
is non-eager, which means that it tries the continuation first for each character. That adds one backtrack. If the next character is also a whitespace, the backtrack triggers immediately, gets popped, and it can continue matching that whitespace. If it reaches the end of whitespace, and it's not {{-
or {%-
, then it's a success, and the next iteration can continue.
So, all in all, it's written to ensure that any character that is matched will stay matched, there is no backtracking that goes back and tries the same character again in a different way, because the compiler can see, with one character lookeahead, that no other continuation is possible.
This tracker is for issues related to:
Dart Version: 2.16.1 Environment: MacOS (it shall be the same on Linux since it is simplified from our production system)
After compiling the following code into executable (
dart compile exe test.dart
), executing the executable will cause StackOverflow:On the other hand, it works fine with Dart VM (i.e.,
dart test.dart
).