Upgrade regular expression engine to support features of major flavours like PCRE2, ECMAScript etc.

[mcarans]

Regular expressions that work in online tools like https://regex101.com/ do not in TRegExpr which is rather frustrating.

For example, I wanted to match a string that does not start with something. I looked for examples and found this 2009 Stackoverflow answer. I tried the first two options in the answer (one using negative lookahead and one using negative lookbehind) via CudaText and neither work. Both give error: lookaround brackets must be at the very beginning/ending

I checked the documentation and TRegExpr seems to have some major limitations:

Limitations:

Brackets for lookahead must be at the very ending of expression, and brackets for lookbehind must be at the very beginning. So assertions between choices |, or inside groups, are not supported.
For lookbehind (?<!foo)bar, regex “foo” must be of fixed length, ie contains only operations of fixed length matches. Quantifiers are not allowed, except braces with the repeated numbers {n} or {n,n}. Char-classes are allowed here, dot is allowed, \b and \B are allowed. Groups and choices are not allowed.

I would like to be able to construct a regular expression in an online regex tester and have it work in TRegExpr. Is it possible to fix these limitations and bring the regular expression engine up to date so that it supports the full range of features of major flavours like PCRE2, ECMAScript etc.?

Help is needed to resolve this issue which relates to one on CudaText

[User4martin]

@Alexey-T

Currently working on some part of this issue. Improved support for look-around. => unlimited and nested positive and negative look-ahead and behind. Including variable length look-behind. Well almost.

On that part the question is what is acceptable

In Look behind (as far as I can deduct from some tests) loops (greedy or non-greedy) are evaluated backwards.

https://regex101.com/r/h0TzgP/1 (?<=(A??a*))X => AaaaaX

• The greedy "a" goes all the way back
• The non-greedy "A" is not needed
• Match ="X" Group="aaaa"

If you match that left-to-right, there is no (reasonable) way to find that.

• Trying to match left most first: Matching from pos 1: The "A" would be included in the group Since the match succeeded, no further tries
• Trying to match right most first: Matching from pos 6: Neither "a" nor "A" would be included in the group Since the match succeeded, no further tries

https://regex101.com/r/oldaRx/1 (?<=(a*)(a*))X => aaaaX It is group 2 that gets all the "a" (forward would have been group 1)

• "Backward" compiling is not just reversing the order of entries.
• captures numbers must be kept
• back-references \1 => well actually I couldn' make them work well in the online reg ex tester...

To match those particular pattern, one would need to compile the reg-ex sub=expression backwards, and apply it backwards.

In any case currently we match from left to right.

• The result in terms of match or not-match is the same. (as far as I know)
• The differences are only visible if there are both:
  • captures
  • variable length parts (in or outside the capture)

In that case, the content of the capture may not be what it should be. Actually:

• If there is only one atom of variable length in the lookbehind, then its greedy-attribute is known. And the result is correct. However, this is "the user must ensure there is only one" => the engine does not check. (too much code for a border case)
• If there are only non-greedy, it should also work (and the engine will know/check)

But

• if there is a mixture containing at least one greedy, and one other loop (greedy or non-greedy) then it may in many cases get the wrong capture.

So, is that acceptable?

Or should look-behind be restricted to be any of

• fixed length
• variable length without any greedy parts
• variable length without any captures

And fail to compile for (having all of):

• variable length
• with capture
• and something greedy

--- EDIT Actually multiple non-greedy can also get a capture to the wrong pos.

So "capture" and "variable len" don't work together.

[User4martin]

Current preview https://github.com/User4martin/TRegExpr/tree/lookaround (does not yet detect fixed len)

[Alexey-T]

Or should look-behind be restricted to be any of

If your talent+time will allow to run matches backward, it will be good. if not, I can live with restrictions: fail to compile certain operators in the lookbehind.

[User4martin]

I already got them running (var-len, and backward) => just if captures are used together with var len, the capture could match at the wrong pos.

So that may need to be disabled. Or at least be made an option (for many patterns they will still work correct).

[Alexey-T]

So that may need to be disabled

right...

[Alexey-T]

@CaptainFlint You can suggest new version 1.163 to Ghisler.

[Alexey-T]

@andgineer Let's close this? look-around now can be anywhere.