andreasvc / pyre2

Python wrapper for RE2
BSD 3-Clause "New" or "Revised" License
99 stars 33 forks source link

Tests failing #11

Closed clarkadamp closed 4 years ago

clarkadamp commented 5 years ago

I am attempting to build this package for my use.

The versions that I am using are the following:

https://github.com/google/re2.git@2019-09-01 https://github.com/andreasvc/pyre2.git@master(3e01eba6ba3eabd1359ef5e16c938c8866deea70)

The error messages seem to be from https://github.com/google/re2/blob/567327038cdd01ed8f2a0e3247b6cd8138dc6356/re2/regexp.cc#L508

Is there a specific version of re2 that is needed?

Test outcomes

======================================================================
ERROR: test_bug_3629 (test_re.ReTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/local/home/cladam/workspaces/hcs/src/Python-pyre2/third-party-src/tests/test_re.py", line 128, in test_bug_3629
    re.compile("(?P<quote>)(?(quote))")
  File "src/compile.pxi", line 6, in re2.compile
    p = _compile(pattern, flags, max_mem)
  File "src/compile.pxi", line 87, in re2._compile
    raise RegexError(error_msg)
RegexError: invalid perl operator: (?(

======================================================================
ERROR: test_bug_725149 (test_re.ReTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/local/home/cladam/workspaces/hcs/src/Python-pyre2/third-party-src/tests/test_re.py", line 589, in test_bug_725149
    self.assertEqual(re.match('(a)(?:(?=(b)*)c)*', 'abb').groups(),
  File "src/re2.pyx", line 162, in re2.match
    return compile(pattern, flags).match(string)
  File "src/compile.pxi", line 6, in re2.compile
    p = _compile(pattern, flags, max_mem)
  File "src/compile.pxi", line 87, in re2._compile
    raise RegexError(error_msg)
RegexError: invalid perl operator: (?=

======================================================================
ERROR: test_flags (test_re.ReTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/local/home/cladam/workspaces/hcs/src/Python-pyre2/third-party-src/tests/test_re.py", line 476, in test_flags
    self.assertNotEqual(re.compile('^pattern$', flag), None)
  File "src/compile.pxi", line 6, in re2.compile
    p = _compile(pattern, flags, max_mem)
  File "src/compile.pxi", line 42, in re2._compile
    return fallback(original_pattern, flags, "re.LOCALE not supported")
  File "src/compile.pxi", line 19, in re2._compile.fallback
    raise RegexError(error_msg)
RegexError: re.LOCALE not supported

======================================================================
ERROR: test_ignore_case (test_re.ReTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/local/home/cladam/workspaces/hcs/src/Python-pyre2/third-party-src/tests/test_re.py", line 417, in test_ignore_case
    self.assertEqual(re.match(r"((a)\s\2)", "a a", re.I).group(1), "a a")
  File "src/re2.pyx", line 162, in re2.match
    return compile(pattern, flags).match(string)
  File "src/compile.pxi", line 6, in re2.compile
    p = _compile(pattern, flags, max_mem)
  File "src/compile.pxi", line 56, in re2._compile
    return fallback(original_pattern, flags, "Backreferences not supported")
  File "src/compile.pxi", line 19, in re2._compile.fallback
    raise RegexError(error_msg)
RegexError: Backreferences not supported

======================================================================
ERROR: test_inline_flags (test_re.ReTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/local/home/cladam/workspaces/hcs/src/Python-pyre2/third-party-src/tests/test_re.py", line 671, in test_inline_flags
    p = re.compile('(?iu)' + upper_char)
  File "src/compile.pxi", line 6, in re2.compile
    p = _compile(pattern, flags, max_mem)
  File "src/compile.pxi", line 87, in re2._compile
    raise RegexError(error_msg)
RegexError: invalid perl operator: (?iu

======================================================================
ERROR: test_non_consuming (test_re.ReTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/local/home/cladam/workspaces/hcs/src/Python-pyre2/third-party-src/tests/test_re.py", line 397, in test_non_consuming
    self.assertEqual(re.match("(a(?=\s[^a]))", "a b").group(1), "a")
  File "src/re2.pyx", line 162, in re2.match
    return compile(pattern, flags).match(string)
  File "src/compile.pxi", line 6, in re2.compile
    p = _compile(pattern, flags, max_mem)
  File "src/compile.pxi", line 87, in re2._compile
    raise RegexError(error_msg)
RegexError: invalid perl operator: (?=

======================================================================
ERROR: test_re_groupref (test_re.ReTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/local/home/cladam/workspaces/hcs/src/Python-pyre2/third-party-src/tests/test_re.py", line 293, in test_re_groupref
    self.assertEqual(re.match(r'^(\|)?([^()]+)\1$', '|a|').groups(),
  File "src/re2.pyx", line 162, in re2.match
    return compile(pattern, flags).match(string)
  File "src/compile.pxi", line 6, in re2.compile
    p = _compile(pattern, flags, max_mem)
  File "src/compile.pxi", line 56, in re2._compile
    return fallback(original_pattern, flags, "Backreferences not supported")
  File "src/compile.pxi", line 19, in re2._compile.fallback
    raise RegexError(error_msg)
RegexError: Backreferences not supported

======================================================================
ERROR: test_re_groupref_exists (test_re.ReTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/local/home/cladam/workspaces/hcs/src/Python-pyre2/third-party-src/tests/test_re.py", line 268, in test_re_groupref_exists
    self.assertEqual(re.match('^(\()?([^()]+)(?(1)\))$', '(a)').groups(),
  File "src/re2.pyx", line 162, in re2.match
    return compile(pattern, flags).match(string)
  File "src/compile.pxi", line 6, in re2.compile
    p = _compile(pattern, flags, max_mem)
  File "src/compile.pxi", line 87, in re2._compile
    raise RegexError(error_msg)
RegexError: invalid perl operator: (?(

======================================================================
ERROR: test_special_escapes (test_re.ReTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/local/home/cladam/workspaces/hcs/src/Python-pyre2/third-party-src/tests/test_re.py", line 360, in test_special_escapes
    "abcd abc bcd bx", re.LOCALE).group(1), "bx")
  File "src/re2.pyx", line 156, in re2.search
    return compile(pattern, flags).search(string)
  File "src/compile.pxi", line 6, in re2.compile
    p = _compile(pattern, flags, max_mem)
  File "src/compile.pxi", line 42, in re2._compile
    return fallback(original_pattern, flags, "re.LOCALE not supported")
  File "src/compile.pxi", line 19, in re2._compile.fallback
    raise RegexError(error_msg)
RegexError: re.LOCALE not supported

======================================================================
FAIL: test_dollar_matches_twice (test_re.ReTests)
$ matches the end of string, and just before the terminating
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/local/home/cladam/workspaces/hcs/src/Python-pyre2/third-party-src/tests/test_re.py", line 682, in test_dollar_matches_twice
    self.assertEqual(pattern.sub('#', 'a\nb\n'), 'a\nb#\n#')
AssertionError: 'a\nb\n#' != 'a\nb#\n#'
andreasvc commented 5 years ago

I'll have to look into this. I'm using the libre2-dev package on Debian stable (i.e., version 20190101+dfsg-2), so I suggest you try that. Let me know how it goes.

clarkadamp commented 5 years ago

Unfortunately I'm not debian

Same result if I import https://github.com/google/re2.git@2019-01-01

There are three patches that debian apply listed in https://launchpad.net/ubuntu/+source/re2/20190101+dfsg-2

  1. 0001-Parallel-friendly-test-runner.patch; seems debian specific
  2. 0002-Use-Debian-s-Unicode-data.patch; get unicode data from /usr/share/unicode rather than downloading it from https://www.unicode.org/Public/11.0.0/ucd
  3. 0003-Upstream-doesn-t-manage-a-SONAME-so-we-do.patch;
    • changes the SONAME variable from 0 to 5, making it build libre2.so.5 instead of libre2.so.0
    • which if I apply breaks pyre2 builds as it is looking for libre2.so.0:
      ImportError: libre2.so.0: cannot open shared object file: No such file or directory

      Symlinking it so it can found doesn't change anything.

Nothing weird looking in dependencies https://packages.debian.org/sid/libre2-5

So I am at a loss as to what is causing this issue

clarkadamp commented 5 years ago

Just for fun I spun up an instance with Ubuntu 19.04 (GNU/Linux 5.0.0-31-generic x86_64) and experienced different results

$ apt-get install libre2-dev cython cython3
...
Unpacking libre2-5:amd64 (20190101+dfsg-2) ...
...
$ git clone https://github.com/andreasvc/pyre2.git
cd ~/pyre2

make test

======================================================================
FAIL: test_dollar_matches_twice (__main__.ReTests)
$ matches the end of string, and just before the terminating
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_re.py", line 682, in test_dollar_matches_twice
    self.assertEqual(pattern.sub('#', 'a\nb\n'), 'a\nb#\n#')
AssertionError: 'a\nb\n#' != 'a\nb#\n#'

---------------------------------------------------------------------

make test3

======================================================================
ERROR: test_flags (__main__.ReTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_re.py", line 476, in test_flags
    self.assertNotEqual(re.compile('^pattern$', flag), None)
  File "src/compile.pxi", line 6, in re2.compile
  File "src/compile.pxi", line 42, in re2._compile
  File "src/compile.pxi", line 23, in re2._compile.fallback
  File "/usr/lib/python3.7/re.py", line 234, in compile
    return _compile(pattern, flags)
  File "/usr/lib/python3.7/re.py", line 286, in _compile
    p = sre_compile.compile(pattern, flags)
  File "/usr/lib/python3.7/sre_compile.py", line 764, in compile
    p = sre_parse.parse(p, flags)
  File "/usr/lib/python3.7/sre_parse.py", line 940, in parse
    p.pattern.flags = fix_flags(str, p.pattern.flags)
  File "/usr/lib/python3.7/sre_parse.py", line 907, in fix_flags
    raise ValueError("cannot use LOCALE flag with a str pattern")
ValueError: cannot use LOCALE flag with a str pattern

======================================================================
ERROR: test_special_escapes (__main__.ReTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_re.py", line 360, in test_special_escapes
    "abcd abc bcd bx", re.LOCALE).group(1), "bx")
  File "src/re2.pyx", line 156, in re2.search
  File "src/compile.pxi", line 6, in re2.compile
  File "src/compile.pxi", line 42, in re2._compile
  File "src/compile.pxi", line 23, in re2._compile.fallback
  File "/usr/lib/python3.7/re.py", line 234, in compile
    return _compile(pattern, flags)
  File "/usr/lib/python3.7/re.py", line 286, in _compile
    p = sre_compile.compile(pattern, flags)
  File "/usr/lib/python3.7/sre_compile.py", line 764, in compile
    p = sre_parse.parse(p, flags)
  File "/usr/lib/python3.7/sre_parse.py", line 940, in parse
    p.pattern.flags = fix_flags(str, p.pattern.flags)
  File "/usr/lib/python3.7/sre_parse.py", line 907, in fix_flags
    raise ValueError("cannot use LOCALE flag with a str pattern")
ValueError: cannot use LOCALE flag with a str pattern

======================================================================
FAIL: test_dollar_matches_twice (__main__.ReTests)
$ matches the end of string, and just before the terminating
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_re.py", line 682, in test_dollar_matches_twice
    self.assertEqual(pattern.sub('#', 'a\nb\n'), 'a\nb#\n#')
AssertionError: 'a\nb\n#' != 'a\nb#\n#'
  a
- b
+ b#
?  +
  #
andreasvc commented 5 years ago

Hm, curious. I tested on CentOS 6 without problems. What system are you running? (distro/version)

andreasvc commented 5 years ago

That means everything works. Those tests are expected to fail (except maybe the 2nd), I believe that's in the README.

clarkadamp commented 5 years ago

Hm, curious. I tested on CentOS 6 without problems. What system are you running? (distro/version)

It's internal/proprietary unfortunately, I might need to hit up the maintainers.

That means everything works. Those tests are expected to fail (except maybe the 2nd), I believe that's in the README.

Had a quick look at the readme, couldn't see anything explicit about failing tests If they are expected to fail, then it potentially they could be marked as skipped or expected failures

https://docs.python.org/3/library/unittest.html#skipping-tests-and-expected-failures

andreasvc commented 5 years ago

On 31/10/19 04:12, clarkadamp wrote:

Had a quick look at the readme, couldn't see anything explicit about failing tests

See here https://github.com/andreasvc/pyre2/#current-status

If they are expected to fail, then it potentially they could be marked as skipped or expected failures

Thanks, I'll look into that.

andreasvc commented 4 years ago

Tests that are expected to fail have now been disabled.