Test issues with current test runners

[sarnold]

For the various repackaging efforts (pep517, cmake, etc) the legacy test driver in setup.py got removed, mainly because all the current "best practice" guidance says "use a test runner like pytest or xxx" instead of through setup.py and I started running tests via nose or python unittest. I think the main issue left is the legacy tests were written to run from the tests subdirectory instead of the top-level project dir.

What works:

  $ python -m unittest discover -v tests/

BUT the python doctest module has no discovery and needs to be run from the tests dir.

  $ nosetests -sx tests/re2_test.py
  $ nosetests -sx tests/test_re.py

will run tests on the *.txt targets and then the same tests as unittest discover but does not run/show the doctests found by python -m doctest when run in the tests directory.

  $ cd tests/
  $ python -m unittest discover -v .
  $ python -m doctest -v *.txt

will find all the working tests, however, using pytest/nosetests to discover any actual tests from the top-level project directory fails on several tests:

pytest --doctest-glob=*.txt tests/

(runs the doc tests)

...

=========================== short test summary info ============================
FAILED tests/test_re.py::ReTests::test_bug_3629 - re.error: invalid perl oper...
FAILED tests/test_re.py::ReTests::test_bug_725149 - re.error: invalid perl op...
FAILED tests/test_re.py::ReTests::test_ignore_case - re.error: Backreferences...
FAILED tests/test_re.py::ReTests::test_inline_flags - re.error: invalid perl ...
FAILED tests/test_re.py::ReTests::test_non_consuming - re.error: invalid perl...
FAILED tests/test_re.py::ReTests::test_re_groupref - re.error: Backreferences...
FAILED tests/test_re.py::ReTests::test_re_groupref_exists - re.error: invalid...
================== 7 failed, 67 passed, 11 warnings in 3.33s ===================

Oddly, pytest can run doctests with an ignore:

$ pytest --ignore=tests/test_re.py --doctest-glob=*.txt tests/
============================= test session starts ==============================
platform linux -- Python 3.8.7rc1, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /home/nerdboy/src/pyre2-sdist
collected 14 items                                                             

tests/charliterals.txt .                                                 [  7%]
tests/count.txt .                                                        [ 14%]
tests/emptygroups.txt .                                                  [ 21%]
tests/findall.txt .                                                      [ 28%]
tests/finditer.txt .                                                     [ 35%]
tests/match_expand.txt .                                                 [ 42%]
tests/mmap.txt .                                                         [ 50%]
tests/namedgroups.txt .                                                  [ 57%]
tests/pattern.txt .                                                      [ 64%]
tests/re2_test.py .                                                      [ 71%]
tests/search.txt .                                                       [ 78%]
tests/split.txt .                                                        [ 85%]
tests/sub.txt .                                                          [ 92%]
tests/unicode.txt .                                                      [100%]

=============================== warnings summary ===============================
tests/charliterals.txt::charliterals.txt
tests/re2_test.py::testall
  <doctest charliterals.txt[10]>:1: DeprecationWarning: invalid escape sequence \9

tests/charliterals.txt::charliterals.txt
tests/re2_test.py::testall
  <doctest charliterals.txt[17]>:1: DeprecationWarning: invalid escape sequence \9

tests/re2_test.py::testall
tests/sub.txt::sub.txt
  <doctest sub.txt[5]>:1: DeprecationWarning: invalid escape sequence \(

tests/re2_test.py::testall
tests/search.txt::search.txt
  <doctest search.txt[2]>:1: DeprecationWarning: invalid escape sequence \d

tests/re2_test.py::testall
tests/search.txt::search.txt
  <doctest search.txt[3]>:1: DeprecationWarning: invalid escape sequence \d

tests/re2_test.py::testall
  <doctest unicode.txt[24]>:1: UserWarning: WARNING: Using re module. Reason: \W and \S not supported inside character classes

-- Docs: https://docs.pytest.org/en/stable/warnings.html
======================= 14 passed, 11 warnings in 1.67s ========================

[sarnold]

What are your thoughts on the "best" way forward for test cleanup? I fixed(?) the first error from "outside tests dir" (see the pickle error commit) but I'm not sure if that's the best path forward (or even what to do with the re.compile errors). Thanks!

[andreasvc]

I would be in favor of having all tests run by pytest.

The test failures are due to the doctests setting the fallback level to "exception" (i.e., raise exception if RE2 cannot handle the expression). I'll add a workaround by re-setting the fallback level to the default.