search

smarie / python-pytest-harvest

Store data created during your `pytest` tests execution, and retrieve it at the end of the session, e.g. for applicative benchmarking purposes.
https://smarie.github.io/python-pytest-harvest/
BSD 3-Clause "New" or "Revised" License
61 stars 8 forks source link

Pytest-harvest's own CI build is failing. A few tests for env='pytest2.x' are failing (on all Python versions). #66

Closed JamesParrott closed 4 months ago

JamesParrott commented 5 months ago

https://github.com/smarie/python-pytest-harvest/actions/runs/5963654515 image

Steps to investigate, and produce errors locally (mostly as per the developer README.md):

git clone --depth=1 https://github.com/smarie/python-pytest-harvest/
cd python-pytest-harvest
conda create -n noxenv python="3.7"
conda activate noxenv

(noxenv)

pip install -r .\noxfile-requirements.txt

(noxenv)

nox -s tests-3.8(env='pytest2.x')

produces:

 pytest_harvest/tests/test_all_raw_with_meta_check.py::test_run_all_tests[test_all_together.py] FAILED
 pytest_harvest/tests/test_all_raw_with_meta_check.py::test_run_all_tests[test_default_fixtures.py] PASSED
 pytest_harvest/tests/test_all_raw_with_meta_check.py::test_run_all_tests[test_get_session_results.py] PASSED
 pytest_harvest/tests/test_all_raw_with_meta_check.py::test_run_all_tests[test_get_session_results_indirect_and_noparam.py] PASSED
 pytest_harvest/tests/test_all_raw_with_meta_check.py::test_run_all_tests[test_results_bags.py] FAILED
 pytest_harvest/tests/test_all_raw_with_meta_check.py::test_run_all_tests[test_saved_fixture_basic.py] FAILED
 pytest_harvest/tests/test_all_raw_with_meta_check.py::test_run_all_tests[test_saved_fixture_in_fixture.py] FAILED
 pytest_harvest/tests/test_all_raw_with_meta_check.py::test_run_all_tests[test_saved_fixture_in_global_var.py] FAILED
 pytest_harvest/tests/test_all_raw_with_meta_check.py::test_run_all_tests[test_saved_fixture_module_params.py] FAILED
 pytest_harvest/tests/test_all_raw_with_meta_check.py::test_run_all_tests[test_saved_fixture_session.py] FAILED
 pytest_harvest/tests/test_all_raw_with_meta_check.py::test_run_all_tests[test_saved_fixture_session_module.py] FAILED
 pytest_harvest/tests/test_doc_basic_dct.py::test_foo[world] PASSED
 pytest_harvest/tests/test_doc_basic_dct.py::test_foo[self] PASSED
 pytest_harvest/tests/test_doc_basic_dct.py::test_synthesis PASSED
 pytest_harvest/tests/test_doc_basic_df.py::test_foo[world] PASSED
 pytest_harvest/tests/test_doc_basic_df.py::test_foo[self] PASSED
 pytest_harvest/tests/test_doc_basic_df.py::test_synthesis PASSED
 pytest_harvest/tests/test_doc_basic_df_all.py::test_foo[0-False] ERROR
 pytest_harvest/tests/test_doc_basic_df_all.py::test_foo[0-True] ERROR
 pytest_harvest/tests/test_doc_basic_df_all.py::test_foo[1-False] ERROR
 pytest_harvest/tests/test_doc_basic_df_all.py::test_foo[1-True] ERROR
 pytest_harvest/tests/test_doc_basic_df_all.py::test_synthesis FAILED
 pytest_harvest/tests/test_doc_basic_results_bag.py::test_foo[world] ERROR
 pytest_harvest/tests/test_doc_basic_results_bag.py::test_foo[self] ERROR
 pytest_harvest/tests/test_doc_basic_results_bag.py::test_synthesis FAILED
 pytest_harvest/tests/test_doc_basic_saved_fixture.py::test_foo[0] ERROR
 pytest_harvest/tests/test_doc_basic_saved_fixture.py::test_foo[1] ERROR
 pytest_harvest/tests/test_doc_basic_saved_fixture.py::test_synthesis FAILED
 pytest_harvest/tests/test_doc_basic_saved_fixture_views.py::test_foo[0] ERROR
 pytest_harvest/tests/test_doc_basic_saved_fixture_views.py::test_foo[1] ERROR
 pytest_harvest/tests/test_doc_basic_saved_fixture_views.py::test_synthesis FAILED
 pytest_harvest/tests/test_doc_example.py::test_my_app_bench[A-1] ERROR
 pytest_harvest/tests/test_doc_example.py::test_my_app_bench[A-2] ERROR
 pytest_harvest/tests/test_doc_example.py::test_my_app_bench[B-1] ERROR
 pytest_harvest/tests/test_doc_example.py::test_my_app_bench[B-2] ERROR
 pytest_harvest/tests/test_doc_example.py::test_my_app_bench[C-1] ERROR
 pytest_harvest/tests/test_doc_example.py::test_my_app_bench[C-2] ERROR
 pytest_harvest/tests/test_doc_example.py::test_basic PASSED
 pytest_harvest/tests/test_doc_example.py::test_synthesis PASSED
 pytest_harvest/tests/test_doc_example_custom.py::test_my_app_bench[A-1] ERROR
 pytest_harvest/tests/test_doc_example_custom.py::test_my_app_bench[A-2] ERROR
 pytest_harvest/tests/test_doc_example_custom.py::test_my_app_bench[B-1] ERROR
 pytest_harvest/tests/test_doc_example_custom.py::test_my_app_bench[B-2] ERROR
 pytest_harvest/tests/test_doc_example_custom.py::test_my_app_bench[C-1] ERROR
 pytest_harvest/tests/test_doc_example_custom.py::test_my_app_bench[C-2] ERROR
 pytest_harvest/tests/test_doc_example_custom.py::test_basic PASSED
 pytest_harvest/tests/test_doc_example_custom.py::test_synthesis PASSED
 pytest_harvest/tests/test_issue_45.py::test_a[a::0] PASSED
 pytest_harvest/tests/test_issue_45.py::test_b PASSED
 pytest_harvest/tests/test_issue_45.py::TestC::test_c PASSED
 pytest_harvest/tests/test_issue_45.py::TestC::test_d[a::0] PASSED
 pytest_harvest/tests/test_issue_45.py::test_synthesis PASSED
 pytest_harvest/tests/test_lazy_and_harvest.py::test_foo[foo] PASSED
 pytest_harvest/tests/test_lazy_and_harvest.py::test_foo2[foo[0]-foo[1]] PASSED
 pytest_harvest/tests/test_lazy_and_harvest.py::test_foo3 ERROR
 pytest_harvest/tests/test_lazy_and_harvest.py::test_synthesis FAILED
 pytest_harvest/tests/test_results_bag_basic.py::test_results_bag_basic PASSED
 pytest_harvest/tests/test_unicode_literals.py::test_filterset PASSED

See attched file for full output. The errors seem basic, and they all look to be TypeErrors within PyTest, plus a handful of KeyErrors, but for missing positional args and keys, a lot of whose names relate to pytest-harvest. . So perhaps PyTest's API changed after v2, or some dependency isn't being installed, that results in the required items not being added into **kwargs?

Examples:

 request = <FixtureRequest for <Function 'test_synthesis'>>
 fixture_store = OrderedDict()

     def test_synthesis(request, fixture_store):
 >       print(fixture_store['dummy'])
 E       KeyError: 'dummy'

 test_run_all_tests.py:20: KeyError
_____________________ ERROR at setup of test_foo[0-False] _____________________

 self = <CallInfo when='setup' exception: person() missing 1 required positional argument: 'fixture_store'>
 func = <function call_runtest_hook.<locals>.<lambda> at 0x0000029316426700>
 when = 'setup'

     def __init__(self, func, when):
         #: context of invocation: one of "setup", "call",
         #: "teardown", "memocollect"
         self.when = when
         self.start = time()
         try:
 >           self.result = func()

 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\runner.py:150:
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\runner.py:138: in <lambda>
     return CallInfo(lambda: ihook(item=item, **kwds), when=when)
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\vendored_packages\pluggy.py:724: in __call__
     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\vendored_packages\pluggy.py:338: in _hookexec
     return self._inner_hookexec(hook, methods, kwargs)
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\vendored_packages\pluggy.py:333: in <lambda>
     _MultiCall(methods, kwargs, hook.spec_opts).execute()
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\vendored_packages\pluggy.py:595: in execute
     return _wrapped_call(hook_impl.function(*args), self.execute)
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\vendored_packages\pluggy.py:253: in _wrapped_call
     return call_outcome.get_result()
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\vendored_packages\pluggy.py:278: in get_result
     raise ex[1].with_traceback(ex[2])
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\vendored_packages\pluggy.py:264: in __init__
     self.result = func()
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\vendored_packages\pluggy.py:595: in execute
     return _wrapped_call(hook_impl.function(*args), self.execute)
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\vendored_packages\pluggy.py:253: in _wrapped_call
     return call_outcome.get_result()
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\vendored_packages\pluggy.py:278: in get_result
     raise ex[1].with_traceback(ex[2])
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\vendored_packages\pluggy.py:264: in __init__
     self.result = func()
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\vendored_packages\pluggy.py:596: in execute
     res = hook_impl.function(*args)
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\runner.py:87: in pytest_runtest_setup
     item.session._setupstate.prepare(item)
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\runner.py:408: in prepare
     col.setup()
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\python.py:1419: in setup
     fillfixtures(self)
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\python.py:819: in fillfixtures
     request._fillfixtures()
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\python.py:1567: in _fillfixtures
     item.funcargs[argname] = self.getfuncargvalue(argname)
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\python.py:1610: in getfuncargvalue
     return self._get_active_fixturedef(argname).cached_result[0]
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\python.py:1627: in _get_active_fixturedef
     result = self._getfuncargvalue(fixturedef)
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\python.py:1673: in _getfuncargvalue
     val = fixturedef.execute(request=subrequest)
 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\python.py:2134: in execute
     result = call_fixture_func(fixturefunc, request, kwargs,
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

 fixturefunc = <function person at 0x0000029314493670>
 request = <SubRequest 'person' for <Function 'test_foo[0-False]'>>
 kwargs = {'request': <SubRequest 'person' for <Function 'test_foo[0-False]'>>}
 yieldctx = False

     def call_fixture_func(fixturefunc, request, kwargs, yieldctx):
         if yieldctx:
             if not is_generator(fixturefunc):
                 fail_fixturefunc(fixturefunc,
                     msg="yield_fixture requires yield statement in function")
             iter = fixturefunc(**kwargs)
             next = getattr(iter, "__next__", None)
             if next is None:
                 next = getattr(iter, "next")
             res = next()
             def teardown():
                 try:
                     next()
                 except StopIteration:
                     pass
                 else:
                     fail_fixturefunc(fixturefunc,
                         "yield_fixture function has more than one 'yield'")
             request.addfinalizer(teardown)
         else:
             if is_generator(fixturefunc):
                 fail_fixturefunc(fixturefunc,
                     msg="pytest.fixture functions cannot use ``yield``. "
                         "Instead write and return an inner function/generator "
                         "and let the consumer call and iterate over it.")
 >           res = fixturefunc(**kwargs)
 E           TypeError: person() missing 1 required positional argument: 'fixture_store'

 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\python.py:2056: TypeError

41*TypeErrors (82 hits)

6*

TypeError: person() missing 1 required positional argument: 'fixture_store'

 .nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\python.py:2056: TypeError

also TypeError: my_fix() missing 1 required positional argument: 'store' and TypeError: dataset() missing 1 required positional argument: 'my_store'

10*

.nox\tests-3-8-env-pytest2-x\lib\site-packages\_pytest\python.py:2036: TypeError

for TypeError: _results_bag() missing 2 required positional arguments: 'request' and 'store' and TypeError: my_yield_fix() missing 1 required positional argument: 'request'

python-pytest-harvest_nox_py3.8_pytest2.x_stdout.txt

smarie commented 4 months ago

Indeed, thanks @JamesParrott for the notice ! I'll drop pytest 2 from Ci for now