tests fail in test_synthesis_id_formatting

[keszybz]

After applying the work-around in #24, I still get this:

Testing that running pytest on file test_get_session_results.py results in {'passed': 16, 'skipped': 1, 'failed': 1}
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.2, py-1.8.0, pluggy-0.12.0
rootdir: /tmp/pytest-of-zbyszek/pytest-5/test_run_all_tests9
plugins: harvest-1.7.2.dev2+g0a88f11c9c, cov-2.7.1, celery-4.3.0
collected 18 items

test_run_all_tests.py .....F.FsF....F...                                 [100%]

=================================== FAILURES ===================================
______ test_foo_synthesis_all_options[duration_in_ms=False-flatten=True] _______

request = <FixtureRequest for <Function test_foo_synthesis_all_options[duration_in_ms=False-flatten=True]>>
flatten = True, durations_in_ms = False

    @pytest.mark.parametrize('flatten', [False, True], ids="flatten={}".format)
    @pytest.mark.parametrize('durations_in_ms', [False, True], ids="duration_in_ms={}".format)
    def test_foo_synthesis_all_options(request, flatten, durations_in_ms):
        """
        Tests that the synthesis is ok

        :param request:
        :return:
        """
        # Get the synthesis dictionary concerning `test_foo`
        synth_dct = get_session_synthesis_dct(request.session, status_details=True, flatten=flatten, filter=test_foo,
                                              durations_in_ms=durations_in_ms)

        # from pprint import pprint
        # pprint(dict(synth_dct))

        durations_unit = ('ms' if durations_in_ms else 's')

        # Check the returned dictionary contents
        prefix = '' if flatten else 'pytest_'
        expected_keys = {'pytest_obj',
                         prefix + 'status',
                         prefix + 'duration_' + durations_unit}
        stages = ['setup', 'call', 'teardown']
        if not flatten:
            expected_keys.update({prefix + 'status_details', prefix + 'params'})
        else:
            expected_keys.update({(prefix + 'status__' + stage) for stage in stages})
            # add parameters
>           expected_keys.update({mark.args[0] for mark in test_foo.parametrize})
E           AttributeError: 'function' object has no attribute 'parametrize'

test_run_all_tests.py:69: AttributeError
_______ test_foo_synthesis_all_options[duration_in_ms=True-flatten=True] _______

request = <FixtureRequest for <Function test_foo_synthesis_all_options[duration_in_ms=True-flatten=True]>>
flatten = True, durations_in_ms = True

    @pytest.mark.parametrize('flatten', [False, True], ids="flatten={}".format)
    @pytest.mark.parametrize('durations_in_ms', [False, True], ids="duration_in_ms={}".format)
    def test_foo_synthesis_all_options(request, flatten, durations_in_ms):
        """
        Tests that the synthesis is ok

        :param request:
        :return:
        """
        # Get the synthesis dictionary concerning `test_foo`
        synth_dct = get_session_synthesis_dct(request.session, status_details=True, flatten=flatten, filter=test_foo,
                                              durations_in_ms=durations_in_ms)

        # from pprint import pprint
        # pprint(dict(synth_dct))

        durations_unit = ('ms' if durations_in_ms else 's')

        # Check the returned dictionary contents
        prefix = '' if flatten else 'pytest_'
        expected_keys = {'pytest_obj',
                         prefix + 'status',
                         prefix + 'duration_' + durations_unit}
        stages = ['setup', 'call', 'teardown']
        if not flatten:
            expected_keys.update({prefix + 'status_details', prefix + 'params'})
        else:
            expected_keys.update({(prefix + 'status__' + stage) for stage in stages})
            # add parameters
>           expected_keys.update({mark.args[0] for mark in test_foo.parametrize})
E           AttributeError: 'function' object has no attribute 'parametrize'

test_run_all_tests.py:69: AttributeError
_________________________________ test_failing _________________________________

    def test_failing():
>       pytest.fail("normal, intended failure here")
E       Failed: normal, intended failure here

test_run_all_tests.py:116: Failed
_________________________ test_synthesis_id_formatting _________________________

request = <FixtureRequest for <Function test_synthesis_id_formatting>>

    def test_synthesis_id_formatting(request):
        """
        Note: we could do this at many other places (hook, teardown of a session-scope fixture...)

        Note2: we could provide helper methods in pytest_harvest to perform the code below more easily
        :param request:
        :param store:
        :return:
        """
        # Get session synthesis filtered on the test function of interest
        # -- to debug the filter:
        # assert pytest_item_matches_filter(request.session.items[28], filter={TestX.test_easy})
        fmt = 'function'
        results_dct = get_session_synthesis_dct(request.session, filter=TestX.test_easy, test_id_format=fmt)
        assert list(results_dct.keys())[0] == 'test_easy[True]'

        fmt = 'class'
        results_dct = get_session_synthesis_dct(request.session, filter=TestX.test_easy, test_id_format=fmt)
>       assert list(results_dct.keys())[0] == 'TestX::()::test_easy[True]'
E       AssertionError: assert 'TestX::test_easy[True]' == 'TestX::()::test_easy[True]'
E         - TestX::test_easy[True]
E         + TestX::()::test_easy[True]
E         ?      ++++

test_run_all_tests.py:157: AssertionError
=============================== warnings summary ===============================
/home/zbyszek/.local/lib/python3.7/site-packages/_pytest/config/__init__.py:817
  /home/zbyszek/.local/lib/python3.7/site-packages/_pytest/config/__init__.py:817: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: celery
    self._mark_plugins_for_rewrite(hook)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
============= 4 failed, 13 passed, 1 skipped, 1 warnings in 0.08s ==============
Error while asserting that test_get_session_results.py results in {'passed': 16, 'skipped': 1, 'failed': 1}

[smarie]

Could that be related to pytest 5 ? I'll try to add pytest 5 in travis so see if I reproduce your errors

EDIT: running here https://travis-ci.org/smarie/python-pytest-harvest/builds/584557750

EDIT2: yes, confirmed by travis. This happens with pytest > 4

[smarie]

After investigating:

• one of the issues seems to be caused by a regression in pytest: https://github.com/pytest-dev/pytest/issues/5846, I will fix it quickly for now

• the other 2 issues came from pytest API changes (one on how to get the parametrization list for a function, and the other on test ids for tests tefined in classes)

I fixed both.

[smarie]

Waiting for https://travis-ci.org/smarie/python-pytest-harvest/builds/584612820 to verify and close

[smarie]

Fixed in https://github.com/smarie/python-pytest-harvest/commit/11456064cd31d315f8e3145c87fcfe51fd3b6bc4

[keszybz]

Yep, everything passes with those two patches. Thanks!

[smarie]

Thanks for the feedback!