Question: How to collect test information without modifying the test themself? In particular the start and end time of tests ?

[leinher]

Hi Sylvain,

I just found your pytest-plugin and have a question to it.

I'm writing integration tests with pytest. For this reason I need to collect the start and end time of each test. One test will check which log messages were generated by running all tests.

In the documentation of your module you set the @saved_fixture decorator. But I don't understand if and how I could extract meta-data from each test without adding the decorator to each test.

Here the current situation:

def test_something():
    # check something
    # test cases from 'update firmware', 'check usb ports' to 'a single REST requests'
    assert 1, "something doesn't work"

def test_log():
    # get all log messages the device under test created (and stored) while testing
    # apply a whitelist on the log list.

The problem here: It's hard to find the test which created the log entry.

Desired solution:

def test_something():
    # check something
    # test cases from 'update firmware', 'check usb ports' to 'a single REST requests'
    assert 1, "something doesn't work"

def test_log(fixture_store):
    # get all log messages the device under test created (and stored) while testing
    # create a test specific whitelist
    # - Test1: can create logmsg 'foobar'
    # On Fail: TestX created log message: 'something is wrong here'

Could you please tell me if this use case is possible with your pytest_harvest?

Many thanks Hermann

[smarie]

First you have to make sure that you perfectly understand the concept of pytest fixtures. This is extremely important, as most questions are due to this point.

From your message I do not understand if your need is

• to get the list of all tests that were executed, together with their start/stop time and status (https://smarie.github.io/python-pytest-harvest/#c-collecting-a-synthesis)

or (and/or ?)

• to store results in each test, so that you can retrieve them later on (typically at the end of test session). (https://smarie.github.io/python-pytest-harvest/#b-collecting-test-artifacts)

In both cases it does not seem that you have your own fixtures, so you do not need to save them :) so @saved_fixtures is definitely not for you. However pytest_harvest provides you with pre-baked fixtures to answer the two above use cases, I included the appropriate parts of the doc in the respective bullets above.

[leinher]

First you have to make sure that you perfectly understand the concept of pytest fixtures.

I think I understand how fixtures work. We use them in several tests to collect expected values, setup connections to a database file or initialize connection handlers.

to get the list of all tests that were executed, together with their start/stop time and status

Could work for this use case here. But I could not run this examples on my machine.

ci_tools/requirements-test.txt does not exist. What I've done so far:

$ git clone https://github.com/smarie/python-pytest-harvest.git
$ cd python-pytest-harvest
$ python3 -m venv venv
$ . venv/bin/activate
(venv) $ python3 -m pip install -r ci_tools/requirements-test.txt
Could not open requirements file: [Errno 2] No such file or directory: 'ci_tools/requirements-test.txt'

to store results in each test [...]

Case 2 doesn't fit for my use case. It would be necessary to add the fixture results_bag to every test in testcode.

[smarie]

Is there any reason why you do not install it using pip install pytest-harvest (i.e. as a "normal" user) ?

Concerning the requirements file for "pytest-harvest developer mode", indeed the requirements file name changed, I need to update the readme for developpers (thanks for telling me ! )

[smarie]

(I updated the readme for contributors with the correct requirements file).

So for your use case: after installing the plugin (again, I recommend pip install), simply create an additional test in your test suite, like this:

def test_synthesis(xxx):
    ...

where xxx is either session_results_dict or session_results_df depending on your preferences. (see https://smarie.github.io/python-pytest-harvest/#c-collecting-a-synthesis to understand how to read the contents)

[leinher]

Is there any reason why you do not install it using pip install pytest-harvest (i.e. as a "normal" user) ?

not really. I thought using your code here would be easier to show you what I've done.

So for your use case: after installing the plugin (again, I recommend pip install), simply create an additional test in your test suite,

Did that now. Hint: The pandas package was missing. This gives me information about the test duration. Now I understand c-collecting-a-synthesis and d-collecting all at once :smiley:. I just missed the fact that session_results_dct collects some default data and its optional to add things.

So far so good. Nevertheless, this doesn't solve the use case to get the test's start and end time. Is it possible to add that to session_results_dct

sure it would be possible to add this to each test (which in this case here is not an option):

# Let's store some things in the results bag
results_bag.nb_letters = len(person)
results_bag.current_time = datetime.now().isoformat()

[smarie]

The pandas package was missing

Yes this is an optional package (as explained in the doc:)

So far so good. Nevertheless, this doesn't solve the use case to get the test's start and end time. Is it possible to add that to session_results_dct

I guess that this would be easily doable, at least the start time (since the duration is already there). Note that most pytest users do not care about the start time, especially in case of parallel/distributed calls with xdist. But you are a living example of someone needing this info :)

It is done here If you are in a hurry and wish to have a try for a PR: https://github.com/smarie/python-pytest-harvest/blob/ab460bc04a6d340490e10a06bd640f64847b6657/pytest_harvest/results_session.py#L161

Otherwise you'll have to wait a bit as I'm under production-related pressure these days Thanks for your understanding !

[smarie]

I had a look again today, and unfortunately pytest does not make this information available by default in the test reports. Only the duration and outcome are provided. So I guess that the only way for you to store this additional information is to store it manually in the result bag as you suggested, at the begining and end of each test.

To automate things, I guess that you could create a function-scoped fixture depending on results_bag, where you would do this once and for all :

@pytest.fixture(scope='function')
def my_results_bag(results_bag):
    results_bag.start_time= datetime.now().isoformat()
    yield results_bag  # this executes the test
    results_bag.end_time = datetime.now().isoformat()

Then you'll just have to use my_results_bag instead of results_bag in all of your tests. Does that workaround work for you ? Let me know, and I'll create a dedicated documentation section on this.

[smarie]

@leinher can you please have a look at my past answers and let me know if this did the trick for you ?

[leinher]

Sorry for not replying. This got lost in my inbox. Unfortunately this did not do the trick for me. I already have a lot of tests and don't want to run through all of them to add another fixture.

My trick was to add this to a pytest hock

# conftest.py
def pytest_runtest_logstart(nodeid, location):
    writeCostomLog(str( datetime.now() ), 'start', str(nodeid))

def pytest_runtest_logfinish(nodeid, location):
    writeCostomLog(str( datetime.now() ), 'stop', str(nodeid))

This change enables to log the start and end timestamps without changing the test base. My hack stores this data to a file to analyze.

[smarie]

Thanks very much @leinher for answering. Glad to read that you found a solution to your problem.

I rename the title for clarity.

As a note for future readers, as of version 5.4.1 pytest does not propagate the start and end time from the CallInfo internal objects into the TestReport objects that plugins can retrieve in the reporting hook. This is why it is not easy to add the start and end time to the fixtures provided by pytest-harvest.

See TestReport.from_item_and_call, this is where the CallInfo is transformed into the TestReport and the call.start and call.stop attributes are not propagated.