CrayLabs / SmartSim

SmartSim Infrastructure Library.
BSD 2-Clause "Simplified" License
227 stars 36 forks source link

Add support to keep files after test suite passes #500

Open al-rigazzi opened 6 months ago

al-rigazzi commented 6 months ago

Description

Add the support to keep files in test_output after the execution of pytest even if the tests pass.

Justification

Currently, all files generated at test time are stored in tests/test_output/ for the duration of the test run and deleted in case all tests pass, kept if at least one test failed. In some cases, anyhow, it could be useful to keep said files even when tests pass. For example, when developing one feature and implementing the corresponding tests, it can be useful to look at output files. The current workaround is to make a test fail intentionally, but I think developers may want a programmatic way of enabling such behavior.

Implementation Strategy

I would suggest to add an(other) environment variable to control whether test files should be kept, and check its value in the following function:


def pytest_sessionfinish(
    session: pytest.Session, exitstatus: int  # pylint: disable=unused-argument
) -> None:
    """
    Called after whole test run finished, right before
    returning the exit status to the system.
    """
    if exitstatus == 0 and not os.environ.get("SMARTSIM_TEST_KEEPFILES") is not None:
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        shutil.rmtree(test_output_root)
    else:
        # kill all spawned processes in case of error
        kill_all_test_spawned_processes()

where I underlined the suggest change.

mellis13 commented 5 months ago

An alternative would be to just keep test outputs by default and consider if we should have an option to remove test output after a run.